Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a combination of several implementation details for vision transformers that can be evaluated efficiently (with low latency) on iphones. These details include: 
- Clever management of the shape of the tensors to avoid expensive reshape operations.
- Using BN instead of LN to save latency by folding in BN parameters into the final model.
- Using GELU nonlinearity (eg. instead of swish)
and most importantly a pruning methodology to prune the least essential blocks by training a categorical gating mechanism via gradient descent. 
The resulting network vastly outperforms previous solutions that have similar latency on both ImageNet classification and semantic segmentation.
strengths_and_weaknesses: Originality: Weak-Medium.
The paper presents a combination of methods, most of which is well-studied and understood in different contexts:
- Shape management and alignment has been one of the most optimized aspect of highly tuned numerical linear algebra software for many decades. For example, this has been the focus of optimizing BLAS libraries over many decades.
- It is well-known that BN has a slight edge over other normalization methods when it comes to latency of the final model, due to the possibility of folding in the normalization parameters.
- It is surprising that swish is so inefficient on CoreML, but it seems like a special artifact of the current state of the library.
- While the pruning methodology seems very efficient, it resembles very much to MorphNet [Gordon, Ariel, et al. "Morphnet: Fast & simple resource-constrained structure learning of deep networks." CVPR. 2018], while failing to cite that prior work.

Quality: Medium
While it is impressive that 79% top-1 accuracy on ImageNet is possible within 1.2ms on an iphone, the papers sole focus on this particular library and hardware raises the question of the general applicability of the methods. Also several of the methods just circumvent the special deficiencies of the library (esp swish vs GELU accounts for a large parts of improvements over LeViT 256, which might become competitive with this single change. 

Clarity: High
The paper is well-written with clear ablation analyses and explaining the motivation of all the applied techniques. The methods are evaluated on reasonable benchmarks on both classification and segmentation tasks.

Significance: Medium
This work gives a strong SoTA baseline for vision models on current IPhone hardware. However, it is unclear how/whether these methods generalize to other types of hardware/libraries. It seems unlikely that similar gains could be achieved on android systems, given the difference and maturity of the employed machine learning infrastructure.
questions: - Are the dimension-consistent layers relevant in other libraries/hardware, like pytorch, TF or jax on nvidia GPUs? (As the model was trained on  pytorch/GPU), this should not be hard to answer.
limitations: Obviously, the paper focuses on a very specific proprietary hardware/software stack, and therefore the applicability of the methods is somewhat limited. The paper does not reference very similar pruning methods presented in 2018 MorphNet paper.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: In this paper, the authors propose a new ViT-based architecture named EfficientViT. In detail, this paper tries to build hybrid designs for MobileNet block and ViT architectures. Finally, this algorithm performs a latency-driven slimming for a series of final models.
strengths_and_weaknesses: Strengths:
1. This paper is easy to read.
2. The proposed observations in this paper are interesting and useful for the designation of architectures. 

Weaknesses:
1. Many papers focus on the combination of CNNs and Transformers.
2. The proposed MB blocks have similar architectures to some existing papers.
3. This paper does not compare with some baseline methods[1][2], and these methods perform better than this paper.

[1] Mobile-Former: Bridging MobileNet and Transformer
[2] CSWin Transformer: A General Vision Transformer Backbone with Cross-ShapedWindows
questions: As shown in the Strengths And Weaknesses.
limitations: I do not find any limitations and potential negative societal impact of this work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper benchmarks the latency of recent efficient vision transformer designs on iPhone 12 through CoreML support and makes observations on the latency vs. accuracy trade-off of different operators. The authors then proposed a super-net search space that bakes in multiple latency-favorable designs and performs gradient-based architecture search followed by a heuristic latency-driven slimming process to obtain sub-net under given latency constraint. The authors evaluate the searched architectures as the backbone across image classification, object detection, and semantic segmentation tasks.
strengths_and_weaknesses: - *Originality:* To the best of my knowledge, the paper adequately cites related works such as MetaFormer and outlines the differences. While the authors mention that the design space is partly inspired by MetaFormer, the latency vs. accuracy trade-off improvement is novel and largely driven by new insights through latency benchmark on actual hardware.
- *Quality*: The submission is technically solid in building the EfficientViT design space based on observations from latency analysis on real hardware. The authors clearly addressed two relevant limitations that come to mind when reading the paper: 1) limited insight on how the observations and designs hold across other hardware, and 2) the latency-driven slimming procedure could invite further investigation. Testing the searched architectures across different tasks is a good practice as well.
- *Clarity*: The submission is clearly written and well organized. The observations from latency analysis naturally build up into respective considerations for the design space.
- *Significance:* The paper provides comprehensive empirical insights on optimizing vision transformer design space for a given hardware, and achieving MobileNetV2 level latency is meaningful for practitioners. While the latency analysis workflow is general and could serve as a good example of devising design decisions for other hardware, it is not immediately clear how well the insights and the design space will transfer.
questions: Looking at the comparison in Table 1, at a given latency, EfficientViT tends to have a higher parameter count or GMACs (i.e. EfficientViT-L1 vs. MobileNetV2, EfficientViT-L7 vs. MobileViT-XS). From reading the paper, it seems the improvement could come from two sources: 1) better hardware utilization so that a more complex model can run under the same latency constraint (latency analysis guided design space), and 2) when hardware utilization saturates, finding architecture with better compute distribution over operators for higher accuracy (NAS + latency-driven slimming). Is there a way to perform ablation and quantify the performance gain from each source? Understanding the contribution from 1) would be valuable for practitioners optimizing architecture for specific hardware, while understanding 2) will shed light on the value of the proposed design space and search algorithm.

limitations: The authors clearly addressed some potential limitations of the work: 1) Some observations and subsequent design decisions might be hardware and software dependent; 2) The NAS procedure, specifically the latency-driven slimming procedure is less involved and could be a direction for future exploration.
review_title: null

Meta-review: metareview: This work proposes a purely transformer-based vision model for mobile vision purposes.

This proposition is somewhat surprising, since transformers did not excel at low-latency inference on resource-constrained hardware, especially compared to convolutional networks.

This is achieved by using a clever design that allows for reshape operations without the actual need of copying data as well as new techniques for latency-optimized network pruning.

While the methods itself are very technical and engineering-oriented, the overall result: a purely transformer-based low-latency, high-quality vision network is of general interest and is worth being shared with the wider community. Therefore I propose this paper to be accepted for NeurIPS 2022.


confidence: Certain
Decision: Accept