Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Initial variants of ViTs borrow their architecture from the seminal ‘attention is all you need’ work. Some recent methods amend the default transformer architecture to incorporate convolutions in the design (e.g., CCT: Compact Convolutional Transformers), or a hierarchical architecture with feature pyramids (e.g., Swin, PyramidViT etc) to make ViTs suitable for dense prediction tasks (detection and segmentation). 
Instead of relying on hand-designed architectures, the paper proposes to automatically search for a ViT architecture, that is both efficient & accurate. 
main_review: 

- A two-stage approach is followed for architecture search. First, there is a search for seed architecture using a training-free strategy. The seed ViT is found from existing designs. Second, the seed is scaled upon along width and depth.

- the paper also proposes elastic-tokens: progressive re-tokenization for fast ViT training.

- While the design search is efficient: requires only 7+5 GPU hours for seed+scaling stages. Compared with existing NAS based approaches, the proposed method requires less time to search for the optimal architecture. However, the final searched architecture by the proposed method has more FLOPs than existing methods e.g., ViT-ResNAS-t with 1.8B FLOPs compared with 8.9B of the proposed architecture, with slightly better performance.
summary_of_the_review: 
- It is unclear if the proposed strategy works to search a ViT architecture suitable for small-scale datasets?

- The advantages of automatically searched ViT architectures are not immediately clear. Their major limitation for ViTs is the time they require for training, and the number of FLOPs introduced by quadratic complexity in self-attention. Compared with existing “hand-designed” ViT architectures, the proposed automatically searched ViT architecture does not significantly reduce compute training time. 

- I believe an automatically searched ViT architecture, that is efficient to train, will help democratize ViTs research, and make it accessible to broader resource-constrained academic labs.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the author proposes auto-scaling ViT, which is to search seed ViT topology based on number of kernels, attention splits, expansion ratio, depth and width jointly. The searched ViT is trained with a progressive re-tokenization scheme that saves ~40% training time and preserves the accuracy with less parameters. Experiments on ImageNet and MSCOCO show that AS-ViT can reach strong performances on classification and detection.
main_review: Strengths:

+The problem is significant

ViT training is time-consuming and the input resolution is fixed for a specific ViT model, which makes it important to have a proper scaling rule for choosing the size of ViT. The search-based auto-scaling rule proposed in this paper is a good way to solve this issue.

+NAS on ViT

The author uses neural architecture search on ViT-like framework on the number of kernels, attention splits, expansion ratio, depth and width jointly. The searched architecture are stage-wise and can be used for down-stream tasks like detection, which is another technical contribution of this paper.

Weaknesses:

-Comparison with other methods

Compared with original single-scale ViT or a multi-scale vision transformer like Swin, the proposed AS-ViT does not outperform them (ViT-S 81.2%, Swin-T 81.3%, AS-S 81.2%; Swin-S 83.0%, AS-B 82.5%; Swin-B 83.3%, AS-L 83.5%). Swin here can be viewed as an augmented multi-scale vision transformer. The author probably needs to propose a Multi-scale ViT baseline for comparison here and show the effectiveness of the searched AS-ViT. 
summary_of_the_review: The paper AS-ViT is focusing on solving the scaling issue of ViT with searched topology and auto-scaling rules. However, it does not show significant improvement compared with other multi-scale vision transformers and my concern is that a multi-scale ViT baseline for fair comparison and evaluation is needed here. I will increase my rating if the concerns are well addressed here.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposed As-ViT to automate the principled design of vision transformers without tedious human efforts. To my best knowledge, it is the first framework that unifies efficient search, scaling and training in ViTs. The empirical performance is in general satisfactory. 
main_review: Pros: 
-1- As-ViT is a novel all-in-one efficient framework: it first finds a promising “seed” topology for ViT of minimal depths and widths, then progressively “grow” it into different s capacities to meet different needs, and finally “train” the desired architecture. All three steps are coherently designed to be very efficient, which is important for designing and verifying ViTs.
-2- For searching ViTs, the seed topology space is relaxed from recent vanilla ViT designs. To compare different topologies, the authors automate this process by a training-free architecture search approach via measuring the network complexity, which is extremely fast and efficient. 
While several previous works already explored NAS for CNN with gradient features or linear regions, transformer has more nonlinear structures and different components. The authors introduced a new general complexity metric of manifold propagation that can be applicable to transformers. The proposed training-free search is experimentally supported by a comprehensive study involving measure time costs and Kendall-tau correlations.
-3- The “seed” ViT topology is then progressively scaled up from a small to a large network, generating a series of ViT variants in a single run. At each step, how to increase the depth and width is automatically determined by comparing network complexities of different scaling choices, being both principled and efficient. The authors claimed they discovered ViT scaling principles that echo the empirical observation of (Zhai 2021).
-4- Further, to address the heavy training costs of ViTs, the authors proposed to make ViT tokens elastic, an idea similar to dynamically downsampling image resolution for efficient CNN training. They then propose a progressive re-tokenization method for efficient ViT training. The idea is not particularly novel but is the first time of this type in ViTs.
-5- The authors made comprehensive comparisons with various SOTA ViTs and their NAS methods, on not only ImageNet-1K classification but also COCO detection. As-ViT performs better than or competitively to most of them, meanwhile largely saving training FLOPs and time so that is also impressive. However, I have some reservations regarding their discussion of results, as to be laid out below.

The paper writing is overall good. The results and findings suggest potentially broad impact of this work.

Cons:
-1- Comparison to Prior Arts of NAS for ViTs
My main issue is that, from table 5, I cannot conclude the same as the authors stated “As-ViT framework significantly outperforms concurrent NAS works for ViTs in both performances and search efficiency”. For example, BossNet has marginal inferior accuracies but also smaller models; the more recent AutoFormer has also the same accuracies for Small/Base, only costing more FLOPs at Base level. 
I understand that their search space comparison is not apple-to-apple (e.g.,  BossNet used conv layers too); and As-ViT has other selling points such as efficient search and training. However, the authors are urged to put their accuracy comparison in the fairer and more detailed context, and the current conclusions look a bit over-rush and misleading to me.
-2- Comparison to Swin Transformers
For image classification, the reported performance of As-Vit is slight less competitive than Swin at comparable costs. Again, I understand the two are not fully apple-to-apple comparable due to vastly different building blocks, and I am not asking As-ViT to outperform Swin since the former’s current search space is more vanilla.  Yet, as Swin follows a different scaling philosophy to be narrower and deeper, how can you then be convinced that your discovered “shallower but wider” principle is optimal?
Also, in object detection, As-ViT Large seems to outperform Swin-B, but the comparison is questionable to me: why the image resolutions are not aligned between the two? If Swin on averages sees smaller image inputs, it can possibly explain why its FLOPs and AP are both lower than As-ViT, and a fairer comparison should be appended.
Further, the current As-ViT space searches for attention splits, which seems to be a simplified sliding window without overlapping. Then, could you expand your search space to include Swin-type architecture into your search space? To be fair, I am not asking the authors to perform so during the rebuttal timeframe. I’m instead asking to address the above fairness issues.

summary_of_the_review: The biggest merit of this paper is to unify efficient search, scaling and training as one -- that is unique and strong. The results prove their concepts and can be considered as promising since the adopted search space is vanilla. However, the authors are expected to clarify a number of experiment and comparison issues as aforementioned.
review_title: null

Meta-review: The paper introduces As-ViT, an interesting framework for searching and scaling ViTs without training. Overall, the paper received positive reviews. On the other hand, R1 rated the paper as marginally below the threshold, raising concerns about search on small datasets and issues regarding the comparison in terms of FLOPS/accuracy with other methods. The authors adequately addressed these concerns in the rebuttal, and helped clarify other questions by R2 and R3. R1 did not participate in the discussion after the author response nor updated his/her review. The AC agrees with R2 and R3 that the paper passes the acceptance bar of ICLR, as the unified approach for efficient search/scaling/training is novel and should be interesting to the ICLR audience.
Decision: Accept (Poster)