Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a NAS algorithm for vision transformers. The search space contains both CNN blocks and transformer blocks. The authors solve the gradient conflict dilemma in this process and find out  a good architecture. Experiments on classification and segmentation show its effectiveness over other vison transformers like swin, cvt, etc.
main_review: ## Strengths:

1. The paper does not only transfer the NAS algorithms from CNN to transformer, but also observes the gradient conflict dilamma and resolves it partly.

2. The searched architecture seems over-performs swin, cvt, and some other vision transformers.

3. Besides gradient conflict, the authors also improve the performance by changing data augmentation.

## Weaknesses & Questions:

1. I am not sure what "Group" in Table 6 means. Does it mean the number of parameters? As I know, most vision transformers have parameters less than 100M.

2. The improvement on segmentation tasks seems a bit marginal.

3. It is recommended to explore how other data augmentation methods affect the NAS performance, e.g., AutoAugment, color jitter, cropping, etc. All these are common data augmentation methods for vision transformers.
summary_of_the_review: Though there are still some minor flaws in the paper, I think it is a good one and is nearly ready for publication.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper investigates applying a SOTA NAS method (from AlphaNet) to a SOTA ViT search-space (from LeViT) and notices disappointing results. Following an investigation, conflicting gradients are identified as the root cause, alleviated via three components: gradient projection, learnable scaling factors, and heavily reduced augmentations.
main_review: I really like the exposition of the paper, going through hypotheses and presenting experiments supporting or disproving the hypotheses. This was great to read. I also really appreciate the links to literature in different fields, speicifically multi-task learning regarding conflicting gradients.

1. My main concern is that the paper is about finding efficient and well-performing models, however it does not include a single timing anywhere I could see. I would really appreciate if Table6 included a latency column, and there was a variant of Figure4 with latency as x-axis. Especially the small attention head size has me concerned, whether the good FLOPs will actually translate to good efficiency.
2. What about gradient agreement between individual random subnetworks? Is it better, equal, or worse than the super network vs average f sub networks?
summary_of_the_review: I find the paper generally of high quality, but the lack of any timings is concerning. If this can be fixed, it would be a pretty good paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: To solve the gradient conflict issue that affect the performance，this work propose a series of techniques, including a gradient projection algorithm, a switchable layer scaling design, and a simplified data augmentation and regularization training recipe to improve the convergence and the performance of all sub-networks.
main_review: + The writing is easy to follow and read.
+ The evaluation results are promising.
-  The motivation of this work is clear, but the explanation of poor performance (gradient conflict issue) is somewhat  not convincible.
-  The architecture in the search space consists of MBConv and transformer, it is better to compare with the existing works that also combine the CNN and transformers, like PiT, etc.

summary_of_the_review: It is a good work with clear motivation and  promising performance, the reason of poor performance by directly applying the supernet-based NAS to optimize ViTs need to be discussed more.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work presents the gradient conflict issue in ViT training, i.e., the gradients of sub-networks conflict with that of the supernet, leads to inferior performance of ViT supernet training. The authors fix this issue by 1) a gradient project method to prioritize the sub-network update; 2) Use switchable layers to increase the model capacities of sub-networks; 3) Simplify the training recipe. The proposed NASViT shows the state-of-the-art top-1 accuracy v.s. FLOPs trade-offs on ImageNet.
main_review: Strengths:
1.	The motivation is clear, and the three proposed solutions are simple and generalizable.
2.	The proposed NASViT achieves promising performance in classification and segmentation with low computational cost.

Weakness:
1.	It is an interesting phenomenon that ViT supernet training suffers from the gradient conflict issue. But I am curious about the performance of other supernet training methods that don’t utilize KD from supernet, like once-for-all (ICLR20), DNANAS (CVPR20) and Cream (NeurIPS20).
2.	I am not clear why the authors introduce the switchable scaling. I don’t see the relationship between it and the gradient conflict issue, also the improvement in Table 7 looks marginal. Also Figure 3 is not very clear, e.g., what does c1 and c2 means?
3.	There is no experiment to support the statement “the supernet and the sub-networks are more likely to conflict with each other in the presence of stronger data augmentations and strong regularization.”. There is performance improvement shown in Table 7, but it is not clear whether this is caused by the conflict between supernet and sub-networks. And if so, I am wondering is there still an improvement for weak DA & Reg if the gradient projection is already applied.
4.	In Table 8, it seems that AlphaNets trained with alpha-divergence doesn’t benefit from your method. Maybe you should show the gradient conflicts ratio for it to give some insights.
5.    Since the gradient projection leads to slow convergence, the authors should provide the training time cost before and after using the proposed gradient projection, as the training time cost is also important.
6.	How about the training time consuming after using the other two techniques.  Does it reduce the training time caused by gradient projection for its slow convergence? Detailed information should be presented.
7.	Why do the authors use EfficientNet-B5 as the teacher? Why not B6，B7 or other networks? The reasons are not mentioned in this paper. It is confusing.
8.    To be more general and convincing, though the evaluations of the smallest and the largest networks are provided, I think the results of the middle-size network in needed experiments are also critical. For example, in Fig 2, Tab 3, and so on.
9.    The authors only adopt gradient projection on CNN-based supernet while conducting ablation studies. The generalizability of the other two techniques in CNN should be verified as well like Tab.7. 



summary_of_the_review: The motivation of this paper is not very convincing, need to provide more evidence to prove. For more details, pls see the weaknesses.
Also, there are some recent works on NAS+ViT missing in the related works, such as "CATE: Computation-aware Neural Architecture Encoding with Transformers", "Vision Transformer Architecture Search" and "AutoFormer: Searching Transformers for Visual Recognition". It is suggested to add them into comparison and discussion, especially for Autoformer which might be the first ViT+NAS work and has been published in ICCV21. I will re-consider the rate if all the concerns are well addressed or not.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper aims at applying one-shot Neural Architecture Search (NAS) to Vision Transformers (ViTs). The authors claim that directly using existing CNN based NAS method to ViTs will lead to a gradient conflict issue. In order to tackle this issue, the authors propose three techniques, including a gradient projection, a switchable layer scaling, and a data augmentation. The experimental results demonstrate the effectiveness of the proposed method to some extent. 
main_review: #Strength
1. The goal of introducing NAS to ViTs sounds reasonable.
2. The discovered gradient conflict issue looks interesting.
3.The experimental results seems solid.

#Weakness
1. It seems that the authors only focus on the one-shot NAS. Could other types of NAS methods be applied to ViTs? From this perspective, the discovered gradient conflict issue seems not universal.

2. The proposed three techniques, i.e., the gradient projection, the switchable layer scaling and the data augmentation, look like a “A+B+C” pattern. There seems to be no obvious connection among the three techniques. 

3. Why do the authors remove the component of the super-net gradient that is conflict with the sub-network gradient? This operation is not intuitive. What are the negative effects of this operation? 

4. The introducing of the weak data augmentation scheme is also sudden. It is highly suggested to give more intuitive explanations in Section 1.

5. What is the motivation of the raised three investigations? 



summary_of_the_review: In general, the writing of this paper makes it difficult to follow the authors’ core thinking.
review_title: null

Meta-review: This paper tackles a very timely problem. 
Scores of 5,6,6,8 put it in the borderline region, but in the private discussion the more negative reviewer noted that they would also be OK with the paper being accepted. I therefore recommend acceptance. 
Going through the paper I missed any mention of available source code. I strongly recommend that the authors make code available; this would greatly increase the paper's impact.
Decision: Accept (Poster)