Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This work applies NAS to the vision transformer architecture. Besides searching the architectures, it also automatically adjusts the search space guided by the E-T error that combines the error distribution and top model errors to evaluate the quality of different choices in a search dimension. By fitting a linear function to approximate the E-T error, the new search space is updated through a simple formula. They used the supernet to efficiently compute the E-T errors, and use evolutionary search to search for the best algorithm in a given search space. The discovered architecture is shown to be competitive to the state-of-the-art models, and also generalize to other vision tasks.

main_review: Strength:

(1) The idea of automatically iterating the search space and using super-net to make the calculation of the E-T error are interesting. It combines existing ideas in designing the search space and weight sharing NAS and proposes a new method to update the search space by fitting a linear function to the E-T error. 

(2) The evaluation result is competitive and additional experiments are conducted on several vision tasks to verify the discovered architectures' generality.

(3) Analysis are performed on how the search space changes in different iterations, however I had some questions regarding the reliability of the results. 

(4) The paper is well written and easy to follow. (A minor issue in formatting, the caption of Figure 2 is covered a bit by Figure 3.)

Weakness:

(1) This paper combines several existing ideas and applies them to a new domain (Vision Transformer), and the novel contribution in the method is the way to update the search space using the linear function fitting of the search choices and the E-T error. However, there should be more analysis or supporting evidence for this simplifying assumption in the experiments. For example, it would help to add some plots and calculate the R-squared to verify how good this linear approximation assumption is. 

(2) Some of the hyperparameter choices need more discussion. For example, how is the value of tau (0.4) picked? Was it tuned over some range? How sensitive is the result with respect to this value? Same for the number of search space iterations. Why does it stop at 3, is it due to compute limits or maybe 3 is the sweet spot? It would help to add some analysis regarding how the accuracy changes with the number of search space iterations.

(3) The analysis of the trends in Section 3 is quite interesting. Since the conclusions are draw from the results in Figure 2, I wonder how much variance is in the result. For example, if you repeat the experiments multiple times, how much would the search space evolution differ between different repeats. This would help evaluate the reliability of the analysis.  



limitations_and_societal_impact: I listed a few additional analysis in the "Weakness" section in the Main Review that would help shed more light on the limitations of the proposed methods.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper introduces a neural architecture search method called S3 that first searches the search space before identifying good architectures from the discovered search space.  The authors focus on applying S3 to vision transformers and design a search space modeled after ViT.  They evaluate S3 discovered architectures on various computer vision tasks and show good performance on image classification, object detection, and semantic segmentation.

main_review: ## Originality 
The main points of novelty introduced in this paper are:
- Applying one-shot NAS approaches to vision transformers. The authors note that in contrast to Once-for-all and BigNAS style approaches for CNNs, inplace distillation did not help when training the vision transformer supernetwork.
- Defining the search space for vision transformers.
- Searching the search space using the E-T Error heuristic along with a linear parameterization to guide the search.
The novelty though is fairly small in scope so I lean towards a neutral rating here.  Additionally, there is a point to be made that the search space should be as large as possible and include all possible sub-spaces of interest.  In this case, the search algorithm would be responsible for efficiently identifying the most suitable architectures from the space. 

## Quality
The E-T error and linear parameterization are not well motivated and I want to see further justification for choosing these heuristics. 

I liked the discussion provided in Section 3 of the search phase results but the writing con be improved significantly.  

The empirical results in Section 4 are promising and demonstrate S3 to yield highly performant architectures.  

However, I have quite a few questions that I hope the authors can address:
- What do you mean by compose the evolved search space in Figure 1?
- It appears that the relationships between E-T error and a search dimension is not always linear.  Have you observed failure cases where using a linear parameterization didn't learn a good search space? 
- How sensitive is the search process to the parameter \tau?  Have you tried other values?  
- What is the search cost for 3 iterations?  Have you conducted search space refinement to convergence? 
- Have you tried comparing S3 to a one-shot model trained on a larger search space that includes the resulting search spaces from all iterations of the search process?
- Have you tried extending the search space refinement process to other NAS benchmarks like the popular MobileNetV3 search space?  Does S3 transfer to these other search spaces?

## Clarity
The paper is poorly written and errors are littered across the entire text.  I have enumerated a subset of issues I noticed but there are certainly more.  

_Need clarification_
- L45: "the third stage is the most important..." what is the third stage? this is not introduced until much later
- There are 4 steps in Figure 1 but there is nothing corresponding to these four steps in the actual text.  I see steps 1 and 2 in section 2.3 but not 3 and 4.  

_Typos & Grammar_
- L37-38: "Different from [34] that trains..." -> In contrast to [34], which trains...
- L40-41: "We decouple the search space by multiple dimensions..." -> We characterize search spaces by multiple dimensions...
- L42-43: "Particularly, we use linear parameterization to model the tendencies of different dimensions so as to guide the search." -> In particular, we parameterize the model along different search dimensions...
- L44: "we analyze the search process of search space and provide some interesting observations" -> needs to be rephrased
- L49: "we hope these observations could help..." -> we hope these observations will help
- L:104: "Different form RegNet [34] that trains..." -> needs to be rephrased
- Figure 2 caption cut off
- L145: "Fig. 3 shows an direct..." -> Fig. 3 shows a direct...
- L147: "The third stage is the most important and increasing blocks of it contribute to better performance" -> needs to be rephrased
- L151-153: needs to be rephrased
- L153: "should include less layers" -> should include fewer layers
- L154: use a different verb from prefer perhaps "more compatible" or "more suitable"
- L156: "deep layers attend to" -> deep layers tend to
- L157: "We assume that transformers increase focusing region with deeper layers" -> needs to be rephrased
- L161: "which illustrated from" -> which is illustrated from
- "focusing regions" -> regions of focus
- L164: "Deeper layer, larger MLP ratio" -> phrase as a sentence to mirror other points.
- L166: "It gives our insight to design different MLP ratio" -> needs to be rephrased
- L167: "Q-K-V dimension could smaller than the embedding dimension" -> Q-K-V dimension can be smaller than the embedding dimension
- L171-173: needs to be rephrased
- L172: "less heads" -> fewer heads
- L215: "we could get model..." -> we found a model...

## Significance
Despite the impressive empirical results, the significance of the paper is greatly limited by the numerous grammatical errors that litter the text. I would evaluate its potential for impact to be low in its current state.  

===============Post Author Response===============

After reading the authors' response, I have decided to change my score to a 5.  I am hesitant to recommend acceptance given the unpolished state of the paper at submission.  I do see the merit of the method proposed and believe it would benefit tremendously from improved  writing and additional experiments evaluating S3 on different search spaces in a future submission.
limitations_and_societal_impact: The authors do not adequately evaluate their proposed S3 method for sensitivity to internal parameters and robustness/transferability to other search spaces.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a method for searching the search space of vision. The authors propose decomposing the search space by the search dimensions and evaluating them with the proposed E-T error to explore the optimal search space. Then the new subspace is formulated as Equation (8). The results reported in this paper have achieved state-of-the-art results.
main_review: (+) This paper is easy to follow.			

(+) State-of-the-art performance. This paper conducts extensive experiments and achieves impressive performance improvements.

(+) Searching the search space is a good problem in NAS methods.

(+) This paper provides lots of interesting and useful observations in section 3, which may help the development of the research of vision transformer.

(-) The proposed method seems very time-consuming in searching search space. Simply calculating each search space's expected error for all the search space is computationally inefficient (as in the 5 line in Algorithm 1). I wonder how many search spaces are considered in the searching process.

(-) I am not satisfied with the Experimental part. The authors did not provide the final searched search space nor the searched architecture. If the authors can provide detailed evidence about this issue, I will support accepting this paper. I may change my rate according to the rebuttal.

limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This work propose to first search for the search space for transformers and then applies once-for-all NAS methods to search for architectures within the searched search space. In particular, the work proposes an E-T error metric to evaluate the quality of the search spaces. From the results of the whole search process, this work also summarizes some design guidelines for vision transformers.
The experiments are extensive and they show the superiority of the proposed search method. 
main_review: Pros:
- The paper is easy to follow.
- The idea of search the search space first is intuitive.
- The experiments on different downstream tasks are extensive.
- The design guidelines for transformer are good for the research community.

Cons:
- The improvements over Swin transformers on object detection and semantic segmentation are not so obvious, comparing to the gap between Swin and the other methods.
- What is the search cost (GPU-days) of each step?
- The ablation study seems not enough. Are there any ablations on the E-T error, such as changing the weighting of E and T, or varying the number of sampled architectures for computing E and for computing T.
limitations_and_societal_impact: Yes, the limitations are addressed.
review_title: null

Meta-review: Four experts reviewed this paper and gave ratings 6, 6, 6, and 5, respectively --- Reviewer 3mCn decided to change 7 to 6 in a private discussion. The reviewers expressed concerns about novelty and writing. Particularly, some reviewers commented that the E-T error heuristic and linear approximation were not well-motivated, and the novelty was small. Reviewer W7fM felt strongly about the “poor state of the writing.” However, the reviewers generally appreciated the study of NAS for Transformers, and a reviewer considered the design of the Transformer-tailored search space original. AC agreed that such a study could benefit future research on Transformers and felt that the concerns can be addressed in a reasonable revision of the paper. Hence, the decision is to recommend the paper for acceptance. The authors are encouraged to make necessary changes in the camera-ready to address the reviewers' questions to the best of their ability. Especially, the authors may consider finding a native English speaker to help polish the writing. We congratulate the authors on the acceptance of their paper!
Decision: Accept (Poster)