Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: -- Short Summary --

This paper proposes a new policy for selecting the optimal architecture in neural architecture search (NAS), in particular for methods that involve a one-shot model and that deploy gradient-based methods for the search. The proposed algorithm sequentially prunes and fine-tunes the one-shot (aka supernet or weigh-sharing) model until the operations that contribute the most in the one-shot validation performance remain at each edge of the cell (represented as a DAG).

-- Detailed comments --

Positive points: 
- The structuring of the sections.
- The paper investigates an interesting aspect of gradient-based NAS algorithms.
- I liked the theoretical justification on the issue with the high \alpha values for skip connections in many DARTS [1] and the experimental backup for that in table 1.

Issues and concerns:
- The proposed algorithm does not seem that elegant and still uses another proxy for true performances of stand-alone architectures, even though indirectly in a lower level, i.e. the drop in performance of the one-shot model after removing one operation at a time in each edge is used to assess the optimal operations operation that will be present in the final architecture. Furthermore, this procedure seems to induce quite some variance by randomly sampling the edges in the cell and then executing the operation drop & fine-tuning steps. Did the authors investigate this by running multiple times their proposed procedure on the same one-shot model?
- Another issue with the algorithms seems to be that it does not scale well with the number of operation and edges in the cell.
- I am not fully convinced by the experiment conducted in 3.1. It seems the authors discretize only one edge of the supernet and fine-tune it further with the discretized edge. In this case the other edges (not discretized) contribute to the network performance together with the single discretized edge and this might be misleading when assessing the importance of an operation. Ultimately, we care about the final network performance, and how this single operation in the discretized edge would perform when combined with all other possible choices in the other edges (still intractable to compute of course). Another potential issue is that the fine-tuned supernet might not correlate well with the discretized stand-alone architecture performance. Can the authors please elaborate on this experiment further in more details?
- Do the theoretical and experimental backup for that hold for other sampling-based methods, e.g. GDAS [2] or SNAS [3]?
- I think the reported search costs in Table 2 are inaccurate. The authors should also add on top of their method the costs of the base algorithm (e.g. 1GPU day for DARTS 2nd order on one GTX 1080Ti).

Minor:
- There are a lot of grammatical errors throughout the text. I would recommend to do a detailed proof read of the paper.
- section 3.2, second paragraph: The acronym VGG does not stand for vanilla networks
- The data in Fig. 1 would look better as a scatter plot, which would also more nicely show the miscorrelation between the 2 quantities that are being compared.
- I like the structuring of the paper, however I think the experiment in the conclusion should not be there, but in the benchmark tables.
- In conclusions: why do you fix \alpha = 0? Did you mean \alpha = 1?

-- References --

[1] Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. In ICLR 2019

[2] Xuanyi Dong and Yi Yang. Searching for a robust neural architecture in four gpu hours. In CVPR 2019

[3] Sirui Xie, Hehui Zheng, Chunxiao Liu, and Liang Lin. SNAS: stochastic neural architecture search. In ICLR 2019
review_title: Interesting analysis, however the proposed method not that exiting

final_score: rating: 10: Top 5% of accepted papers, seminal paper
review_content: Summary:

In one-shot differentiable NAS, a supergraph is usually trained (via bilevel optimization as in DARTS, or other approximations to bilevel such as gumbel softmax, etc). After supergraph training, a final architecture is obtained by taking the operator at each edge which has the highest architecture weight magnitude. This step is usually termed as the 'finalization' step. (In DARTS the finalization step actually orders incoming edges by the max of the architecture weight magnitudes at each edge and selects the top two edges and the corresponding maximum architecture weight in them as the final operators.). This paper examines this quite ad hoc step very closely. It finds that the magnitude of architecture weights (alphas commonly in this niche literature) are misleading. It shows by careful ablation experiments that alpha magnitudes are very much not useful in selecting good operators. 

By taking inspiration from the "unrolled estimation" viewpoint of ResNet prior work it shows that DARTS converging to degenerate architectures where alphas over parameters operators like skipconnect is actually to be expected when finalization step relies on the magnitude of alpha. 

The paper proposes a much more intuitive finalization step which just picks the operator at each edge which if removed from the supergraph results in the largest drop in validation accuracy. To bring back the supergraph to convergence a few epochs of further training is carried out between operator selection.  

Experiments show that just by carefully thinking about the finalization step in differentiable one-shot NAS, one can obtain much better performance. In fact, one does not even need architecture weights at all! Don't worry about complicated bilevel optimization, gumbel softmax approximation, etc. Just train a supergraph and pick operators progressively. 

Comments:

- The paper is wonderfully written! Thanks!

- As I read a paper I try to think without looking at the experiments, what set of experiments I would try to run to prove/disprove the hypotheses proposed. Afterwards I go through the experiments and see if those experiments were actually run (or if they differed why). In this case, every experiment and more were already run. Particularly towards the end I was thinking what if we just got rid of all the alphas and just trained a supergraph as usual and did the PT finalization as proposed. And lo and behold, it actually works better!

- This paper is actually throwing a big wrench in one-shot differentiable NAS literature. Many papers are being written which try to improve/fix DARTS and DARTS-like methods. If I were to believe the experiments, I don't actually need to do any of that. I have some questions I hope to discuss with the authors:

1. Is all the complicated bilevel optimization (often popular as 'metalearning' currently) not useful in the case of NAS? (This is not really the authors' burden to answer but I am just hoping to see if they have any insights.)

2. Can we view the PT finalization step as a progressive pruning step?  So if I were to turn this into a method which produces a pareto-frontier of models (e.g. accuracy vs. memory/flops/latency etc), we first train a big supergraph and then progressively prune out operators one at a time as proposed here and take a snapshot of the supergraph and plot it on the x-y plot (where say x is latency and y is accuracy) and pick the ones clearly on the pareto-frontier and train them from scratch? (Again not really authors' burden but curious if they have any insights)

3. Figure 4 suggests that training the supergraph anymore than 20 epochs only hurts performance (no matter which finalization procedure is used, of course PT has far less of a drop). Does bilevel optimization actually hurt with weight sharing?

review_title: Finalization step is crucial! 

final_score: rating: 7: Good paper, accept
review_content: # post rebuttal

I have no further concerns and increase the rate to accept.

# Summary

This paper identifies an interesting phenomenon that on DARTS based method, the operation can not be simply chosen based on the maximum value of trained weights. The authors propose a new selection paradigm. For each operation, one should first discretize the soft weights encoding into a one-hot, then fine-tune the network for some iterations, and use the performance as a metric to select the operation. 
It provides an interesting and novel analysis on the question of why DARTS always converges to skip-connections. The experiments are conducted on CIFAR-10, 100, and SVHN on DARTS space and NASBench-201 over three datasets.


# Strength

I enjoyed reading this paper although there are some flaws in terms of presentation. It tackles one of the most critical problems in the DARTS domain, why the skip-connection dominates after the super-net training converges. All earlier works propose some solutions in an ad-hoc manner, by early stopping. This is the first time I have seen a simple and reasonable explanation of this phenomenon, and by itself is a great contribution to the NAS community. The toy example constructed in section 3 Figure 3 also evidences this explanation, though the theoretical part only discussed a simple case with two operations. 
 

# Weakness
I am looking forwards to hearing back from the authors and improve my scores if these issues are fixed/explained. 

## Effectiveness of the method 
The proposed solution is simple yet effective, though there is not another analysis of why discretization and fine-tune will triumph except comparison with baselines. On page 6 we can see the perturbation based method, test accuracy drops for both your method and the baseline after 20 epochs. Does it mean the current approach is still not working well? Training for longer simply destroys the search and without proper training, the network is essentially in a random search state, i.e. it goes back to the game that DARTS do not out-perform random search. This trend is consistent over three datasets. 

## Other questions

- Ablation on the finetuning. In the algorithm part, you mentioned to train for a few epochs, but do you have some rule for that? I saw you doing this on DARTS space (Figure 6), but will this be more reasonable on NASBench-201?

- Table 1:  Will it because the swapped edges belong to the same node? For example, you could control the swap that it only happens, e.g. edge 0->2 and 2->4?


- Discretization accuracy is not clearly defined. Say for the operation weights in DARTS, [0.2, 0.4, 0.2, 0.2], does this mean you will directly evaluate the accuracy over [0, 1, 0, 0] given the supernet? Or first, make it one-hot and train to convergence while using the original super-net as a warmup? (I think I found it on page 3 after reading it again but please also help me to confirm.)

- Figure 1: NAS methods are not stable over one set of training, could authors provide another visualization, say training the darts super-net for multiple times, and shows the average of these training the alpha and its discretization accuracy? Showing three selected edges is not convincing.

## suggestions on the presentation
- Section 2.1, 2.2 and 2.3 can be a `\paragraph` instead of subsections. They are too short.
- Grammar issues
	- Abstract: one of the most ... methods (s)

review_title: Very interesting analysis on DARTS skip-connection problem, but with some issues on the proposed method

final_score: rating: 7: Good paper, accept
review_content: Pros
- The work analyzes the differential NAS methods from a new perspective that the value alpha is not suitable for selecting edges. Based on this observation, the author proposes a new method to evaluate the edge strength.
- The motivation is clear and the finding of the residual path is interesting.
- The proposed method is effective and shows good results.

Cons
- All experiments are based on the Cifar10 dataset, the author may consider extending the proposed method to a larger dataset such as ImageNet.
- The efficiency of the proposed method, what is the computational cost?
- Some descriptions and figures are not very clear.  E.g., in Figure 1, what do the three figures stand for respectively?
- Except for the residual connection, can the same rules be found in other layers?

The paper is interesting and has found some values for the NAS community by rethinking the representation ability of the important factor $\alpha$. The paper may need proof-reading and do some experiments on a more widely used large-scale dataset. I tend to accept this paper.
review_title: The work analyzes the differential NAS methods from a new perspective that the value alpha is not suitable for selecting edges. Based on this observation, the author proposes a new method to evaluate the edge strength.

Meta-review: This paper proposes a new selection paradigm for selecting the optimal architecture in neural architecture search (NAS), in particular for methods that involve a one-shot model and that deploy gradient-based methods for the search. Basically, the paper focuses on examining the max selection very closely and found the magnitude of architecture weights are misleading. Instead, the paper proposes much more intuitive finalization step, pick the operator that has the largest drop in validation if the edge is removed. All reviewers agreed that the idea is interesting, the paper is well-written, and the results found in the paper are interesting. In addition, author response satisfactorily addressed most of the points raised by the reviewers, and most of them increased their original score. Therefore, I recommend acceptance.
Decision: Accept (Oral)