Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 3: Clear rejection
review_content: Title: FTSO: EFFECTIVE NAS VIA FIRST TOPOLOGY SECOND OPERATOR

Summary of the paper:

The goal is to fit the hyper-parameters of the neural network namely topology (i.e. network architecture) and operator (e.g. skip connection or convolution) rather than just fit the parameters (i.e. weights). The approach is similar to DARTS which relaxes the architecture choice to obtain a continuous optimisation.

The main difference from DARTS is that rather than jointly selecting operator and topology, first the topology and then the operator is selected. It's roughly a form of coordinate descent. 

The paper is empirical in nature and claims good results. 

Pros:

The idea of optimising topology then operator appears to work well in practice. 

Cons:

The presentation is far from top tier conference standard. 

Examples include, this sentence from the introduction (yes, it's one sentence): 

"We first mathematically prove that, by greatly shrinking the graph of the search space, reducing the operators’ complexity in magnitude, lowering the required searching period from 50 epochs to one iteration and significantly easing the Matthew effect, namely that the complex operators may never get the chance to be well tuned, thus the found architecture only contains very simple operators, and performs poorly on the testing set, FTSO reduces the required parameters by a factor of 0.53×108, decreases the FLOPs per iteration by a factor of 2×105 and significantly promotes the accuracy compared to the baseline, PC-DARTS."

(Anyway, the "mathematical proof" aluded to is an informal argument for an example, essentially saying that searching one coordinate then the other is cheaper than searching both jointly.)

Also figures 2 and 3 are unreadable on printout. 

Also, there is no code to check the results. This is a significant factor considering the empirical nature.

Recommendation:

The paper is surely interesting to some, but needs a lot more work.
review_title: inadequate presentation quality

final_score: rating: 5: Marginally below acceptance threshold
review_content: This work researches the issue of neural architecture search (NAS), which is of significance for practical applications of deep neural networks and has become an active research topic in the past several years. Many methods on NAS have been developed recently. The computational efficiency of search has been one of the obstacles for this line of research.

Strengths of this work: 

The key idea of this work is to decouple the search of network topology and the search of operators. By doing so, the computational efficiency can be substantially increased (say, reducing the search time from several GPU-days to less than one second), while well maintaining the classification performance or even slightly improving it. Experimental study on CIFAR10 and ImageNet demonstrates the advantage of the proposed method, especially the improvement on computational cost. The paper is overall well written.   

Weaknesses:

1. This work can do better on theoretical analysis. Currently, it mainly describes how the decoupled search (FTSO) is implemented (in Section 3). Considering the significance of the improvement on computational efficiency, it will be more valuable to discuss whether such a kind of decoupled search can be generally applied, or it can only work for certain kinds of network architecture, data, or tasks. This will provide more insights on this interesting method and theoretical contribution.

2. This paper has described several interesting observations. It will be better if they are further explained and discussed. 

2-1. When conducting topology search, this paper only uses a simple operator (say, skip connection). It is indicated that this supernet can hardly overfit the dataset, which is understandable. Meanwhile, with such a simple operator, will the supernet underfit the data? This needs to be clarified.

2-2. Similarly, in Section 4.1, it is stated that FTSO contains very few parameters and therefore can even achieve comparable performance to PC-DARTS. Why could very few parameters lead to this comparable performance? Please clarify it.

2-3. At the end of Section 4.2, it is found that the best architecture obtained by the search is the shallowest and widest one. The current explanation (provided at the end of this Section) is too brief and vague for such an interesting and surprising finding. This need to be further clarified. 

2-4. The part at the top of page 7 discusses the performance with respect to iteration and epoch. It is not very clear. It is stated that "although one iteration cannot surpass one epoch, it is better than a few iterations." However, considering that one epoch consists of multiple iterations, this statement seems to contradict to itself and is a bit hard to follow. Please clarify.   

3. The second paragraph on page 4 compares the computational complexity of DARTS and the proposed method. Since this is the key part of this work, more details shall be provided on how the complexity is worked out. 

4. Minor issues:

4-1. The first sentence of the second paragraph on page 2 needs to be revised.
4-2. Although the investigation of correlation among different architectures is mentioned as the key motivation for this work, how is the correlation of architectures considered in this work is not presented. This shall be improved; 
4-3. In Section 4.2, the resolution of images in ImageNet is reduced to 28x28. Does this imply that the proposed method is not effective enough to deal with images of normal size?
4-4. Figures 1 and 2 shall be enlarged.

--- Thank the authors for the detailed response. After reading the response and the comments of peer reviewers, the rating is altered as follows. 
review_title: An interesting paper with very promising result

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper proposes a method for Neural Architecture Search (NAS) with two stages of search. In the first  stage, the topology of the cell is searched with only one operator (skip connection) using graph pruning through gradient descents. In the later stage, there are two ways to search the operators. In the first approach, the found topology is equipped with some operators (e.g., 3x3 convolution,  skip connection, and 3x3 dilated convolution) and then the architecture parameters are optimized. Another approach is to replace all operators with one single operator e.g. convolution. The experiments show that the searching time is reduced significantly compared to DARTS and the results on CIFAR-10 and ImageNet are very competitive.


Strengths:

- The search time is reduced significantly from normal DARTS and PC-DARTS.
- The results are competitive to the prior differentiable NAS methods with less searching time.

Weaknesses:

- It is not very clear how exactly the topology search algorithm ensures that each node is connected?
- The choice of skip connection as the operator for topology search is not well-justified. Max-pooling has no learnable parameters too. Is there any particular reason why skip connection?
- The experiments are lacking because there is no evaluation on current benchmarks such as NAS-101, NAs1Shot1, or NAS-201 that can explicitly show how the search algorithm performs. In current experimental results, it is very difficult to judge if the proposed approach has benefits for searching 'good' architectures.
- The comparison with other differentiable NAS algorithms (e.g., DARTS and PC-DARTS) can be delivered in the form of a table. The information of this comparison is all scattered in the manuscript.
- In Section 3 "In FTSO, this problem is almost nonexistent because, the skip connection has no kernel
weights to tune", this is not very clear how the problem exists in the DARTS formulation and how the formulation in the proposed approach can mitigate this issue. There is no discussion about the bi-level program in DARTS.
- It is very difficult to follow the overall approach, it would be better to have an algorithm/pseudocode.
- ", we take the second strategy as the default configuration because, it is much more efficient, and the large amount of kernel weights, contained in the first strategy, may lead to over-fitting,
and finally leads to worse performance than the second strategy. ". How do exactly large amount of kernel weights lead to overfit in NAS? Is there any proof/citation about it? The second strategy has no clear evidence to always give better results compared to the first strategy.
- There is no empirical results how the algorithm works by varying the number of nodes in a cell. This will show that the limit of the proposed approach and also its capacity to handle various number of nodes.

- The manuscript is poorly written,  at least these sentences/paragraphs must be clarified and revised:
** In Section 3,  "it might be possible to cluster the architectures according to their connection topologies"
** In Subsection 4.3, "When we do not search for the operators, after the topology search, we assign all the remaining
edges a fixed operator" and the rest of the paragraph is difficult to follow.
** In Section 3, "Secondly. Because our supernet contains very few parameters, it can hardly over-fit the dataset. Thus, the found architecture may generalize better on new datasets.". This sentence is a claim without any evidence.
** Table 3 is not clear, some codes are not described in its caption.
** In Section 3, please fix "Firstly. It allows the algorithm to converge within extremely few iterations... and so on.."
review_title: The paper needs more experiments and the claims are not well-justified by either empirical evidences or theories.

Meta-review: Three reviewers have reviewed this manuscript, and they had severe reservations regarding the presentation quality and the lack of sufficient theoretical support behind empirical observations. Even after rebuttal, the reviewers maintained that the above issues are not fully resolved. Unfortunately, this paper cannot be accepted in its current form.
Decision: Reject