Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: The paper is well written and flows very well. The idea is straightforward and easy to understand. Here are my feedbacks:

--Method--

Methodology-wise, the novelty is somewhat limited. The main technical contributions are: 1) formulate linear programming to reduce energy costs, and the formulation of linear programming is straightforward. 2) the claimed main contribution is an application of Rayleigh-Quotient gradient descent to approximate the eigenvalue calculations in the model proposed by Wu et al (2019). 

I believe computing the eigenvalue shall not be the only way to solve Eq.(3), e.g. using gradient-based method or Lagrange. My main question is whether computing the global optimum really matters? Since your method is still essentially an approximation algorithm, which may break the optimality condition here. From the experimental results, it seems that your approximation is quite close to the analytical solution. Therefore, it is unclear computing a global optimum really matters here.

I tried to buy the idea of escaping the saddle point with splitting (it indeed sounds straightforward and reasonable). It will be great if the author can add some empirical experiments to verify it.

--Content--
The entire section2 is at reviewing prior works, and I believe you should cut down the content here.

--Experiments--
1. Diversity of your tests: the author main uses MobileNet in testing their ideas. It seems that their method starts at MobileNet, then tries to improve it. I suggest authors adding other recent works, e.g. FBNet, MobileNetV3, into their tests to diversify the types of networks. 

2. Results variations: all figures, e.g. fig.3, fig.4, in the paper lack plotting their results variations. Since the optimization may converge to different local optimum, it is more persuasive to show their performance variations.

3. The final results are not surprising: in table.1 and table.2, as far as I'm aware, the mainstream accuracy for ImageNet under the mobile setting should be 75% top-1. And the SOTA top-1 accuracy on ImageNet is around 85.5%. Table.1 and table.2 do somewhat show the effectiveness of your method, but its significance is limited especially considering the limited novelty of methodology.

Minors:
In Fig.3 k = 6, it seems vanilla splitting is better than accuracy and parameters, except for 0.2 log higher flops. I don't think this makes a compelling case here.

In Fig.4, from flops 7 ~ 9, your results are similar to Bn especially accuracy > 0.6. When accuracy < 0.6, it is less interesting, and I believe the improvement should be huge.

Fig.5, could you please compare the time using MAGMA from NVIDIA? I had some experiences in implementing the LAPACK and BLAS on GPUs, and it should not be that slow.

Overall, this paper has some interesting results, which shows the eigenvalue can be approximated by Rayleigh-Quotient gradient descent, and show positive improvement. However, the methodological and experimental results can definitely be strengthened. The author may consider proving that achieving the global optimum on Eq.(3) really matters, then motivate the methodology. Thank you. 

review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: Summary:
This paper builds on a recently proposed algorithm ("splitting steepest descent", Wu et al 2019) for guiding the growth of a smaller network into a larger one in architecture search. The algorithm in Wu, et al. alternates between two steps, (i) optimization of parameters for a fixed model and (ii) modification of the architecture by identifying a subset of neurons to split into more neurons, based on the "splitting index" of each neuron (amounting to evaluating the smallest eigenvalue of a matrix). This work builds on that in two ways: (i) it incorporates an energy budget into the optimization procedure for choosing which subset of neurons to split, which it approximately solves by a continuous relaxation, and (ii) avoids doing exact eigendecomposition to extract the minimum eigenvalue (splitting index) but instead replaces it with a more efficient SGD on the Rayleigh quotient. 

Evaluation:
--Evaluations are done on variants of MobileNet on CIFAR-100 and ImageNet (the latter would be infeasible without the approximation scheme). The proposed approach appears to get better tradeoff between accuracy and FLOPs in these cases. In practice the non-energy aware "vanilla" networks do tend towards models that are small in size (fewer parameters) but are not necessarily low in energy consumption.
--There is new material here, although I find the novelty a bit limited (e.g. only an additional constraint compared to the original approach of Wu et al and addressing a clear scalability issue with the original work, i.e. eigendecomposition of a matrix, with what seem straightforward approximations, ). The empirical results in Table 1 and 2 seem solid, but I'm not familiar enough with past results in this area  to evaluate their significance. 
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This paper is based on a prior work which proposed Splitting Steepest Descent to search better network structures via splitting existing neurons to multiple off-springs. As an improvement, the authors (1) incorporate the energy cost to better guide the splitting process and (2) reduce the time and space complexity by approximating the original computation process with Rayleigh-Quotient Gradient Descent. They conduct experiments on public image classification datasets using lightweight networks as backbones to show that their algorithm outperforms existing methods. The paper is well written and easy to follow. The experiments are comprehensive and the evaluation results shows good properties of proposed method.

In brief, this paper is an improvement to a splitting algorithm in a previous work, achieving good efficiency and enabling application on large datasets. However, the theory needs more justification and the experiments results are not sufficient to show the significance of their contribution. Therefore, this paper may not be accepted unless more experiment results are given.

For the theory & modeling, the following should be addressed.
1.	The mathematical justification of the optimization on energy cost is not very sound, and the definition of optimal splitting set seems arbitrary.
2.	Given that the experiments are conducted on convolutional networks, it would be more illustrative if the paper describe the process of applying the algorithm on common convolutional operators.
3.	The novelty is limited by the prior work.

For the experiment, the following should be addressed.
1.	The experiments are mainly conducted on MobileNet network. It would be more convincing if more experiment is done on other lightweight or normal convolutional networks.
2.	This paper reduces time and space complexity of the algorithm in a previous work, but there is no running time or memory footprint statistics to support this argument.
3.	The paper only lists one pruning-based method as comparison in the experiments. It would be more convincing if more pruning and splitting methods are presented.

review_title: Official Blind Review #1

Meta-review: This paper extends previous work on searching for good neural architectures by iteratively growing a network, including energy-aware metrics during the process. There was discussion about the extent of the novelty of this work and how well it was evaluated, and in the end the reviewers felt it was not quite ready for publicaiton.
Decision: Reject