Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: this paper presents the Surgeon, a ANN/evolutionary algorithm hybrid optimization de- signed for neural architecture search. On SVHN and CIFAR-10, the network generated by the Surgeon is able to outperform the baseline in case of suboptimal topologies, or reach comparable accuracies while pruning the underlying network structure to less resource-intensive topologies. My major concern is the novelty, as the SVD technique and net2net techniques have all be proposed before. The experiment is not solid. The accuracy on Cifar is far from the state of the art. There is no large scale datasets. There's no performance comparison with related work, making the paper less convincing. 
review_title: novelty & experiment

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper presents a neural network training framework based on a genetic algorithm called Surgeon. The idea consists of two modules. First, a series of network structure changes that aims to minimize the network input and output. Second, a heuristic (with regard to the objective function) for ranking and selecting the potentially good network modifications. The algorithm works iteratively between the former propose candidate network changes, and the later decides which candidate to accept and evolve into.

The paper is well written and easy to follow. As the authors stated, the paper is the first of the series of work. The framework presented is rigorous and flexible. I am fairly optimistic that the idea itself is pointing to a promising direction for exploring the neural network structure. However, I think the theoretical basis and experiments are a bit weak in their current form.

I have two main concerns. First, the heuristic used by Surgeon, either by weights or by BIC, does not establish direct association to the objective function that the network is optimizing. Therefore there is no any form of guarantee that the evolution will converge to some optima. Second, the experiments are initiated from rather simple baseline, without comparing against more sophisticated work or discussing why other state-of-the-art results are not relevant in this context.

Overall I like this paper, but for the two reasons listed above I think it may fall short to be accepted.
review_title: Good framework, weak theoretical ground and experiments 

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper seems to devise a set of modification modules and develops an overall genetic algorithm that utilizes the recommendation module to identify optimized topologies for networks.

The paper is a combination of genetic algorithm with recommendation rule which builds on some tools like BIC. While these tools are organized in a reasonable manner, the overall idea seems rather simplistic and the contributions seem rather humble.

Importantly, the main concerns about the paper revolves around the evaluation. For example, the surgeon is tested on very small networks. As such, the evaluation seems very primitive. Without the results of networks for large datasets like ImageNet, it is hard to give credit to their evaluation nor gain confidence about the performance of this method in real scenarios.

Also, it would be very interesting to see the evolution of the topology. Maybe some figures may help the readers visualize the overall procedure. While this may be excessive for small networks like the ones authors rely on, I believe this may lead to new and interesting observations.

Furthermore, it would be very inspiring if the paper presents some prospect of the work in improving the mainstream neural architecture search algorithms. I believe such evaluation with some empirical results would make this a much more relevant paper for people working on improving neural architecture search.
review_title: Official Blind Review #2

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary:
This study aims to search for topology jointly with the network training. The topology is optimized by a heuristic search in structural modifications including adding/removing neurons/layers. Experiments on SVHN and CIFAR-10 are conducted to show the effectiveness. 

Pros:
The paper is well organized. The proposed method enjoys good interpretability because the decision of modification is made based on the weight matrix properties by SVD and projecting onto its lower-rank subspaces. 

Cons:
1.	The major problem of this study is its extendibility. The method is only tested on fully-connected networks with very limited network depths and layer sizes. When the network is deep with much more non-linear activations, is the proposed method still valid? Besides, the authors only claim that the method can be generalized to convolutional network. But there is no corresponding description about how to generalize to convolutional network and its experimental results. 
2.	The current experiments are not sufficient. The network and datasets used to test are too simple and the performance is far from being acceptable. It is weird to use fully-connected network on real-image datasets, CIFAR-10 and SVHN. Besides, the training time, computational and memory cost should also be reported and compared with the baseline method. 
3.	The novelties are limited. The methods proposed in this paper are mainly heuristic implementations. 

review_title: suggest a rejection

Meta-review: This work proposes a framework to search for the topology of an artificial neural network jointly with the network training, via a genetic algorithm that can decide structural actions, such as addition or removal of neurons and layers. An extra heuristic based on Bayesian information criterion helps the optimization process decide on its decisions about the topology. They demonstrate improvements over baseline fully-connected networks on SVHN and (augmented) CIFAR-10.

Reviewers and myself agree that this is an interesting idea, and that the paper is easy to follow. While I may not agree that we need to achieve SOTA on these datasets, or see large scale ImageNet-type experiments for novel ideas, I agree with the reviewers, esp R1's point that the current experiments are not satisfactory to meet the bar for acceptance at ICLR.

CIFAR-10 and SVHN are well-established tasks, and showing baseline accuracy of 75%/48% on them respectively doesn't seem to do them justice, especially when most methods (even with low compute requirements) can get > 95% on both, for the past few years. For this work to be of interest to the broader community, it needs to be improved to incorporate at least respectable baselines on these small datasets, and perhaps be improved to work beyond fully connected networks.

At this stage, we need to see a revision of the method and see improvements before an acceptance decision can be made.
Decision: Reject