Venue: ICLR 2018 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: In the paper titled "Faster Discovery of Neural Architectures by Searching for Paths in a Large Model", the authors proposed an efficient algorithm which can be used for efficient (less resources and time) and faster architecture design for neural networks. The motivation of the new algorithm is by sharing parameters across child models in the searching of archtecture. The new algorithm is empirically evaluated on two datasets (CIFAR-10 and Penn Treeback) --- the new algorithm is 10 times faster and requires only 1/100 resources, and the performance gets worse only slightly.

Overall, the paper is well-written. Although the methodology within the paper appears to be incremental over previous NAS method, the efficiency got improved quite significantly. 
 
review_title: an incremental work, but the results are not bad

final_score: rating: 5: Marginally below acceptance threshold
review_content: In this paper, the authors look to improve Neural Architecture Search (NAS), which has been successfully applied to discovering successful neural network architectures, albeit requiring many computational resources. The authors propose a new approach they call Efficient Neural Architecture Search (ENAS), whose key insight is parameter sharing. In NAS, the practitioners have to retrain for every new architecture in the search process, but in ENAS this problem is avoided by sharing parameters and using discrete masks. In both approaches, reinforcement learning is used to  learn a policy that maximizes the expected reward of some validation set metric. Since we can encode a neural network as a sequence, the policy can be parameterized as an RNN where every step of the sequence corresponds to an architectural choice. In their experiments, ENAS achieves test set metrics that are almost as good as NAS, yet require significantly less computational resources and time.

The authors present two ENAS models: one for CNNs, and another for RNNs. Initially it seems like the controller can choose any of B operations in a fixed number of layers along with choosing to turn on or off ay pair of skip connections. However, in practice we see that the search space for modeling both skip connections and choosing convolutional sizes is too large, so the authors use only one restriction to reduce the size of the state space. This is a limitation, as the model space is not as flexible as one would desire in a discovery task. Moreover, their best results (and those they choose to report in the abstract) are due to fixing 4 parallel branches at every layer combined with a 1 x 1 convolution, and using ENAS to learn the skip connections. Thus, they are essentially learning the skip connections while using a human-selected model. 

ENAS for RNNs is similar: while NAS searches for a new architecture, the authors use a recurrent highway network for each cell and use ENAS to find the skip connections. Thus, it seems like the term Efficient Neural Architecture Search promises too much since in both tasks they are essentially only using the controller to find skip connections. Although finding an appropriate architecture for skip connections is an important task, finding an efficient method to structure RNN cells seems like a significantly more important goal.

Overall, the paper is well-written, and it brings up an important idea: that parameter sharing is important for discovery tasks so we can avoid re-training for every new architecture in the search process. Moreover, using binary masks to control network path (essentially corresponding to training different models) is a neat idea. It is also impressive how much faster their model performs on tasks without sacrificing much performance. The main limitation is that the best architectures as currently described are less about discovery and more about human input -- finding a more efficient search path would be an important next step.
review_title: Improving Neural Architecture Search by Parameter Sharing

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary: 
The paper presents a method for learning certain aspects of a neural network architecture, specifically the number of output maps in certain connections and the existence of skip connections. The method is relatively efficient, since it searches in a space of similar architectures, and uses weights sharing between the tested models to avoid optimization of each model from scratch. Results are presented for image classification on Cifar 10 and for language modeling.

Page 3: “for each channel, we only predict C/S binary masks”   -> this seems to be a mistake. Probably “for each operation, we only predict C/S binary masks” is the right wording
Page 4: Stabilizing Stochastic Skip Connections: it seems that the suggested configuration does not enable an identity path, which was found very beneficial in (He. et al., 2016). Identity path does not exist since layers are concatenated and go through 1*1 conv, which does not enable plain identity unless learned by the 1*1 conv.
Page 5: 
-	The last paragraph in section 4.2 is not clear to me. What does a compilation failure mean in this context and why does it occur? And: if each layer is connected to all its previous layers by skip connections, what remains to be learned w.r.t the model structure? Isn’t the pattern of skip connection the thing we would like to learn?
-	Some  details of the policy LSTM network are also not clear to me:
o	How is the integer mask (output of the B channel steps) encoded? Using 1-hot encoding over 2^{C/S} output neurons? Or maybe C/S output neurons, used for sampling the C/S bits of the mask? this should be reported in some detail.
o	How is the mask converted to an input embedding for the next step? Is it by linear multiplication with a matrix? Something more complicated? And are there different matrices used/trained for each mask embedding (one for 1*1 conv, one for 3*3 conv, etc..)?
o	What is the motivation for using equation 5 for the sampling of skip connection flags? What is the motivation for averaging the winning anchors as the average embedding for the next stage (to let it ‘know’ who is connected to the previous?). Is anchor j also added to the average?
o	How is equation 5 normalized? That is: the probability is stated to be proportional to an exponent of an inner product, but it is not clear what the constant is and how sampling is done.

Page 6: 
-	  Section 4.4: what is the fixed policy used for generating models in the stage of training the shared W parameters? (this is answered at page 7
Experiments:
-	The accuracy figures obtained are impressive, but I’m not convinced the ENAS learning is the important ingredient in obtaining them (rather than a very good baseline)
-	Specifically, in the Cifar -10 example it does not seem that the networks chooses the number of maps in a way which is diverse or different from layer to layer. Therefore we do not have any evidence that the LSTM controller has learnt any interesting rule regarding block type, or relation between block type width and layer index. All we see is that the model does not chose too many maps, thus avoid significant overfit. The relevant baseline here is a model with 64 or 96 maps on each block, each layer.Such a model is likely to do as well as the ENAS model, and can be obtained easily with slight parameter tuning of a single parameter.
-	 Similarly, I’m not convinced the  skip connection pattern found for Cifar-10 is superior to standard denseNet or Resnet pattern. The found configuration was not compared to these baselines. So again we do not have evidence showing the merit of keeping and tuning many parameters with the RINFORCE
-	The experiments with Penn Treebank are described with too less details: for example, what exactly is the task considered (in terms on input-output mapping), what is the dataset size, etc..
-	Also, for the Penn treebank experiments no baseline is given, so it is not possible to understand if the structure learning here is beneficial. Comparison of the results to an architecture with all skip connections, and with a single skip connection per layer is required to estimate if useful structure is being learnt.

Overall:
-	Pro: the method gives high accuracy results 
-	Cons: 
o	It is not clear if the ENAS search is responsible to the results, or just the strong baseline. The advantage of ENAS over plain hyper parameter choosing was not sufficiently established.
o	The controller was not presented in a clear enough manner. Many of its details stay obscure.
o	The method does not seem to be general. It seems to be limited to choosing a specific set of parameters in very specific scenario (scenario which enable parameter sharing between model. The conditions for this to happen seems to be rather strict, and where not elaborated).

After revision:
The controller is now better presented.
However, the main points were not changed:
   - ENAS seems to be limited to a specific architecture and search space, in which probably the search is already exhausted. For example for the image processing network, it is determining the number of skip connections and structure of a single layer as a combination of several function types. We already know the answers to these search problems (denser skip connection pattern works better, more functions types  in a layer in parallel do better, the number of maps should be adjusted to the complexity and data size to avoid overfit). ENAS does not reveal a new surprising architectures, and it seems that instead of searching in the large space it suggests, one can just tune a 1-2 parameters  (for the image network, it is the number of maps in a layer).
  - Results comparing ENAS results to the simple baseline of just tuning 1-2 hyper parameters were not shown. I hence believe the strong empirical results of ENAS are a property of the search space (the architecture used) and not of the search algorithm.
review_title: Nice idea, but the paper suffers from unclear presentation and the empirical results are not convincing enough.

Meta-review: First off, this was a difficult paper to decide on. There was some vigorous discussion on the paper centering around the choices that were available to the conv-nets.  The author's strongly emphasized the improvements on the PTB task.

For my part, I think the method is very compelling -- sharing weights for all the models we are optimizing on seems like a great idea -- and that we can make it work is even more interesting. So from this point of view, I think its a novel contribution and worth accepting.

On the other hand, I'm likely to agree with some of the motivations behind the questions raised by R3. Are all the choices really necessary ? perhaps the gains came from just a couple of things like number of skip connections and channels, etc. That exploration is useful. On the flip side, I think it may be an irrelevant question -- the model is able to make the correct decisions from a big set.

The authors emphasize the language modelling part, but for me, this was actually less compelling. The authors use some of the tricks from Merity in their model training (perplexity 52.8), and as a result are already using some techniques that produces better results. Further, PTB is a regularization game -- and that's not really the claim of this paper. Although, one could argue that weight sharing between different models can produce an ensembling / regularization effect and those gains may show up on PTB. A much more compelling story would have been to show that this method works on a large dataset where the impact of the architecture cannot be conflated with controlling overfitting better.

As a result, this puts the paper on the fence for me; even though I very much like the idea. Polishing the paper and making a more convincing case for both the CNNs and RNNs will make this paper a solid contribution in the future.
Decision: Invite to Workshop Track