Title: Designing Neural Network Architectures using Reinforcement Learning
Authors: Bowen Baker
Otkrist Gupta
Nikhil Naik
Ramesh Raskar
Venue: ICLR 2017
Keywords: Deep learning
Reinforcement Learning

Abstract: At present, designing convolutional neural network (CNN) architectures requires both human expertise and labor. New architectures are handcrafted by careful experimentation or modified from a handful of existing networks. We introduce MetaQNN, a meta-modeling algorithm based on reinforcement learning to automatically generate high-performing CNN architectures for a given learning task. The learning agent is trained to sequentially choose CNN layers using $Q$-learning with an $\epsilon$-greedy exploration strategy and experience replay. The agent explores a large but finite space of possible architectures and iteratively discovers designs with improved performance on the learning task. On image classification benchmarks, the agent-designed networks (consisting of only standard convolution, pooling, and fully-connected layers) beat existing networks designed with the same layer types and are competitive against the state-of-the-art methods that use more complex layer types. We also outperform existing meta-modeling approaches for network design on image classification tasks.

TL;DR: A Q-learning algorithm for automatically generating neural nets

Review discussion:
How do you position yourself to: https://arxiv.org/abs/1606.02492
"Convolutional Neural Fabrics"
Thanks for pointing us to the Computational Neural Fabrics (CNF) work. We will include a citation and comparison in our paper. CNF bypasses the architecture selection process by creating a much wider network with a homogeneous connection pattern.
In contrast, we introduce a meta-modeling method that can optimize for various architecture design metrics. Improving prediction performance in CNF would require increasing the number of parameters in the model.
On the other hand, improving prediction performance in our method would require a larger state space and/or larger number of model sampling, which is less prone to overfitting.
In addition, our method can be easily extended to optimize for other architecture design metrics, such as size, speed, and accuracy, by modifying the reward function.
Finally, in experimental comparison, we outperform the CNF model on CIFAR-10 by 0.51% with a model with 47% less parameters than the CNF counterpart. On the MNIST dataset, our method beats CNF by 0.05%, even though CNF uses distortions, and we do not.
Hello,
I am the author of CNF. This work is very interesting in its spirit but at the same time is different from our motivation in CNF. The main goal of our work is to remove the practice of hand-crafting CNN architectures. In our work, we propose fabrics which have only 2 hyper-parameters and can essentially represent and implement any existing CNN architecture. During optimization, the right architecture is recovered automatically.
The method proposed here is similar in spirit, in the sense that the optimum architecture is recovered rather than being hand-crafted.
However, the search space is constrained manually with rules which might not generalize across different datasets.
For example, the number of filters in a convolution layer, or to have only two fully connected layers, etc.
Hence, even though the right architecture can be recovered automatically, the process of specifying the search space is manual in nature. Ofcourse, as argued in the paper, this is not limitation of the method which is proposed here. One can specify a larger space, but this comes at a steep price in computational cost.
To illustrate, fabrics were trained in a day or two for MNIST and CIFAR-10 on a single GPU.
In comparison, this work uses 10 GPUs and train for 8-10 days. Looking at the computational time and the performance gain, one can see the advantages proposed by the fabric, where exponential amount of architectures are evaluated in parallel on a single GPU, thanks to the factorization of CNN models in a fabric.
Despite the differences, this work has very interesting ideas which are orthogonal to fabric.
Very interesting work!
Some questions below:
What is the size of the state space (approximately)?
Can you define the action space more formally? What is the size of the action space (is it a fixed dimension)?
For example, a state of type convolution (C) "can transition to any other layer type, but each layer type has a different number of hyperparameters. How do you represent the action as a fixed dimensional vector?
How stable is the proposed Q-learning process? Have you measured the standard deviation from multiple independent runs?
What is the network training objective if softmax is not the top layer?
How practical is the method for large-scale neural network design (e.g., for ImageNet classification) in terms of time consumption? Is the network designed on CIFAR transferable to a larger scale (ImageNet)?
a. What is the size of the state space (approximately)?
RESPONSE: The upper bound on state space size is a multiplication of #layer types, #representation bins, and the maximum layer depth. For the state space described in our paper, the MNIST and SVHN experiments had roughly 700 states.  For the CIFAR-10 experiment, where we increased the maximum depth to 18, there were roughly 1200 states.
b. Can you define the action space more formally? What is the size of the action space (is it a fixed dimension)?
For example, a state of type convolution (C) "can transition to any other layer type, but each layer type has a different number of hyperparameters. How do you represent the action as a fixed dimensional vector?
RESPONSE: Actions take the form of a  6-dimensional tuple with dimensions (type, # receptive fields, receptive field size, stride, fully connected size, terminate). “Type” refers to the layer type, namely ‘conv’, ‘pool’, ‘fully connected’, and ‘global average pooling’. For each layer type, the  unnecessary variables in the action tuple were set to zero  (because for example a fully connected layer does not have a stride parameter). “# receptive fields” is used when the agent is choosing a convolutional layer and refers to how many independent kernels are learned at that layer. “Receptive field size” is  used when the agent is choosing a convolutional or pooling layer and refers to the size of the kernel learned (i.e. 1x1, 3x3, or 5x5). “Stride” is only used for pooling layers (because we set all convolutional layers to have stride 1). “Fully connected size” refers to the number of neurons in the chosen fully connected layer.
Finally, the agent selects “terminate” = 1 if it wishes to end the architecture with a softmax classification layer. If the agent is currently at a ‘global average pooling’ state then it is forced to terminate, which is why we also refer to the ‘global average pooling’ state as a termination state.
The number of state-action pairs in the MNIST and SVHN experiments is roughly 4300, and the number of state-action pairs in the CIFAR-10 experiment is roughly 6900.
c. How stable is the proposed Q-learning process? Have you measured the standard deviation from multiple independent runs?
RESPONSE: Due to resource constraints, we were not able to perform multiple runs for the state space described in the paper (each run takes roughly 100 GPU days). We will include a scaled-down experiment with 10 runs on a smaller state-action space and a smaller dataset to demonstrate stability. Note that the agent was able to improve its model selection performance on three independent experiments: CIFAR-10, SVHN, and MNIST.
d. What is the network training objective if softmax is not the top layer?
RESPONSE: For experiments in this paper, which all involve classification, the agent must always end with a softmax layer, even if the agent selects a ‘global average pooling’ state.
e. How practical is the method for large-scale neural network design (e.g., for ImageNet classification) in terms of time consumption? Is the network designed on CIFAR transferable to a larger scale (ImageNet)?
RESPONSE: The timescale of our method scales linearly with the training time of individual networks. We estimate that, with the state space explored in this paper, our current implementation will take 1000 GPU days on the Imagenet classification dataset.  In the paper, we demonstrate the transferability of the top CIFAR-10 model on CIFAR-100, which has a larger number of classes. We will include the performance of this network on the Imagenet dataset in an updated version soon.
We have completed 5 independent runs on a reduced SVHN dataset (10% of training data) and a smaller state space (max layer depth of 8 instead of 12). The table below shows the mean and standard deviation of accuracy across independent experiments for each epsilon. The mean accuracy improves as epsilon decreases, and the standard deviation at each epsilon is low. We will include detailed results of 10 independent runs in the paper once they are completed.
Epsilon                    1.0       0.9     0.8       0.7     0.6      0.5      0.4      0.3      0.2     0.1
Mean Accuracy     0.489  0.510  0.531  0.548  0.578  0.620  0.659  0.691  0.725  0.769
Std Dev                 0.014  0.040  0.027  0.038  0.024  0.031  0.017  0.019  0.013  0.021
- Tables with top architectures in appendix are hard to read, it would be great to present them visually, in a similar style to Inception graphics.
- A qualitative analysis of the difference between the best existing human-optimized architecture would make a much stronger case: are there some common patterns the Qlearning finds over human designed architectures? are they very similar to human architectures which are then already optimal? What lessons can human designers learn from the machine designer? Do learned models tend to be deeper, wider, etc?
- Have you considered adding penalties in the rewards for adding expensive layers, for too long running time, for too much depth or width?
a. Tables with top architectures in appendix are hard to read, it would be great to present them visually, in a similar style to Inception graphics.
RESPONSE: Please see our project website (https://bowenbaker.github.io/metaqnn/) for a visual representation of top architectures. We are happy to include these in the appendix.
b. A qualitative analysis of the difference between the best existing human-optimized architecture would make a much stronger case: are there some common patterns the Qlearning finds over human designed architectures? are they very similar to human architectures which are then already optimal? What lessons can human designers learn from the machine designer? Do learned models tend to be deeper, wider, etc?
RESPONSE: We found that there were more dissimilarities than similarities between the top agent designed architectures and optimal human designed architectures in all experiments. The agent designed networks showed irregular patterns of `conv’ and `pool’ layers, in contrast to human designed architectures, which have a regular pattern of `conv-pool' layers, or all `conv' layers. The agent chose wider networks in general, preferring the 5x5 filter sizes over 3x3 and 1x1 (Figure A4-(b,d,f)).
As to the depth of our architectures, the top 5 agent designed models for CIFAR-10 have an average depth of 8.8 layers, which is shorter than all human designed optimal models compared in this paper. For SVHN, the average depth of top 5 agent designed models is 12.6, which is comparable to human designed optimal models. Interestingly, in the SVHN experiment, we found that the agent was likely to pick 1x1 convolutional filters for the first layer of several top models, which can be interpreted as a learnable color transformation---similar in spirit to preprocessing of input data from RGB to a different color spaces such as YUV (Sermanet et al., 2012; 2013).
Finally, Section D and Figure A4 of the appendix provides further analysis on network design.
c. Have you considered adding penalties in the rewards for adding expensive layers, for too long running time, for too much depth or width?
RESPONSE: We agree with the reviewer that such constraint-based network design---which optimizes parameters such as size, speed, and accuracy---would be a great future direction.
While we have not yet run experiments, we discuss this idea in Section 7 of the main text.
Authors learn deep architectures on a few small vision problems using Q-learning and obtain solid results, SOTA results when limiting to certain types of layers and competitive against everything else. It would be good to know how well this performs when allowing more complex structures. Paper would be much more convincing on a real-size task such as ImageNet.
Thank you for your review! Due to our limited computation resources we were not able to perform an experiment on a much larger dataset such as Imagenet or with more complex structures. We would like to stress, however, that this is a limitation of our current hardware setup but not a limitation of the method overall.
Yes I agree it is not a limitation of the method, however methods often don't transfer from the small scale to large scale problems, so showing that it does is necessary in my opinion.
The paper looks solid and the idea is natural. Results seem promising as well.
I am mostly concerned about the computational cost of the method. 8-10 days on 10 GPUs for relatively tiny datasets is quite prohibitive for most applications I would ever encounter.
I think the main question is how this approach scales to larger images and also when applied to more exotic and possibly tiny datasets. Can you run an experiment on Caltech-101 for instance? I would be very curious to see if your approach is suitable for the low-data regime and areas where we all do not know right away how a suitable architecture looks like. For Cifar-10/100, MNIST and SVHN, everyone knows very well what a reasonable model initialization looks like.
If you show proof that you can discover a competitive architecture for something like Caltech-101, I would recommend the paper for publication.
Minor:
- ResNets should be mentioned in Table
Thank you for your review!
Computational cost:  The computational cost of our current implementation might be prohibitive for a single researcher with minimal resources, but it is still within the reach of academic or industry research groups. In the foreseeable future when the increase in deep learning computational resources may outstrip the number deep learning engineers/researchers, we believe that metamodeling methods for deep learning will become extremely useful. We are also exploring efficient Q-function approximation methods (as used in Minh et al., Nature, 2015, for example) to reduce the computational cost.
Performance on  tiny datasets and  larger images:  During the rebuttal period, we conducted two experiments to demonstrate that our method is applicable to tiny datasets and  larger images.
First, we used our method to discover network architectures using 10% of the SVHN dataset (containing only 7000 training images) and identified a network architecture with 87.9% validation accuracy, which is only 7.5% less than the best model from the full SVHN experiment (both with short training schedules).
Second, we ran an experiment with the Caltech-101 dataset, according to your suggestion, using the same state-action space as the SVHN experiment. Our best model (a three-layer network) obtained an accuracy of 29.64% over 10 train-test splits when training from scratch on 224x224 size images. We compared our results with Alexnet (trained from scratch with the procedure described by Zeiler and Fergus, ECCV 2014), which obtained an average accuracy of 15.85% over the same train-test splits.
Models for standard datasets: You mention that “For Cifar-10/100, MNIST and SVHN, everyone knows very well what a reasonable model initialization looks like.”  However, it’s worth noting that the top model architectures found by our method (Tables A1-A3) are quite dissimilar to the hand-crafted networks found in literature. Human designers should be able to gain insights into the network design process by analyzing metamodeling methods.
Moreover, metamodeling methods can generate several top-performing models with varied architectures, which is not practical for human design.
Resnet results: Table 4 includes published resnet results. We will be happy to include latest unpublished versions of resnet in our updated manuscript.
When looking into Zeiler and Fergus, ECCV 2014 [1]. I find the following accuracy for Caltech-101 Alexnet in table 3:
Non-pretrained convnet: 15/class - 22.8 ± 1.5 | 30/class - 46.5 ± 1.7
Is this a mistake or a you referring to a different version of the paper / experiment?
Did you run the 15/class or 30/class experiment?
[1] http://www.cs.nyu.edu/~fergus/papers/zeilerECCV2014.pdf
We followed the procedure described by Zeiler and Fergus in Section 5.2 of their paper and performed the 30 images/class experiment. The exact hyperparameters and data augmentation techniques used in their paper to train the model in table 3 were unavailable.
So we trained an Alexnet (obtained from Caffe model zoo) from scratch and tried to optimize the performance with several initial learning rates, learning rate schedules, solvers, and data augmentation techniques. With the same train-test splits, our method obtained an accuracy of 29.64% in comparison to 15.85% obtained by Alexnet in this experiment.
This paper introduces a reinforcement learning framework for designing a neural network architecture. For each time-step, the agent picks a new layer type with corresponding layer parameters (e.g., #filters). In order to reduce the size of state-action space, they used a small set of design choices.
Strengths:
- A novel approach for automatic design of neural network architectures.
- Shows quite promising results on several datasets (MNIST, CIFAR-10).
Weakness:
- Limited architecture design choices due to many prior assumptions (e.g., a set of possible number of convolution filters, at most 2 fully-connected layers, maximum depth, hard-coded dropout, etc.)
- The method is demonstrated in tabular Q-learning setting, but it is unclear whether the proposed method would work in a large state-action space.
Overall, this is an interesting and novel approach for neural network architecture design, and it seems to be worth publication despite some weaknesses.
Thank you for your review!
Limited architecture design choices:  We agree that our current implementation had the weakness of somewhat limited architecture space, which we chose to constrain due to our limited computation resources. We would like to stress, however, that this is a limitation of our current hardware setup but not a limitation of the method overall.
Small state-action space and tabular Q-learning:  While our current state-action space size is fairly small---4296 state-action pairs---we do vary the station-action space size between experiments and still obtain consistent results. For both SVHN/MNIST experiments (with state-action space size 4296) and the larger  CIFAR-10 experiment (with state-action space size 6834), our meta-modeling method was able to progressively improve neural network architectures.
Moreover, we plan to experiment with efficient Q-function approximation methods  (as used in Minh et al., Nature, 2015, for example) instead of tabular Q-learning, for exploring larger state-action spaces.
We have added the results of the stability experiment (as suggested by AnonReviewer1) into the Appendix section D.1. We have also included a citation to the CNF paper and results from the latest ResNet paper.

Meta-review:
role: PC
data: comment: This paper comes up with a novel approach to searching the space of architectures for deep neural networks using reinforcement learning. The idea is straightforward and sensible: use a reinforcement learning strategy to iteratively grow a deep net graph (the space of actions is e.g. adding different layer types) via Q-learning. The reviewers agree that the idea is interesting, novel and promising but are underwhelmed with the execution of the experiments and the empirical results. 
 
 The idea behind the paper and the formulation of the problem are quite similar to a concurrent submission (https://openreview.net/forum?id=r1Ue8Hcxg) which has much higher scores. That paper formulates their reinforcement learning strategy using the REINFORCE algorithm while this one uses Q-learning. The major discrepancy between the papers is in the execution of the experiments. This paper had a much more restricted set of experiments on a smaller search space and thus had less impressive results. The other paper explored a much larger space of architectures and explored recurrent neural networks, achieving state-of-the-art on multiple tasks and finding exciting novel architectures that challenge current standard models (LSTMs). As a result that paper received three 9's and this one three 6's.
 
 The authors argue that they had almost the same idea but did not have access to the same amount of computing resources as the other paper. That certainly is warranted as the other paper used an *extreme* amount of compute resources for their experiments. 
 
 Given the average scores of ICLR and target acceptance rate, three 6s should normally be considered a reject. Specifically, in the absence of the other paper this would be rejected. However, the authors argue that this would be unfair, since the major difference was simply in compute resources. A major concern of the reviewers was that this approach was too computationally expensive, which is strongly contradictory to the scores for the other paper. In reality, the other paper really is just much more interesting because of the empirical results. However, I sympathize with the authors as the other paper essentially demonstrates that their idea is sensible and effective (albeit only with obscene compute resources), and the paper would effectively be "scooped" from further submissions.
 
 (Note a contrarian viewpoint might take an analogy from other fields. e.g. experimental physics papers are likely rejected if they don't have multi-billion dollar equipment to run the right set of exhaustive experiments, which is the difference between e.g. the large hadron collider and most university depts.)
 
 Pros:
 - The paper proposes a novel approach to architecture search using reinforcement learning
 - The approach is technically sound and interesting
 - The authors achieve good results automatically on benchmark tasks using their approach
 - They empirically demonstrate that their method works (via plots demonstrating the behavior of the algorithm while varying how exploratory the algorithm is)
 
 Cons:
 - The search space is very constrained (too much for us to learn anything interesting from the results)
 - The experimental results are good but underwhelming
 - Compared to e.g. Bayesian optimization methods, this approach seems very wasteful in compute resources (e.g. 1500 runs of training before the approach starts exploiting).
 
 The Program Chairs have also reviewed this paper, and taken everything into account, have reccommended this paper for poster presentation at the main conference.

Decision: Accept (Poster)