Illustrated: Efficient Neural Architecture Search | by Raimi Karim | Towards Data Science

(TL;DR the only two animations you need to know are here)
Updated:23 Mar 2020: Erratum — Previously it was mentioned that all convolution cells in micro search are different from one another. This is wrong; the convolution cell is repeated multiple times in the final child models. This information is verified by the author. Thank you Martin Ferianc for the correction!
Designing neural networks for various tasks like image classification and natural language understanding often requires significant architecture engineering and expertise. Enter Neural Architecture Search (NAS), a task to automate the manual process of designing neural networks. NAS owes its growing research interest to the increasing prominence of deep learning models of late.
There are many ways to search for or discover neural architectures. Over the past couple of years, the community has seen different search methods proposed including:
Reinforcement learningNeural Architecture Search with Reinforcement Learning (Zoph and Le, 2016)NASNet (Zoph et al., 2017)ENAS (Pham et al., 2018)
Evolutionary algorithmHierarchical Evo (Liu et al., 2017)AmoebaNet (Real et al., 2018)
Sequential model-based optimisation (SMBO)PNAS (Liu et al., 2017)
Bayesian optimisationAuto-Keras (Jin et al., 2018)NASBOT (Kandasamy et al. 2018)
Gradient-based optimisationSNAS (Xie et al., 2018)DARTS (Liu et al., 2018)
In this post, we will look at Efficient Neural Architecture Search (ENAS) which employs reinforcement learning to build convolutional neural networks (CNNs) and recurrent neural networks (RNNs). The authors Hieu Pham, Melody Guan, Barret Zoph, Quoc V. Le, and Jeff Dean proposed a predefined neural network to generate new neural networks guided by a reinforcement learning framework using macro search and micro search (see the paper here). That’s right – a neural network building another neural network.
This article is a tutorial on how the macro and micro search strategies lead to generating neural networks. While the illustrations and animations serve to guide the readers, the sequence of animations do not necessarily reflect the flow of operations (due to vectorisation etc.).
We shall narrow the scope of this tutorial to neural architecture search for CNNs in an image classification task. This article assumes that the reader is familiar with the basics of RNNs, CNNs, and reinforcement learning. Familiarity with deep learning concepts like transfer learning and skip/residual connections will greatly help as they are heavily used in the architecture search. It is not required to have read the paper, but it would speed up your understanding.
0. Overview1. Search strategy1.1. Macro search1.2. Micro search 2. Notes3. Summary4. Implementations5. References
In ENAS, there are 2 types of neural networks involved:
Controller – a predefined RNN, which is a long short-term memory (LSTM) cell
Child model – the desired CNN for image classification
Like most other NAS algorithms, the ENAS involves 3 concepts:
Search space — all the different possible architectures or child models that can possibly be generatedSearch strategy — a method to generate these architectures or child modelsPerformance evaluation — a method to measure the effectiveness of the generated child models
Search space — all the different possible architectures or child models that can possibly be generated
Search strategy — a method to generate these architectures or child models
Performance evaluation — a method to measure the effectiveness of the generated child models
Let’s see how these five ideas form the ENAS story.
The controller controls or directs the building of the child model’s architecture by “generating a set of instructions” (or, more rigorously, making decisions or sampling decisions) using a certain search strategy. These decisions are things like what types of operations (convolutions, pooling etc.) to perform at a particular layer of the child model. Using these decisions, a child model is built. A generated child model is one of the many possible child models that can be built in the search space.
This particular child model is then trained to convergence (~95% training accuracy) using stochastic gradient descent to minimise the expected loss function between the predicted class and ground truth class (for an image classification task). This is done for a specified number of epochs, what I’d like to call child epochs, say 100. Then, a validation accuracy is obtained from this trained model.
Then, we update the controller’s parameters using REINFORCE, a policy-based reinforcement learning algorithm, to maximise the expected reward function which is the validation accuracy. This parameter update hopes to improve the controller in generating better decisions that give higher validation accuracies.
This entire process (from 3 paragraphs before this) is just one epoch — let’s call it controller epoch. We then repeat this for a specified number of controller epochs, say 2000.
Of all the 2000 child models generated, the one with the highest validation accuracy gets the honour to be the neural network for your image classification task. However, this child model must go through just one more round of training (again specified by the number of child epochs), before it can be used for deployment.
A pseudo algorithm for the entire training is written below:
CONTROLLER_EPOCHS = 2000CHILD_EPOCHS = 100Build controller networkfor i in CONTROLLER_EPOCHS:     1. Generate a child model     2. Train this child model for CHILD_EPOCHS     3. Obtain val_acc     4. Update controller parametersGet child model with the highest val_accTrain this child model for CHILD_EPOCHS
This entire problem is essentially a reinforcement learning framework with the archetypal elements:
Agent — Controller
Action — The decisions taken to build the child network
Reward — Validation accuracy from the child network
The aim of this reinforcement learning task is to maximise the reward (validation accuracy) from the actions taken (decisions taken to build child model architecture) by the agent (controller).
[Back to top]
Recall in the previous section that the controller generates the child model’s architecture using a certain search strategy. There are two questions that you should ask in this statement — (1) how does the controller make decisions and (2) what search strategy?
How does the controller make decisions?
This brings us to the model of the controller, which is an LSTM. This LSTM samples decisions via softmax classifiers, in an auto-regressive fashion: the decision in the previous step is fed as input embedding into the next step.
What are the search strategies?
The authors of ENAS proposed 2 strategies for searching for or generating an architecture.
Macro searchMicro search
Macro search
Micro search
Macro search is an approach where the controller designs the entire network. Examples of publications that use this include NAS by Zoph and Le, FractalNet and SMASH. On the other hand, micro search is an approach where the controller designs modules or building blocks, which are combined to build the final network. Some papers that implement this approach are Hierarchical NAS, Progressive NAS and NASNet.
In the following 2 sub-sections we will see how ENAS implements these 2 strategies.
[Macro search][Micro search]
In macro search, the controller makes 2 decisions for every layer in the child model:
the operation to perform on the previous layer (see Notes for the list of operations)
the previous layer to connect to for skip connections
In this macro search example, we will see how the controller generates a 4-layer child model. Each layer in this child model is colour-coded with red, green, blue and purple respectively.
Convolutional layer 1 (red)
We’ll start with running the first time step of the controller. The output of this time step is softmaxed to get a vector, which translates to a conv3×3 operation.
What this means for the child model is that the we perform a convolution with a 3×3 filter on the input image.
I know I mentioned that the controller needs to make 2 decisions but there’s only 1 here. Since this is the first layer, we can only sample one decision which is the operation to perform, because there’s nothing else to connect to except for the input image itself.
Convolutional layer 2 (green)
To build the subsequent convolutional layers, the controller makes 2 decisions (no more lies): (i) operation and (ii) layer(s) to connect to. Here, we see that it generated 1 and sep5×5.
What this means for the child model is that we first perform a sep5×5 operation on the output of the previous layer. Then, this output is concatenated along the depth together with the output of Layer 1, i.e. the output from the red layer.
Convolutional layer 3 (blue)
We repeat the previous step again to generate the 3rd convolutional layer. Again, we see here that the controller generates 2 things: (i) operation and (ii) layer(s) to connect to. Below, the controller generated 1 and 2, and the operation max3×3.
So, the child model performs the operation max3×3 on the output of the previous layer (Layer 2, green). Then, the result of this operation is concatenated along the depth dimension with Layers 1 and 2.
Convolutional layer 4 (purple)
We repeat the previous step again to generate the 4th convolutional layer. This time the controller generated 1 and 3, and the operation conv5×5.
The child model performs the operation conv5×5 on the output of the previous layer (Layer 3, blue). Then, the result of this operation is concatenated along the depth dimension with Layers 1 and 3.
End
And there you have it — a child model generated using the macro search! Now on to micro search.
[Back to top]
Micro search designs only one building block whose architecture is repeated throughout the final architecture. ENAS calls this building block a convolutional cell and reduction cell. Both are similar — the only thing different about reduction cells is that the operations are applied with a stride of 2, thus reducing the spatial dimensions.
The idea of micro search is to build one single architecture for a convolutional cell and repeat this same architecture through out the final model. In the example below, the final model consists of a convolutional cell whose architecture is repeated N times across 3 blocks. As mentioned in the previous paragraph, the reduction cell is similar to the architecture of the convolutional cell.
Let’s come back to this in a bit.
Building units for networks derived for micro search
There’s sort of a hierarchy in the ‘building units’ of child networks derived from micro search. From biggest to smallest:
block
convolutional cell / reduction cell
node
A child model consists of several blocks. Each block consists of N convolutional cells and 1 reduction cell. Each convolutional/reduction cell comprises B nodes. And each node consists of standard convolutional operations (we’ll see this later). (N and B are hyperparameters that can be tuned by you, the architect.)
Below is a child model with 3 blocks. Each block consists of N=3 convolutional cells and 1 reduction cell. The operations within each cell are not shown here.
So how to generate this child model from micro search, you may ask? Continue reading!
Generate a child model from micro search
For this micro search tutorial, let’s build a child model that has 1 block, for simplicity’s sake. This block comprises N=3 convolutional cells and 1 reduction cell, and each cell comprises B=4 nodes.
Recall that what this means is simply to design one single architecture for a convolutional cell, then repeat it for 3 more times (2 more convolutional cells and 1 reduction cell). This means our generated child model should look like this:
Let’s now build a convolutional cell!
To explain how to build a convolutional cell, let’s assume we already have 2 convolutional cells. Notice that the last operations from each of these 2 cells are add operations.
Recall that a convolutional cell consists of 4 nodes. So where are these nodes?
Node 1 and Node 2
The two previous cells (red and blue) will be considered as Node 1 and Node 2 respectively. The other 2 nodes fall in this very convolutional cell that we are building now.
From this section onwards, you can safely disregard the ‘Convolutional cell’ labels you see on the image above and concentrate on the ‘Nodes’ labels:
Node 1 — red
Node 2 — blue
Node 3 — green
Node 4 — purple
ErratumI̶f̶ ̶y̶o̶u̶’̶r̶e̶ ̶w̶o̶n̶d̶e̶r̶i̶n̶g̶ ̶i̶f̶ ̶t̶h̶e̶s̶e̶ ̶n̶o̶d̶e̶s̶ ̶w̶i̶l̶l̶ ̶c̶h̶a̶n̶g̶e̶ ̶f̶o̶r̶ ̶e̶v̶e̶r̶y̶ ̶c̶o̶n̶v̶o̶l̶u̶t̶i̶o̶n̶a̶l̶ ̶c̶e̶l̶l̶ ̶w̶e̶’̶r̶e̶ ̶b̶u̶i̶l̶d̶i̶n̶g̶,̶ ̶t̶h̶e̶ ̶a̶n̶s̶w̶e̶r̶ ̶i̶s̶ ̶y̶e̶s̶!̶ ̶E̶v̶e̶r̶y̶ ̶c̶e̶l̶l̶ ̶w̶i̶l̶l̶ ̶’̶a̶s̶s̶i̶g̶n̶’̶ ̶t̶h̶e̶ ̶n̶o̶d̶e̶s̶ ̶i̶n̶ ̶t̶h̶i̶s̶ ̶m̶a̶n̶n̶e̶r̶.̶
Node 3
Node 3 is where the building starts. The controller samples 4 decisions (or rather 2 sets of decisions):
2 nodes to connect to
the respective 2 operations to perform on the nodes to connect to
With 4 decisions to make, the controller runs 4 time steps. Have a look below:
From the above we see that the controller sampled Node2, Node 1, avg5×5, and sep5×5 from each of the four time steps. How does this translate to the architecture of the child model? Let’s see:
From the above, we observe three things:
The output from Node2 (blue) undergoes the avg5×5 operation.The output from Node 1 (red) undergoes a sep5×5 operation.Both the results from these two operations undergo an add operation.
The output from Node2 (blue) undergoes the avg5×5 operation.
The output from Node 1 (red) undergoes a sep5×5 operation.
Both the results from these two operations undergo an add operation.
The output from this node is the tensor that undergoes the add operation. This explains why Nodes 1 and 2 end with add operations.
Node 4
Now for Node 4. We repeat the same steps, but now the controller now has three nodes to choose from (Nodes 1, 2 and 3). Below, the controller generated 3, 1, id and avg3×3.
This translates to building the following:
What just happened?
The output from Node3 (green) undergoes an id operation.The output from Node 1 (red) undergoes an avg3×3 operation.Both the results from these two operations undergo an add operation.
The output from Node3 (green) undergoes an id operation.
The output from Node 1 (red) undergoes an avg3×3 operation.
Both the results from these two operations undergo an add operation.
And that’s it! All convolutional cells in the final child model will share the same architecture. Similarly, all reduction cells (whose architecture differ from that of convolutional cell by having stride 2 operations) in the final child model will share the same architecture.
[Back to top]
Because this post mainly shows the macro and micro search strategies, I’ve left out many small details (especially on the concept of transfer learning). Let me briefly cover them:
What’s so ‘efficient’ in ENAS? Answer: transfer learning. If a computation between two nodes has been done (trained) before, the weights from the convolutional filters and 1×1 convolutions (to maintain number of channel outputs; not mentioned in the previous sections) will be reused. This is what makes ENAS faster than its predecessors!
It is possible that the controller samples a decision where no skip connection is needed.
There are 6 operations available for the controller: convolutions with filter sizes 3×3 and 5×5, depthwise-separable convolutions with filter sizes 3×3 and 5×5, max pooling and average pooling of kernel size 3×3.
Do read up on the concatenate operation at the end of each cell which ties up ‘loose ends’ of any nodes.
Do read up briefly on the policy gradient algorithm (REINFORCE) reinforcement learning.
[Back to top]
Macro search (for an entire network)
The final child model is as shown below.
Micro search (for a convolutional cell)
Note that only part of the final child model is shown here.
[Back to top]
TensorFlow implementation by the authors
Keras implementation
PyTorch implementation
[Back to top]
Efficient Neural Architecture Search via Parameter Sharing
Neural Architecture Search with Reinforcement Learning
Learning Transferable Architectures for Scalable Image Recognition
That’s it! Remember to read the ENAS paper Efficient Neural Architecture Search via Parameter Sharing. If you have any questions, please highlight and leave a comment.
General
Counting No. of Parameters in Deep Learning Models
Related to NLP
Animated RNN, LSTM and GRU
Attn: Illustrated Attention
Illustrated: Self-Attention
Line-by-Line Word2Vec Implementation
Related to Computer Vision
Breaking down Mean Average Precision (mAP)
Optimisation
Step-by-Step Tutorial on Linear Regression with Stochastic Gradient Descent
10 Gradient Descent Optimisation Algorithms + Cheat Sheet
Follow me on Twitter @remykarem or LinkedIn. You may also reach out to me via raimi.bkarim@gmail.com. Feel free to visit my website at remykarem.github.io.