Everything you need to know about AutoML and Neural Architecture Search
By: George Seif
Tags: Technology, Towards Data Science, Deep Learning, Artificial Intelligence, Machine Learning

AutoML and Neural Architecture Search (NAS) are the new kings of the deep learning castle. They’re the quick and dirty way of getting great accuracy for your machine learning task without much work. Simple and effective; it’s what we want AI to be all about!

So how does it work? How do you use it? What options do you have to harness that power today?

Here’s everything you need to know about AutoML and NAS.

Neural Architecture Search (NAS)

Developing neural network models often requires significant architecture engineering. You can sometimes get by with transfer learning, but if you really want the best possible performance it’s usually best to design your own network. This requires specialised skills (read: expensive from a business standpoint) and is challenging in general; we may not even know the limits of the current state-of-the-art techniques! It’s a lot of trial and error and the experimentation itself is time consuming and expensive.

This is where NAS comes in. NAS is an algorithm that searches for the best neural network architecture. Most of the algorithms work in this following way. Start off by defining a set of “building blocks” that can possibly be used for our network. For example, the state-of-the-art NASNet paper proposes these commonly used blocks for an image recognition network:

NASNet blocks for image recognition network

In the NAS algorithm, a controller Recurrent Neural Network (RNN) samples these building blocks, putting them together to create some kind of end-to-end architecture. This architecture generally embodies the same style as state-of-the-art networks, such as ResNets or DenseNets, but uses a much different combination and configuration of the blocks.

This new network architecture is then trained to convergence to obtain some accuracy on a held-out validation set. The resulting accuracies are used to update the controller so that the controller will generate better architectures over time, perhaps by selecting better blocks or making better connections. The controller weights are updated with policy gradient. The whole end-to-end setup is shown below.

The NAS algorithm

It’s a fairly intuitive approach! In simple terms: have an algorithm grab different blocks and put those blocks together to make a network. Train and test out that network. Based on your results, adjust the blocks you used to make the network and how you put them together!

Part of the reason this algorithm succeeds and the paper demonstrates such great results is because of the constraints and assumptions made with it. The NAS discovered architecture is trained and tested on a much-smaller-than-real-world dataset. This is done because training on something large, like ImageNet would take a very long time. But, the idea is that a network that performs better on a smaller, yet similarly structured dataset should also perform better on a larger and more complex one, which has generally been true in the deep learning era.

Second, is that the search space itself is quite limited. NAS is designed to build architectures that are very similar in style to the current state-of-the-art. For image recognition, this is to have a set of repeated blocks in the network while progressively downsampling, as shown on the left below. The set of blocks to choose from to build the repeating ones are also quite commonly used in current research. The main novel part of the NAS discovered networks is how the blocks are connected together.

Check out the best discovered blocks and structure for the ImageNet network on the right below. It is interesting to notes how they contain quite a random looking mix of operations, including many separable convolutions.