Randomly wired neural networks and state-of-the-art accuracy? Yes it works.
By: George Seif
Tags: Machine Learning, Education, Deep Learning, Artificial Intelligence, Technology

In an effort to discover even more powerful networks, Google AI’s research team developed a technique for having an algorithm search for good networks. This is known as Neural Architecture Search (NAS).

Neural Architecture Search

NAS is an algorithm that searches for the best neural network architecture. Most of the NAS algorithms work quite similarly.

Start off by defining a set of “building blocks” that can possibly be used for our network. Then try training a network by putting together different combinations of those blocks. Through this trial and error, the NAS algorithm will eventually figure out which blocks and which configurations work best.

As it turns out, this method works quite well and achieves state-of-the-art accuracy. It discovers some pretty peculiar looking architectures too, as you can see from the official paper below:

NASNet blocks discovered by the NAS algorithm. Source

The great thing about NAS is that we get to see some neural net architectures that we might otherwise not have looked for at all.

It does however come with its own limitations. Since we’re sampling from a fixed set of blocks, we won’t ever discover any new blocks. It’s the same case with the skip connections: NASNet only allows for a skip connection across 1 downsampling level, but what if other types of connections might work?

Facebook’s AI Research team took a new stab at NAS by trying out randomly wired neural networks. The intuition was that if the general idea of architecture search worked well for the fixed blocks and connections, perhaps running it with a much wider search space (i.e random connections) will lead to some unexplored configurations.

Randomly Wired Neural Networks

Since the authors of the paper wanted to focus specifically on the wiring i.e the connections of the neural networks, they make a few restrictions on the network structure:

The input is always an image of size 224x224

The blocks are always a ReLU-Conv-BatchNorm triplet. The convolution in the triplet is always a 3x3 separable convolution from Xception.

The aggregation of multiple tensors (for example when fusing a skip connection with other tensors) is always done in the same way — via weighted sum. These weights are learnable and always positive.

The stages of the network always remain the same. The general CNN structure of a multiple convolution blocks, followed by downsampling, repeated over and over until a final softmax classifier, has become standard in state-of-the-art network designs. This paradigm is also taken in these research experiments.

General structure of state-of-the-art CNNs, also adapted for this research. Source

An input size of 224x224 is quite standard in ImageNet competitions — the ImageNet dataset was used for benchmarking against other handcrafted networks as well as NAS-generated networks. The triplet block of ReLU-Conv-BatchNorm is also common and has been extensively proven to deliver state-of-the-art results when repeated throughout deep CNNs.

The way the tensors are aggregated is a small twist. Most state-of-the-art networks do a direct addition or concatenation without weights — but it’s a minor addition that shouldn’t affect performance too drastically. The staged design of CNNs shown in the table above has also become a go-to network structure used in ResNets, DenseNets, and NASNets.

Notice how this isn’t an entirely a “randomised neural network”. There’s no randomisation from complete scratch. Instead, a single component of the CNN design, namely the wiring, is explored while the other components remain fixed.

This is a significant point that the authors really tried to send home with the reader. We’re not yet doing completely randomised neural networks. But we are beginning to do some deep exploration of the component search space — one step at a time.

With these constraints set in place various classical random graph models were used to generate the random wiring of the networks.

And by random, I mean random.

Some of the randomly wired networks. Source

Randomised Networks open the gates of Deep Learning Exploration

The significance of this research lies in its main exploratory idea: expand the search space of NAS algorithms to find new, better network designs. While researchers have discovered some great designs on their own, it’s not practically feasible to go through the entire search space manually.

This also extends to another point: if we are expanding our search space, we’ll need an algorithm, in this case our network generator, to be really good at searching. It has to know what to look for or at the very least have some tendency to move towards it by design, much like optimisation algorithms like gradient descent work.

Architecture Search is the next frontier for Deep Learning research. It allows us to use an algorithm, rather than trial and error, to discover optimal architectures.

For now, it makes sense to fix some network components while searching for others (in this case wiring). This simplifies the problem to something bite-sized, easier to handle. NAS algorithms should have some degree of randomisation, since that’s the only way we can discover truly novel architectures.

The next step will of course be further expanding the search space and random nature of our search algorithms. That means expanding to more and more components of the network until the whole thing is designed algorithmically.

What happens when we let NAS go loose? Does it select non-square convolutions like 2x4? Does it propose to use feedback loops? Does the network become more simple or more complicated?

Neural Architecture Search presents an exciting new field of research. The hope is that letting a search algorithm loose becomes a way of using randomization to our advantage, discovering creative and never before thought of architectures.