From Nodes to Networks: Evolving Recurrent Neural Networks

1 INTRODUCTION . In many areas of engineering design , the systems have become so complex that humans can no longer optimize them , and instead , automated methods are needed . This has been true in VLSI design for a long time , but it has also become compelling in software engineering : The idea in ” programming by optimization ” is that humans should design only the framework and the details should be left for automated methods such as optimization ( Hoos , 2012 ) . Recently similar limitations have started to emerge in deep learning . The neural network architectures have grown so complex that humans can no longer optimize them ; hyperparameters and even entire architectures are now optimized automatically through gradient descent ( et al. , 2016 ) , Bayesian parameter optimization ( Malkomes et al. , 2015 ) , reinforcement learning ( Zoph & Le , 2016 ; Baker et al. , 2016 ) , and evolutionary computation ( Miikkulainen et al. , 2018 ; Real , 2017 ; Fernando , 2017 ) . Improvements from such automated methods are significant : the structure of the network matters . This paper shows that the same approach can be used to improve architectures that have been used essentially unchanged for decades . The case in point is the Long Short-Term Memory ( LSTM ) network Hochreiter & Schmidhuber ( 1997 ) . It was originally proposed in 1992 ; with the vastly increased computational power , it has recently been shown a powerful approach for sequential tasks such as speech recognition , language understanding , language generation , and machine translation , in some cases improving performance 40 % over traditional methods ( Bahdanau et al. , 2015 ) . The basic LSTM structure has changed very little in this process , and thorough comparisons of variants concluded that there ’ s little to be gained by modifying it further ( Klaus et al. , 2014 ; Jozefowicz et al. , 2015 ) . However , very recent studies on metalearning methods such as neural architecture search and evolutionary optimization have shown that LSTM performance can be improved by complexifying it further ( Zoph & Le , 2016 ; Miikkulainen et al. , 2018 ) . This paper develops a new method along these lines , recognizing that a large search space where significantly more complex node structures can be constructed could be beneficial . The method is based on a tree encoding of the node structure so that it can be efficiently searched using genetic programming . Indeed , the approach discovers significantly more complex structures than before , and they indeed perform significantly better : Performance in the standard language modeling benchmark , where the goal is to predict the next word in a large language corpus , is improved by 6 perplexity points over the standard LSTM ( Zaremba et al. , 2014 ) , and 0.9 perplexity points over reinforcement-learning based neural architecture search ( Zoph & Le , 2016 ) . These improvements are obtained by constructing a homogeneous layered network architecture from a single gated recurrent node design . A second innovation in this paper shows that further improvement can be obtained by constructing such networks from multiple different designs . As a first step , allocation of different kinds of LSTM nodes into slots in the network is shown to improve performance by another 0.5 perplexity points . This result suggests that further improvements are possible with more extensive network-level search . A third contribution of this paper is to show that evolution of neural network architectures in general can be speeded up significantly by using an LSTM network to predict the performance of candidate neural networks . After training the candidate for a few epochs , such a Meta-LSTM network predicts what performance a fully trained network would have . That prediction can then be used as fitness for the candidate , speeding up evolution fourfold in these experiments . A fourth contribution is to encourage exploration by using an archive of already-explored areas . The effect is similar to that of novelty search , but does not require a separate novelty objective , simplifying the search . Interestingly , when the recurrent node evolved for language modeling was applied to another task , music modeling , it did not perform well . However , it was possible to evolve another solution for that task that did . As a fifth contribution , the results in this paper demonstrate that it is not simply the added complexity in the nodes that matter , but that it is the right kind , i.e . complexity customized for each task . Thus , evolutionary optimization of complex deep learning architectures is a promising approach that can yield significant improvements beyond human ability to do so . 2 BACKGROUND AND RELATED WORK . In recent years , LSTM-based recurrent networks have been used to achieve strong results in the supervised sequence learning problems such as in speech recognition [ 10 ] and machine translation ( Bahdanau et al. , 2015 ) . Further techniques have been developed to improve performance of these models through ensembling ( Zaremba et al. , 2014 ) , shared embeddings ( Zilly et al. , 2016 ) and dropouts ( Gal , 2015 ) . In contrast , previous studies have shown that modifying the LSTM design itself did not provide any significant performance gains ( Bayer et al. , 2009 ; Cho et al. , 2014 ; Jozefowicz et al. , 2015 ) . However , a recent paper from Zoph & Le ( 2016 ) showed that policy gradients can be used to train a LSTM network to find better LSTM designs . The network is rewarded based on the performance of the designs it generates . While this approach can be used to create new designs that perform well , its exploration ability is limited ( as described in more detail in Section 3.3 ) . The setup detailed in Zoph & Le ( 2016 ) is used for comparison in this paper . In a subsequent paper Pham et al . ( 2018 ) , the same policy gradient approach is used to discover new recurrent highway networks to achieve even better results . Neuroevolution methods like NEAT ( Stanley & Miikkulainen , 2002 ) are an alternative to policy gradient approaches , and have also been shown to be sucessful in the architecture search problem ( Miikkulainen et al. , 2018 ; Real , 2017 ) . For instance , Cartesian genetic programming was recently used to achieve state of the art results in CIFAR-10 ( Suganuma et al. , 2017 ) . Along similar lines , a tree based variant of genetic programming is used in this paper to evolve recurrent nodes . These trees can grow in structure and can be pruned as well , thus providing a flexible representation . Novelty search is a particularly useful technique to increase exploration in evolutionary optimization ( Lehman , 2012 ) . Novelty is often cast as a secondary objective to be optimized . It allows searching in areas that do not yield immediate benefit in terms of fitness , but make it possible to discover stepping stones that can be combined to form better solutions later . This paper proposes an alternative approach : keeping an archive of areas already visited and exploited , achieving similar goals without additional objectives to optimize . Most architecture search methods reduce compute time by evaluating individuals only after partial training ( Suganuma et al. , 2017 ; Real , 2017 ) . This paper proposes a meta LSTM framework to predict final network performance based on partial training results . These techniques are described in detail in the next section . 3 METHODS . Evolving recurrent neural networks is an interesting problem because it requires searching the architecture of both the node and the network . As shown by recent research ( Zoph & Le , 2016 ) ( Zilly et al. , 2016 ) , the recurrent node in itself can be considered a deep network . In this paper , Genetic Programming ( GP ) is used to evolve such node architectures . In the first experiment , the overall network architecture is fixed i.e . constructed by repeating a single evolved node to form a layer ( Figure1 ( b ) ) . In the second , it is evolved by combining several different types of nodes into a layer ( Figure1 ( c ) ) . In the future more complex coevolution approaches are also possible . Evaluating the evolved node and network is costly . Training the network for 40 epochs takes two hours on a 1080 NVIDIA GPU . A sequence to sequence model called meta-LSTM is developed to speed up evaluation . Following sections describe these methods in detail . 3.1 GENETIC PROGRAMMING FOR RECURRENT NODES . As shown in Figure1 ( a ) , a recurrent node can be represented as a tree structure , and GP can therefore be used to evolve it . However , standard GP may not be sufficiently powerful to do it . In particular , it does not maintain sufficient diversity in the population . Similar to the GP-NEAT approach by Trujillo et al . Tujillo et al . ( 2015 ) , it can be augmented with ideas from NEAT speciation . A recurrent node usually has two types of outputs . The first , denoted by symbol h in Figure1 ( a ) , is the main recurrent output . The second , often denoted by c , is the native memory cell output . The h value is weighted and fed to three locations : ( 1 ) to the higher layer of the network at the same time step , ( 2 ) to other nodes in the network at the next time step , and ( 3 ) to the node itself at the next time step . Before propagation , h are combined with weighted activations from the previous layer , such as input word embeddings in language modeling , to generate eight node inputs ( termed as base eight by Zoph & Le ( 2016 ) ) . In comparison , the standard LSTM node has four inputs ( see Figure5 ( a ) ) . The native memory cell output is fed back , without weighting , only to the node itself at the next time step . The connections within a recurrent cell are not trainable by backpropagation and they all carry a fixed weight of 1.0 . Thus , even without an explicit recurrent loop , the recurrent node can be represented as a tree . There are two type of elements in the tree : ( 1 ) linear activations with arity two ( add , multiply ) , and ( 2 ) non-linear activations with arity one ( tanh , sigmoid , relu , sin , cos ) . There are three kind of mutation operations in the experiments : ( 1 ) Mutation to randomly replace an element with an element of the same type , ( 2 ) Mutation to randomly inserts a new branch at a random position in the tree . The subtree at the chosen position is used as child node of the newly created subtree . ( 3 ) Mutation to shrink the tree by choosing a branch randomly and replacing it with one of the branch ’ s arguments ( also randomly chosen ) . One limitation of standard tree is that it can have only a single output : the root . This problem can be overcome by using a modified representation of a tree that consists of Modi outputs ( Zhang & Zhang , 2004 ) . In this approach , with some probability p ( termed modirate ) , non-root nodes can be connected to any of the possible outputs . A higher modi rate would lead to many sub-tree nodes connected to different outputs . A node is assigned modi ( i.e . connected to memory cell outputs c or d ) only if its sub-tree has a path from native memory cell inputs . This representation allows searching for a wide range of recurrent node structures with GP .

Summary: The authors apply (tree-based) genetic programming (GP) to RNN search, or more specifically RNNs with memory cells, with the foremost example of this being the LSTM. GP provide a structured search that seems appropriate for designing NN modules, and has previously been applied successfully to evolving CNNs. However, the authors fail to mention that (tree-based) GP has been applied to evolving RNN topologies as far back as 2 decades ago, with even multiple cells in a single RNN unit [1]. The selection of more advanced techniques is good though - use of Modi for allowing multiple outputs, and neat-GP for more effective search (though a reference to the "hall of fame" [2] is lacking).