Title: A FLEXIBLE APPROACH TO AUTOMATED RNN ARCHITECTURE GENERATION

Abstract: The process of designing neural architectures requires expert knowledge and extensive trial and error. While automated architecture search may simplify these requirements, the recurrent neural network (RNN) architectures generated by existing methods are limited in both flexibility and components. We propose a domain-specific language (DSL) for use in automated architecture search which can produce novel RNNs of arbitrary depth and width. The DSL is flexible enough to define standard architectures such as the Gated Recurrent Unit and Long Short Term Memory and allows the introduction of non-standard RNN components such as trigonometric curves and layer normalization. Using two different candidate generation techniques, random search with a ranking function and reinforcement learning, we explore the novel architectures produced by the RNN DSL for language modeling and machine translation domains. The resulting architectures do not follow human intuition yet perform well on their targeted tasks, suggesting the space of usable RNN architectures is far larger than previously assumed.

1. INTRODUCTION:

Developing novel neural network architectures is at the core of many recent AI advances (Szegedy et al., 2015; He et al., 2016; Zilly et al., 2016) . The process of architecture search and engineering is slow, costly, and laborious. Human experts, guided by intuition, explore an extensive space of potential architectures where even minor modifications can produce unexpected results. Ideally, an automated architecture search algorithm would find the optimal model architecture for a given task.

Many explorations into the automation of machine learning have been made, including the optimization of hyperparameters (Bergstra et al., 2011; Snoek et al., 2012) and various methods of producing novel model architectures (Stanley et al., 2009; Baker et al., 2016; Zoph and Le, 2017) . For architecture search, ensuring these automated methods are able to produce results similar to humans usually requires traversing an impractically large search space, assuming high quality architectures exist in the search space at all. The choice of underlying operators composing an architecture is further typically constrained to a standard set across architectures even though recent work has found promising results in the use of non-standard operators (Vaswani et al., 2017) .

We propose a meta-learning strategy for flexible automated architecture search of recurrent neural networks (RNNs) which explicitly includes novel operators in the search. It consists of three stages, outlined in Figure 1 , for which we instantiate two versions.

1. A candidate architecture generation function produces potential RNN architectures using a highly flexible DSL. The DSL enforces no constraints on the size or complexity of the generated tree and can be incrementally constructed using either a random policy or with an RL agent. 2. A ranking function processes each candidate architecture's DSL via a recursive neural network, predicting the architecture's performance. By unrolling the RNN representation, the ranking function can also model the interactions of a candidate architecture's hidden state over time.

Figure 1 : A generator produces candidate architectures by iteratively sampling the next node (either randomly or using an RL agent trained with REINFORCE). Full architectures are processed by a ranking function and the most promising candidates are evaluated. The results from running the model against a baseline experiment are then used to improve the generator and the ranking function.

3. An evaluator, which takes the most promising candidate architectures, compiles their DSLs to executable code and trains each model on a specified task. The results of these evaluations form architecture-performance pairs that are then used to train the ranking function and RL generator.

2. A DOMAIN SPECIFIC LANGUAGE FOR DEFINING RNNS:

In this section, we describe a domain specific language (DSL) used to define recurrent neural network architectures. This DSL sets out the search space that our candidate generator can traverse during architecture search. In comparison to Zoph and Le (2017) , which only produced a binary tree with matrix multiplications at the leaves, our DSL allows a broader modeling search space to be explored.

When defining the search space, we want to allow for standard RNN architectures such as the Gated Recurrent Unit (GRU) (Cho et al., 2014) or Long Short Term Memory (LSTM) (Hochreiter and Schmidhuber, 1997) to be defined in both a human and machine readable manner.

The core operators for the DSL are 4 unary operators, 2 binary operators, and a single ternary operator:

[MM , Sigmoid , Tanh, ReLU ] [Add , Mult] [Gate3 ].

MM represents a single linear layer with bias, i.e. MM (x) := W x + b. Similarly, we define:

Sigmoid (x) = σ(x).

The operator Mult represents element-wise multiplication: Mult(x, y) = x • y. The Gate3 operator performs a weighted summation between two inputs, defined by

Gate3 (x, y, f ) = σ(f ) • x + (1 -σ(f )) • y.

These operators are applied to source nodes from the set

[x t , x t-1 , h t-1 , c t-1 ]

, where x t and x t-1 are the input vectors for the current and previous timestep, h t-1 is the output of the RNN for the previous timestep, and c t-1 is optional long term memory.

The Gate3 operator is required as some architectures, such as the GRU, re-use the output of a single Sigmoid for the purposes of gating. While allowing all possible node re-use is out of scope for this DSL, the Gate3 ternary operator allows for this frequent use case.

Using this DSL, standard RNN cell architectures such as the tanh RNN can be defined: tanh(Add (MM (x t ), MM (h t-1 ))). To illustrate a more complex example that includes Gate3 , the GRU is defined in full in Appendix A.

2.1. ADDING SUPPORT FOR ARCHITECTURES WITH LONG TERM MEMORY:

With the operators defined above it is not possible to refer to and re-use an arbitrary node. The best performing RNN architectures however generally use not only a hidden state h t but also an additional hidden state c t for long term memory. The value of c t is extracted from an internal node computed while producing h t .

The DSL above can be extended to support the use of c t by numbering the nodes and then specifying which node to extract c t from (i.e. c t = Node 5 ). We append the node number to the end of the DSL definition after a delimiter.

As an example, the nodes in bold are used to produce c t , with the number appended at the end indicating the node's number. Nodes are numbered top to bottom (h t will be largest), left to right.

Mult(Sigmoid (MM (x t )), Tanh(Add (MM (h t-1 ), Mult(MM (c t-1 ), MM (x t )))))|5 Mult(Sigmoid (MM (x t )), Tanh(Add(MM (h t-1 ), Mult(MM (c t-1 ), MM (x t )))))|6 Mult(Sigmoid (MM (x t )), Tanh(Add (MM (h t-1 ), Mult(MM (c t-1 ), MM (x t )))))|7

2.2. EXPRESSIBILITY OF THE DOMAIN SPECIFIC LANGUAGE:

While the domain specific language is not entirely generic, it is flexible enough to capture most standard RNN architectures. This includes but is not limited to the GRU, LSTM, Minimal Gate Unit (MGU) (Zhou et al., 2016) , Quasi-Recurrent Neural Network (QRNN) (Bradbury et al., 2017) , Neural Architecture Search Cell (NASCell) (Zoph and Le, 2017) , and simple RNNs.

2.3. EXTENDING THE DOMAIN SPECIFIC LANGUAGE:

While many standard and non-standard RNN architectures can be defined using the core DSL, the promise of automated architecture search is in designing radically novel architectures. Such architectures should be formed not just by removing human bias from the search process but by including operators that have not been sufficiently explored. For our expanded DSL, we include:

[Sub, Div ] [Sin, Cos] [PosEnc] [LayerNorm, SeLU ].

These extensions add inverses of currently used operators (Sub(a, b) = ab instead of addition, Div (a, b) = a b instead of multiplication), trigonometric curves (Sin and Cos are sine and cosine activations respectively, PosEnc introduces a variable that is the result of applying positional encoding (Vaswani et al., 2017) according to the current timestep), and optimizations (LayerNorm applies layer normalization (Ba et al., 2016) to the input while SeLU is the activation function defined in Klambauer et al. (2017) ).

2.4. COMPILING A DSL DEFINITION TO EXECUTABLE CODE:

For a given architecture definition, we can compile the DSL to code by traversing the tree from the source nodes towards the final node h t . We produce two sets of source code -one for initialization required by a node, such as defining a set of weights for matrix multiplication, and one for the forward call during runtime. For details regarding speed optimizations, refer to Appendix A2.

3. CANDIDATE ARCHITECTURE GENERATION:

The candidate architecture generator is responsible for producing candidate architectures that are then later filtered and evaluated. Architectures are grown beginning at the output h t and ordered to prevent multiple representations for equivalent architectures:

Growing architectures from h t up Beginning from the output node h t , operators are selected to be added to the computation graph, depicted in Figure 2 . Whenever an operator has one or more children to be filled, the children are filled in order from left to right. If we wish to place a limit on the height (distance from h t ) of the tree, we can force the next child to be one of the source nodes when it would otherwise exceed the maximum height.

Preventing duplicates through canonical architecture ordering Due to the flexibility allowed by the DSL, there exist many DSL specifications that result in the same RNN cell. To solve the issue of commutative operators (i.e. Add (a, b) = Add (b, a)), we define a canonical ordering of an architecture by sorting the arguments of any commutative nodes. Special consideration is required for non-commutative operators such as Gate3 , Sub, or Div. For full details, refer to Appendix A3.

3.1. INCREMENTAL ARCHITECTURE CONSTRUCTION USING REINFORCEMENT LEARNING:

Architectures in the DSL are constructed incrementally a node at a time starting from the output h t .

The simplest agent is a random one which selects the next node from the set of operators without internalizing any knowledge about the architecture or optima in the search space. Allowing an intelligent agent to construct architectures would be preferable as the agent can learn to focus on promising directions in the space of possible architectures.

For an agent to make intelligent decisions regarding which node to select next, it must have a representation of the current state of the architecture and a working memory to direct its actions. We propose achieving this with two components:

1. a tree encoder that represents the current state of the (partial) architecture.

2. an RNN which is fed the current tree state and samples the next node.

The tree encoder is an LSTM applied recursively to a node token and all its children with weights shared, but the state reset between nodes. The RNN is applied on top of the encoded partial architecture and predicts action scores for each operation. We sample with a multinomial and encourage exploration with an epsilon-greedy strategy. Both components of the model are trained jointly using the REINFORCE algorithm (Williams, 1992) .

As a partial architecture may contain two or more empty nodes, such as h t = Gate3(∅, ∅, σ(∅)), we introduce a target token, T , which indicates which node is to next be selected. Thus, in h t = Gate(T, ∅, σ(∅)), the tree encoder understands that the first argument is the slot to be filled.

3.2. FILTERING CANDIDATE ARCHITECTURES USING A RANKING FUNCTION:

Even with an intelligent generator, understanding the likely performance of an architecture is difficult, especially the interaction of hidden states such as h t-1 and c t-1 between timesteps. We propose to approximate the full training of a candidate architecture by training a ranking network through regression on architecture-performance pairs. This ranking function can be specifically constructed to allow a richer representation of the transitions between c t-1 and c t .

As the ranking function uses architecture-performance samples as training data, human experts can also inject previous best known architectures into the training dataset. This is not possible for on-policy reinforcement learning and when done using off-policy reinforcement learning additional care and complexity are required for it to be effective (Harutyunyan et al., 2016; Munos et al., 2016) .

Given an architecture-performance pair, the ranking function constructs a recursive neural network that reflects the nodes in a candidate RNN architecture one-to-one. Sources nodes are represented by a learned vector and operators are represented by a learned function. The final vector output then passes through a linear activation and attempts to minimize the difference between the predicted and real performance. The source nodes (x t , x t-1 , h t-1 , and c t-1 ) are represented by learned vector representations. For the operators in the tree, we use TreeLSTM nodes (Tai et al., 2015) . All Unrolling the graph for accurately representing h t-1 and c t-1 : A strong assumption made above is that the vector representation of the source nodes can accurately represent the contents of the source nodes across a variety of architectures. This may hold true for x t and x t-1 but is not true for h t-1 or c t-1 . The value of h t and c t are defined by the operations within the given architecture itself.

To remedy this assumption, we can unroll the architecture for a single timestep, replacing h t-1 and c t-1 with their relevant graph and subgraph. This would allow the representation of h t-1 to understand which source nodes it had access to and which operations were applied to produce h t-1 .

While unrolling is useful for improving the representation of h t-1 , it is essential for allowing an accurate representation of c t-1 . This is as many small variations of c t-1 are possible -such as selecting a subgraph before or after an activation -that may result in substantially different architecture performance.

4. EXPERIMENTS:

We evaluated our architecture generation on two experiments: language modeling (LM) and machine translation (MT). Due to the computational requirements of the experiments, we limited each experiment to one combination of generator components. For language modeling, we explore the core DSL using randomly constructed architectures (random search) directed by a learned ranking function. For machine translation, we use the extended DSL and construct candidate architectures incrementally using the RL generator without a ranking function.

4.1. LANGUAGE MODELING USING RANDOM SEARCH WITH A RANKING FUNCTION:

For evaluating architectures found during architecture search, we use the WikiText-2 dataset (Merity et al., 2017b) . When evaluating a proposed novel RNN cell c, we construct a two layer c-RNN with a 200 unit hidden size. Aggressive gradient clipping is performed to ensure that architectures such as the ReLU RNN would be able to train without exploding gradients. The weights of the ranking network were trained by regression on architecture-perplexity pairs using the Adam optimizer and mean squared error (MSE). Further hyperparameters and training details are listed in Appendix B1.

Explicit restrictions on generated architectures During the candidate generation phase, we filter the generated architectures based upon specific restrictions. These include structural restrictions and restrictions aimed at effectively reducing the search space by removing likely invalid architectures. For Gate3 operations, we force the input to the forget gate to be the result of a sigmoid activation.

We also require the cell to use the current timestep x t and the previous timestep's output h t-1 to satisfy the requirements of an RNN. Candidate architectures were limited to 21 nodes, the same number of nodes as used in a GRU, and the maximum allowed distance (height) from h t was 8 steps.

We also prevent the stacking of two identical operations. While this may be an aggressive filter it successfully removes many problematic architectures. These problematic architectures include when two sigmoid activations, two ReLU activations, or two matrix multiplications are used in succession -the first of which is unlikely to be useful, the second of which is a null operator on the second activation, and the third of which can be mathematically rewritten as a single matrix multiplication.

If a given candidate architecture definition contained c t-1 , the architecture was queried for valid subgraphs from which c t could be generated. The subgraphs must contain c t-1 such that c t is recurrent and must contain three or more nodes to prevent trivial recurrent connections. A new candidate architecture is then generated for each valid c t subgraph.

Random architecture search directed by a learned ranking function Up to 50,000 candidate architecture DSL definitions are produced by a random architecture generator at the beginning of each search step. This full set of candidate architectures are then simulated by the ranking network and an estimated perplexity assigned to each. Given the relative simplicity and small training dataset, the ranking function was retrained on the previous full training results before being used to estimate the next batch of candidate architectures. Up to 32 architectures were then selected for full training. 28 of these were selected from the candidate architectures with the best perplexity while the last 4 were selected via weighted sampling without replacement, prioritizing architectures with better estimated perplexities.

c t architectures were introduced part way through the architecture search after 750 valid h t architectures had been evaluated with h t architectures being used to bootstrap the c t architecture vector representations. Figure 3 provides a visualization of the architecture search over time, showing valid h t and c t architectures.

Analyzing the BC3 cell After evaluating the top 10 cells using a larger model on WikiText-2, the top performing cell BC3 (named after the identifying hash, bc3dc7a. . .) was an unexpected layering of two Gate3 operators,

f = σ(W f x t + U f h t-1 ) (1) z = V z (X y c t-1 • U z x t ) • W z x t (2) c t = tanh(f • W g x t + (1 -f ) • z) (3) o = σ(W o x t + U o h t-1 ) (4) h t = o • c t + (1 -o) • h t-1 (5)

where • is an element-wise multiplication and all weight matrices W, U, V, X ∈ R H×H .

Equations 1 to 3 produce the first Gate3 while equations 4 and 5 produce the second Gate3 . The output of the first Gate3 becomes the value for c t after passing through a tanh activation.

While only the core DSL was used, BC3 still breaks with many human intuitions regarding RNN architectures. While the formulation of the gates f and o are standard in many RNN architectures, the rest of the architecture is less intuitive. The Gate3 that produces c t (equation 3) is mixing between a matrix multiplication of the current input x t and a complex interaction between c t-1 and x t (equation 2). In BC3, c t-1 passes through multiple matrix multiplications, a gate, and a tanh activation before becoming c t . This is non-conventional as most RNN architectures allow c t-1 to become c t directly, usually through a gating operation. The architecture also does not feature a masking output gate like the LSTM, with outputs more similar to that of the GRU that does poorly on language modeling. That this architecture would be able to learn without severe instability or succumbing to exploding gradients is not intuitively obvious.

4.1.1. EVALUATING THE BC3 CELL:

For the final results on BC3, we use the experimental setup from Merity et al. (2017a) and report results for the Penn Treebank (Table 1 ) and WikiText-2 (Table 2 ) datasets. To show that not any standard RNN can achieve similar perplexity on the given setup, we also implemented and tuned a GRU based model which we found to strongly underperform compared to the LSTM, BC3, NASCell, or Recurrent Highway Network (RHN). Full hyperparameters for the GRU and BC3 are in Appendix Table 2 : Model perplexity on validation/test sets for the WikiText-2 language modeling task.

B4. Our model uses equal or fewer parameters compared to the models it is compared against. While BC3 did not outperform the highly tuned AWD-LSTM (Merity et al., 2017a) or skip connection LSTM (Melis et al., 2017) , it did outperform the Recurrent Highway Network (Zilly et al., 2016) and NASCell (Zoph and Le, 2017) on the Penn Treebank, where NASCell is an RNN found using reinforcement learning architecture search specifically optimized over the Penn Treebank.

4.2. INCREMENTAL ARCHITECTURE CONSTRUCTION USING RL FOR MACHINE TRANSLATION:

For our experiments involving the extended DSL and our RL based generator, we use machine translation as our domain. The candidate architectures produced by the RL agent were directly used without the assistance of a ranking function. This leads to a different kind of generator: whereas the ranking function learns global knowledge about the whole architecture, the RL agent is trimmed towards local knowledge about which operator is ideal to be next.

Training details Before evaluating the constructed architectures, we pre-train our generator to internalize intuitive priors. These priors include enforcing well formed RNNs (i.e. ensuring x t , h t-1 , and one or more matrix multiplications and activations are used) and moderate depth restrictions (between 3 and 11 nodes deep). The full list of priors and model details are in Appendix C1.

For the model evaluation, we ran up to 28 architectures in parallel, optimizing one batch after receiving results from at least four architectures. As failing architectures (such as those with exploding gradients) return early, we needed to ensure the batch contained a mix of both positive and negative results. To ensure the generator yielded mostly functioning architectures whilst understanding the negative impact of invalid architectures, we chose to require at least three good architectures with a maximum of one failing architecture per batch.

For candidate architectures with multiple placement options for the memory gate c t , we evaluated all possible locations and waited until we had received the results for all variations. The best c t architecture result was then used as the reward for the architecture.

Baseline Machine Translation Experiment Details:

To ensure our baseline experiment was fast enough to evaluate many candidate architectures, we used the Multi30k English to German (Elliott et al., 2016 ) machine translation dataset. The training set consists of 30,000 sentence pairs that briefly For the hyper-parameters in our baseline experiment, we use a hidden and word encoding size of 300, 2 layers for the encoder and decoder RNNs, batch size of 64, back-propagation through time of 35 timesteps, dropout of 0.2, input feeding, and stochastic gradient descent. The learning rate starts at 1 and decays by 50% when validation perplexity fails to improve. Training stops when the learning rate drops below 0.03.

Analysis of the Machine Translation Architecture Search Figure 4 shows the relative frequency of each operator in the architectures that were used to optimize the generator each batch. For all the architectures in a batch, we sum up the absolute number that each operator occurs and divide by the total number of operators in all architectures of the batch. By doing this for all batches (x-axis), we can see which operators the generator prefers over time.

Intriguingly, the generator seems to rely almost exclusively on the core DSL (MM , Gate3 , Tanh, Sigmoid , x t , h t-1 ) when generating early batches. The low usage of the extended DSL operators may also be due to these operators frequently resulting in unstable architectures, thus being ignored in early batches. Part way through training however the generator begins successfully using a wide variety of the extended DSL (Sub, Div , Sin, Cos, . . .). We hypothesize that the generator first learns to build robust architectures and is only then capable of inserting more varied operators without compromising the RNN's overall stability. Since the reward function it is fitting is complex and unknown to the generator, it requires substantial training time before the generator can understand how robust architectures are structured. However, the generator seems to view the extended DSL as beneficial given it continues using these operators.

Overall, the generator found 806 architectures that out-performed the LSTM based on raw test BLEU score, out of a total of 3450 evaluated architectures (23%). The best architecture (determined by the validation BLEU score) achieved a test BLEU score of 36.53 respectively, compared to the standard LSTM's 34.05. Multiple cells also rediscovered a variant of residual networks (Add (Transformation(x t ), x t )) (He et al., 2016) or highway networks (Gate3 (Transformation(x t ), x t , Sigmoid (. . .))) (Srivastava et al., 2015) . Every operation in the core and extended DSL made their way into an architecture that outperformed the LSTM and many of the architectures found by the generator would likely not be considered valid by the standards of current architectures. Figure 5 highlights how often the full range of operators occur in architectures that out-performed the LSTM. These results suggest that the space of successful RNN architectures might hold many unexplored combinations with human bias possibly preventing their discovery.

In Table 3 we take the top five architectures found during automated architecture search on the Multi30k dataset and test them over the IWSLT 2016 (English to German) dataset (Cettolo et al., 2016) . Operator frequency of architectures that out-perform LSTM on multi30k

Figure 5 : Operator frequency of architectures that out-perform LSTM on Multi30k (colored like Fig. 4 ). For every architecture with a BLEU score higher than LSTM, we count if an operator occurs in its architecture. While variables x t and h t-1 are inherent to every architecture, the generator also picked up on the Gate3 -Sigmoid combination for every one of its top architectures. Intriguingly, even operators that are less commonly used in the field such as sine curves and positional encoding occur in a large number of architectures and thus seem to contribute to successful architectures. that cover a wide variety of topics with more conversational language than in the Multi30k dataset. This dataset is larger, both in number of sentences and vocabulary, and was not seen during the architecture search. While all six architectures achieved higher validation and test BLEU on Multi30k than the LSTM baseline, it appears the architectures did not transfer cleanly to the larger IWSLT dataset. This suggests that architecture search should be either run on larger datasets to begin with (a computationally expensive proposition) or evaluated over multiple datasets if the aim is to produce general architectures. We also found that the correlation between loss and BLEU is far from optimal: architectures performing exceptionally well on the loss sometimes scored poorly on BLEU. It is also unclear how these metrics generalize to perceived human quality of the model (Tan et al., 2015) and thus using a qualitatively and quantitatively more accurate metric is likely to benefit the generator. For hyper parameters of the IWSLT model, refer to Appendix C3.

5. RELATED WORK:

Architecture engineering has a long history, with many traditional explorations involving a large amount of computing resources and an extensive exploration of hyperparamters (Jozefowicz et al., 2015; Greff et al., 2016; Britz et al., 2017) . The approach most similar to our work is Zoph and Le (2017) which introduces a policy gradient approach to search for convolutional and recurrent neural architectures. Their approach to generating recurrent neural networks was slot filling, where element-wise operations were selected for the nodes of a binary tree of specific size. The node to produce c t was selected once all slots had been filled. This slot filling approach is not highly flexible in regards to the architectures it allows. As opposed to our DSL, it is not possible to have matrix multiplications on internal nodes, inputs can only be used at the bottom of the tree, and there is no complex representation of the hidden states h t-1 or c t-1 as our unrolling ranking function provides. Many other similar techniques utilizing reinforcement learning approaches have emerged such as designing CNN architectures with Q-learning (Baker et al., 2016) .

Neuroevolution techniques such as NeuroEvolution of Augmenting Topologies (NEAT) (Stanley and Miikkulainen, 2002) and HyperNEAT (Stanley et al., 2009) evolve the weight parameters and structures of neural networks. These techniques have been extended to producing the non-shared weights for an LSTM from a small neural network (Ha et al., 2016) and evolving the structure of a network (Fernando et al., 2016; Bayer et al., 2009) .

6. CONCLUSION:

We introduced a flexible domain specific language for defining recurrent neural network architectures that can represent most human designed architectures. It is this flexibility that allowed our generators to come up with novel combinations in two tasks. These architectures used both core operators that are already used in current architectures as well as operators that are largely unstudied such as division or sine curves. The resulting architectures do not follow human intuition yet perform well on their targeted tasks, suggesting the space of usable RNN architectures is far larger than previously assumed. We also introduce a component-based concept for architecture search from which we instantiated two approaches: a ranking function driven search which allows for richer representations of complex RNN architectures that involve long term memory (c t ) nodes, and a Reinforcement Learning agent that internalizes knowledge about the search space to propose increasingly better architectures. As computing resources continue to grow, we see automated architecture generation as a promising avenue for future research.

Given the differences in generated architectures, and the usage of components likely to impact the long term hidden state of the RNN models, we began to explore the progression of the hidden state over time. Each of the activations differs substantially from those of the other architectures even though they are parsing the same input.

As the input features are likely to not only be captured in different ways but also stored and processed differently, this suggests that ensembles of these highly heterogeneous architectures may be effective. 

7.. INFRASTRUCTURE DETAILS:

We ran the architecture search for 5 days on one CPU head node and several worker nodes with a total of 28 GPUs on AWS. 24 GPUs were of the type Tesla K80 and 4 GPUs were of the type Tesla M40. The best architecture (Table 3 , bottom row) was found after 40 hours. However, as evident in Figure 6 , the generator created well-performing architectures more and more consistently with more training.

:



Figures and Tables:

Figure 3: Visualization of the language modeling architecture search over time. Lower log perplexity (y-axis) is better.

Figure 4: Distribution of operators over time. Initially the generator primarily uses the core DSL (faded colors) but begins using the extended DSL as the architecture representation stabilizes.

Figure 6: Visualization of the hidden state over time for a variety of different generated architectures.

Figure 2: An example of generating an architecture from h t up. Nodes which have an empty child (∅) are filled left to right. A source node such as x t can be selected at any time if max depth is exceeded.

Table content: <table><tr><td/><td/><td/><td>; ;</td><td/><td>x t x t</td><td/><td>x t x t</td><td>; ;</td><td colspan="2">x t x t h t 1 h t 1</td></tr><tr><td/><td>; ;</td><td>; ;</td><td>MM</td><td>; ;</td><td>MM</td><td>; ;</td><td>MM</td><td>MM</td><td>MM</td><td>MM</td></tr><tr><td/><td>; ;</td><td>Add</td><td>Add</td><td/><td>Add</td><td/><td colspan="2">Add</td><td colspan="2">Add</td></tr><tr><td>; ;</td><td>Tanh</td><td>Tanh</td><td colspan="2">Tanh</td><td colspan="2">Tanh</td><td colspan="2">Tanh</td><td colspan="2">Tanh</td></tr><tr><td>h t h t</td><td>h t h t</td><td>h t h t</td><td>h t h t</td><td/><td>h t h t</td><td/><td/><td>h t h t</td><td/><td>h t h t</td></tr></table>

Model perplexity on validation/test sets for the Penn Treebank language modeling task.

Table content: <table><tr><td>Model</td></tr></table>

The training set consists of 209,772 sentence pairs from transcribed TED presentations

Table content: <table><tr><td>800</td><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>700</td><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>600</td><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>500</td><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>400</td><td/><td/><td/><td/><td/><td>42%</td><td/><td/><td/></tr><tr><td>0 300 200 100</td><td>Sigmoid</td><td>Var('hm1')</td><td>Gate3</td><td>Var('x')</td><td>MM</td><td>Var('cm1')</td><td>Sin 28% 24% 23% Var('posenc') Tanh</td><td>Var('xm1') 16%</td><td>Cos 9% 8% 7% 6% 6% 6% 4% 4% 1% Selu Mult Sub LayerNorm Add PosEncAdd Relu Div</td></tr></table>

Model loss and BLEU on the Multi30k and IWSLT'16 MT datasets. All architectures were generated on the Multi30k dataset other than the LSTM and BC3 from the LM architecture search. We did not perform any hyperparameter optimizations on either dataset to avoid unfair comparisons, though the initial OpenNMT hyperparameters likely favored the baseline LSTM model.

Table content: <table/>

Peer review (iclr2018):
 The authors introduce a new method to generate RNNs architectures. The authors propose a domain-specific language two types of generators (random and RL-based) together with a ranking function and evaluator. The results are promising, and this research introduces a framework that might enable the community to find new interesting models. However, it*s not clear how these framework compare with previous ones (see below). Also, the clarity of the text could be improved. Pros: 1. An interesting automatic method for generating RNNs is introduced by the authors (but is not entirely clear how does ir compare with previous different approaches) 2. The approach is tested in a number of tasks: Language modelling (PTB and wikipedia-2) and machine translation ( 3. In these work the authors tested a wide range of different RNNs 3. It is interesting that some of the best performing architectures (e.g. LSTM, residual nets) are found during the automatic search Cons: 1. It would be nice if the method didn’t rely on defining a specific set of functions and operators upon which the proposed method works. 2. The text has some typos: for example: “that were used to optimize the generator each batch” 3. In section 5, the authors briefly discuss other techniques using RL and neuroevolution, but they never contrast these approaches with theirs. Overall, it would be nice if the authors had made a more direct comparison with other methods for generating RNNs. 4. The description of the ranking function is not clear. What kind of networks were used? This appears to introduce a ranking-network-specific bias in the search process. Minor comments: 1. The authors study the use of subtractive operators. Recently a new model has considered the use of subtractive gates in LSTMs as a more biologically plausible implementation (Cortical microcircuits as gated-recurrent neural networks, NIPS 2017). 2. Figure 4 missing label on x-axis 3. End of Related Work period is missing. 3. The authors state that some of the networks generated do not follow human intuition, but this doesn’t appear to discussed. What exactly do the authors mean? 4. Not clear what happens in Figure 4 in epoch 19k or so, why such an abrupt change? 5. Initial conditions are key for such systems, could the init itself be included in this framework?