Learning Algorithmic Solutions to Symbolic Planning Tasks with a Neural Computer
Authors: Daniel Tanneberg
Elmar Rueckert
Jan Peters
Venue:  (2020)

Abstract: A key feature of intelligent behavior is the ability to learn abstract strategies that transfer to unfamiliar problems. Therefore, we present a novel architecture, based on memory-augmented networks, that is inspired by the von Neumann and Harvard architectures of modern computers. This architecture enables the learning of abstract algorithmic solutions via Evolution Strategies in a reinforcement learning setting. Applied to Sokoban, sliding block puzzle and robotic manipulation tasks, we show that the architecture can learn algorithmic solutions with strong generalization and abstraction: scaling to arbitrary task configurations and complexities, and being independent of both the data representation and the task domain.

Under review as a conference paper at ICLR 2020
LEARNING ALGORITHMIC SOLUTIONS TO SYMBOLIC PLANNING TASKS WITH A NEURAL COMPUTER
Anonymous authors Paper under double-blind review
ABSTRACT
A key feature of intelligent behavior is the ability to learn abstract strategies that transfer to unfamiliar problems. Therefore, we present novel architecture, based geemrry-aggment
networks, that is inspired by the von Neumann and Har- vard architectures of modern computers. This architecture enables the learning abstract algorithmic solutions via Evolution Strategies in a reinforcement learning setting. Applied to Sokoban, sliding block puzzle and robotic manipulation tasks, we show that the architecture can learn algorithmic solutions with strong general- ization and abstraction: scaling to arbitrary task configurations and complexities, and being independent of both the data representation and the task domain.
INTRODUCTION
Transferring solution strategies from one problem to another is a crucial ability for intelligent be- havior (Silver et al., 2013). Current learning systems can learn a multitude of specialized tasks, but extracting the underlying structure of the solution for effective transfer is an open research prob- lem (Taylor & Stone 2009p. Abstraction is key enable these transfers (Tenenbaum et al.] 201T) and the concept of algorithms in computer science is an ideal example for such transferable abstract strategies. An algorithm a sequence of instructions, which solves a given problem when executed, independent of the specific instantiation of the problem. For example, consider the task of sort- ing a set of objects. The algorithmic solution, specified as the sequence of instructions, is able to sort any number of arbitrary classes of objects in any order, c.g., toys by color, waste by type, or numbers by value, by using the same sequence of instructions, as long as the features and compare operations defining the order specified. Learning such structured, abstract strategies enables the transfer to new domains and representations (Tenenbaum et al.] 201T). Moreover, abstract strategies as algorithms have built-in generalization capabilities to new task configurations and complexities.
Here, we present a novel architecture for learning abstract strategies in the form of algorithmic so- lutions. Based on the Differential Neural Computer (Graves al. (016)) and inspired by the von Neumann and Harvard architectures of modern computers, the architectures modular structure al- lows for straightforward transfer by reusing learned modules instead of relearning, prior knowledge can be included, and the behavior of the modules can be examined and interpreted. Moreover, the individual modules of the architecture can be learned with different learning settings and strate- gies or be hardcoded if applicable allowing to split the overall task into easier subproblems, contrary to the end-to-end learning philosophy of most deep learning architectures. Building on memory-augmented neural networks (Graves et al. 2016 Neelakantan et al. 2016: Weston et al.) 2013 Joulin& Mikolov| 2015), we propose a flexible architecture for learning abstract strategies as algorithmic solutions and show the learning and transferring of such in symbolic planning tasks.
1.1 THE PROBLEM OF LEARNING ALGORITHMIC SOLUTIONS
We investigate the problem of learning algorithmic solutions which are characterized by three quirements: R1 generalization to different and unseen task configurations and task complexities, R2 independence of the data representation, and R3 independence of the task domain.
Picking up the sorting algorithm example again, R1 represents the ability to sort lists of arbitrary length and initial order, while R2 and R3 represent the abstract nature of the solution. This ab- straction enables the algorithm, for example, to sort a list of binary numbers while being trained only hexadecimal numbers (R2). Furthermore, the algorithm trained numbers is able sort

---

Under review as a conference paper at ICLR 2020
data control
Memory
Algorithmic Modulee
Dala
Transformp
previous timestep
4
Compuiationl
A A
A M
4
Output
A
Input
Transform.
ALU
d
Figure I: The proposed architecture with its modules inspired by computer architectures. In this work the modules are based on neural networks. Information flow is divided into data and control streams. The modules inside the highlighted area are learning the algorithmic solution a reinforcement learning setting, whereas the others (data modules) are learned independently in a supervised setting or can use hardcoded information.
lists of strings (R3). If R3 are fulfilled, the algorithmic solution does not need to be retrained or adapted to solve unforeseen task instantiations only the data specific operations need to be adjusted.
Research on learning algorithms typically focuses on identifying algorithmic generated patterns or solving algorithmic problems (Neelakantan et al., 2016:; Zaremba & Sutskever] [004 Kaiser & Sutskever 2016; Kaiser & Bengio 2016), less on finding algorithmic solutions (Joulin & Mikolov] 2015] et al. (016) fulfilling the three discussed requirements R1 R3. While RI is typi- cally tackled, as it represents the overall goal of generalization in machine learning, the abstraction abilities from R2 and R3 are missing. Additionally, most algorithms require a form of feedback, using computed intermediate results from one computational step in subsequent steps, and vari- able number of computational steps to solve problem instance. Thus, it necessary to be able to cope with varying numbers of steps and determining when to stop, in contrast to using fixed number of steps (Neelakantan et al. 2016: Sukhbaatar et al. 2013), making the learning problem more challenging in addition.
A crucial feature for algorithms is the ability to save and retrieve data. Therefore, augmenting neural networks with different forms of external memory, c.g., matrices, stacks, tapes grids, to increase their expressiveness and to separate computation from memory, especially in long time
dependencies setups, is active research direction (Graves al.
2016 | Weston al. 2015 Joulin
& Mikolov| 2015 Zaremba et al. 2016 Sukhbaatar et al. [2015] 2016 with earlier work in the field of grammar learning (Das al.
Kumar et al.] 2016 Greve et al.)
1992 Mozer & Das
1993 Zeng
et al. 1994). These memory-augment
networks improve performance on a variety of tasks like
reasoning and inference in natural language (Graves et al. 2016; Weston al. 2015 Sukhbaatar et al. 2015, Kumar et al., 2016), learning of simple algorithms and algorithmic patterns (Joulin &
Mikolov 201 Zaremba et al. 2016 Graves al. 2014), and navigation tasks (Wayne et al. 2018).
The contribution of this paper is novel modular architecture building on a memory-augmented neural network (DNC (Graves et al.) (016)) for learning algorithmic solutions reinforcement learning setting. We show that the learned solutions fulfill all three requirements RI R3 for an algorithmic solution and the architecture can process a variable number of computational steps.
2 A NEURAL COMPUTER ARCHITECTURE FOR ALGORITHMIC SOLUTIONS
In this section, we introduce the novel modular architecture for learning algorithmic solutions, shown in Figure [ The architecture builds on the Differential Neural Computer (DNC) (Graves al.) (016)) and its modular design inspired by modern computer architectures, related to (Nee- lakantan et al. 2016 Weston et al. 2015).
The DNC augments a controller neural network with a differentiable autoassociative external mem- ory to separate computation from memory, as memorization is usually done in the networks weights. The controller network learns to write and read information from that memory by emitting an inter- face vector which is mapped onto different vectors by linear transformations. These vectors control the read and write operations of the memory, called read and write heads. For writing and read- ing, multiple attention mechanisms are employed, including content lookup, temporal linkage and memory allocation. Due to the design of the interface and the attention mechanisms, the DNC is in- dependent of the memory size and fully differentiable, allowing gradient-based end-to-end learning.
2

---

Under review as a conference paper at ICLR 2020
Our architecture. In order to learn algorithmic solutions, the computations need to be decoupled from the specific data and task. To enable such data and task independent computations, we propose multiple alterations and extensions to the DNC, inspired by modern computer architectures.
First, information flow is divided into two streams, data and control. This separation allows to disen- tangle data representation dependent manipulations from data independent algorithmic instructions. Due to this separation, the algorithmic modules need to be extended to include two memories, a data and a computational memory. The data memory stores and retrieves the data stream, whereas the computational memory works on information generated by the control signal flow through the learn- able controller and memory transformations. The two memories are coupled, operating on the same locations, and these locations are determined by the computational memory, and hence by the con- trol stream, As with the DNC, multiple read and write heads can be used. In our experiments, one read and two write heads are used, with one write head constrained to the previously read location.
In contrast to the DNC, but in line with the computer rechitecture-insprre design and the goal of learning deterministic algorithms, writing and reading uses hard attentions instead of soft atten- tions. Hard attention means that only one memory location can be written to and read from (unique addresses), instead of an weighted average over all locations with soft attentions. Such hard at- tention was shown to be beneficial for generalization (Greve et al. 2016). We also employed an additional attention mechanism for reading, called usage linkage, similar to the temporal linkage of the DNC, but instead of capturing temporal relations, it captures usage relations, i.e., the relation between written memory location and previously read location. With both linkages in two direc- tions and the content look up, the model has five attention mechanisms for reading. While the final read memory location is determined by weighted combination of these attentions (see attention in Figure 3 in the Appendix), each attention mechanism itself uses hard decisions, returning only one memory location. See Appendix Cfo the effect of the introduced modifications and extensions.
For computing the actual solution, operating only on the control stream is not enough, as the model still needs to manipulate the data. Therefore, we added several modules operating on the data stream, inspired by the architecture of computers. In particular, an Input, Transformp. ALU (arithmetic logic unit) and Output module were added (more details in Section 2.2 These modules manipulate the data, steered by the algorithmic modules. The full architecture shown in Figure] I
As algorithms typically involve recursive or iterative data manipulation, the model receives its own output as input in the next computation step, making the whole architecture output-input model. With all aforementioned extensions, algorithmic solutions fulfilling R1 R3 can be learned.
2.1 THE ALGORITTMIC MODULESS
The algorithmic modules consist of the Controller, the Memory and the Transforme module and build the core of the model. These modules learn the algorithmic solution operating on the control stream. With t as the current computational step and e as the control stream (see Figure I.. the input-output of the modules are C(Ci,t= 1,,t-1, Cf.t-1 1 C,,t-1, C,, m Ce.t (C Co,t)
Cm,t; dm, and Te(Ce,t, Cm.t, (i.e) > Cj,t- The algorithmic modules are based on the DNC with the alterations and extensions described before. Next we discuss how these algorithmic modules can be learned before looking into the data-dependent modules.
2.1.1 LEARNING OF THE ALGORITTMIC MODULES
Learning the algorithmic modules, and hence the algorithmic solution, is done in reinforcement learning setting using Natural Evolution Strategies (NES) (Wierstra et 20.4). NES is a blackbox optimizer that does not require differentiable models, giving more freedom to the model design, e.g., the hard attention mechanisms are not differentiable. NES updates search distribution of the parameters to be learned by following the natural gradient towards regions of higher fitness using a population of offsprings (altered parameters) for exploration. Let A be the parameters to be learned and using an isotropic multivariate Gaussian search distribution with fixed variance o², the stochastic natural gradient at iteration & is given by
P Σ
Vo, Vo, Iu(O, + ae)] R Po
u(0))e
3

---

Under review as a conference paper at ICLR 2020
where P is the population size and u(+) is the rank transformed fitness
(Wierstra et al. 2014). The
parameters are updated by
P u (0)).
0e+1 =0= + Po
with learning rate Recent research showed that NES and related approaches like Random Search (Mania et al., 2018) or NEAT (Stanley & Mikkkulanenn, 2002) are powerful alternatives in reinforeement learning. They are easier to implement and scale, perform better with sparse rewards and credit assignment over long time scales, have fewer hyperparameters (Salimans et al. 2017) and
were used to train memory-augmentt networks (Greve al. 2016 Merrild et al. 2018
For robustness and learning efficiency, weight decay for regularization (Krogh & Hertz 1992) and
automatic restarts of runs stuck in local optima used in (Wierstra et al. 2014). This restarting can be seen as another level of evolution, where some lineages die out. Another way of dealing with early converged or stuck líneages to add intrinsic motivation signals like novelty, that help get attracted by another local optima, as in NSRA-ES (Conti et al. (018)) In the experiments however, we found that within our setting, restarting or having an additional survival of the fittest on the lineages - was more effective, see Appendix C for a comparison.
The algorithmic solutions are learned in curriculum learning setup (Bengio et al.) 2009) with sampling from old lessons (Zaremba & Sutskever, 2014) to prevent unlearning and to foster gener- alization. Furthermore, we created bad memories, a learning from mistakes strategy, similar to the idea of AdaBoost (Freund & Schapire| 1997), which samples previously failed tasks to encourage focusing on the hard tasks. This can also seen as form of experience replay (Mnih et al. 2015] Lin 1992), but only using the task configurations, the initial input to the model, not the full gen- crated sequence. Bad memories were developed for training the data-dependent modules to ensure their robustness and 100% accuracy, which is crucial to learn algorithmic solutions. If the individual modules do not have 100% accuracy, no stable algorithmic solution can be learned even if the algo- rithmic modules are doing the correct computations. For example, if one module has accuracy of 99%, the 1% error prevents learning an algorithmic solution that works always. This problem is even reinforced computation steps using its own output as the new input meaning the overall accuracy drops to 36.6% for 100 computation steps. Therefore using the bad memories strategy, and thus focusing on the mistakes, helps significantly in achieving robust results when learning the modules, enabling the learning of algorithmic solutions. While the bad memories strategy was crucial to achieve 100% ro- bustness when training the data-dependent modules, the effect on learning the algorithmic solutions was less significant (see Appendix C for an evaluation).
the proposed model is output-input architecture that works over multiple
2.2 DATA-DEPENDENT MODULES
The data-dependent modules (Input, ALU, Transformp and Output) are responsible for all operations that involve direct data contact, such as receiving the input data from the outside or manipulating a data word with an operation chosen by the algorithmic modules. Thus, these modules need to be learned or designed for specific data representation and task. However, as all modules only have to perform a certain subtask, these modules are typically easier train.
As learning the algorithmic modules via NES does not rely on gradients and due to the infor- mation flow split, the data-dependent modules can be instantiated arbitrarily, e.g., can have non- differentiable parts, do not need to be neural networks or can be hardcoded. Therefore, prior knowl- edge can be incorporated by implementing it directly into these modules. The modular design facilitates the transfer of learned modules, e.g., using the same algorithmic solution in new domain without retraining the algorithmic modules or learning a new algorithm within the same domain without retraining the data modules. Next the general functionality of the modules will be explained.
The Input module is the interface to the external world and responsible for data preprocessing. Therefore, it receives the external input data and the data from the previous computational step. It sends data to the memory and control signals to the subsequent modules with information about the presented data or the state of the algorithm formally as I(da,,, do, > Ci,e, di,--
The ALU module performs the basic operations which the architecture can use to modify data. Therefore, it receives the data and control signal indicating which operation to apply and outputs
4

---

Under review as a conference paper at ICLR 2020
sturt configaatooo
9
dech
degth 0000
92
9g
dict
94
caddmaadeng
destt
%
812
goel contiumaiin
(a) Complexity that triggered learning.
(b) Complexity solved by the algorithmic solution.
Figure 2: Examples of search trees that the architecture implicitly learned to generate to solve given symbolic planning task, where s; corresponds to task configurations and az to ALU operations that transform the task configuration. (a) corresponds to a task from curriculum level 3 with a maximum number of computations steps of 15 including backtracking. (b) shows the tree for task that required 330.631 computation steps (corresponds level 82.656) that was solved by an algorithmic solution that triggered learning only until 15 steps, the complexity shown in (a).
the modified data and control signals about the operation A(Cf,t; dx,a)) - Ca,t, du, As in many applications the basic operations only modify a part of the data and to reduce the complexity of the ALU, Transformp module extracts the relevant part from the data beforehand -DD (dm.t) + dz or just transfers the unmodified data if no transformation is required for the task.
The Output module combines the result of the data manipulation operation from the ALU module and the data before the manipulation. It inserts the local change done by the ALU into the original data word OCa.,s da.t, d > Co.t, do,t. As before with the Transformation module, depending on the task, the Output module can also be designed to just pass on the received data.
EXPERIMENTS
We investigate the learning of symbolic planning tasks, where task complexity is measured as the number of computational steps required solve task, i.e., the size of the corresponding search tree (see Figure 2). Learning is done in the Sokoban domain, whereas the generalization and abstraction requirements RI R3 are shown by transferring (1) longer planning tasks, (2) bigger Sokoban worlds, (3) a different data representation, and (4) two different task domains sliding block puzzle and robotic manipulation.
In Sokoban, an agent interacts in grid world with four actions moving up, right, down or left. Therefore, the ALU can perform four operations and additionally a nop operation that leaves the given configuration unchanged. The world contains empty spaces that can be entered, walls that block movement and boxes that can be pushed onto empty space. task given by a start configu- ration of the world and the desired goal configuration. For learning, we use world of size 6 6 that is enclosed by walls. A world is represented with binary vectors and cour-dimensional one-hot en- codings for each position, resulting in 1 144-dimensional data words. The configuration of each world inner walls, boxes and agent position is sampled randomly. Each world is generated by sampling uniformly the number of additional inner walls from [0, 2] and boxes from [1, 5]. The positions of these walls, boxes and the position of the agent are sampled uniformly from the empty spaces. An example task and the learned solution is shown in the Appendix in Figure 5 the penguin is the agent, icebergs are boxes, iceblocks are walls and water is empty space.
3.1 ALGORIHHMICC MODULES
In the experiments we use a feedforward neural network as Controller with a layer size of 16 neurons and tanh activation. The Transforme is linear layer projecting its 27-dimensional input onto the operations of the ALU using leaky-ReIJ activation and one-hot encoding. The computational memory has word size of 8 bit, the Input module generates 3 control signals (2 for Learning to Search), and the ALU and Output module control signal feedback is not used here. Thus, the input to the Controller consists of 16 control signals and in total there are about 1600 parameters.
3.1.1 LEARNING OF THE DATA- DATA-DEPENDEN MODULES
All data-dependent modules are trained in a supervised setting and consist of feedforward networks. They optimize cross entropy loss using Adam (Kingma & Ba 2015) mini-batch size of
5

---

Under review as a conference paper at ICLR 2020
20. To improve their generalization and robustness, the bad memories mechanism described in Section 2.1.1 is used with a buffer size of 200 and 50% of the samples within a mini-batch are sampled from that. The following task-dependent instantiations of the data-dependent modules are examples used for the Sokoban domain.
The Input module learns an equality function using differential rectifier units inductive bias (Weyde& & Kopparti 2018) and consists of a feedforward network with 10 hidden units and leaky-ReIl activation. Using the learned binary equality signal Le, step t, it produces three bi-
[2] [2] = Less + i,t-11 and [i] = le,ec, f -
nary control signals according to Ci. = (1-1e,)) f indicating the different phases of the algorithm. For the Learning to Search experiment only the first two signals are used.
The Transformp module extracts a different view on the data, if required by the ALU, as de- scribed in Section 2.2 Here, it consists of feedforward network with 500 hidden neurons and uses leaky-Retan activation. For the Sokoban domain, the actions that the agent can take - and therefore the operation the ALU can apply only change the world locally. Thus, the Transformp module extracts a local observation of the world dy. i... the agent and the two adjacent locations in all four directions, these are the only locations where action can produce change.
The ALU module receives the data view extracted by Transformp and the control signal from Transformc, that encodes the operation to apply. It learns to apply the operations, i.e., it learns an action model by learning preconditions and effects, and outputs the (potential) local change together with control signal indicating if the action changed the world not. The local change is encoded as the direction of the change and the three according spaces. The module consists of two feedforward networks, one for the control signal Ca and one for applying the actions producing the manipulated data da- The learned C, is used to gate the output between the output of the action network and the data input without change. The control network has two hidden layers with sizes [64, 64], the action network has hidden layers with [128, 64) neurons and both use leaky-Re activations.
The Output module inserts the (locally) changed data from the ALU into the data stream. It receives the data from the memory dmn, the data d, and control stream Ca from the ALU. It consists of two feedforward networks for learning the data d, and the control signal e, stream. The control network has two hidden layers with sizes [500, 250]. the data network has hidden layers with [500, 500] neurons and both use leaky-Reeee ReUU activations. The control signal Co is used for gating between the data with the inserted change and the original data dm. To ensure that the Output module uses the manipulated data of the ALU and is not learning to manipulate the data itself, it is constrained to learn binary mask that indicates where the change needs to be inserted. This binary map indicates for each position in da where to insert it in dm and can be seen as a structured prediction problem. Note, the training data only consists of data and control signals, the true binary mask is not known.
3.2 LEARNING ALGORIHHIIC SOLUTIONS
We investigate the learning of two algorithms,, (1) a search algorithm and (2) search-based planning algorithm. The data-dependent modules do not need be retrained for the different algorithms. For evaluating that the learned strategy is an abstract algorithmic solution, we show that it fulfills the three requirements RI R3 discussed in Section 1.1
3.2.1 LEARNING SEARCH
In the first task, the model has to learn For that purpose, the initial input the model is the start and goal configuration and subsequent inputs are the goal configuration and the output of the model from the previous computation step. To solve the task, the model has to learn to produce the correct search tree and recognizing that the goal configuration is reached by choosing the nop operation for the correct computation step.
to find the desired goal configuration.
For the curriculum learning the levels are defined as the number of nodes from the search tree that have to be fully explored, e.g., for Level 1, up to five correct computation steps have to be performed on the initial configuration; for Level 3 the initial configuration as well as the two subsequently found configurations need to be fully explored (see Figure a)). This requires up to 13 correct computational steps. Curriculum levels are specified up to Level 21 that involves up to 85 correct computation steps to be solved. An additional Level 22 is activated afterwards that consists of new samples from all 21 levels for evaluation. To prevent unlearning of previous levels, 20% of the samples in the mini-batch are sampled uniformly from previous levels. As in (Wierstra et al. 2014)

---

Under review as a conference paper at ICLR 2020
restarting, but here the run automatically restarts if the maximum fitness of a level is not reached within 2500 iterations. All experiments have a total budget of 10.000 iterations.
The fitness function f uses step-wire wise binary losses computed as comparison to the correct solution over mini-batches of N samples and is defined as
f!") fem + s!m
if N E s!a < otherwise
100
N
with
(I)
f!yl
100
Σ T
1=1 I(cf)) = 2/rl) + 21 2I = d
and fgm 20(( l.1!= = nop)
3T!
where T. is the number of steps required for constructing the search tree when the first mistake occurs, Cf.t is the operation chosen to be applied by the ALU from Transforme: at step l, dm, the data word read from the memory, and ēÿ,, and dm,t are the correct choices respectively. The exploration fitness s!() captures the fraction correct computation steps until the goal configuration is found, scaled to 0-100%. Note that, NES therefore only uses a single scalar value that summarizes the performance of the parameters over N samples and all computational steps. The learning rate a is to 0.01, the a of the search distribution to 0.1, weight decay is applied with 0.9995, mini-bated size N 20 and the population size is 20.
We use gini coefficient based ranking that gives more importance to samples with higher fit- ness (Schaul et al., 2010). The maximum fitness 120 for all levels and level is solved when 250 subsequent iterations have the maximum fitness, i.e., 5000 samples are solved correctly. The bad memories consist 200 samples and 25% of the samples within a mini-batch are sampled uni- formly from those. Whenever 10 subsequent iterations achieve the maximum fitness, the buffer is cleared and no learning is performed.
3.2.2 LEARNING TO PLAN (SEARCH + BACKTRACK)
In the second task, the model has to learn, in addition to the wreatth-frst-algorith that computes search tree the goal configuration, to also extract the path from the search tree that encodes the solution to the given planning problem (see Figure 2 and Figure in the Appendix). Therefore, the model has to not only learn to encode and perform two different algorithms, but also to switch between them at the correct computation step.
The initial input the model is the start and goal configuration and subsequent inputs are the goal configuration and the output of the model from the previous computation step, as before. When the goal configuration is found by the model, the input is the start configuration and the previous output. To solve the task, the model has to learn to produce the search tree and recognizing that the goal configuration reached as before. In addition, after recognizing the goal configuration, the model needs to switch behavior and output the path of the search tree encoding the planning solution. This solution consists of the states from the initial the goal configuration and nop operations in reverse order. Therefore, the number of maximum computation steps increases up to 89 in Level 21. The fitness function is defined as in Equation equation | but with
50
E T! = nop) + 2d = Jin
Σ
+T!""
37, mn
where T3 is the number of steps required for backtracking the solution or when the first mistakes occurs. The maximum fitness 150 and all other settings remain before.
3.3 R1 GENERALIZATION TO UNSEEN TASK CONFIGURATION AND COMPLEXITIES
A main goal in all learning tasks, is to achieve generalization to not only learn to solve seen situations, but to learn a solution that generalizes to unseen situations. One evaluation of this gen- eralization ability is built into our learning process itself. A curriculum level is solved after 250 subsequent iterations (5000 samples) with maximum fitness and iterations with maximum fitness do not trigger learning. Thus, if presenting new level that involves more complex tasks, the fitness stays at maximum and no learning is triggered, the previously learned solution generalizes to the new setting generalizes to more complex tasks (see Figure
7

---

Under review as a conference paper at ICLR 2020
Dalk slvie
33000
10000
Iteeatons 21393 trenings
Reritiees 12583 trannee
(a) Learning to Search (ours).
(b) Learning to Plan (ours).
Leirtr ta bata laant kg bo 3100 4953
100
Last solvet
50
927
carl om beall
(c) Learning to Search comparison.
(d) Evaluation over 15 runs (ours).
Figure 3: (a-c) The gray dashed line marks the maximum fitness and the colored lines show the fitness. The light colors indicate that the maximum fitness is achieved and no learning is triggered. The colored dashed lines indicate when curriculum level was solved successfully. When no learning triggered after a new level unlocked, the model generalized to more complex tasks. The top numbers indicate the number of computational steps the model needs to perform correctly to solve samples of the associated curriculum level. (c) Comparison with the original DNC and a auemmeeeee neural network on the Learning to Search task over 10 runs. In contrast entropy loss, i.e.. have a richer and localized training signal. For comparison the mean and standard deviation of the same fitness function that our model uses for training is shown. Both are not able to successfully solve Level I within considerably more iterations. (d) 15 runs of the two learning tasks, highlighting that learning happens during the first levels and generalizes to the subsequent levels. Bar plot shows mean and standard deviation of the number of learning iterations, numbers top of the bars show the number of runs that triggered learning that level. Lower plot shows the number runs that solved the according curriculum level, i.e., where they ended after the budget of 10.000 iterations.
architecture, both methods trained in supervised setup with gradient descent and cross-
This generalization is shown in Figure 3] For example, in the Learning to Plan setup (Figure ()),, after 3 levels the algorithmic solution is found and no learning is triggered anymore during the run. Moreover, the last triggered learning was for curriculum Level 3 - meaning a complexity of 15 computational steps and the found solution generalizes up to the highest specified curriculum Level 21 with 89 computational steps. Learning the algorithmic solution is done within 3 levels and 2563 iterations. Figure d)) shows the evaluation of learning to solve the two tasks over 15 runs each. In contrast, the original DNC (Graves et al., 2016) model and a aageeeee recurrent neural network for algorithmic patterns (Joulin & Mikolov 2015) are not able to solve Level 1 when trained in a supervised setup with gradient descent and considerably more training iterations, see Figure 3 ee) and the Appendix B for implementation details.
Task complexity. Additionally, we evaluated the learned algorithmic solution with task complexities far beyond the specified currieulum learning levels, i.e., complexities experienced during training. Therefore, we used the run shown in Figure 3b)) and solved tasks requiring 330.631 computational steps (corresponds level 82.656), having been trained only up to 15 steps (see Figure 2 for the complexities) and having been tested during training only up to 89 steps. Remember the models recurrent output-input structure, given the initial task input, the model performs 330.631 compu- tational steps, i.e., learns to build a search tree with over 330.600 nodes, autonomously correct to compute and output the solution. Moreover, the solution learned x 6 environments, successfully solved all tasks within 8 x 8 environments. Thus, the learned strategy represents an abstract algorith- mic solution that generalizes and scales to arbitrary task configurations and complexities, fulfilling R1. The learned algorithmic solution is explained with an example in the Appendix A
3.4 R2 INDEPENDENCE OF THE DATA REPRESENTATION
Algorithmic solutions are independent of the data representation, meaning the abstract strategy is still working if the encoding is changed, long the data-dependent operations are adjusted. Consider again a sorting algorithm. Its algorithmic behavior stays the same independent of has to sort a list of numbers encoded binary hexadecimally, as long the compare operators are defined. To show that our learned algorithmic solutions have this feature and fulfill R2, we change the representation of the data, but reuse the learned algorithmic modules and the model can
8

---

Under review as a conference paper at ICLR 2020
PWW
cord
Di D4 D
150
17455 2013
FENIS Evikatee
Krke ndaar Invel
CUITICA ana SOLVEE
W/s.miaaar Lover
OLITIUAA livdd GiEre
Ieeacions For (maninn)
Inear oms traranid
Persffone (3 for traniingt
Figure 4: Transferring the learned algorithmic solution (left) to new data representation (R2) and (middle & right) to two new task domains (R3). In all setups, all 200.000 samples over all curriculum levels are solved correctly without triggering learning, indicated by the constant maximum fitness, showing the straightforward transfer the learned solution the abstract features R2 and of the learned solution.
still solve all tasks without retraining. The data-dependent modules are adapted and relearned. The changed representation, e.g., the penguin represents a wall instead of the agent, and results over 10.000 iterations (200.000 samples) over all curriculum levels are shown in Figure 4 (left). The fitness is at maximum from the start, showing that all samples in all levels are successfully solved without triggering learning while operating on the new data representation and hence, R2 is fulfilled.
3.5 R3- INDEPENDENCE OF THE TASK DOMAIN
Requirement R3 states that an algorithmic solution is independent of the task domain. Consider again the sorting algorithm example: as long as the compare operators are defined, it is able sort arbitrary objects. Therefore, the data-dependent modules are adapted and relearned but we reuse the learned algorithmic solution on two new task domains.
As new domains, 3 x sliding block puzzles and a robotic manipulation task are used (Figure Configurations are represented with binary vectors as described for Sokoban in Section 3 For the puzzle domain, actions are sliding adjacent tiles onto the free (white) space from four directions. A task configuration is given a start and goal board configuration. In the robotic manipulation domain, a task is given as start and goal configuration of the objects. The available actions are the four locations gripper to the position and place the grasped object on top, or to pick up the top object if no object grasped. The maximum stacking height is 3 boxes, resulting in discrete representation of the object configuration with a 3 x 4 grid. As with the new data representation, the learned algorithmic solution able solve all 200.000 presented samples from all curriculum levels in the new domains without triggering learning (Figure 4), showing the independence of the task domain, fulfilling R3,
which objects can be stacked, e.g., the action posl encodes to move the
CONCLUSION
We present novel architecture for learning algorithmic solutions and showed how it can learn abstract strategies that generalize and scale to arbitrary task configurations and complexities (RI) (Section 3.3 and are independent to both, the data representation (R2) (Section 3.4) and the task domain (R3) (Section 3.5). Such algorithmic solutions represent abstract strategies that can be transferred directly to novel problem instantiations, a crucial ability for intelligent behavior.
To show that our architecture is capable of learning strategies fulfilling the algorithm requirements R1 R3 in symbolic planning tasks, we performed experiments with complexities orders of magni- tude higher than seen during training (15 vs. 330.631 steps, and Figure P & 33. and transferred the learned solution bigger state spaces, a new data representation and two new task domains (Fig- 4- 1 showing, to the best of our knowledge, for the first time how such abstract strategies can be represented and learned with memory-augmented networks. The learned algorithmic solution can be applied to any problem that can be framed as such symbolic search or planning problem.
The modular structure and the information flow of the architecture enable the learning of algo- rithmic solutions. the transfer of those, and the incorporation of prior knowledge. Using Natural Evolution Strategies for learning removes constraints on the individual modules, allowing for arbi- trary module instantiations and combinations, and the beneficial use of non-differentiable memory module (Greve et al. 2016).. As the complexity and structure of the algorithmic modules need to be specified, it is an interesting road for future work to learn these in addition, building on the ideas from Greve et al. (2166); Merrild et al. (2018). Showing how algorithmic solutions characterized by RI - R3 can be represented and learned with augmente networks sets the foundation for future work, extending beyond symbolic planning and incorporating intrinsic motivation (Oudeyer Baldassarre & Mirolli 2013) to discover new and unexpected strategies.
& Kaplan 2009

---

Under review as a conference paper at ICLR 2020
REFERENCES
Gianluca Baldassarre and Marco Mirolli. Intrinsically motivated learning systems: an overview. In Intrinsically motivated learning in natural and artificial systems, pp. 1-14. Springer, 2013.
Yoshua Bengio. Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In International Conference on Machine Learning, pp. 41 41-48. ACM, 2009.
Edoardo Conti, Vashisht Madhavan, Felipe Petroski Such, Joel Lehman, Kenneth Stanley, and Jeff Clune. Improving exploration in evolution strategies for deep reinforcement learning via pop- ulation of novelty-seeking agents. In Advances in Neural Information Processing Systems, pp. 5027-5038, 2018.
Sreerupa Das, C Lee Giles, and Guo-Zheng Sun. Learning context-free grammars: Capabilities and limitations of recurrent neural network with an external stack memory. In Conference of Cognitive Science Socieryy, pp. 14, 1992.
Yoav Freund and Robert E Schapire. A docision-theoretie generalization of online learning and an application to boosting. Journal of Computer and System Sciences, 15(1):119-15)) 1997.
Alex Graves, Greg Wayne, and Ivo Danihelka. Neural turing machines. arXi::141054011, 2014.
Alex Graves, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka Grabska- Barwińska, Sergio Gómez Colmenarejo, Edward Grefenstette, Tiago Ramalho, John Agapiou, et al. Hybrid computing using neural network with dynamic external memory. Nature, 538 (7626):471, 2016.
Rasmus Boll Greve, Emil Juul Jacobsen, and Sebastian Risi. Evolving neural turing machines for reward- seward-based learning. In Proceedings of the Genetic and Evolutionary Computation Conference 2016, pp. 117-124. ACM, 2016.
Armand Joulin and Tomas Mikoloy. Inferring algorithmic patterns with stack-augmented recurrent nets. In Advances in Neural Information Processing Systems, pp. 190-198, 2015.
Eukasz Kaiser and Samy Bengio. Can active memory replace attention? In Advances in Neural Information Processing Systems, pp. 3781-3789, 2016.
Lukasz Kaiser and Ilya Sutskever. Neural GPUs learn algorithms. International Conference on Learning Representations, 2016.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. International
Conference on Learning Representations, 2015.
Anders Krogh and John Hertz. A simple weight decay can improve generalization. In Advances in Neural Information Processing Systems, pp. 950-957, 1992.
Ankit Kumar, Ozan Irsoy, Peter Ondruska, Mohit lyyer, James Bradbury, Ishaan Gulrajani, Victor Zhong, Romain Paulus, and Richard Socher. Ask me anything: Dynamic memory networks for natural language processing. In International Conference Machine Learning, pp. 1378-1387, 2016.
Long-Ji Lin. Self-improving reactive agents based on reinforcement learning, planning and teaching. Machine Learning, 8(3-4):293-321, 1992.
Horia Mania, Aurelia Guy, and Benjamin Recht. Simple random search of static linear policies is competitive for reinforcement learning. Advances in Neural Information Processing Systems, pp. 1880-1882.. 2018.
Jakob Merrid,, Mikkel Angaju Rasmussen, and Sebastian Risi. HyperNTM: evolving scalable neu- ral turing machines through HyPerNEAT. In International Conference on the Applications of Evolutionary Computation, pp. 750-766. Springer, 2018
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529, 2015.
10

---

Under review as a conference paper at ICLR 2020
Michael C Mozer and Sreerupa Das. connectionist symbol manipulator that discovers the structure of context- free languages. In Advances Neural Information Processing Systems, pp. 563-870, 1993.
Arvind Neelakantan, Quoc V Le, and Ilya Sutskever. Neural programmer: Inducing latent programs with gradient descent. International Conference on Learning Representations, 2016.
Pierre Yves Oudeyer and Frederic Kaplan. What is intrinsic motivation? typology of computa- tional approaches. Frontiers in neurorobotics, 1:6, 2009.
Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and Ilya Sutskever. Evolution strategies as a scalable alternative reinforcement learning. arXiv: (tt:::/07303444 2017.
Tom Schaul, Justin Bayer, Daan Wierstra, Yi Sun, Martin Felder, Frank Sehnke, Thomas RückstieBß and Jürgen Schmidhuber. Pybrain. Journal of Machine Learning Research, 11(Feb):743-466 2010.
Daniel Silver, Qiang Yang. and Lianghao Li. Lifelong machine learning systems: Beyond learning algorithms. In AAAI Spring Symposium: Lifelong Machine Leurning, volume 13, pp. 05, 2013.
Kenneth O Stanley and Risto Mikkkulainen. Evolving neural networks through augmenting topolo- gies. Evolutionary computation, 10(2):99-127, 2002.
Sainbayar Sukhbaatar, Jason Weston, Rob Fergus, et al. End-to-end memory networks. In Advances in Neural Information Processing Systems, pp. 2440-2448, 2015.
Matthew Tayllor and Peter Stone. Transfer learning for reinforcement learning domains: survey. Journal of Machine Learning Research, 100ul):1633-1683 2009.
Joshua B Tenenbaum, Charles Kemp, Thomas L Griffiths, and Noah D Goodman. How to grow mind: Statistics, structure, and abstraction. Science, (61(622)) 1285, 2011.
Greg Wayne, Chia-Chun Hung, David Amos, Mehdi Mirza, Arun Ahuja, Agnieszka Grabska- Barwinska, Jack Rae, Piotr Mirowski, Joel Z Leibo, Adam Santoro, al. Unsupervised predictive memory in goal-directed agent. arXiv: 10760, 2018.
Jason Weston, Sumit Chopra, and Antoine Bordes. Memory networks. In International Conference on Learning Representutions, 2015.
Tillman Weyde and Radha Manisha Kopparti. Feed-forward neural networks need inductive bias to learn equality relations. crXiv:18/2.01663) 2018.
Daan Wierstra, Tom Schaul, Tobias Glasmachers, Yi Sun, Jan Peters, and Jürgen Schmidhuber. Natural evolution strategies. Journal of Machine Learning Research, 15())949-9880 2014.
Wojciech Zaremba and Ilya Sutskever. Learning to execute. arXiv: 140.44541..4
Wojciech Zaremba, Tomas Mikolov, Armand Joulin, and Rob Fergus. Learning simple algorithms from examples. In International Conference on Machine Learning, pp. 421 421-429, 2016.
Zheng Zeng, Rodney M Goodman, and Padhraic Smyth. Discrete recurrent neural networks for grammatical inference. IEEE Transactions Neural Networks, 5(2):320-330, 1994.

---

Under review as a conference paper at ICLR 2020
A BEHAVIOR OF THE LEARNED ALGORITHMIC SOLUTION
Figure 5 highlights the learned algorithmic behavior one memory location is read with content lookup attention repeatedly until all operations have been applied, the node is fully explored. Then attention shifts towards temporal linkage to read the лех! data to be explored. This pattern contin- uous until the goal configuration is found in step 11. After that, behavior changes output the backtracking solution by switching to usage linkage attention and nop operations until reaching the initial configuration.
conTe
03
11
Output e
corresponding search tree backtracking the found solution
search
backtrack
Figure 5: The behavior of the learned model on task from Level 3 (see Sec. 3.2 for details) and the corre- sponding search tree that is constructing implicitly. In the search phase, the model fully explores node by successively applying all operations, before reading the node, until the goal is found. Then behavior changes the backtrack phase, where solution of the planning task is emitted as states from start to goal in reverse order along with nop operations. The algorithmic behavior can also be seen in the repetitive patterns of the attention vector. showing the five attention mechanisms for reading (temporal and usage linkage both directions, and content lookup), that represents how strong each mechanism for reading used in each computation step.
B DETAILS ON THE IMPLEMENTATIONS OF THE COMPARISON METHODS
Both models, the orignal Differential Neural Computer (DNC) (Graves et al. 2016) and the stack- augmented recurrent network (Joulin & Mikolov 2015) are trained in supervised setting with cross-entropy losses for 500.000 iterations to compensate the pretraining of the data modules. They use the same output-input loop our architecture, i.e., receiving their next computation step in addition to the goal configuration. The loss is computed based on the correct sequences of configurations and the control signal indicating that the goal has been reached, similar like the fitness function from our architecture in equation | Both use LSTM network with 256 hidden units as controller and the memory word size is set to 152, equal our model. Like our architecture, the DNC has one read and two write heads. The augmened model uses four stacks with the three actions PUSH, POP, and NO.OP.
output input in the
Controtoo
Monoy
and tempors lina
Ingut
outout
V
Texd
Salt
$100
stack(t-1]
stack())
(a) Differential Neural Computer, reprinted with permission from (Graess et al. 2016).
(b) Stack-augmented recurrent net-
work, reprinted with permission
from (Joulin & Mikolov 2015).
Figure
12

---

Under review as a conference paper at ICLR 2020
C EVALUATION OF THE LEARNING PROCESS AND MODEL COMPONENTS
For evaluation the effect of the individual modifications and extensions we compared our architec- ture with and without them on the Learning Search task. In all setups all runs had budget of 10.000 iterations. The bar plots show mean and standard deviation of the number of learning itera- tions, numbers on top the bars show the number runs that triggered learning in that level. Plots below the bar plot show the number of runs that successfully solved the according curriculum level, ie., where they ended after the budget of 10.000 iterations. All comparisons are done without the restarting mechanisms, except in the evaluation for that mechanism.
NOVELTY AND RESTARTS
Here two mechanisms to face the problem getting stuck in local optima are evaluated, namely the automatic restart in the original NES (Wierstra al. 2014) and the of additional novelty signal as in NSRA-ES (Conti et al. 2018). For the novelty calculation, we defined the behavior as the sequence of read memory locations and applied ALU operations. The bascline model does not use either of the two mechanisms. While we did not observe an improvement using novelty, the automatic restarts reduced the number of learning iterations, see Figure 7 Note that the baseline and novelty model are also able to learn algorithmic solutions, but they require more iterations and, hence, they die out before final curriculum level due to reaching the budget of 10.000 iterations.
aclive learning iterations
4000
baseline (total learning iterations: 4515 # 1684 )
with novelty (total learning iterations: 4509 = 2680 )
with restarts (total learning iterations: 2601 1253)
2000
1000
1080
0
00
080
000
000
100
000
000
100
001
000
000
001
000
000 010 010 000
1
3
4
5
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
curriculum levels
Figure 7: Evaluation of additional novelty signal and automatic restarts.
CONSTRAINED WRITE HEAD
Here evaluated the introduced constrained write head, that updates the previously read memory location. We compared against two models without this constrained head, one with one write head and with two write heads compensate the missing constrained head. The constrained head was a necessary modification to enable the efficient learning of algorithmic solutions, see Figure
aclive learning Iterations
full model (total learning iterations: 4515 1684
5000
constrained write head write head) (total learning iterations: 6231 2373
8000
constrained write head write heads) (total learning iterations: 4597 3086
3000
2000
L0
1000
200 000 000 000 100 000 000 100 000 000
000 000
000 000 000 000 000
10
1
3
4
5
7
8
10
11
12
13
14
15
16
17
18
19
20
21
22
curriculum levels
Figure Evaluation the introduced constrained write head.
13

---

Under review as a conference paper at ICLR 2020
USNGE-L AND HARD ATTENTION VS. SOFT ATTENTION
Here the introduced usage-linkage and hard attention mechanism for memory access are evaluated. While using hard attention instead of soft attention
necessary modification enable efficient learning of algorithmic solutions, the introduced usage-linkage had a smaller impact on the Learning Search task, as shown in Figure When applied the Learning Plan setup however, the usage- linkage improved the learning of algorithmic solutions significantly, see Figure TT] Both results show that the model learns use the attention mechanisms that are required for the algorithmic solution, i.e., the usage-linkage is especially useful for the backtracking in the Learning to Plan setup compared to the Learning to Search setup where no backtracking required.
aclive learning iterations
full model (total learning iterations: 4515 = 1684) )
3000
no usage-linkage (total learning iterations: 4407 = 1941
180
soft attention (total learning iterations: 5649 1055
2000
10
1000
0
0
000
000 000
100
000
000
100 000 000 000 000 000 000 000 000 000
10
1
3
4
5
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
curriculum levels
Figure 9: Evaluation of the introduced usage-linkage attention and the hard attention memory access.
learning iterations
4000
full model (total learning iterations: 6426 = 1675)
no usage-linkage (total leamingg Ittrationn: 3706 3326
3000
2000
10
10
1000
active runs
10
3
7
8
13 curriculum levels
10
11
12
15
16
17
19
20
21
22
Figure 10: Evaluation of the Lingellinage attention on the Learning to Plan setup.
BAD MEMORIES
The bad memories approach was developed while learning the data-dependent modules and was necessary mechanism to learn robust and generalized modules with 100% accuracy, as explained in Section 2.1.1 For learning the algorithmic solutions, the impact of this learning from mistakes
strategy was less significant, see Figure II
aclive learning Iterations
3000
with bad memories (total leaminng iterations: 4515 = 1684 )
without bad memories (total learning iter fterations: 4830 2327
2000
1000
10
10,0
0
0
0
0
0
0
0
0
0
D
0
0
0
0
0
2
3
4
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
curriculum levels
Figure 11: Evaluation of the bad memories mechanism.
14


Reviews:
[{"review_id": "rygiuvKJ9B", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents an approach called a neural computer, which has a Differential Neural Computer (DNC) at its core that is optimised with an evolutionary strategy. In addition to the typical DNC architecture, the system proposed in this paper has different modules that transfer different domain representations into the same representation, which allows the system to generalise to different and unseen tasks.\n\nThe paper is interesting and well written but I found that the contributions of this paper could be made more clear. \n\nFirst, the idea of evolving a Neural Turing machine was first proposed in Greve et al. 2016, which the authors cite, but only in passing in the conclusion. Greve et al. paper introduced the idea of hard attention mechanisms in an NTM through evolution and the benefits of having a memory structure that does not have to be differentiable. However, if the reader of this paper is not careful, they would miss this fact. I, therefore, suggest featuring Greve’s paper more prominently and highlighting the differences/similarities to the current paper earlier in the introduction.\n\nSecond, the idea of learned modules to allow the approach to work across different domains is interesting, but I’m wondering how novel it really is? Isn’t this basically just like feature engineering and changing the underlying representation, something that we have been doing for a long time? Also, the domains that this approach can be applied to seem potentially limited in that the two problems have to already be very similar; In fact, I probably wouldn’t call them different tasks but the same task with a different visual representation. \n\nI also had a question about the DNC training. Is the DNC version also trained with NES? It would be good to know how much of the difference between the proposed approach and the DNC is because of the training method (NES vs SGD) or other factors.  \n\nOnce the points raised above and the specific contributions of this paper are made more clear, I would suggest accepting it. \n\n####After rebuttal####\nThe authors' response and the revised paper address most of my concerns now. Since I do believe the approach could be more novel, I'm keeping my weak accept, but do think it is very interesting work. ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SyllmIQAtB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a neural controller architecture for learning abstract algorithmic solutions to search and planning problems. By combining abstract and domain-specific components, the model is able to mimic two classical algorithms quite closely across several domains. The precision of the learning is very high; verified by generalization to substantially larger problem sizes and different domains. One notable conclusion is that Evolutionary Strategies is able to learn algorithmic solutions whose precision is on par with deterministic algorithms. The method of triggering learning based on curriculum level performance is a notable feature that nicely couples generalization progress with learning, and yields insightful learning curves.\n\nAlthough the paper shows that learning abstract algorithmic solutions is possible, it is not clear that the framework is able to learn such solutions when the training signals have not already been broken down into their smallest constituent parts. In other words, it is not clear what this framework could be used for, since it appears the experimenter must already possess a complete specification of the target algorithm and breakdown of the domain. Is there a setting where this framework could be used to learn something the experimenter is not already aware of? Or is the main point that it is technically possible to get an NN to learn this behavior?\n\nAlthough it is clear that models are achieved that satisfy R1-R3, it is not clear exactly what problem formulation is being considered. It would be very helpful if the paper included a formal problem definition so that the purpose of each framework component and differences w.r.t. prior work are clear. \n\nSimilarly, the motivation for each of the data dependent modules is not clear. Is there something fundamental about this particular decomposition into modules? Or are these just the modules that were necessary given the specifics of the algorithms that were learned in experiments? How does this framework generalize to other kinds of algorithms?\n\nAre the comparison methods (DNC and stackRNN) unable to generalize to larger problem sizes? Including the full comparisons on generalization would give a more complete picture. Similarly, the figures are missing the comparisons for Learning to Plan for DNC and stackRNN.\n\nIs the comparison w.r.t. training time in Figure 3c fair, since the proposed framework pretrains the submodules?\n\nIs there a fundamental problem of DNC being addressed here? E.g., are there some critical types of submodules where making them differentiable is not an option?\n\nIs the algorithm limited to cases where the number of actions at each state is equal? I.e., could it be applied to algorithms like shortest path in the DNC paper?\n\nFinally, the last line talks about intriguing applications to the real world, but the running example in the paper is sorting. Is there some hypothetical but concrete example of how this framework could help in the real world, and do something better than a hard-coded classical algorithm? Or discover a new algorithm?\n\n----------\n\nAfter the rebuttal and discussion, I am convinced the paper is a useful contribution, and have increased my rating. However, I still think the presentation can be much improved to improve the clarity of the contribution. I would hope to see the following addressed in the final version:\n\n1. More detailed and precise discussion of how the approach relates to prior work. As is, this discussion is informal and scattered throughout the paper. Enumerating the distinctions in one place would make the contribution much more clear.\n\n2. The above would also help make the explicit contribution more clear. E.g., the \"Contribution\" paragraph currently does not contain the fact that the machine has only been applied to \"planning\" problems. This should be included to avoid making the contribution seem overly general.\n\n3. More formal description of what each module does. Right now, they are described informally. Actually seeing the equations of what each computes would make it easier to understand how and why they all fit together.\n\nI still see the main contribution as \"It is technically possible to train the abstract controller of a neural computer for planning using NES, so that R1-R3 are satisfied.\" Ideally, the above clarifications would make it more clear that this is the main contribution, or that there are some other key contributions beyond this.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BylAVtWwFr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes modifications and modular extensions to the differential neural computer (DNC). The approach is nicely modular, decoupling the data modules from algorithmic modules. This enables the authors to pretrain the data modules with supervised learning and to train the small algorithmic modules with neural evolution strategies (NES). NES is a global optimization method (which may be understood as policy gradients where the parameters of the neural policy are the actions) and consequently this enables the authors to use discrete selection mechanisms instead of the soft attention mechanisms of DNC.\n\nThe authors correctly argue for generalization to different and unseen tasks (R1), independence of data representation (R2) and independence of task environment (R3). These are important points, and the authors are able to achieve these by using a modular approach where the interface modules are pretrained.\n\nWhile what can be achieved with modularity is important, the modules themselves are rather simple and leave open the question of scalability. NES clearly does not scale to higher dimensional spaces, and the complexity or real say vision modules is the problem for many people --- one does not just learn the vision module and freeze it. \n\nThe idea that modularity can be used to attain greater generality and domain independence has already been explored at ICLR to some extent. In fact, some authors have shown theoretically provable generalisation via verification approaches: https://openreview.net/forum?id=BkbY4psgg\n\nIn summary, this paper demonstrated the advantages of modular neural systems, but fails to address the important issue of making sure the modules scale to real problems.\n\n\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors present a method that optimizes a differentiable neural computer with evolutionary search, and which can transfer abstract strategies to novel problems.  The reviewers all agreed that the approach is interesting, though were concerned about the magnitude of the contribution / novelty compared to existing work, clarity of contributions, impact of pretraining, and simplicity of examples.  While the reviewers felt that the authors resolved the many of their concerns in the rebuttal, there was remaining concern about the significance of the contribution.  Thus, I recommend this paper for rejection at this time.
Decision: Reject