Learning Algorithmic Solutions to Symbolic Planning Tasks with a Neural Computer

1 INTRODUCTION . Transferring solution strategies from one problem to another is a crucial ability for intelligent behavior ( Silver et al. , 2013 ) . Current learning systems can learn a multitude of specialized tasks , but extracting the underlying structure of the solution for effective transfer is an open research problem ( Taylor & Stone , 2009 ) . Abstraction is key to enable these transfers ( Tenenbaum et al. , 2011 ) and the concept of algorithms in computer science is an ideal example for such transferable abstract strategies . An algorithm is a sequence of instructions , which solves a given problem when executed , independent of the specific instantiation of the problem . For example , consider the task of sorting a set of objects . The algorithmic solution , specified as the sequence of instructions , is able to sort any number of arbitrary classes of objects in any order , e.g. , toys by color , waste by type , or numbers by value , by using the same sequence of instructions , as long as the features and compare operations defining the order are specified . Learning such structured , abstract strategies enables the transfer to new domains and representations ( Tenenbaum et al. , 2011 ) . Moreover , abstract strategies as algorithms have built-in generalization capabilities to new task configurations and complexities . Here , we present a novel architecture for learning abstract strategies in the form of algorithmic solutions . Based on the Differential Neural Computer ( Graves et al. , 2016 ) and inspired by the von Neumann and Harvard architectures of modern computers , the architectures modular structure allows for straightforward transfer by reusing learned modules instead of relearning , prior knowledge can be included , and the behavior of the modules can be examined and interpreted . Moreover , the individual modules of the architecture can be learned with different learning settings and strategies – or be hardcoded if applicable – allowing to split the overall task into easier subproblems , contrary to the end-to-end learning philosophy of most deep learning architectures . Building on memory-augmented neural networks ( Graves et al. , 2016 ; Neelakantan et al. , 2016 ; Weston et al. , 2015 ; Joulin & Mikolov , 2015 ) , we propose a flexible architecture for learning abstract strategies as algorithmic solutions and show the learning and transferring of such in symbolic planning tasks . 1.1 THE PROBLEM OF LEARNING ALGORITHMIC SOLUTIONS . We investigate the problem of learning algorithmic solutions which are characterized by three requirements : R1 – generalization to different and unseen task configurations and task complexities , R2 – independence of the data representation , and R3 – independence of the task domain . Picking up the sorting algorithm example again , R1 represents the ability to sort lists of arbitrary length and initial order , while R2 and R3 represent the abstract nature of the solution . This abstraction enables the algorithm , for example , to sort a list of binary numbers while being trained only on hexadecimal numbers ( R2 ) . Furthermore , the algorithm trained on numbers is able to sort lists of strings ( R3 ) . If R1 – R3 are fulfilled , the algorithmic solution does not need to be retrained or adapted to solve unforeseen task instantiations – only the data specific operations need to be adjusted . Research on learning algorithms typically focuses on identifying algorithmic generated patterns or solving algorithmic problems ( Neelakantan et al. , 2016 ; Zaremba & Sutskever , 2014 ; Kaiser & Sutskever , 2016 ; Kaiser & Bengio , 2016 ) , less on finding algorithmic solutions ( Joulin & Mikolov , 2015 ; Zaremba et al. , 2016 ) fulfilling the three discussed requirements R1 – R3 . While R1 is typically tackled , as it represents the overall goal of generalization in machine learning , the abstraction abilities from R2 and R3 are missing . Additionally , most algorithms require a form of feedback , using computed intermediate results from one computational step in subsequent steps , and a variable number of computational steps to solve a problem instance . Thus , it is necessary to be able to cope with varying numbers of steps and determining when to stop , in contrast to using a fixed number of steps ( Neelakantan et al. , 2016 ; Sukhbaatar et al. , 2015 ) , making the learning problem more challenging in addition . A crucial feature for algorithms is the ability to save and retrieve data . Therefore , augmenting neural networks with different forms of external memory , e.g. , matrices , stacks , tapes or grids , to increase their expressiveness and to separate computation from memory , especially in long time dependencies setups , is an active research direction ( Graves et al. , 2016 ; Weston et al. , 2015 ; Joulin & Mikolov , 2015 ; Zaremba et al. , 2016 ; Sukhbaatar et al. , 2015 ; Kumar et al. , 2016 ; Greve et al. , 2016 ) with earlier work in the field of grammar learning ( Das et al. , 1992 ; Mozer & Das , 1993 ; Zeng et al. , 1994 ) . These memory-augmented networks improve performance on a variety of tasks like reasoning and inference in natural language ( Graves et al. , 2016 ; Weston et al. , 2015 ; Sukhbaatar et al. , 2015 ; Kumar et al. , 2016 ) , learning of simple algorithms and algorithmic patterns ( Joulin & Mikolov , 2015 ; Zaremba et al. , 2016 ; Graves et al. , 2014 ) , and navigation tasks ( Wayne et al. , 2018 ) . The contribution of this paper is a novel modular architecture building on a memory-augmented neural network ( DNC ( Graves et al. , 2016 ) ) for learning algorithmic solutions in a reinforcement learning setting . We show that the learned solutions fulfill all three requirements R1 – R3 for an algorithmic solution and the architecture can process a variable number of computational steps . 2 A NEURAL COMPUTER ARCHITECTURE FOR ALGORITHMIC SOLUTIONS . In this section , we introduce the novel modular architecture for learning algorithmic solutions , shown in Figure 1 . The architecture builds on the Differential Neural Computer ( DNC ) ( Graves et al. , 2016 ) and its modular design is inspired by modern computer architectures , related to ( Neelakantan et al. , 2016 ; Weston et al. , 2015 ) . The DNC augments a controller neural network with a differentiable autoassociative external memory to separate computation from memory , as memorization is usually done in the networks weights . The controller network learns to write and read information from that memory by emitting an interface vector which is mapped onto different vectors by linear transformations . These vectors control the read and write operations of the memory , called read and write heads . For writing and reading , multiple attention mechanisms are employed , including content lookup , temporal linkage and memory allocation . Due to the design of the interface and the attention mechanisms , the DNC is independent of the memory size and fully differentiable , allowing gradient-based end-to-end learning . Our architecture . In order to learn algorithmic solutions , the computations need to be decoupled from the specific data and task . To enable such data and task independent computations , we propose multiple alterations and extensions to the DNC , inspired by modern computer architectures . First , information flow is divided into two streams , data and control . This separation allows to disentangle data representation dependent manipulations from data independent algorithmic instructions . Due to this separation , the algorithmic modules need to be extended to include two memories , a data and a computational memory . The data memory stores and retrieves the data stream , whereas the computational memory works on information generated by the control signal flow through the learnable controller and memory transformations . The two memories are coupled , operating on the same locations , and these locations are determined by the computational memory , and hence by the control stream . As with the DNC , multiple read and write heads can be used . In our experiments , one read and two write heads are used , with one write head constrained to the previously read location . In contrast to the DNC , but in line with the computer architecture-inspired design and the goal of learning deterministic algorithms , writing and reading uses hard attentions instead of soft attentions . Hard attention means that only one memory location can be written to and read from ( unique addresses ) , instead of an weighted average over all locations as with soft attentions . Such hard attention was shown to be beneficial for generalization ( Greve et al. , 2016 ) . We also employed an additional attention mechanism for reading , called usage linkage , similar to the temporal linkage of the DNC , but instead of capturing temporal relations , it captures usage relations , i.e. , the relation between written memory location and previously read location . With both linkages in two directions and the content look up , the model has five attention mechanisms for reading . While the final read memory location is determined by a weighted combination of these attentions ( see attention in Figure 5 in the Appendix ) , each attention mechanism itself uses hard decisions , returning only one memory location . See Appendix C for the effect of the introduced modifications and extensions . For computing the actual solution , operating only on the control stream is not enough , as the model still needs to manipulate the data . Therefore , we added several modules operating on the data stream , inspired by the architecture of computers . In particular , an Input , TransformD , ALU ( arithmetic logic unit ) and Output module were added ( more details in Section 2.2 ) . These modules manipulate the data , steered by the algorithmic modules . The full architecture is shown in Figure 1 . As algorithms typically involve recursive or iterative data manipulation , the model receives its own output as input in the next computation step , making the whole architecture an output-input model . With all aforementioned extensions , algorithmic solutions fulfilling R1 – R3 can be learned . 2.1 THE ALGORITHMIC MODULES . The algorithmic modules consist of the Controller , the Memory and the TransformC module and build the core of the model . These modules learn the algorithmic solution operating on the control stream . With t as the current computational step and c as the control stream ( see Figure 1 ) , the input-output of the modules are C ( ci , t , cm , t−1 , cf , t−1 , ca , t−1 , co , t−1 ) 7−→ cc , t , M ( ci , t , cc , t ) 7−→ cm , t , dm , t and TC ( cc , t , cm , t , ci , t ) 7−→ cf , t . The algorithmic modules are based on the DNC with the alterations and extensions described before . Next we discuss how these algorithmic modules can be learned before looking into the data-dependent modules . 2.1.1 LEARNING OF THE ALGORITHMIC MODULES . Learning the algorithmic modules , and hence the algorithmic solution , is done in a reinforcement learning setting using Natural Evolution Strategies ( NES ) ( Wierstra et al. , 2014 ) . NES is a blackbox optimizer that does not require differentiable models , giving more freedom to the model design , e.g. , the hard attention mechanisms are not differentiable . NES updates a search distribution of the parameters to be learned by following the natural gradient towards regions of higher fitness using a population of offsprings ( altered parameters ) for exploration . Let θ be the parameters to be learned and using an isotropic multivariate Gaussian search distribution with fixed variance σ2 , the stochastic natural gradient at iteration t is given by ∇θtE ∼N ( 0 , I ) [ u ( θt + σ ) ] ≈ 1 Pσ P∑ i=1 u ( θit ) i , where P is the population size and u ( · ) is the rank transformed fitness ( Wierstra et al. , 2014 ) . The parameters are updated by θt+1 = θt + α Pσ P∑ i=1 u ( θit ) i , with learning rate α . Recent research showed that NES and related approaches like Random Search ( Mania et al. , 2018 ) or NEAT ( Stanley & Miikkulainen , 2002 ) are powerful alternatives in reinforcement learning . They are easier to implement and scale , perform better with sparse rewards and credit assignment over long time scales , have fewer hyperparameters ( Salimans et al. , 2017 ) and were used to train memory-augmented networks ( Greve et al. , 2016 ; Merrild et al. , 2018 ) . For robustness and learning efficiency , weight decay for regularization ( Krogh & Hertz , 1992 ) and automatic restarts of runs stuck in local optima are used as in ( Wierstra et al. , 2014 ) . This restarting can be seen as another level of evolution , where some lineages die out . Another way of dealing with early converged or stuck lineages is to add intrinsic motivation signals like novelty , that help to get attracted by another local optima , as in NSRA-ES ( Conti et al. , 2018 ) . In the experiments however , we found that within our setting , restarting – or having an additional survival of the fittest on the lineages – was more effective , see Appendix C for a comparison . The algorithmic solutions are learned in a curriculum learning setup ( Bengio et al. , 2009 ) with sampling from old lessons ( Zaremba & Sutskever , 2014 ) to prevent unlearning and to foster generalization . Furthermore , we created bad memories , a learning from mistakes strategy , similar to the idea of AdaBoost ( Freund & Schapire , 1997 ) , which samples previously failed tasks to encourage focusing on the hard tasks . This can also be seen as a form of experience replay ( Mnih et al. , 2015 ; Lin , 1992 ) , but only using the task configurations , the initial input to the model , not the full generated sequence . Bad memories were developed for training the data-dependent modules to ensure their robustness and 100 % accuracy , which is crucial to learn algorithmic solutions . If the individual modules do not have 100 % accuracy , no stable algorithmic solution can be learned even if the algorithmic modules are doing the correct computations . For example , if one module has an accuracy of 99 % , the 1 % error prevents learning an algorithmic solution that works always . This problem is even reinforced as the proposed model is an output-input architecture that works over multiple computation steps using its own output as the new input – meaning the overall accuracy drops to 36.6 % for 100 computation steps . Therefore using the bad memories strategy , and thus focusing on the mistakes , helps significantly in achieving robust results when learning the modules , enabling the learning of algorithmic solutions . While the bad memories strategy was crucial to achieve 100 % robustness when training the data-dependent modules , the effect on learning the algorithmic solutions was less significant ( see Appendix C for an evaluation ) .

Summary: This paper proposes modifications and modular extensions to the differential neural computer (DNC). The approach is nicely modular, decoupling the data modules from algorithmic modules. This enables the authors to pretrain the data modules with supervised learning and to train the small algorithmic modules with neural evolution strategies (NES). NES is a global optimization method (which may be understood as policy gradients where the parameters of the neural policy are the actions) and consequently this enables the authors to use discrete selection mechanisms instead of the soft attention mechanisms of DNC.