An investigation of model-free planning

The field of reinforcement learning ( RL ) is facing increasingly challenging domains with combinatorial complexity . For an RL agent to address these challenges , it is essential that it can plan effectively . Prior work has typically utilized an explicit model of the environment , combined with a specific planning algorithm ( such as tree search ) . More recently , a new family of methods have been proposed that learn how to plan , by providing the structure for planning via an inductive bias in the function approximator ( such as a tree structured neural network ) , trained end-to-end by a model-free RL algorithm . In this paper , we go even further , and demonstrate empirically that an entirely model-free approach , without special structure beyond standard neural network components such as convolutional networks and LSTMs , can learn to exhibit many of the hallmarks that we would typically associate with a model-based planner . We measure our agent ’ s effectiveness at planning in terms of its ability to generalize across a combinatorial and irreversible state space , its data efficiency , and its ability to utilize additional thinking time . We find that our agent has the characteristics that one might expect to find in a planning algorithm . Furthermore , it exceeds the state-ofthe-art in challenging combinatorial domains such as Sokoban and outperforms other model-free approaches that utilize strong inductive biases toward planning . 1 INTRODUCTION . One of the aspirations of artificial intelligence is a cognitive agent that can adaptively and dynamically form plans to achieve its goal . Traditionally , this role has been filled by model-based RL approaches , which first learn an explicit model of the environment ’ s system dynamics or rules , and then apply a planning algorithm ( such as tree search ) to the learned model . But model-based approaches have been challenging to scale with learned models in complex environments . More recently , a variety of approaches have been proposed that learn to plan implicitly , solely by model-free training . These model-free planning agents utilize a special neural architecture that mirrors the structure of a particular planning algorithm . For example the neural network may be designed to represent search trees ( Farquhar et al. , 2017 ; Oh et al. , 2017 ; Guez et al. , 2018 ) , forward simulations ( Racanière et al. , 2017 ; Silver et al. , 2016 ) , or dynamic programming ( Tamar et al. , 2016 ) . The main idea is that , given the appropriate inductive bias for planning , the function approximator can learn to leverage these structures to learn its own planning algorithm . This kind of algorithmic function approximation may be more flexible than an explicit model-based approach , allowing the agent to customize the nature of planning to the specific environment . In this paper we explore the hypothesis that planning may occur implicitly , even when the function approximator has no special inductive bias toward planning . Previous work ( Pang & Werbos , 1998 ; Wang et al. , 2018 ) have supported the idea that model-based behavior could be learned with general recurrent architectures , with planning computation amortized over multiple discrete steps ( Schmidhuber , 1990 ) , but comprehensive demonstrations of its effectiveness are still missing . Inspired by the successes of deep learning and the universality of neural representations , our main idea is simply to furnish a neural network with a high capacity and flexible representation , rather than mirror any particular planning structure . Given such flexibility , the network can in principle learn its own algorithm for approximate planning . Specifically , we utilize a family of neural networks based on a widely used function approximation architecture : the stacked convolutional LSTMs ( ConvLSTM by Xingjian et al . ( 2015 ) ) . It is perhaps surprising that a purely model-free reinforcement learning approach can be so successful in domains that would appear to necessitate explicit planning . This raises a natural question : what is planning ? Can a model-free RL agent truly be considered to be planning , without any explicit model of the environment , and without any explicit simulation of that model ? In this paper we take a behaviourist approach . Here , planning will rather be considered to be a measurable property of the agent ’ s interactions . In particular , we consider three key properties that an agent equipped with planning should exhibit . First , an effective planning algorithm should be able to generalize to different situations . The intuition here is that a simple function approximator can not predict accurately across a combinatorial space of possibilities ( for example the value of all chess positions ) , but a planning algorithm can perform a local search to dynamically compute predictions ( for example by tree search ) . We measure this property using procedural environments ( such as random gridworlds , Sokoban ( Racanière et al. , 2017 ) , Boxworld ( Zambaldi et al. , 2018 ) ) with a massively combinatorial space of possible layouts . We find that our model-free planning agent achieves state-of-the-art performance , and significantly outperforms more specialized model-free planning architectures . We also investigate extrapolation to a harder class of problems beyond those in the training set , and again find that our architecture performs effectively – especially with larger network sizes . Second , a planning agent should be able to learn efficiently from relatively small amounts of data . Model-based RL is frequently motivated by the intuition that a model ( for example the rules of chess ) can often be learned more efficiently than direct predictions ( for example the value of all chess positions ) . We measure this property by training our model-free planner on small data-sets , and find that our model-free planning agent still performs well and generalizes effectively to a heldout test set . Third , an effective planning algorithm should be able to make good use of additional thinking time . Put simply , the more the algorithm thinks , the better its performance should be . This property is likely to be especially important in domains with irreversible consequences to wrong decisions ( e.g . death or dead-ends ) . We measure this property in Sokoban by adding additional thinking time at the start of an episode , and find that our model-free planning agent solves considerably more problems . Together , our results suggest that a model-free agent , without planning-inspired network structure , can learn to exhibit many of the behavioural characteristics of planning . The architecture presented in this paper serves to illustrate this point , and shows the surprising power of one simple approach . We hope our findings broaden the search for more general architectures that can tackle an even wider range of domains . 2 METHODS . We first motivate and describe the main network architecture we use in this paper . Then we briefly explain our training setup . More details can be found in Appendix C . 2.1 MODEL ARCHITECTURES . We desire models that can represent and learn powerful but unspecified planning procedures . Rather than encode strong inductive biases toward particular planning algorithms , we choose high-capacity neural network architectures that are capable of representing a very rich class of functions . As in many works in deep RL , we make use of convolutional neural networks ( known to exploit the spatial structure inherent in visual domains ) and LSTMs ( known to be effective in sequential problems ) . Aside from these weak but common inductive biases , we keep our architecture as general and flexible as possible , and trust in standard model-free reinforcement learning algorithms to discover the functionality of planning . 2.1.1 BASIC ARCHITECTURE . The basic element of the architecture is a ConvLSTM ( Xingjian et al. , 2015 ) – a neural network similar to an LSTM but with a 3D hidden state and convolutional operations . A recurrent network fθ stacks together ConvLSTM modules . For a stack depth of D , the state s contains all the cell states cd and outputs hd of each module d : s = ( c1 , . . . , cD , h1 , . . . , hD ) . The module weights θ = ( θ1 , . . . , θD ) are not shared along the stack . Given a previous state and an input tensor i , the next state is computed as s′ = fθ ( s , i ) . The network fθ is then repeated N times within each timestep ( i.e. , multiple internal ticks per real time-step ) . If st−1 is the state at the end of the previous time-step , we obtain the new state given the input it as : st = gθ ( st−1 , it ) = fθ ( fθ ( . . . fθ ( st−1 , it ) , . . . , it ) , it ) ︸ ︷︷ ︸ N times ( 1 ) The elements of st all preserve the spatial dimensions of the input it . The final output ot of the recurrent network for a single time-step is hD , the hidden state of the deepest ConvLSTM module after N ticks , obtained from st. We describe the ConvLSTM itself and alternative choices for memory modules in Appendix C. The rest of the network is rather generic . An encoder network e composed of convolutional layers processes the input observation xt into a H ×W × C tensor it — given as input to the recurrent module g. The encoded input it is also combined with ot through a skip-connection to produce the final network output . The network output is then flattend and an action distribution π and a state-value V are computed via a fully-connected MLP . The diagram in Fig 1 illustrates the full network . From here on , we refer to this architecture as Deep Repeated ConvLSTM ( DRC ) network architecture , and sometimes followed explicitly by the value of D and N ( e.g. , DRC ( 3 , 2 ) has depth D = 3 and N = 2 repeats ) . 2.1.2 ADDITIONAL DETAILS . Less essential design choices in the architectures are described here . Ablation studies show that these are not crucial , but do marginally improve performance ( see Appendix F ) . Encoded observation skip-connection The encoded observation it is provided as an input to all ConvLSTM modules in the stack . Top-down skip connection As described above , the flow of information in the network only goes up ( and right through time ) . To allow for more general computation we add feedback connection from the last layer at one time step to the first layer of the next step . Pool-and-inject To allow information to propagate faster in the spatial dimensions than the size of the convolutional kernel within the ConvLSTM stack , it is useful to provide a pooled version of the module ’ s last output h as an additional input on lateral connections . We use both max and mean pooling . Each pooling operation applies pooling spatially for each channel dimension , followed by a linear transform , and then tiles the result back into a 2D tensor . This is operation is related to the pool-and-inject method introduced by Racanière et al . ( 2017 ) and to Squeeze-and-Excitation blocks ( Hu et al. , 2017 ) . Padding The convolutional operator is translation invariant . To help it understand where the edge of the input image is , we append a feature map to the input of the convolutional operators that has ones on the boundary and zeros inside .

Summary: In this paper, the authors provide an interesting view of 'planning' from the behaviorist approach. Specifically, they raise three properties as the criteria to define the planning algorithm. They exploit the convLSTM as the building block cell to construct the neural network for planning. The neural network for planning is learned with some actor critic algorithms in reinforcement learning setting. The learned policy is empirically verified to be an implicit planning module under the proposed criteria.