Title: AN INVESTIGATION OF MODEL-FREE PLANNING Anonymous authors Paper under double-blind review

Abstract: The field of reinforcement learning (RL) is facing increasingly challenging domains with combinatorial complexity. For an RL agent to address these challenges, it is essential that it can plan effectively. Prior work has typically utilized an explicit model of the environment, combined with a specific planning algorithm (such as tree search). More recently, a new family of methods have been proposed that learn how to plan, by providing the structure for planning via an inductive bias in the function approximator (such as a tree structured neural network), trained end-to-end by a model-free RL algorithm. In this paper, we go even further, and demonstrate empirically that an entirely model-free approach, without special structure beyond standard neural network components such as convolutional networks and LSTMs, can learn to exhibit many of the hallmarks that we would typically associate with a model-based planner. We measure our agent's effectiveness at planning in terms of its ability to generalize across a combinatorial and irreversible state space, its data efficiency, and its ability to utilize additional thinking time. We find that our agent has the characteristics that one might expect to find in a planning algorithm. Furthermore, it exceeds the state-ofthe-art in challenging combinatorial domains such as Sokoban and outperforms other model-free approaches that utilize strong inductive biases toward planning.

1. INTRODUCTION:

One of the aspirations of artificial intelligence is a cognitive agent that can adaptively and dynamically form plans to achieve its goal. Traditionally, this role has been filled by model-based RL approaches, which first learn an explicit model of the environment's system dynamics or rules, and then apply a planning algorithm (such as tree search) to the learned model. But model-based approaches have been challenging to scale with learned models in complex environments.

More recently, a variety of approaches have been proposed that learn to plan implicitly, solely by model-free training. These model-free planning agents utilize a special neural architecture that mirrors the structure of a particular planning algorithm. For example the neural network may be designed to represent search trees (Farquhar et al., 2017; Oh et al., 2017; Guez et al., 2018) , forward simulations (Racanière et al., 2017; Silver et al., 2016) , or dynamic programming (Tamar et al., 2016) . The main idea is that, given the appropriate inductive bias for planning, the function approximator can learn to leverage these structures to learn its own planning algorithm. This kind of algorithmic function approximation may be more flexible than an explicit model-based approach, allowing the agent to customize the nature of planning to the specific environment.

In this paper we explore the hypothesis that planning may occur implicitly, even when the function approximator has no special inductive bias toward planning. Previous work (Pang & Werbos, 1998; Wang et al., 2018) have supported the idea that model-based behavior could be learned with general recurrent architectures, with planning computation amortized over multiple discrete steps (Schmidhuber, 1990) , but comprehensive demonstrations of its effectiveness are still missing. Inspired by the successes of deep learning and the universality of neural representations, our main idea is simply to furnish a neural network with a high capacity and flexible representation, rather than mirror any particular planning structure. Given such flexibility, the network can in principle learn its own algorithm for approximate planning. Specifically, we utilize a family of neural networks based on a widely used function approximation architecture: the stacked convolutional LSTMs (ConvLSTM by Xingjian et al. (2015) ).

It is perhaps surprising that a purely model-free reinforcement learning approach can be so successful in domains that would appear to necessitate explicit planning. This raises a natural question: what is planning? Can a model-free RL agent truly be considered to be planning, without any explicit model of the environment, and without any explicit simulation of that model? In this paper we take a behaviourist approach. Here, planning will rather be considered to be a measurable property of the agent's interactions. In particular, we consider three key properties that an agent equipped with planning should exhibit.

First, an effective planning algorithm should be able to generalize to different situations. The intuition here is that a simple function approximator cannot predict accurately across a combinatorial space of possibilities (for example the value of all chess positions), but a planning algorithm can perform a local search to dynamically compute predictions (for example by tree search). We measure this property using procedural environments (such as random gridworlds, Sokoban (Racanière et al., 2017) , Boxworld (Zambaldi et al., 2018) ) with a massively combinatorial space of possible layouts. We find that our model-free planning agent achieves state-of-the-art performance, and significantly outperforms more specialized model-free planning architectures. We also investigate extrapolation to a harder class of problems beyond those in the training set, and again find that our architecture performs effectively -especially with larger network sizes.

Second, a planning agent should be able to learn efficiently from relatively small amounts of data. Model-based RL is frequently motivated by the intuition that a model (for example the rules of chess) can often be learned more efficiently than direct predictions (for example the value of all chess positions). We measure this property by training our model-free planner on small data-sets, and find that our model-free planning agent still performs well and generalizes effectively to a heldout test set.

Third, an effective planning algorithm should be able to make good use of additional thinking time. Put simply, the more the algorithm thinks, the better its performance should be. This property is likely to be especially important in domains with irreversible consequences to wrong decisions (e.g. death or dead-ends). We measure this property in Sokoban by adding additional thinking time at the start of an episode, and find that our model-free planning agent solves considerably more problems.

Together, our results suggest that a model-free agent, without planning-inspired network structure, can learn to exhibit many of the behavioural characteristics of planning. The architecture presented in this paper serves to illustrate this point, and shows the surprising power of one simple approach. We hope our findings broaden the search for more general architectures that can tackle an even wider range of domains.

2. METHODS:

We first motivate and describe the main network architecture we use in this paper. Then we briefly explain our training setup. More details can be found in Appendix C.

2.1. MODEL ARCHITECTURES:

We desire models that can represent and learn powerful but unspecified planning procedures. Rather than encode strong inductive biases toward particular planning algorithms, we choose high-capacity neural network architectures that are capable of representing a very rich class of functions. As in many works in deep RL, we make use of convolutional neural networks (known to exploit the spatial structure inherent in visual domains) and LSTMs (known to be effective in sequential problems). Aside from these weak but common inductive biases, we keep our architecture as general and flexible as possible, and trust in standard model-free reinforcement learning algorithms to discover the functionality of planning.

2.1.1. BASIC ARCHITECTURE:

The basic element of the architecture is a ConvLSTM (Xingjian et al., 2015) -a neural network similar to an LSTM but with a 3D hidden state and convolutional operations. A recurrent network f θ stacks together ConvLSTM modules. For a stack depth of D, the state s contains all the cell states c d and outputs h d of each module d: s = (c 1 , . . . , c D , h 1 , . . . , h D ). The module weights θ = (θ 1 , . . . , θ D ) are not shared along the stack. Given a previous state and an input tensor i, the next state is computed as s = f θ (s, i). The network f θ is then repeated N times within each timestep (i.e., multiple internal ticks per real time-step). If s t-1 is the state at the end of the previous time-step, we obtain the new state given the input i t as:

s t = g θ (s t-1 , i t ) = f θ (f θ (. . . f θ (s t-1 , i t ), . . . , i t ), i t ) N times (1)

The elements of s t all preserve the spatial dimensions of the input i t . The final output o t of the recurrent network for a single time-step is h D , the hidden state of the deepest ConvLSTM module after N ticks, obtained from s t . We describe the ConvLSTM itself and alternative choices for memory modules in Appendix C.

The rest of the network is rather generic. An encoder network e composed of convolutional layers processes the input observation x t into a H × W × C tensor i t -given as input to the recurrent module g. The encoded input i t is also combined with o t through a skip-connection to produce the final network output. The network output is then flattend and an action distribution π and a state-value V are computed via a fully-connected MLP. The diagram in Fig 1 illustrates the full network.

From here on, we refer to this architecture as Deep Repeated ConvLSTM (DRC) network architecture, and sometimes followed explicitly by the value of D and N (e.g., DRC(3, 2) has depth D = 3 and N = 2 repeats).

2.1.2. ADDITIONAL DETAILS:

Less essential design choices in the architectures are described here. Ablation studies show that these are not crucial, but do marginally improve performance (see Appendix F).

Encoded observation skip-connection:

The encoded observation i t is provided as an input to all ConvLSTM modules in the stack.

Top-down skip connection As described above, the flow of information in the network only goes up (and right through time). To allow for more general computation we add feedback connection from the last layer at one time step to the first layer of the next step.

Pool-and-inject:

To allow information to propagate faster in the spatial dimensions than the size of the convolutional kernel within the ConvLSTM stack, it is useful to provide a pooled version of the module's last output h as an additional input on lateral connections. We use both max and mean pooling. Each pooling operation applies pooling spatially for each channel dimension, followed by a linear transform, and then tiles the result back into a 2D tensor. This is operation is related to the pool-and-inject method introduced by Racanière et al. (2017) and to Squeeze-and-Excitation blocks (Hu et al., 2017) .

Padding The convolutional operator is translation invariant. To help it understand where the edge of the input image is, we append a feature map to the input of the convolutional operators that has ones on the boundary and zeros inside.

2.2. TRAINING:

We used a distributed framework to train an RL agent using the IMPALA V-trace actor-critic algorithm (Espeholt et al., 2018) . While we found this training regime to help for training heavier networks, we also ran experiments which demonstrate that the DRC architecture can be trained effectively with A3C (Mnih et al., 2016) . More details on the setup can be found in Appendix D.2.

3. PLANNING DOMAINS:

Our domains are formally specified as RL problems, where agents must learn via reward feedback obtained by interacting with the environment (Sutton et al., 1998) . We focus on combinatorial domains for which episodes are procedurally generated. In these domains each episode is instantiated in a pseudorandom configuration, so solving an episode typically requires some form of reasoning. Most of the environments are fully-observable and have simple 2D visual features. The domains are illustrated and explained in Appendix A. In addition to the planning domains listed below, we also run control experiments on a set of Atari 2600 games (Bellemare et al., 2013) .

Gridworld A simple navigation domain following (Tamar et al., 2016) , consisting of a grid filled with obstacles. The agent, goal, and obstacles are randomly placed for each episode.

Sokoban A difficult puzzle domain requiring an agent to push a set of boxes onto goal locations (Botea et al., 2003; Racanière et al., 2017) . Irreversible wrong moves can make the puzzle unsolvable. We describe how we generate a large number of levels (for the fixed problem size 10x10 with 4 boxes) at multiple difficulty levels in Appendix B, and then each split into a training and test set. We are releasing these levels in the standard Sokoban format1 . Unless otherwise specified, we ran experiments with the easier unfiltered set of levels.

Boxworld Introduced in (Zambaldi et al., 2018) , the aim is to reach a goal target by collecting coloured keys and opening colour-matched boxes until a target is reached. The agent can see the keys (i.e., their colours) locked within boxes; thus, it must carefully plan the sequence of boxes that should be opened so that it can collect the keys that will lead to the target. Keys can only be used once, so opening an incorrect box can lead the agent down a dead-end path from which it cannot recover.

MiniPacman (Racanière et al., 2017) . The player explores a maze that contains food while being chased by ghosts. The aim of the player is to collect all the rewarding food. There are also a few power pills which allow the player to attack ghosts (for a brief duration) and earn a large reward. See Appendix A.2 for more details.

4. RESULTS:

We first examine the performance of our model and other approaches across domains. Then we report results aimed at understanding how elements of our architecture contribute to observed performance. Finally, we study evidence of iterative computation in Section 4.2 and generalization in Section 4.3.

4.1. COMPARISONS:

In general, across all domains listed in Section 3, the DRC architecture performed very well with only modest tuning of hyper-parameters (see Appendix D). The DRC(3,3) variant was almost always the best in terms both of data efficiency (early learning) and asymptotic performance. 

4.1.1. INFLUENCE OF NETWORK ARCHITECTURE:

We studied the influence of stacking and repeating the ConvLSTM modules in the DRC architecture, controlled by the parameters D (stack length) and N (number of repeats) as described in Section 2.1. These degrees of freedom allow our networks to compute its output using shared, iterative, computation with N > 1, as well as computations at different levels of representation and more capacity with D > 1. We found that the DRC(3,3) (i.e, D = 3, N = 3) worked robustly across all of the tested domain. We compared this to using the same number of modules stacked without repeats (DRC(9,1)) or only repeated without stacking (DRC(1,9)). In addition, we also look at the same smaller capacity versions D = 2, N = 2 and D = 1, N = 1 (which reduces to a standard ConvL-STM). Figure 3a shows the results on Sokoban for the different network configurations. In general, the versions with more capacity performed better. When trading-off stacking and repeating (with total of 9 modules), we observed that only repeating without stacking was not as effective (this has the same number of parameters as the DRC(1,1) version), and only stacking was slower to train in the early phase but obtained a similar final performance. We also confirmed that DRC(3,3) performed better than DRC(1,1) in Boxworld, MiniPacman, and Gridworld (see Figure 15a in Appendix).

On harder Sokoban levels (Medium-difficulty dataset), we trained the DRC(3,3) and the larger capacity DRC(9,1) configurations and found that, even though DRC(9,1) was slower to learn at first, it ended up reaching a better score than DRC(3,3) (94% versus 91.5% after 1e9 steps). See Fig 9 in appendix. We tested the resulting DRC(9,1) agent on the hardest Sokoban setting (Hard-difficulty), and found that it solved 80% of levels in less than 48 minutes. In comparison, running a powerful tree search algorithm, Levin Tree Search (Orseau et al., 2018) , with a DRC(1,1) as policy prior solves 94%, but in 10 hours.

In principle, deep feedforward models should support iterative procedures within a single time-step and perhaps match the performance of our recurrent networks (Jastrzebski et al., 2017) . In practice, deep ResNets performed poorly versus our recurrent models (see Figure 3b ), and are in any case incapable of caching implicit iterative planning steps over time steps. Finally, we note that recurrence by itself was also not enough: replacing the ConvLSTM modules with flat 1-D LSTMs performed poorly (see Figure 3b ). Across experiments and domains, our results suggests that both the network capacity and the iterative aspect of a model drives the agent's performance.

4.2. ITERATIVE COMPUTATION:

One desirable property for planning mechanisms is that their performance can scale with additional computation without seeing new data. Although RNNs (and more recently ResNets) can in principle learn a function that can be iterated to obtain a result (Graves, 2016; Jastrzebski et al., 2017; Greff et al., 2016) , it is not clear whether the networks trained in our RL domains learn to amortize computation over time in this way. To test this, we took trained networks in Sokoban (unfiltered) and tested post hoc their ability to improve their results with additional steps. To do so we introduced 'no-op' actions at the start of each episode -up to 10 extra steps. We observed clear performance improvements on Medium difficulty levels of up to 5% for DRC networks (see Figure 4 ). We did not find such improvements for the simpler fully-connected LSTM architecture. This suggests that the networks have learned a scaleable strategy for the task which is computed and refined through a series of identical steps, thereby exhibiting one of the essential properties of a planning algorithm.

4.3. GENERALIZATION:

In combinatorial domains generalization is a central issue. Given limited exposure to configurations in an environment, how well can a model perform on unseen scenarios? In the supervised setting, large flexible networks are capable of over-fitting. Thus, one concern when using high-capacity networks is that they may over-fit to the task, for example by memorizing, rather than learning a strategy that can generalize to novel situations. Recent empirical work in SL (Supervised Learning) has shown that the generalization of large networks is not well understood (Zhang et al., 2016; Arpit et al., 2017) . Generalization in RL is even less well studied, though recent work by (Zhang et al., 2018a; b) has begun to explore the effect of training data diversity.

We explored two main axes in the space of generalization. We varied both the diversity of the environments as well as the size of our models. We trained the DRC architecture in various data regimes, by restricting the number of unique Sokoban levels -during the training, similar to SL, the training algorithm iterates on those limited levels many times. We either train on a Large (900k Under review as a conference paper at ICLR 2019 levels), Medium (10k) or Small (1k) set. For each dataset size, we compared a larger version of the network, DRC(3,3), to a smaller version DRC(1,1)2 . Results are shown in Figure 5 .

In all cases, the larger DRC(3,3) network generalized better than its smaller counterpart, both in absolute terms and in terms of generalization gap. In particular, in the Medium regime, the generalization gap 3 is 6.482% for DRC(3,3) versus 33.398% for DRC(1, 1). Figure 6a shows the results when tested after training on both unfiltered and Medium test sets. We performed an analogous experiment in the Boxworld environment and observed remarkably similar results (see Appendix Fig 12 ).

Looking across these domains and experiments there are two findings that are of particular note. First, unlike analog SL experiments, reducing the number of training levels does not necessarily improve performance on the train set. Networks trained on 1k levels perform worse in terms of fraction of level solved. We believe this is due to the exploration problem in low-diversity regime: With more levels, the training agent faces a natural curriculum to help it progress toward harder levels. Another view of this is that larger networks can overfit the training levels, but only if they experience success on these levels at some point. While local minima for the loss in SL are not practically an issue, local minima in policy space can be problematic.

From a classic optimization perspective, a surprising finding is that the larger networks in our experiment (both Sokoban & Boxworld) suffer less from over-fitting in the low-data regime than their smaller counterparts (see Figure 6 ). However, this is in line with recent findings Zhang et al. (2016) in SL that the generalization of a model is driven by the architecture and nature of the data, rather than simply as a results of the network capacity and size of the dataset. Indeed, we also trained the same networks in a purely supervised fashion through imitation learning of an expert policy. 4 We observed a similar result when comparing the classification accuracy of the networks on the test set, with the DRC(3,3) better able to generalize -even though both networks had similar training errors on small datasets.

Extrapolation:

Another facet of generality in the strategy found by the DRC network is how it performs outside the training distribution. In Sokoban, we tested the DRC(3,3) and DRC(1,1) networks on levels with a larger number of boxes than those seen in the training set. Figure 13a shows that DRC was able to extrapolate with little loss in performance to up to 7 boxes (for a a fixed grid size). The performance degradation for DRC(3,3) on 7 boxes was 3.5% and 18.5% for DRC(1,1). In comparison, the results from Racanière et al. (2017) report a loss of 34% when extrapolating to 7 boxes in the same setup. 

5. DISCUSSION:

We aspire to endow agents with the capacity to plan effectively in combinatorial domains where simple memorization of strategies is not feasible. An overarching question is regarding the nature of planning itself. Can the computations necessary for planning be learned solely using model-free RL, and this can be achieved by a general-purpose neural network with weak inductive biases? Or is it necessary to have dedicated planning machinery -either explicitly encoding existing planning algorithms, or implicitly mirroring their structure? In this paper, we studied a variety of different neural architectures trained using model-free RL in procedural planning tasks with combinatorial and irreversible state spaces. Our results suggest that general-purpose, high-capacity neural networks based on recurrent convolutional structure, are particularly efficient at learning to plan. This approach yielded state-of-the-art results on several domains -outperforming all of the specialized planning architectures that we tested. Our generalization and scaling analyses, together with the procedural nature of the studied domains, suggests that these networks learn an algorithm for approximate planning that is tailored to the domain. The algorithmic function approximator appears to compute its plan dynamically, amortised over many steps, and hence additional thinking time can boost its performance.

Recent work in the context of supervised learning is pushing us to rethink how large neural network models generalize (Zhang et al., 2016; Arpit et al., 2017) . Our results further demonstrate the mismatch between traditional views on generalisation and model size. The surprising efficacy of our planning agent, when trained on a small number of scenarios across a combinatorial state space, suggests that any new theory must take into account the algorithmic function approximation capabilities of the model rather than simplistic measures of its complexity. Ultimately, we desire even more generality and scalability from our agents, and it remains to be seen whether model-free planning will be effective in reinforcement learning environments of real-world complexity.

time-step, each ghost moves with a probability of 0.95. This simulates, in a gridworld, the fact that ghosts move slower than the player. An implementation of this game is available under the name Pill Eater at https://github.com/vasiloglou/mltrain-nips-2017/tree/ master/sebastien_racaniere, where 5 different reward structures (or modes) are available.

A.3 BOXWORLD:

In all our experiments we use branch length of 3 and maximum solution length of 4 . Each episode is capped at 120 frames. Unlike Sokoban, Boxworld levels are generated by the game engine. Unlike Sokoban we do not used sprite-based images; the environment state is provided as a (14, 14, 3) tensor.

A.4 ATARI:

We use the standard Atari setup of Mnih et al. (2015) with up to 30 no-ops at random, episodes capped at 30 mins, an action repeat of 4, and the full set of 18 actions. We made little effort to tune the hyperparameters for Atari; only the encoder size was increased. See Appendix E.2 for details about the experiments, and Figure 10 for the learning curves.

A.5 GRIDWORLD:

We generate 32x32-shaped levels by sampling a number of obstacles between 12 and 24. Each obstacle is a square, with side in [2, 10] . These obstacles may overlap. The player and goal are placed on random different empty squares. We use rejection sampling to ensure the player can reach the goal. Episodes terminate when reaching a goal or when stepping on an obstacle, with a reward of 1 and -1 respectively. There is a small negative reward of -0.01 at each step otherwise.

B DATASET GENERATION DETAILS:

The unfiltered Sokoban levels are generated by the procedure described in Racanière et al. (2017) . Simple hashing is used to avoid repeats. To generate medium levels, we trained a DRC(1,1) policy whose LSTM state is reset at every time step. It achieves around 95% of level solved on the easy set. Then, we generate medium difficulty levels by rejection sampling: we sample levels using the procedure from Racanière et al. (2017) , and accept a level if the policy cannot solve it after 10 attempts. The levels are not subsampled from the easy levels dataset but from fresh generation instead, so overlap between easy and medium is minimal (though there are 120 levels in common in the easy and medium datasets). To generate hard levels, we train a DRC(3,3) policy on a mixture of easy and medium levels, until it reached a performance level of around 85%. We then select the first 3332 levels of a separate medium dataset which are not solved by that policy after 10 attempts (so there is no overlap between hard and medium sets).

Unfiltered Medium Hard 99% 95% 80% 

C MODEL:

We denoted g θ (s t-1 , i t ) = f θ (f θ (. . . f θ (s t-1 , i t ), . . . , i t ), i t ) as the computation at a full time-step of the DRC(D, N) architecture. Here θ = (θ 1 , . . . , θ D ) are the parameters of D stacked memory modules, and i t = e(x t ) is the agent's encoded observation.

Let s 1 t-1 , . . . , s N t-1 be the state of the D stacked memory modules at the N ticks at time-step t -1. Each s n t-1 = (c n 1 , . . . , c n d , h n 1 , . . . , h n d ). We then have the "repeat" recurrence below: 

E.2 ATARI:

We train DRC(3,3) and DRC(1,1) on five Atari 2600 games: Alien, Asteroid, Breakout, Ms. Pac-Man, and Up'n Down. We picked these for their planning focus. Figure 10 shows learning curves comparing DRC(3,3) against DRC(1,1) and ApeX-DQN (Horgan et al., 2018) on the above games. We improve on ApeX-DQN on three out of five games.

Our scores generally improve with more training. For example, if we let the DRC(3,3) run for longer, it reaches scores above 75000 in Ms. Pacman.

F EXTRA ABLATION STUDIES:

In Figure 11 , we compare the Sokoban test-set performance of our baseline DRC(3,3) agent against various ablated versions. These results justify our choice of the ConvLSTM memory module and the architectural improvements described in Section 2.1.2.

F.1 MEMORY TYPE:

The ConvLSTM is the top-performing memory module among those we tested. The Gated Con-vRNN module comes out very close with half as many parameters. We surmise that the gap between these two types could be larger for other domains.

The Simple ConvRNN suffers from instability and high variance as expected, asserting that some gating is essential for the DRC's performance.

F.2 ADDITIONAL IMPROVEMENTS:

Without pool-and-inject, the network learns significantly slower at early stages but nevertheless converges to the same performance as the baseline. The vision shortcut seems to have little influence on the baseline model. This is likely because we already feed the encoded observation to every (d,n)-location of the DRC grid (see Appendix C). Without the top-down skip connection, the model exhibits larger performance variance and also slightly lower final performance. On the whole, none of these are critical to the model's performance. We remove our extra implementation details, namely pool-and-inject, the vision shortcut, and the top-down skip connection, from the model.

• The environment model is similar to the one used in Racanière et al. (2017) . It consists of a deterministic model that given a frame and one-hot encoded action, outputs a predicted next frame and reward. The input frame is first reduced in size using a convolutional layer with kernel size 8, stride 8 and 32 channels. The action is then combined with the output of this layer using pool-and-inject (see (Racanière et al., 2017) ). This is followed by 2 size preserving residual blocks with convolutions of shape 3. The output of the last residual block is then passed to two separate networks to predict frame and reward. The frame prediction head uses a convolution layer with shape 3, stride 1, followed by a deconvolution with stride 8, kernel shape 8 and 3 channels. The reward prediction head uses a convolution layer with shape 3, stride 1, followed by a linear layer with output size 3. We predict rewards binned in 3 categories (less than -1, between [-1, 1] and greater than 1), reducing it to a classification problem. Racanière et al. (2017) .

The agent was trained with the V-trace actor-critic algorithm described by Espeholt et al. (2018) . Unlike in Racanière et al. (2017) , the environment model was not pre-trained. Instead it was trained online, using the same data that was used to train the RL loss.

I DETAILS ABOUT FIGURES:

Unless otherwise noted, each curve is the average of five independent runs (with identical parameters). We also show a 95% confidence interval (using a shaded area) around each averaged curve. Before independent runs are averaged, we smoothen them with a window size of 5 million steps.

Steps in RL learning curves refer to the total number of environment steps seen by the actors.

:

Link to the dataset will be provided after the review process.

DRC(3,3) has around

300K more parameters, and it requires around 3 times more computation3 We compute the generalization gap by subtracting the performance (ratio of levels solved) on the training set from performance on the test set.4 Data was sampled on-policy from the expert policy executed on levels from the training datasets.

Figures and Tables:

Figure 1: Illustration of the agent's network architecture. This diagram shows DRC(2,3) for two time steps. Square boxes denote ConvLSTM modules and the rectangle box represents an MLP. Boxes with the same color share parameters.

Figure 3: a) Learning curves for various configurations of DRC in Sokoban-Unfiltered. b) Comparison with other network architectures tuned for Sokoban. Results are on test-set levels.

Figure 4: Forcing extra computation steps after training improves the performance of DRC on Sokoban-Medium set (5 networks, each tested on the same 5000 levels). Steps are performed by overriding the policy with no-op actions at the start of an episode.

Figure 5: Comparison of DRC(3,3) (Top, Large network) and DRC(1,1) (Bottom, Small network) when trained with RL on various train set sizes (subsets of the Sokoban-unfiltered training set). Left column shows the performance on levels from the corresponding train set, right column shows the performance on the test set (the same set across these experiments).

Figure 6: (a) Generalization results from a trained model on different training set size (Large, Medium and Small) for Sokoban. Left figure shows results on the unfiltered test set, right figure shows results on the medium test set. (b) Similar generalization results for trained models in Boxworld.

Figure 9: Learning curves in Sokoban for DRC architectures tested on the medium-difficulty test set. The dashed curve shows DRC(3,3) trained on the easier unfiltered dataset. The solid curves show DRC(3,3) and DRC(9,1) trained directly on the medium-difficulty train set.

Figure 10: Learning curves comparing the DRC(3,3) and DRC(1,1) network configurations in 5 Atari 2600 games. Results are averaged over two independent runs. We also provide ApeX-DQN (no-op regime) results from Horgan et al. (2018) as a reference.

Best results obtained on the test set for the different difficulty levels across RL agents within 2e9 steps of training (averaged across 5 independent runs).

Table content: <table/>

Peer review (iclr2019):
 The authors hypothesize that, under appropriate conditions, neural networks without specific architectural biases trained by model-free reinforcement learning algorithms are capable of learning procedures that are analogous to planning. This is certainly an important area of research in reinforcement learning. Unfortunately, the approach employed to demonstrate this hypothesis seems flawed, which is why this submission should be rejected in its present form. The authors suggest that the presence of planning should be accompanied by three observable characteristics: generalization of desired behavior to radically different situations, learning of desired behavior from small amounts of data, and ability to benefit from additional "thinking" time. Instead of trying to identify how an environmental model is represented by a network and how it is used for planning, the authors focus on checking for the aforementioned characteristics. Even after conceding their strong claim despite weak argumentation provided by the authors, there are fundamental experimental issues that make the conclusions of this study unwarranted. Regarding the first two characteristics, the concepts of "radically different situations" or "small amounts of data" are extremely vague. Basically the authors assume that their problems are difficult enough to require planning. Having solved these problems with their proposed architecture, they conclude that planning must have occurred. Regarding the use of additional "thinking time," the authors claim that the improvement in performance caused by providing additional micro-steps to a recurrent neural network is clear evidence that something analogous to planning is happening, which is obviously not the case. While it would not be surprising if there was indeed something analogous to planning happening inside the networks under consideration, this paper presents no stronger evidence for this claim than most other deep reinforcement learning papers that tackle complex environments. Perhaps the most important contribution of this submission is the architecture based on ConvLSTMs proposed by the authors, which apparently surpasses many alternatives, including some biased towards planning. However, surpassing planning models is not strong evidence of planning. When stripped of unwarranted claims made by the authors regarding implicit planning, the proposed architecture does not seem sufficiently novel to warrant acceptance. The authors should be commended for what was certainly very demanding experimental work, even though it does not support their core claims. Their second most important contribution is the experimental comparison between several recent architectures in a diverse selection of environments. The writing is clear and accessible, except possibly for the architectural details described in Section 2.1.2, which do not seem very important. There are also several typos in Appendix D.2. Regarding related work, the authors mention that Pang and Werbos [1] "advanced the approach." But they do not explain how they advanced this approach. In fact, we could not find much about this in the 1998 paper. Also, to our knowledge, "additional thinking time" was first proposed in the context of reinforcement learning and planning with two interacting RNNs by Schmidhuber [2, Section: "more network ticks than environmental ticks"]. [1] Xiaozhong Pang & Paul J. Werbos (1998): Neural Network Design for J Function Approximation in Dynamic Programming [2] J. Schmidhuber. Making the world differentiable: On using fully recurrent self-supervised neural networks for dynamic reinforcement learning and planning in non-stationary environments. TR FKI-126-90, TU Munich, November 1990. Perhaps a strongly revised version the paper might become more acceptable if the authors addressed the issues above and especially toned down their claims about having experimentally identified the emergence of planning. Instead they should be extremely careful here, perhaps present this as "intriguing results," and address all possible counter arguments.