Title: BABYAI: A PLATFORM TO STUDY THE SAMPLE EFFI-CIENCY OF GROUNDED LANGUAGE LEARNING

Abstract: Allowing humans to interactively train artificial agents to understand language instructions is desirable for both practical and scientific reasons. Though, given the lack of sample efficiency in current learning methods, reaching this goal may require substantial research efforts. We introduce the BabyAI research platform, with the goal of supporting investigations towards including humans in the loop for grounded language learning. The BabyAI platform comprises an extensible suite of 19 levels of increasing difficulty. Each level gradually leads the agent towards acquiring a combinatorially rich synthetic language, which is a proper subset of English. The platform also provides a hand-crafted bot agent, which simulates a human teacher. We report estimated amount of supervision required for training neural reinforcement and behavioral-cloning agents on some BabyAI levels. We put forward strong evidence that current deep learning methods are not yet sufficiently sample-efficient in the context of learning a language with compositional properties.

1. INTRODUCTION:

How can a human train an intelligent agent to understand natural language instructions? We believe that this research question is important from both technological and scientific perspectives. No matter how advanced AI technology becomes, human users will likely want to customize their intelligent helpers to better understand their desires and needs. On the other hand, developmental psychology, cognitive science and linguistics study similar questions but applied to human children, and a synergy is possible between research in grounded language learning by computers and research in human language acquisition.

In this work, we present the BabyAI research platform, whose purpose is to facilitate research on grounded language learning. In our platform we substitute a simulated human expert for a real human; yet our aspiration is that BabyAI-based studies enable substantial progress towards putting an actual human in the loop. The current domain of BabyAI is a 2D gridworld in which synthetic natural-looking instructions (e.g. "put the red ball next to the box on your left") require the agent to navigate the world (including unlocking doors) and move objects to specified locations. BabyAI improves upon similar prior setups (Hermann et al., 2017; Chaplot et al., 2018; Yu et al., 2018) by supporting simulation of certain essential aspects of the future human in the loop agent training: curriculum learning and interactive teaching. The usefulness of curriculum learning for training machine learning models has been demonstrated numerous times in the literature (Bengio et al., 2009; Kumar et al., 2010; Zaremba and Sutskever, 2015; Graves et al., 2016) , and we believe that gradually increasing the difficulty of the task will likely be essential for achieving efficient humanmachine teaching, as in the case of human-human teaching. To facilitate curriculum learning studies, BabyAI currently features 19 levels in which the difficulty of the environment configuration and the complexity of the instruction language are gradually increased. Interactive teaching, i.e. teaching differently based on what the learner can currently achieve, is another key capability of human teachers. Many advanced agent training methods, including DAGGER (Ross et al., 2011) , TAMER (Warnell et al., 2017) and learning from human preferences (Wilson et al., 2012; Christiano et al., 2017) , assume that interaction between the learner and the teacher is possible. To support interactive experiments, BabyAI provides a bot agent that can be used to generate new demonstrations on the fly and advise the learner on how to continue acting.

Arguably, the main obstacle to language learning with a human in the loop is the amount of data (and thus human-machine interactions) that would be required. Deep learning methods that are used in the context of imitation learning or reinforcement learning paradigms have been shown to be very effective in both simulated language learning settings (Mei et al., 2016; Hermann et al., 2017) and applications (Sutskever et al., 2014; Bahdanau et al., 2015; Wu et al., 2016) . These methods, however, require enormous amounts of data, either in terms of millions of reward function queries or hundreds of thousands of demonstrations. To show how our BabyAI platform can be used for sample efficiency research, we perform several case studies. In particular, we estimate the number of demonstrations/episodes required to solve several levels with imitation and reinforcement learning baselines. As a first step towards improving sample efficiency, we additionally investigate to which extent pretraining and interactive imitation learning can improve sample efficiency.

The concrete contributions of this paper are two-fold. First, we contribute the BabyAI research platform for learning to perform language instructions with a simulated human in the loop. The platform already contains 19 levels and can easily be extended. Second, we establish baseline results for all levels and report sample efficiency results for a number of learning approaches. The platform and pretrained models are available online. We hope that BabyAI will spur further research towards improving sample efficiency of grounded language learning, ultimately allowing human-in-the-loop training.

2. RELATED WORK:

There are numerous 2D and 3D environments for studying synthetic language acquistion. (Hermann et al., 2017; Chaplot et al., 2018; Yu et al., 2018; Wu et al., 2018) . Inspired by these efforts, BabyAI augments them by uniquely combining three desirable features. First, BabyAI supports world state manipulation, missing in the visually appealing 3D environments of Hermann et al. (2017) , Chaplot et al. (2018) and Wu et al. (2018) . In these environments, an agent can navigate around, but cannot alter its state by, for instance, moving objects. Secondly, BabyAI introduces partial observability (unlike the gridworld of Bahdanau et al. (2018) ). Thirdly, BabyAI provides a systematic definition of the synthetic language. As opposed to using instruction templates, the Baby Language we introduce defines the semantics of all utterances generated by a context-free grammar (Section 3.2). This makes our language richer and more complete than prior work. Most importantly, BabyAI provides a simulated human expert, which can be used to investigate human-in-the-loop training, the aspiration of this paper.

Currently, most general-purpose simulation frameworks do not feature language, such as PycoLab (DeepMind, 2017), MazeBase (Sukhbaatar et al., 2015) , Gazebo (Koenig and Howard, 2004) , Viz-Doom (Kempka et al., 2016) , DM-30 (Espeholt et al., 2018) , and AI2-Thor (Kolve et al., 2017) . Using a more realistic simulated environment such as a 3D rather than 2D world comes at a high computational cost. Therefore, BabyAI uses a gridworld rather than 3D environments. As we found that available gridworld platforms were insufficient for defining a compositional language, we built a MiniGrid environment for BabyAI.

General-purpose RL testbeds such as the Arcade Learning Environment (Bellemare et al., 2013) , DM-30 (Espeholt et al., 2018) , and MazeBase (Sukhbaatar et al., 2015) do not assume a humanin-the-loop setting. In order to simulate this, we have to assume that all rewards (except intrinsic PutNextLocal: "put the blue key next to the green ball" (c) BossLevel: "pick up the grey box behind you, then go to the grey key and open a door". Note that the green door near the bottom left needs to be unlocked with a green key, but this is not explicitly stated in the instruction.

Figure 1 : Three BabyAI levels built using the MiniGrid environment. The red triangle represents the agent, and the light-grey shaded area represents its field of view (partial observation). rewards) would have to be given by a human, and are therefore rather expensive to get. Under this assumption, imitation learning methods such as behavioral cloning, Searn (Daumé Iii et al., 2009) , DAGGER (Ross et al., 2011) or maximum-entropy RL (Ziebart et al., 2008) are more appealing, as more learning can be achieved per human-input unit.

Similar to BabyAI, studying sample efficiency of deep learning methods was a goal of the bAbI tasks (Weston et al., 2016) , which tested reasoning capabilities of a learning agent. Our work differs in both of the object of the study (grounded language with a simulated human in the loop) and in the method: instead of generating a fixed-size dataset and measuring the performance, we measure how much data a general-purpose model would require to get close-to-perfect performance.

There has been much research on instruction following with natural language (Tellex et al., 2011; Chen and Mooney, 2011; Artzi and Zettlemoyer, 2013; Mei et al., 2016; Williams et al., 2018) as well as several datasets including SAIL (Macmahon et al., 2006; Chen and Mooney, 2011) and Room-to-Room (Anderson et al., 2018) . Instead of using natural language, BabyAI utilises a synthetic Baby language, in order to fully control the semantics of an instruction and easily generate as much data as needed.

Finally, Wang et al. (2016) presented a system that interactively learned language from a human. We note that their system relied on substantial amounts of prior knowledge about the task, most importantly a task-specific executable formal language.

3. BABYAI PLATFORM:

The BabyAI platform that we present in this work comprises an efficiently simulated gridworld environment (MiniGrid) and a number of instruction-following tasks that we call levels, all formulated using subsets of a synthetic language (Baby Language). The platform also includes a bot that can generate successful demonstrations for all BabyAI levels. All the code is available online at https://github.com/mila-iqia/babyai/tree/iclr19.

3.1. MINIGRID ENVIRONMENT:

Studies of sample efficiency are very computationally expensive given that multiple runs are required for different amounts of data. Hence, in our design of the environment, we have aimed for a minimalistic and efficient environment which still poses a considerable challenge for current general-purpose agent learning methods. We have implemented MiniGrid, a partially observable 2D gridworld environment. The environment is populated with entities of different colors, such as the agent, balls, boxes, doors and keys (see Figure 1 ). Objects can be picked up, dropped and moved around by the agent. Doors can be unlocked with keys matching their color. At each step, the agent receives a 7x7 representation of its field of view (the grid cells in front of it) as well as a Baby Language instruction (textual string).

The MiniGrid environment is fast and lightweight. Throughput of over 3000 frames per second is possible on a modern multi-core laptop, which makes experimentation quicker and more accessible.

The environment is open source, available online, and supports integration with OpenAI Gym. For more details, see Appendix B.

3.2. BABY LANGUAGE:

We have developed a synthetic Baby Language to give instructions to the agent as well as to automatically verify their execution. Although Baby Language utterances are a comparatively small subset of English, they are combinatorially rich and unambigously understood by humans. The language is intentionally kept simple, but still exhibits interesting combinatorial properties, and contains 2.48 × 10 19 possible instructions. In this language, the agent can be instructed to go to objects, pick up objects, open doors, and put objects next to other objects. The language also expresses the conjunction of several such tasks, for example "put a red ball next to the green box after you open the door". The Backus-Naur Form (BNF) grammar for the language is presented in Figure 2 and some example instructions drawn from this language are shown in Figure 3 . In order to keep the resulting instructions readable by humans, we have imposed some structural restrictions on this language: the and connector can only appear inside the then and after forms, and instructions can contain no more than one then or after word. The BabyAI platform includes a verifier which checks if an agent's sequence of actions successfully achieves the goal of a given instruction within an environment. Descriptors in the language refer to one or to multiple objects. For instance, if an agent is instructed to "go to a red door", it can successfully execute this instruction by going to any of the red doors in the environment. The then and after connectors can be used to sequence subgoals. The and form implies that both subgoals must be completed, without ordering constraints. Importantly, Baby Language instructions leave 

3.3. BABYAI LEVELS:

There is abundant evidence in prior literature which shows that a curriculum may greatly facilitate learning of complex tasks for neural architectures (Bengio et al., 2009; Kumar et al., 2010; Zaremba and Sutskever, 2015; Graves et al., 2016) . To investigate how a curriculum improves sample efficiency, we created 19 levels which require understanding only a limited subset of Baby Language within environments of varying complexity. Formally, a level is a distribution of missions, where a mission combines an instruction within an initial environment state. We built levels by selecting competencies necessary for each level and implementing a generator to generate missions solvable by an agent possessing only these competencies. Each competency is informally defined by specifying what an agent should be able to do:

• Room Navigation (ROOM): navigate a 6x6 room.

• Ignoring Distracting Boxes (DISTR-BOX): navigate the environment even when there are multiple distracting grey box objects in it. • Ignoring Distractors (DISTR): same as DISTR-BOX, but distractor objects can be boxes, keys or balls of any color. • Maze Navigation (MAZE): navigate a 3x3 maze of 6x6 rooms, randomly inter-connected by doors. • Unblocking the Way (UNBLOCK): navigate the environment even when it requires moving objects out of the way. • Unlocking Doors (UNLOCK): to be able to find the key and unlock the door if the instruction requires this explicitly. • Guessing to Unlock Doors (IMP-UNLOCK): to solve levels that require unlocking a door, even if this is not explicitly stated in the instruction. • Go To Instructions (GOTO): understand "go to" instructions, e.g. "go to the red ball".

• Open Instructions (OPEN): understand "open" instructions, e.g. "open the door on your left". • Pickup Instructions (PICKUP): understand "pick up" instructions, e.g. "pick up a box".

• Put Instructions (PUT): understand "put" instructions, e.g. "put a ball next to the blue key". • Location Language (LOC): understand instructions where objects are referred to by relative location as well as their shape and color, e.g. "go to the red ball in front of you". 

x GoToRedBallGrey x x GoToRedBall x x x GoToLocal x x x x PutNextLocal x x x x PickupLoc x x x x x GoToObjMaze x x GoTo x x x x x Pickup x x x x x UnblockPickup x x x x x x Open x x x x x Unlock x x x x x x PutNext x x x x x Synth x x x x x x x x x x SynthLoc x x x x x x x x x x x GoToSeq x x x x x x SynthSeq x x x x x x x x x x x x GoToImpUnlock x x x x x x BossLevel x x x x x x x x x x x x x

the BossLevel, which requires mastering all competencies. The definitions of competencies are informal and should be understood in the minimalistic sense, i.e. to test the ROOM competency we have built the GoToObj level where the agent needs to reach the only object in an empty room. Note that the GoToObj level does not require the GOTO competency, as this level can be solved without any language understanding, since there is only a single object in the room. However, solving the GoToLocal level, which instructs the agent to go to a specific object in the presence of multiple distractors, requires understanding GOTO instructions.

3.4. THE BOT AGENT:

The bot is a key ingredient intended to perform the role of a simulated human teacher. For any of the BabyAI levels, it can generate demonstrations or suggest actions for a given environment state.

Whereas the BabyAI learner is meant to be generic and should scale to new and more complex tasks, the bot is engineered using knowledge of the tasks. This makes sense since the bot stands for the human in the loop, who is supposed to understand the environment, how to solve missions, and how to teach the baby learner. The bot has direct access to a tree representation of instructions, and so does not need to parse the Baby Language. Internally, it executes a stack machine in which instructions and subgoals are represented (more details can be found in Appendix C). The stackbased design allows the bot to interrupt what it is currently doing to achieve a new subgoal, and then resume the original task. For example, going to a given object will require exploring the environment to find that object.

The subgoals which the bot implements are:

• Open: Open a door that is in front of the agent.

• Close: Close a door that is in front of the agent.

• Pickup: Execute the pickup action (pick up an object).

• Drop: Execute the drop action (drop an object being carried).

• GoNextTo: Go next to an object matching a given (type, color) description or next to a cell at a given position.

• Explore: Uncover previously unseen parts of the environment.

All of the Baby Language instructions are decomposed into these internal subgoals which the bot knows how to solve. Many of these subgoals, during their execution, can also push new subgoals on the stack. A central part of the design of the bot is that it keeps track of the grid cells of the environment which it has and has not seen. This is crucial to ensure that the bot can only use information which it could realistically have access to by exploring the environment. Exploration is implemented as part of the Explore subgoal, which is recursive. For instance, exploring the environment may require opening doors, or moving objects that are in the way. Opening locked doors may in turn require finding a key, which may itself require exploration and moving obstructing objects. Another key component of the bot's design is a shortest path search routine. This is used to navigate to objects, to locate the closest door, or to navigate to the closest unexplored cell.

4. EXPERIMENTS:

We assess the difficulty of BabyAI levels by training a behavioral cloning baseline for each level. Furthermore, we estimate how much data is required to solve some of the simpler levels and study to which extent the data demands can be reduced by using basic curriculum learning and interactive teaching methods. All the code that we use for the experiments, as well as containerized pretrained models, is available online.

4.1. SETUP:

The BabyAI platform provides by default a 7x7x3 symbolic observation x t (a partial and local egocentric view of the state of the environment) and a variable length instruction c as inputs at each time step. We use a basic model consisting of standard components to predict the next action a based on x and c. In particular, we use a GRU (Cho et al., 2014) to encode the instruction and a convolutional network with two batch-normalized (Ioffe and Szegedy, 2015) FiLM (Perez et al., 2017) layers to jointly process the observation and the instruction. An LSTM (Hochreiter and Schmidhuber, 1997) memory is used to integrate representations produced by the FiLM module at each step. Our model is thus similar to the gated-attention model used by Chaplot et al. (2018) , inasmuch as gated attention is equivalent to using FiLM without biases and only at the output layer.

We have used two versions of our model, to which we will refer as the Large model and the Small model. In the Large model, the memory LSTM has 2048 units and the instruction GRU is bidirectional and has 256 units. Furthermore, an attention mechanism (Bahdanau et al., 2015) is used to focus on the relevant states of the GRU. The Small model uses a smaller memory of 128 units and encodes the instruction with a unidirectional GRU and no attention mechanism.

In all our experiments, we used the Adam optimizer (Kingma and Ba, 2015) with the hyperparameters α = 10 -4 , β 1 = 0.9, β 2 = 0.999 and = 10 -5 . In our imitation learning (IL) experiments, we truncated the backpropagation through time at 20 steps for the Small model and at 80 steps for the Large model. For our reinforcement learning experiments, we used the Proximal Policy Optimization (PPO, Schulman et al., 2017) algorithm with parallelized data collection. Namely, we performed 4 epochs of PPO using 64 rollouts of length 40 collected with multiple processes. We gave a non-zero reward to the agent only when it fully completed the mission, and the magnitude of the reward was 1 -0.9n/n max , where n is the length of the successful episode and n max is the maximum number of steps that we allowed for completing the episode, different for each mission. The future returns were discounted with a factor γ = 0.99. For generalized advantage estimation (Schulman et al., 2015) in PPO we used λ = 0.99.

In all our experiments we reported the success rate, defined as the ratio of missions of the level that the agent was able to accomplish within n max steps.

Running the experiments outlined in this section required between 20 and 50 GPUs over two weeks. At least as much computing was required for preliminary investigations. 

4.2. BASELINE RESULTS:

To obtain baseline results for all BabyAI levels, we have trained the Large model (see Section 4.1) with imitation learning using one million demonstration episodes for each level. The demonstrations were generated using the bot described in Section 3.4. The models were trained for 40 epochs on levels with a single room and for 20 epochs on levels with a 3x3 maze of rooms. Table 2 reports the maximum success rate on a validation set of 512 episodes. All of the single-room levels are solved with a success rate of 100.0%. As a general rule, levels for which demonstrations are longer tend to be more difficult to solve.

Using 1M demonstrations for levels as simple as GoToRedBall is very inefficient and hardly ever compatible with the long-term goal of enabling human teaching. The BabyAI platform is meant to support studies of how neural agents can learn with less data. To bootstrap such studies, we have computed baseline sample efficiencies for imitation learning and reinforcement learning approaches to solving BabyAI levels. We say an agent solves a level if it reaches a success rate of at least 99%. We define the sample efficiency as the minimum number of demonstrations or RL episodes required to train an agent to solve a given level. To estimate the thus defined sample efficiency for imitation learning while staying within a reasonable computing budget, we adopt the following procedure. For a given level, we first run three experiments with 10 6 demonstrations. In the remaining M experiments we use

k 1 = 2 l0 , k 2 = 2 l0+d , . . . , k M = 2 l0+(M -1)d demonstrations respectively.

We use different values of l 0 , M for each level to ensure that we run experiments with not enough, just enough and more than enough demonstrations. Same value of d = 0.2 is used in all imitation learning experiments. For each experiment i, we measure the best smoothed online validation performance s i that is achieved during the first 2T training steps, where T = (T 1 + T 2 + T 3 )/3 is the average number of training steps required to solve the level in the three runs with 10 6 demonstrations. We then fit a Gaussian Process (GP) model (Rasmussen and Williams, 2005) with noisy observations using (k i , s i ) as training data in order to interpolate between these data points. The GP posterior is fully tractable, which allows us to compute analytically the posterior distribution of the expected success rate, as well as the posterior over the minimum number of samples k min that is sufficient to solve the level. We report the 99% credible interval for k min . We refer the reader to Appendix A for a more detailed explanation of this procedure.

We estimate sample efficiency of imitation learning on 6 chosen levels. The results are shown in Table 3 (see "IL from Bot" column). In the same table (column "RL") we report the 99% confidence interval for the number of episodes that were required to solve each of these levels with RL, and as expected, the sample efficiency of RL is substantially worse than that of IL (anywhere between 2 to 10 times in these experiments).

To analyze how much the sample efficiency of IL depends on the source of demonstrations, we try generating demonstrations from agents that were trained with RL in the previous experiments.

The results for the 3 easiest levels are reported in the "IL from RL Expert" column in Table 5 . Interestingly, we found that the demonstrations produced by the RL agent are easier for the learner to imitate. The difference is most significant for GoToRedBallGrey, where less than 2K and more than 8K RL and bot demonstrations respectively are required to solve the level. For GoToRedBall and GoToLocal, using RL demonstrations results in 1.5-2 times better sample efficiency. This can be explained by the fact that the RL expert has the same neural network architecture as the learner.

4.3. CURRICULUM LEARNING:

To demonstrate how curriculum learning research can be done using the BabyAI platform, we perform a number of basic pretraining experiments. In particular, we select 5 combinations of base levels and a target level and study whether pretraining on base levels can help the agent master the target level with fewer demonstrations. The results are reported in Table 4 . In four cases, using GoToLocal as one of the base levels reduces the number of demonstrations required to solve the target level. However, when only GoToObjMaze was used as the base level, we have not found pretraining to be beneficial. We find this counter-intuitive result interesting, as it shows how current deep learning methods often can not take the full advantage of available curriculums.

4.4. INTERACTIVE LEARNING:

Lastly, we perform a simple case study of how sample efficiency can be improved by interactively providing more informative examples to the agent based on what it has already learned. We experiment with an iterative algorithm for adaptively growing the agent's training set. In particular, we start with 2 10 base demonstrations, and at each iteration we increase the dataset size by a factor of 2 1/4 by providing bot demonstrations for missions on which the agent failed. After each dataset increase we train a new agent from scratch. We perform such dataset increases until the dataset reaches the final size is clearly sufficient to achieve 99% success rate. We repeat the experiment 3 times for levels GoToRedBallGrey, GoToRedBall and GoToLocal and then estimate how many interactively provided demonstrations would be required for the agent be 99% successful for each of these levels. To this end, we use the same GP posterior analysis as for regular imitation learning experiments.

The results for the interactive imitation learning protocol are reported in Table 5 . For all 3 levels that we experimented with, we have observed substantial improvement over the vanilla IL, which is most significant (4 times less demonstrations) for GoToRedBallGrey and smaller (1.5-2 times less demonstrations) for the other two levels.

5. CONCLUSION & FUTURE WORK:

We present the BabyAI research platform to study language learning with a human in the loop. The platform includes 19 levels of increasing difficulty, based on a decomposition of tasks into a set of basic competencies. Solving the levels requires understanding the Baby Language, a subset of English with a formally defined grammar which exhibits compositional properties. The language is minimalistic and the levels seem simple, but empirically we have found them quite challenging to solve. The platform is open source and extensible, meaning new levels and language concepts can be integrated easily.

The results in Section 4 suggest that current imitation learning and reinforcement learning methods scale and generalize poorly when it comes to learning tasks with a compositional structure. Hundreds of thousands of demonstrations are needed to learn tasks which seem trivial by human standards. Methods such as curriculum learning and interactive learning can provide measurable improvements in terms of sample efficiency, but, in order for learning with an actual human in the loop to become realistic, an improvement of at least three orders of magnitude is required.

An obvious direction of future research to find strategies to improve sample efficiency of language learning. Tackling this challenge will likely require new models and new teaching methods. Approaches that involve an explicit notion of modularity and subroutines, such as Neural Module Networks (Andreas et al., 2016) or Neural Programmer-Interpreters (Reed and de Freitas, 2015) , seem like a promising direction. It is our hope that the BabyAI platform can serve as a challenge and a benchmark for the sample efficiency of language learning for years to come.

aware of an analytic expression for p(k min |D), and hence we compute a numerical approximation as follows. We sample a dense log-scale grid of M points k

1 , k 2 , . . . , k M in the range [k 1 ; k M ].

For each number of demonstrations k i we approximate the probability p(k i-1 < k min < k i |D) that s(k) crosses the 99% threshold somewhere between k i-1 and k i as follows:

EQUATION

Equation 5 is an approximation because the posterior s is not necessarily monotonic. In practice, we observed that the monotonic nature of the observed data D shapes the posterior accordingly. We use the probabilities p i to construct the following discrete approximation of the posterior p(k min |D):

EQUATION

where δ(k i ) are Dirac delta-functions. Such a discrete approximation is sufficient for the purpose of computing 99% credible intervals for k min that we report in the paper.

B MINIGRID ENVIRONMENTS FOR OPENAI GYM:

The environments used for this research are built on top of MiniGrid, which is an open source gridworld package. This package includes a family of reinforcement learning environments compatible with the OpenAI Gym framework. Many of these environments are parameterizable so that the difficulty of tasks can be adjusted (e.g. the size of rooms is often adjustable).

B.1 THE WORLD:

In MiniGrid, the world is a grid of size NxN. Each tile in the grid contains exactly zero or one object, and the agent can only be on an empty tile or on a tile containing an open door. The possible object types are wall, door, key, ball, box and goal. Each object has an associated discrete color, which can be one of red, green, blue, purple, yellow and grey. By default, walls are always grey and goal squares are always green.

B.2 REWARD FUNCTION:

Rewards are sparse for all MiniGrid environments. Each environment has an associated time step limit. The agent receives a positive reward if it succeeds in satisfying an environment's success criterion within the time step limit, otherwise zero. The formula for calculating positive sparse rewards is 1 -0.9 * (step_count/max_steps). That is, rewards are always between zero and one, and the quicker the agent can successfully complete an episode, the closer to 1 the reward will be. The max_steps parameter is different for each mission, and varies depending on the size of the environment (larger environments having a higher time step limit) and the length of the instruction (more time steps are allowed for longer instructions).

B.3 ACTION SPACE:

There are seven actions in MiniGrid: turn left, turn right, move forward, pick up an object, drop an object, toggle and done. The agent can use the turn left and turn right action to rotate and face one of 4 possible directions (north, south, east, west) . The move forward action makes the agent move from its current tile onto the tile in the direction it is currently facing, provided there is nothing on that tile, or that the tile contains an open door. The agent can open doors if they are right in front of it by using the toggle action.

B.4 OBSERVATION SPACE:

Observations in MiniGrid are partial and egocentric. By default, the agent sees a square of 7x7 tiles in the direction it is facing. These include the tile the agent is standing on. The agent cannot see through walls or closed doors. The observations are provided as a tensor of shape 7x7x3. However, note that these are not RGB images. Each tile is encoded using 3 integer values: one describing the

:



Figures and Tables:

Figure 2: BNF grammar productions for the Baby Language

Figure 3: Example Baby Language instructions

BabyAI Levels and the required competencies

Table content: <table><tr><td>ROOM</td><td>DISTR-BOX</td><td>DISTR</td><td>MAZE</td><td>UNBLOCK</td><td>UNLOCK</td><td>IMP-UNLOCK</td><td>GOTO</td><td>OPEN</td><td>PICKUP</td><td>PUT</td><td>LOC</td><td>SEQ</td></tr><tr><td>GoToObj</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr></table>

Baseline imitation learning results for all BabyAI levels. Each model was trained with 1M demonstrations from the respective level. For reference, we also list the mean and standard deviation of demonstration length for each level.

Table content: <table><tr><td>Model</td><td colspan="2">Success Rate (%) Demo Length (Mean ± Std)</td></tr><tr><td>GoToObj</td><td>100</td><td>5.18±2.38</td></tr><tr><td>GoToRedBallGrey</td><td>100</td><td>5.81±3.29</td></tr><tr><td>GoToRedBall</td><td>100</td><td>5.38±3.13</td></tr><tr><td>GoToLocal</td><td>99.8</td><td>5.04±2.76</td></tr><tr><td>PutNextLocal</td><td>99.2</td><td>12.4±4.54</td></tr><tr><td>PickupLoc</td><td>99.4</td><td>6.13±2.97</td></tr><tr><td>GoToObjMaze</td><td>99.9</td><td>70.8±48.9</td></tr><tr><td>GoTo</td><td>99.4</td><td>56.8±46.7</td></tr><tr><td>Pickup</td><td>99</td><td>57.8±46.7</td></tr><tr><td>UnblockPickup</td><td>99</td><td>57.2±50</td></tr><tr><td>Open</td><td>100</td><td>31.5±30.5</td></tr><tr><td>Unlock</td><td>98.4</td><td>81.6±61.1</td></tr><tr><td>PutNext</td><td>98.8</td><td>89.9±49.6</td></tr><tr><td>Synth</td><td>97.3</td><td>50.4±49.3</td></tr><tr><td>SynthLoc</td><td>97.9</td><td>47.9±47.9</td></tr><tr><td>GoToSeq</td><td>95.4</td><td>72.7±52.2</td></tr><tr><td>SynthSeq</td><td>87.7</td><td>81.8±61.3</td></tr><tr><td>GoToImpUnlock</td><td>87.2</td><td>110±81.9</td></tr><tr><td>BossLevel</td><td>77</td><td>84.3±64.5</td></tr></table>

The sample efficiency of imitation learning (IL) and reinforcement learning (RL) as the number of demonstrations (episodes) required to solve each level. All numbers are thousands. For the imitation learning results we report a 99% credible interval. For RL experiments we report the 99% confidence interval. See Section 4 for details.

Table content: <table><tr><td>Level</td><td>IL from Bot</td><td>RL</td></tr><tr><td colspan="2">GoToRedBallGrey 8.431 -12.43</td><td>15.9 -17.4</td></tr><tr><td>GoToRedBall</td><td colspan="2">49.67 -62.01 261.1 -333.6</td></tr><tr><td>GoToLocal</td><td>148.5 -193.2</td><td>903 -1114</td></tr><tr><td>PickupLoc</td><td colspan="2">204.3 -241.2 1447 -1643</td></tr><tr><td>PutNextLocal</td><td colspan="2">244.6 -322.7 2186 -2727</td></tr><tr><td>GoTo</td><td>341.1 -408.5</td><td>816 -1964</td></tr></table>

The sample efficiency results for pretraining experiments. For each pair of base levels and target levels that we have tried, we report how many demonstrations (in thousands) were required, as well as the baseline number of demonstrations required for training from scratch. In both cases we report a 99% credible interval, see Section 4 for details. Note how choosing the right base levels (e.g. GoToLocal instead of GoToObjMaze) is crucial for pretraining to be helpful.

Table content: <table><tr><td>Base Levels</td><td colspan="3">Target Level Without Pretraining With Pretraining</td></tr><tr><td>GoToLocal</td><td>GoTo</td><td>341 -409</td><td>183 -216</td></tr><tr><td>GoToObjMaze</td><td>GoTo</td><td>341 -409</td><td>444 -602</td></tr><tr><td>GoToLocal-GoToObjMaze</td><td>GoTo</td><td>341 -409</td><td>173 -216</td></tr><tr><td>GoToLocal</td><td>PickupLoc</td><td>204 -241</td><td>71.2 -88.9</td></tr><tr><td>GoToLocal</td><td>PutNextLocal</td><td>245 -323</td><td>188 -231</td></tr></table>

The sample efficiency of imitation learning (IL) from an RL-pretrained expert and interactive imitation learning defined as the number of demonstrations required to solve each level. All numbers are in thousands. 99% credible intervals are reported in all experiments, see Section 4 for details.

Table content: <table><tr><td>Level</td><td colspan="3">IL from Bot IL from RL Expert Interactive IL from Bot</td></tr><tr><td colspan="2">GoToRedBallGrey 8.43 -12.4</td><td>1.53 -2.11</td><td>1.71 -1.88</td></tr><tr><td>GoToRedBall</td><td>49.7 -62</td><td>36.6 -44.5</td><td>31.8 -36</td></tr><tr><td>GoToLocal</td><td>148 -193</td><td>74.2 -81.8</td><td>93 -107</td></tr></table>

Peer review (iclr2019):
 This paper focuses on grounded language learning with a human in the loop, in the sense where the language is synthetic, the environment is a 2D grid world, and the human is a simulated human teacher implemented using heuristics. This setup is dubbed the BabyAI platform, and includes curriculum learning over 19 levels of increasing difficulty. Overall, the BabyAI platform is conceptually similar to numerous previous works that seek to learn grounded language in simulation environments. These efforts differ along various axes, for example visual realism, 2D vs 3D, partially vs. fully observed, different tasks, world-state manipulation or not, etc. The main original aspect of the BabyAI platform is the simulated human-teacher. Strengths - Learning with a human in the loop is an extremely important problem to study, although currently efforts are hampered by cost, lack or reproducibility, and the sample inefficiency of existing learning methods. This paper addresses all three of these issues, albeit by removing the human and natural language. This is simultaneously the greatest weakness of this approach. The contribution of this paper therefore rests on the quality/interestingness/utility of the provided synthetic language and the synthetic teacher. - Fortunately, the synthetic language does exhibit interesting compositional properties, it is readily extensible, it has the appealing property that it can be readily interpreted as a subset of english, and it is accompanied by a verifier to check if the specified actions were completed. Weaknesses - If the ultimate goal is learning with a human in the loop, the usefulness of the synthetic teacher is not clear, particularly as it is apparently easier to imitate from an RL trained agent than the teacher. The explanation *This can explained by the fact that the RL expert has the same neural network architecture as the learner* does no seem obvious to me. - Regarding the human in the loop, since this is aspirational and not an aspect of the paper, the title of the paper does not seem reflective of its content (even with the *First steps* qualifier). - If the main unique aspect is the simulated human-teacher, it is not clear why it is necessary to create a new environment, rather than re-using an existing environment. The effect of this is to limit comparisons with recent work and an increasing fragmentation of research across tasks that are related but can’t be compared. Summary: This paper represents an important direction, in that it provides a testbed for studying the sample efficiency of grounded language learning in a simplified (yet still challenging and compositional) environment. I believe the environment and the provided synthetic language and verifier will prove useful to the community, and despite some reservations about the title and the simulated human-teacher, I recommend acceptance.