BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learning

1 INTRODUCTION . How can a human train an intelligent agent to understand natural language instructions ? We believe that this research question is important from both technological and scientific perspectives . No matter how advanced AI technology becomes , human users will likely want to customize their intelligent helpers to better understand their desires and needs . On the other hand , developmental psychology , cognitive science and linguistics study similar questions but applied to human children , and a synergy is possible between research in grounded language learning by computers and research in human language acquisition . In this work , we present the BabyAI research platform , whose purpose is to facilitate research on grounded language learning . In our platform we substitute a simulated human expert for a real human ; yet our aspiration is that BabyAI-based studies enable substantial progress towards putting an actual human in the loop . The current domain of BabyAI is a 2D gridworld in which synthetic natural-looking instructions ( e.g . “ put the red ball next to the box on your left ” ) require the agent to navigate the world ( including unlocking doors ) and move objects to specified locations . BabyAI improves upon similar prior setups ( Hermann et al. , 2017 ; Chaplot et al. , 2018 ; Yu et al. , 2018 ) by supporting simulation of certain essential aspects of the future human in the loop agent training : ∗Equal contribution . †Work done during an internship at Mila . ‡Work done during a post-doc at Mila . curriculum learning and interactive teaching . The usefulness of curriculum learning for training machine learning models has been demonstrated numerous times in the literature ( Bengio et al. , 2009 ; Kumar et al. , 2010 ; Zaremba and Sutskever , 2015 ; Graves et al. , 2016 ) , and we believe that gradually increasing the difficulty of the task will likely be essential for achieving efficient humanmachine teaching , as in the case of human-human teaching . To facilitate curriculum learning studies , BabyAI currently features 19 levels in which the difficulty of the environment configuration and the complexity of the instruction language are gradually increased . Interactive teaching , i.e . teaching differently based on what the learner can currently achieve , is another key capability of human teachers . Many advanced agent training methods , including DAGGER ( Ross et al. , 2011 ) , TAMER ( Warnell et al. , 2017 ) and learning from human preferences ( Wilson et al. , 2012 ; Christiano et al. , 2017 ) , assume that interaction between the learner and the teacher is possible . To support interactive experiments , BabyAI provides a bot agent that can be used to generate new demonstrations on the fly and advise the learner on how to continue acting . Arguably , the main obstacle to language learning with a human in the loop is the amount of data ( and thus human-machine interactions ) that would be required . Deep learning methods that are used in the context of imitation learning or reinforcement learning paradigms have been shown to be very effective in both simulated language learning settings ( Mei et al. , 2016 ; Hermann et al. , 2017 ) and applications ( Sutskever et al. , 2014 ; Bahdanau et al. , 2015 ; Wu et al. , 2016 ) . These methods , however , require enormous amounts of data , either in terms of millions of reward function queries or hundreds of thousands of demonstrations . To show how our BabyAI platform can be used for sample efficiency research , we perform several case studies . In particular , we estimate the number of demonstrations/episodes required to solve several levels with imitation and reinforcement learning baselines . As a first step towards improving sample efficiency , we additionally investigate to which extent pretraining and interactive imitation learning can improve sample efficiency . The concrete contributions of this paper are two-fold . First , we contribute the BabyAI research platform for learning to perform language instructions with a simulated human in the loop . The platform already contains 19 levels and can easily be extended . Second , we establish baseline results for all levels and report sample efficiency results for a number of learning approaches . The platform and pretrained models are available online . We hope that BabyAI will spur further research towards improving sample efficiency of grounded language learning , ultimately allowing human-in-the-loop training . 2 RELATED WORK . There are numerous 2D and 3D environments for studying synthetic language acquistion . ( Hermann et al. , 2017 ; Chaplot et al. , 2018 ; Yu et al. , 2018 ; Wu et al. , 2018 ) . Inspired by these efforts , BabyAI augments them by uniquely combining three desirable features . First , BabyAI supports world state manipulation , missing in the visually appealing 3D environments of Hermann et al . ( 2017 ) , Chaplot et al . ( 2018 ) and Wu et al . ( 2018 ) . In these environments , an agent can navigate around , but can not alter its state by , for instance , moving objects . Secondly , BabyAI introduces partial observability ( unlike the gridworld of Bahdanau et al . ( 2018 ) ) . Thirdly , BabyAI provides a systematic definition of the synthetic language . As opposed to using instruction templates , the Baby Language we introduce defines the semantics of all utterances generated by a context-free grammar ( Section 3.2 ) . This makes our language richer and more complete than prior work . Most importantly , BabyAI provides a simulated human expert , which can be used to investigate human-in-the-loop training , the aspiration of this paper . Currently , most general-purpose simulation frameworks do not feature language , such as PycoLab ( DeepMind , 2017 ) , MazeBase ( Sukhbaatar et al. , 2015 ) , Gazebo ( Koenig and Howard , 2004 ) , VizDoom ( Kempka et al. , 2016 ) , DM-30 ( Espeholt et al. , 2018 ) , and AI2-Thor ( Kolve et al. , 2017 ) . Using a more realistic simulated environment such as a 3D rather than 2D world comes at a high computational cost . Therefore , BabyAI uses a gridworld rather than 3D environments . As we found that available gridworld platforms were insufficient for defining a compositional language , we built a MiniGrid environment for BabyAI . General-purpose RL testbeds such as the Arcade Learning Environment ( Bellemare et al. , 2013 ) , DM-30 ( Espeholt et al. , 2018 ) , and MazeBase ( Sukhbaatar et al. , 2015 ) do not assume a humanin-the-loop setting . In order to simulate this , we have to assume that all rewards ( except intrinsic rewards ) would have to be given by a human , and are therefore rather expensive to get . Under this assumption , imitation learning methods such as behavioral cloning , Searn ( Daumé Iii et al. , 2009 ) , DAGGER ( Ross et al. , 2011 ) or maximum-entropy RL ( Ziebart et al. , 2008 ) are more appealing , as more learning can be achieved per human-input unit . Similar to BabyAI , studying sample efficiency of deep learning methods was a goal of the bAbI tasks ( Weston et al. , 2016 ) , which tested reasoning capabilities of a learning agent . Our work differs in both of the object of the study ( grounded language with a simulated human in the loop ) and in the method : instead of generating a fixed-size dataset and measuring the performance , we measure how much data a general-purpose model would require to get close-to-perfect performance . There has been much research on instruction following with natural language ( Tellex et al. , 2011 ; Chen and Mooney , 2011 ; Artzi and Zettlemoyer , 2013 ; Mei et al. , 2016 ; Williams et al. , 2018 ) as well as several datasets including SAIL ( Macmahon et al. , 2006 ; Chen and Mooney , 2011 ) and Room-to-Room ( Anderson et al. , 2018 ) . Instead of using natural language , BabyAI utilises a synthetic Baby language , in order to fully control the semantics of an instruction and easily generate as much data as needed . Finally , Wang et al . ( 2016 ) presented a system that interactively learned language from a human . We note that their system relied on substantial amounts of prior knowledge about the task , most importantly a task-specific executable formal language . 3 BABYAI PLATFORM . The BabyAI platform that we present in this work comprises an efficiently simulated gridworld environment ( MiniGrid ) and a number of instruction-following tasks that we call levels , all formulated using subsets of a synthetic language ( Baby Language ) . The platform also includes a bot that can generate successful demonstrations for all BabyAI levels . All the code is available online at https : //github.com/mila-iqia/babyai/tree/iclr19 . 3.1 MINIGRID ENVIRONMENT . Studies of sample efficiency are very computationally expensive given that multiple runs are required for different amounts of data . Hence , in our design of the environment , we have aimed for a minimalistic and efficient environment which still poses a considerable challenge for current general-purpose agent learning methods . We have implemented MiniGrid , a partially observable 2D gridworld environment . The environment is populated with entities of different colors , such as the agent , balls , boxes , doors and keys ( see Figure 1 ) . Objects can be picked up , dropped and moved around by the agent . Doors can be unlocked with keys matching their color . At each step , the agent receives a 7x7 representation of its field of view ( the grid cells in front of it ) as well as a Baby Language instruction ( textual string ) . The MiniGrid environment is fast and lightweight . Throughput of over 3000 frames per second is possible on a modern multi-core laptop , which makes experimentation quicker and more accessible . The environment is open source , available online , and supports integration with OpenAI Gym . For more details , see Appendix B . 3.2 BABY LANGUAGE . We have developed a synthetic Baby Language to give instructions to the agent as well as to automatically verify their execution . Although Baby Language utterances are a comparatively small subset of English , they are combinatorially rich and unambigously understood by humans . The language is intentionally kept simple , but still exhibits interesting combinatorial properties , and contains 2.48× 1019 possible instructions . In this language , the agent can be instructed to go to objects , pick up objects , open doors , and put objects next to other objects . The language also expresses the conjunction of several such tasks , for example “ put a red ball next to the green box after you open the door '' . The Backus-Naur Form ( BNF ) grammar for the language is presented in Figure 2 and some example instructions drawn from this language are shown in Figure 3 . In order to keep the resulting instructions readable by humans , we have imposed some structural restrictions on this language : the and connector can only appear inside the then and after forms , and instructions can contain no more than one then or after word . The BabyAI platform includes a verifier which checks if an agent ’ s sequence of actions successfully achieves the goal of a given instruction within an environment . Descriptors in the language refer to one or to multiple objects . For instance , if an agent is instructed to `` go to a red door '' , it can successfully execute this instruction by going to any of the red doors in the environment . The then and after connectors can be used to sequence subgoals . The and form implies that both subgoals must be completed , without ordering constraints . Importantly , Baby Language instructions leave details about the execution implicit . An agent may have to find a key and unlock a door , or move obstacles out of the way to complete instructions , without this being stated explicitly .

Summary: This paper presents a research platform with a simulated human (a.k.a bot) in the loop for learning to execute language instructions in which language has compositional structures. The language introduced in this paper can be used to instruct an agent to go to objects, pick up objects, open doors, and put objects next to other objects. MiniGrid is used to build the environments used for this platform. In addition to introducing the platform, they evaluate the difficulty of each level by training an imitation learning baseline using one million demonstration episodes for each level and report results. Moreover, the reported results contain data efficiencies for imitation learning and reinforcement learning based approaches to solving BabyAI levels.