Safe Deep RL in 3D Environments using Human Feedback

Agents should avoid unsafe behaviour during both training and deployment . This typically requires a simulator and a procedural specification of unsafe behaviour . Unfortunately , a simulator is not always available , and procedurally specifying constraints can be difficult or impossible for many real-world tasks . A recently introduced technique , ReQueST , aims to solve this problem by learning a neural simulator of the environment from safe human trajectories , then using the learned simulator to efficiently learn a reward model from human feedback . However , it is yet unknown whether this approach is feasible in complex 3D environments with feedback obtained from real humans - whether sufficient pixel-based neural simulator quality can be achieved , and whether the human data requirements are viable in terms of both quantity and quality . In this paper we answer this question in the affirmative , using ReQueST to train an agent to perform a 3D first-person object collection task using data entirely from human contractors . We show that the resulting agent exhibits an order of magnitude reduction in unsafe behaviour compared to standard reinforcement learning . 1 INTRODUCTION . Many of deep reinforcement learning ’ s recent successes have relied on the availability of a procedural reward function and a simulated environment for the task in question . As a result , research has been largely insulated from many of the difficulties of learning in the real world . One of these issues is safe exploration ( Garcıa & Fernández , 2015 ) . Online reinforcement learning is dependent on first-hand experience in order to learn the constraints of safe behaviour : the agent must drive the car off a cliff to learn not to drive the car off a cliff . While such actions may be fine in simulation , in the real world these actions may have unacceptable consequences , such as injury to humans . Further , such constraints are not always easy to describe using procedural functions . A recently proposed approach to safe exploration is reward query synthesis via trajectory optimization , ReQueST ( Reddy et al. , 2019 ) . In this approach , the agent is trained in a learned dynamics model ( a neural environment simulator ) with rewards from a learned reward model . Given models of sufficient fidelity , this should allow us to train an agent with close to zero instances of unsafe behaviour in the real environment . However , as of Reddy et al . ( 2019 ) , ReQueST has only been demonstrated to work in simple 2D environments – a simple navigation task , and a 2D car racing game – with a dynamics model learned from ( potentially unsafe ) random exploration , and a reward model learned from binary feedback generated by a procedural reward function . In this work , we aim to answer the question : is ReQueST feasible in complex 3D environments , with data used to train both dynamics and reward models sourced from real humans ? In particular , can we learn a pixel-based dynamics model of sufficient quality to enable informative human feedback , and are the data requirements viable , especially in terms of quantity ? Our key contributions in this work are as follows . • We demonstrate that ReQueST is feasible in a complex 3D environment , training a pixelbased dynamics model and reward model from 160 person-hours of safe human exploratory trajectories and 10 person-hours of reward sketches . We also show that performance degrades smoothly when models are trained on smaller amounts of data . • On a 3D first-person object collection task , we show that ReQueST enables training of a competent agent with 3 to 20 times fewer instances of unsafe behaviour during training ( close to zero instances if not counting mistakes by human contractors ) than a traditional RL algorithm . 2 RELATED WORK . Safe exploration Safe exploration has been studied extensively ( Garcıa & Fernández , 2015 ) . Most existing work achieves safety by making strong assumptions about the state space , such as all unsafe states being known in advance ( Geibel & Wysotzki , 2005 ; Luo & Ma , 2021 ) or the state space being reasonably smooth ( Berkenkamp et al. , 2017 ; Dalal et al. , 2018 ) . Other approaches require additional inputs , such as a procedural constraint function ( Altman , 1999 ; Achiam et al. , 2017 ; Ray et al. , 2019 ; Dalal et al. , 2018 ) , a safe baseline policy ( Garcia & Fernández , 2012 ) , or a separate system that can determine whether an action is safe ( Alshiekh et al. , 2018 ) . In contrast , the only assumption we make is that a human can recognise when a trajectory contains or is heading towards an unsafe state . Acceptability of unsafe behaviour Another important dimension is whether safety is treated as a soft or a hard constraint . Most work assumes the former , seeking to minimise time spent in unsafe states ( e.g . Geibel & Wysotzki ( 2005 ) ) . Two notable examples of the latter include Saunders et al . ( 2017 ) , which avoids unsafe behaviour by having a human intervene during training to block unsafe actions , and Luo & Ma ( 2021 ) , which starts with a trivial-but-safe policy and slowly broadens the policy while guaranteeing the policy will avoid a set of unsafe states specified by the user in advance . Learned dynamics models in prior work The broad structure of our approach – learning a dynamics model ( Chiappa et al. , 2017 ) from trajectories , then learning a policy or planning using that model – has been successfully used in simple control tasks ( Hafner et al. , 2019 ) , autonomous helicopter flight ( Abbeel et al. , 2010 ) , fabric manipulation ( Hoque et al. , 2021 ) , Atari games ( Buesing et al. , 2018 ) , and in simple 3D environments ( Ha & Schmidhuber , 2018 ) . Our work shows that this approach is viable even with complex 3D scenes , with a dynamics model learned from humandemonstrated trajectories , and with a reward model learned from human feedback rather than relying on environment rewards ( Hafner et al. , 2019 ; Buesing et al. , 2018 ) , trajectory following ( Abbeel et al. , 2010 ) , or maximisation of episode length ( Ha & Schmidhuber , 2018 ) . Prior work on ReQueST Our work expands on the original ReQueST ( Reddy et al. , 2019 ) in two main ways . First , we source all data from humans , showing that ReQueST is still applicable with imperfect , real-world data . Second , rather than simple 2D environments , we use visually-complex 3D environments , requiring a much more sophisticated dynamics model . Reward modeling As with the original ReQueST , we rely on a learned model of the reward function ( Knox , 2012 ; Leike et al. , 2018 ) . In contrast to the classification-based reward model used in the original , our reward model regresses to a continuous-valued reward , trained using reward sketches ( Cabi et al. , 2019 ) on imagined trajectories . Other forms of feedback on which such reward models can be trained include real-time scalar rewards ( Knox & Stone , 2009 ; Warnell et al. , 2018 ) , goal states ( Bahdanau et al. , 2018 ) , trajectory demonstrations ( Finn et al. , 2016 ) , trajectory comparisons ( Christiano et al. , 2017 ) , and combinations thereof ( Ibarz et al. , 2018 ; Stiennon et al. , 2020 ; Jeon et al. , 2020 ) . 3 METHODS . 3.1 REQUEST . Approach overview For online reinforcement learning algorithms , the only way an agent can learn about unsafe states is by visiting them . Intuitively , ReQueST avoids this problem by allowing agents to explore these states in a simulated model without having to visit them in the real environment . First , the agent learns a model of the environment by watching a human , who already knows how to navigate the environment safely . The agent then uses this model to ‘ imagine ’ various scenarios in the environment both safe and unsafe , asking the human for feedback on those scenarios . This process continues until the human is satisfied with the agent ’ s understanding of the task and its safety constraints , at which point the agent can be deployed in the real environment . If all goes to plan , this agent should avoid unsafe states in the real environment without having needed to visit those states in the first place . For further details , see Reddy et al . ( 2019 ) . ReQueST in this work ReQueST is flexible in i ) the type of feedback used to train the reward model , ii ) the proxies for value of information used to elicit informative feedback , and iii ) the algorithm used to train the agent in the learned simulation . In this work , for i ) we use reward sketching ( Cabi et al. , 2019 ) , one of the highest-bandwidth feedback mechanisms currently available . For ii ) , we use maximisation and minimisation of reward as predicted by the current reward model . Finally , for iii ) we use model predictive control – appealing for its simplicity , requiring no addi- tional training once the dynamics model and reward model are complete . See Fig . 2 for details , and Appendix A for pseudocode . 3.2 PROBLEM SETTING . Our 3D environment consists of an arena with two apples out in the open , and a third apple behind a gate that must be opened by stepping on a button ( shown in Fig . 3 ) . Agents receive 96×72×3 RGB pixel observations from a first-person perspective , and take actions in a two-dimensional continuous action space allowing movement forward and backward and turns left and right . Agent spawn position , positions of the two apples out in the open , and wall and floor colour are all randomised . Task The task is to eat the apples by moving close to them , ideally eating all 3 apples . The episode ends when either the agent eats the apple behind the gate or 900 steps have elapsed . To test ReQueST ’ s ability to avoid unsafe states , we use two variations on this setup . Cliff edge environment In the first environment variant , we remove the walls from the arena , so that it is possible for the agent to fall off the side . Such a fall is considered unsafe , and immediately ends the episode . Dangerous blocks environment The second environment variant models a more subtle case : where some part of the agent ’ s internal mechanisms is exposed in environment , and we would like to use safe exploration to discourage the agent from tampering ( Everitt et al. , 2021 ) with those mechanisms . We use a scenario based on the REALab framework ( Kumar et al. , 2020 ) , where instead of training rewards being communicated to the agent directly , rewards are communicated through a pair of blocks , where the reward given to the agent is based on the distance between these blocks . If the agent happens to bump into one of these blocks , changing the distance between them , this will affect the reward the agent receives , likely interfering with the agent ’ s ability to learn the task . Unsafe behaviour in this environment corresponds to making contact with the blocks . Sized subvariants We further use three subvariants , small , medium and large , of each of these two main variants . In each subvariant , the apples are the same distance from the agent spawn position , but the arena size is different , varying the difficulty . The larger the arena , the easier it is to avoid unsafe states : the edges are further away in the cliff edge variant , and the blocks are further away in the dangerous blocks variant . See Appendix B for details .

Summary:  ### Contributions * This paper proposes a safe model-based deep RL approach where     * No simulator is needed. The dynamic model is learned from data.     * No constraint is specified. * This work is an extension of reward query synthesis via trajectory optimization (ReQueST).  * ReQueSt (previous work)     * Algorithm         * Dynamics learned from (potentially unsafe) random exploration.         * Reward learned from binary human feedback that is generated by a procedural reward function.         * Demonstrated in State-based 2D navigation (non-pixel-based), Image-based Car Racing (pixel-based, 64x64×3).     * Regarding safety         * ReQueST avoids the problem of safe exploration by allowing agents to explore these states in a simulated model without having to visit them in the real environment. * This work     * Algorithm         * Dynamics learned from safe demonstration provided by humans (160 person-hours).           * Similar to ReQueSt, `except that we use a larger encoder network, a deconvolutional decoder network, and train using a simple mean-squared error loss between ground-truth pixel observations and predicted pixel observations.`         * Reward learned from             * Type of feedback from humans: reward sketch (previous work) (10 person-hours).             * The proxies for value of information: maximization and minimization of reward as predicted by the current reward model.             * The algorithm for reward learning: MPC.         * Demonstrated in 3D env           * Task goal = eat all the 3 apples           * Env 1 = Cliff edge environment             * Safety = not fall off the end of the world           * Env 2 = Dangerous blocks environment             * Safety = bumping into blocks           * Each env has 3 Sized subvariants.           * Each env is pixel-based, 96×72×3.     * Regarding safety         * Safety is achieved by focusing the learned dynamic model in the safe scenarios, with only the safe data. `Given models of sufficient fidelity, this should allow us to train an agent with close to zero instances of unsafe behaviour in the real environment.`  ### Results * Env 1 = Cliff edge environment (Fig5)     * Safety violations during training: proposed method << model-free RL     * Safety violations during testing: proposed method = model-free RL << random policy     * Apples eaten: random policy <= proposed method < model-free RL     * => Proposed method vs model-free RL => tradeoff between safety violations during training and apples eaten * Env 2 = Dangerous blocks environment (Fig6)     * Safety violations during training: Proposed method << model-free RL     * Safety violations during testing: Proposed method << model-free RL = random policy     * Apples eaten: random policy = model-free RL < proposed method     * => Proposed method is better than model-free RL and random