Safe Reinforcement Learning with Natural Language Constraints

1 Introduction . Although reinforcement learning ( RL ) has shown promise in several simulated domains such as games [ 1 , 2 , 3 ] and autonomous navigation [ 4 , 5 ] , deploying RL in real-world scenarios remains challenging [ 6 ] . In particular , real-world RL requires ensuring the safety of the agent and its surroundings , which means accounting for constraints during training that are orthogonal to maximizing rewards . For example , a cleaning robot must be careful to not knock the television over , even if the television lies on the optimal path to cleaning the house . Safe RL tackles these challenges with algorithms that maximize rewards while simultaneously minimizing constraint violations during exploration [ 7 , 8 , 9 , 10 , 11 , 12 , 13 , 14 , 15 , 16 ] . However , these algorithms have two key limitations that prevent their widespread use . First , they require us ∗ Equal contribution . 1Code and data are available at https : //github.com/princeton-nlp/SRL-NLC 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . to provide constraints in mathematical or logical forms , which calls for specific domain expertise . Second , a policy trained with a specific set of constraints can not be transferred easily to learn new tasks with the same set of constraints , since current approaches do not maintain an explicit notion of constraints separate from reward-maximizing policies . This means one would have to retrain the policy ( with constraints ) from scratch . We consider the use of natural language to specify constraints ( which are orthogonal to rewards ) on learning . Human languages provide an intuitive and easily-accessible medium for describing constraints–not just for machine learning experts or system developers , but also for potential end users interacting with agents such as household robots . Consider the environment in Fig . 1 for example . Instead of expressing a constraint as ∑T t=0 1st∈lava · 16∃st′∈water , t′∈ [ 0,1 , ... , t−1 ] = 0 , one could simply say “ Do not visit the lava before visiting the water ” . The challenge of course , lies in training the RL agent to accurately interpret and adhere to the textual constraints as it learns a policy for the task . To study this problem , we first create HAZARDWORLD , a collection of grid-world and robotics environments for safe RL with textual constraints ( Fig . 1 ) . HAZARDWORLD consists of separate ‘ safety training ’ and ‘ safety evaluation ’ sets , with disjoint sets of reward functions and textual constraints between training and evaluation . To do well on HAZARDWORLD , an agent has to learn to interpret textual con- straints during safety training and safely adhere to any provided constraints while picking up new tasks during the safety evaluation phase . Built on existing RL software frameworks [ 17 , 18 ] , HAZARDWORLD consists of navigation and object collection tasks with diverse , crowdsourced , free-form text specifying three kinds of constraints : ( 1 ) budgetary constraints that limit the frequency of being in unsafe states , ( 2 ) relational constraints that specify unsafe states in relation to surrounding entities , and ( 3 ) sequential constraints that activate certain states to be unsafe based on past events ( e.g. , “ Make sure you don ’ t walk on water after walking on grass ” ) . Our setup differs from instruction following [ 19 , 20 , 21 , 22 , 23 , 24 ] in two ways . First , instructions specify what to do , while textual constraints only inform the agent on what not to do , independent of maximizing rewards . Second , learning textual constraints is a means for ensuring safe exploration while adapting to a new reward function . In order to demonstrate learning under this setting , we develop Policy Optimization with Language COnstraints ( POLCO ) , where we disentangle the representation learning for textual constraints from policy learning . Our model first uses a constraint interpreter to encode language constraints into representations of forbidden states . Next , a policy network operates on these representations and state observations to produce actions . Factorizing the model in this manner allows the agent to retain its constraint comprehension capabilities while modifying its policy network to learn new tasks . Experiments demonstrate that our approach achieves higher rewards ( up to 11x ) while maintaining lower constraint violations ( up to 1.8x ) compared to several baselines on two different domains within HAZARDWORLD . Nevertheless , HAZARDWORLD remains far from being solved , especially in tasks with high-dimensional observations , complex textual constraints and those requiring high-level planning or memory-based systems . 2 Related Work . Safe RL . Safe RL deals with learning constraint-satisfying policies [ 25 ] , or learning to maximize rewards while minimizing constraint violations.2 This is a constrained optimization problem , and thus different from simply assigning negative reward values to unsafe states . Furthermore , large negative reward values for constraint violations can destabilize training and lead to degenerate behavior , such as the agent refusing to move . In prior work , the agent typically learns policies either by ( 1 ) exploring the environment to identify forbidden behaviors [ 7 , 27 , 8 , 9 , 28 ] , or ( 2 ) using expert demonstration data to recognize safe trajectories [ 29 , 30 , 31 , 10 ] . All these works require a human to specify the cost constraints in mathematical or logical form , and the learned constraints can not be easily reused for new learning tasks . In this work , we design a modular architecture to learn to interpret textual constraints , and demonstrate transfer to new learning tasks . Instruction following . Our work closely relates to the paradigm of instruction following in RL , which has previously been explored in several environments [ 19 , 32 , 20 , 33 , 21 , 34 , 35 , 36 , 37 , 38 , 39 ] . Prior work has also focused on creating realistic vision-language navigation datasets [ 40 , 41 , 4 , 42 ] and proposed computational models to learn multi-modal representations that fuse images with goal instructions [ 43 , 44 , 45 , 46 , 47 , 48 , 49 , 50 , 51 ] . Our work differs from the traditional instruction following setup in two ways : ( 1 ) Instruction following seeks to ( roughly ) ‘ translate ’ an instruction directly into an action policy . This does not apply to our setting since the textual constraints only tell an agent what not to do . To actually obtain rewards , the agent has to explore and figure out optimal policies on its own . ( 2 ) Since constraints are decoupled from rewards and policies , agents trained to understand certain constraints can transfer their understanding to respect these constraints in new tasks , even when the new optimal policy is drastically different . Therefore , we view this work as orthogonal to traditional instruction following–one could of course combine both instructions and textual constraints to simultaneously advise an agent on what to do and what not to do . Connection to Seldonian algorithms . POLCO can also be interpreted as a Seldonian algorithm [ 52 ] . Seldonian algorithms ensure ML safety through three steps : ( 1 ) defining a goal , ( 2 ) defining an interface for users to provide constraints , and ( 3 ) creating an algorithm that satisfies the goal and constraints . Here , we use natural language as the interface for end users and map natural language into optimization constants and vector representations . Thus , POLCO is also a potential step towards widely developing and deploying Seldonian algorithms . Constraints in natural language . Our notion of ‘ constraints ’ in this paper differs from prior work that uses instructions to induce planning constraints [ 33 , 53 , 39 ] –these works again provide instructions for the agent on how to perform the task . Perhaps closest to this paper is the work of Misra et al . [ 5 ] , which proposes datasets to study spatial and temporal reasoning , containing a subset focusing on trajectory constraints ( e.g. , “ go past the house by the right side of the apple ” ) . However , they do not disentangle the rewards from the constraints , which may be orthogonal to each other . Prakash et al . [ 54 ] train a constraint checker to identify whether a constraint ( specified in text ) has been violated in a trajectory . While their motivation is similar , they ultimately convert constraints to negative rewards , whereas we use a modular approach that allows disentangling reward maximization from minimizing constraint violations and is compatible with modern algorithms for safe RL . 3 Preliminaries . Problem formulation . Our learning problem can be viewed as a partially observable constrained Markov decision process [ 55 ] , which is defined by the tuple < S , O , A , T , Z , X , R , C > . Here S is the set of states , O is the set of observations , A is the set of actions , T is the conditional probability T ( s′|s , a ) of the next state s′ given the current state s and the action a , and Z is the conditional probability Z ( o|s ) of the observation o given the state s. In addition , X is the set of textual constraint specifications , R : S ×A → R is the reward function , which encodes the immediate reward provided when the agent takes an action a in state s , and C : S ×A×X → R is the true underlying constraint function described by x ∈ X , which specifies positive penalties for constraint violations due to an action a in a state s. Finally , we assume each x ∈ X corresponds to a specific cost function C. 2In this paper , we consider minimizing constraint violations in expectation [ 9 ] and leave stricter notions such as enforcing zero violations [ 26 ] to future work . RL with constraints . The goal of the learning agent is to acquire a good control policy that maximizes rewards , while adhering to the specified constraints as much as possible during the learning process . Thus , the agent learns a policy π : O ×X → P ( A ) , which is a mapping from the observation space O and constraint specification X to the distributions over actions A . Let γ ∈ ( 0 , 1 ) denote a discount factor , µ ( S ) denote the initial state distribution , and τ denote a trajectory sequence of observations and actions induced by a policy π , i.e. , τ = ( o0 , a0 , o1 , · · · ) . For any given x , we seek a policy π that maximizes the cumulative discounted reward JR while keeping the cumulative discounted cost JC below a specified cost constraint threshold hC ( x ) : max π JR ( π ) . = E τ∼π [ ∞∑ t=0 γtR ( st , at ) ] s.t . JC ( π ) . = E τ∼π [ ∞∑ t=0 γtC ( st , at , x ) ] ≤ hC ( x ) , where τ ∼ π is shorthand for indicating that the distribution over trajectories depends on π : s0 ∼ µ , ot ∼ Z ( ·|st ) , at ∼ π ( ·|ot , x ) , st+1 ∼ T ( ·|st , at ) . We use C ( st , at , x ) and hC ( x ) here to emphasize that both functions depend on the particular constraint specification x . Task setup . Our goal is to show that constraints specified in natural language allow for generalization to new tasks that require similar constraints during learning . With this in mind , we consider the following safety training and safety evaluation setup : ( 1 ) Safety training : During training , we generate random environment layouts and starting states s0 while keeping the reward function R fixed . For each episode , we randomly generate a constraint function C and limit hC . We then sample a constraint text x that describes C and hC from the training set of texts . The constraint text x is an input to the agent ’ s policy . Whenever the agent violates a constraint ( at any step ) , it is provided with a scalar cost penalty learned by the model from C ( s , a , x ) . The agent , therefore , sees a variety of different task layouts and constraints , and learns a policy with respect to the constraints for this task as well as how to interpret textual constraints . ( 2 ) Safety evaluation : During evaluation , we place the agent in new environments with randomly generated layouts , with a different reward function R′ . The set of possible constraints C is the same as seen in training , but the corresponding constraint texts are from an unseen test set . During this phase , the agent is not provided any cost penalties from the task . This setup allows us to measure two things : ( 1 ) how well an agent can learn new tasks while following previously learned textual constraints , and ( 2 ) the applicability of our method when using textual constraints unseen in training .

Summary: The paper is proposing an end-to-end reinforced approach of sequential task learning incorporating constraint using the natural language called POLCO standing for Policy Optimization with Language Constraints.   The authors propose a novel grid-world and robotic manipulation environment, called HazardWorld, to illustrate their proposal. Each world is composed of a variety of cells symbolizing a different level of safe reachability, in the case of robotic manipulation the state is discretized for applying the proposedapproach. Then, a set of three different constraint types expressed in natural language are studied: Budgetary, Relational, Sequential and correspond to different types and levels of compositionality.   Constraints are translated into reachability space through a CNN encoder and transfer into the policy and can be learnt in an end-to-end manner.  Reaching an unsafe place would produce a large negative reward. Proposing this setting, natural language is seamlessly incorporated into a non-reachable space of the considered environment.  The paper introduces a two-step process for safety policy training. First, a random policy explores the environment and grasp reward in accordance to language specified set of constraints. A constraint mask over the environment definition is computed using binary cross-entropy. Second, the policy is trained using PCPO which allow optimizing a policy by maximizing the reward function while satisfying the learnt constraint mask through re-projection in the satisfiability policy subspace.  The overall policy architecture is an original composition of state of the art differentiable building blocks suitable for differentiable training.