Safe Reinforcement Learning with Natural Language Constraints
Year: 2022

Abstract:
While safe reinforcement learning (RL) holds great promise for many practical applications like robotics or autonomous cars, current approaches require specifying constraints in mathematical form. Such specifications demand domain expertise, limiting the adoption of safe RL. In this paper, we propose learning to interpret natural language constraints for safe RL. To this end, we first introduce HAZARDWORLD, a new multi-task benchmark that requires an agent to optimize reward while not violating constraints specified in free-form text. We then develop an agent with a modular architecture that can interpret and adhere to such textual constraints while learning new tasks. Our model consists of (1) a constraint interpreter that encodes textual constraints into spatial and temporal representations of forbidden states, and (2) a policy network that uses these representations to produce a policy achieving minimal constraint violations during training. Across different domains in HAZARDWORLD, we show that our method achieves higher rewards (up to 11x) and fewer constraint violations (by 1.8x) compared to existing approaches. However, in terms of absolute performance, HAZARDWORLD still poses significant challenges for agents to learn efficiently, motivating the need for future work.1

heading: 1 Introduction
text: Although reinforcement learning (RL) has shown promise in several simulated domains such as games [1, 2, 3] and autonomous navigation [4, 5], deploying RL in real-world scenarios remains challenging [6]. In particular, real-world RL requires ensuring the safety of the agent and its surroundings, which means accounting for constraints during training that are orthogonal to maximizing rewards. For example, a cleaning robot must be careful to not knock the television over, even if the television lies on the optimal path to cleaning the house.
Safe RL tackles these challenges with algorithms that maximize rewards while simultaneously minimizing constraint violations during exploration [7, 8, 9, 10, 11, 12, 13, 14, 15, 16]. However, these algorithms have two key limitations that prevent their widespread use. First, they require us
∗ Equal contribution. 1Code and data are available at https://github.com/princeton-nlp/SRL-NLC
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
to provide constraints in mathematical or logical forms, which calls for specific domain expertise. Second, a policy trained with a specific set of constraints cannot be transferred easily to learn new tasks with the same set of constraints, since current approaches do not maintain an explicit notion of constraints separate from reward-maximizing policies. This means one would have to retrain the policy (with constraints) from scratch.
We consider the use of natural language to specify constraints (which are orthogonal to rewards) on learning. Human languages provide an intuitive and easily-accessible medium for describing constraints–not just for machine learning experts or system developers, but also for potential end users interacting with agents such as household robots. Consider the environment in Fig. 1 for example. Instead of expressing a constraint as ∑T t=0 1st∈lava · 16∃st′∈water, t′∈[0,1,...,t−1] = 0, one could simply say “Do not visit the lava before visiting the water”. The challenge of course, lies in training the RL agent to accurately interpret and adhere to the textual constraints as it learns a policy for the task.
To study this problem, we first create HAZARDWORLD, a collection of grid-world and robotics environments for safe RL with textual constraints (Fig. 1). HAZARDWORLD consists of separate ‘safety training’ and ‘safety evaluation’ sets, with disjoint sets of reward functions and textual constraints between training and evaluation. To do well on HAZARDWORLD, an agent has to learn to interpret textual con-
straints during safety training and safely adhere to any provided constraints while picking up new tasks during the safety evaluation phase. Built on existing RL software frameworks [17, 18], HAZARDWORLD consists of navigation and object collection tasks with diverse, crowdsourced, free-form text specifying three kinds of constraints: (1) budgetary constraints that limit the frequency of being in unsafe states, (2) relational constraints that specify unsafe states in relation to surrounding entities, and (3) sequential constraints that activate certain states to be unsafe based on past events (e.g., “Make sure you don’t walk on water after walking on grass”). Our setup differs from instruction following [19, 20, 21, 22, 23, 24] in two ways. First, instructions specify what to do, while textual constraints only inform the agent on what not to do, independent of maximizing rewards. Second, learning textual constraints is a means for ensuring safe exploration while adapting to a new reward function.
In order to demonstrate learning under this setting, we develop Policy Optimization with Language COnstraints (POLCO), where we disentangle the representation learning for textual constraints from policy learning. Our model first uses a constraint interpreter to encode language constraints into representations of forbidden states. Next, a policy network operates on these representations and state observations to produce actions. Factorizing the model in this manner allows the agent to retain its constraint comprehension capabilities while modifying its policy network to learn new tasks.
Experiments demonstrate that our approach achieves higher rewards (up to 11x) while maintaining lower constraint violations (up to 1.8x) compared to several baselines on two different domains within HAZARDWORLD. Nevertheless, HAZARDWORLD remains far from being solved, especially in tasks with high-dimensional observations, complex textual constraints and those requiring high-level planning or memory-based systems.

heading: 2 Related Work
text: Safe RL. Safe RL deals with learning constraint-satisfying policies [25], or learning to maximize rewards while minimizing constraint violations.2 This is a constrained optimization problem, and thus different from simply assigning negative reward values to unsafe states. Furthermore, large negative reward values for constraint violations can destabilize training and lead to degenerate behavior, such as the agent refusing to move. In prior work, the agent typically learns policies either by (1) exploring the environment to identify forbidden behaviors [7, 27, 8, 9, 28], or (2) using expert demonstration data to recognize safe trajectories [29, 30, 31, 10]. All these works require a human to specify the cost constraints in mathematical or logical form, and the learned constraints cannot be easily reused for new learning tasks. In this work, we design a modular architecture to learn to interpret textual constraints, and demonstrate transfer to new learning tasks.
Instruction following. Our work closely relates to the paradigm of instruction following in RL, which has previously been explored in several environments [19, 32, 20, 33, 21, 34, 35, 36, 37, 38, 39]. Prior work has also focused on creating realistic vision-language navigation datasets [40, 41, 4, 42] and proposed computational models to learn multi-modal representations that fuse images with goal instructions [43, 44, 45, 46, 47, 48, 49, 50, 51]. Our work differs from the traditional instruction following setup in two ways: (1) Instruction following seeks to (roughly) ‘translate’ an instruction directly into an action policy. This does not apply to our setting since the textual constraints only tell an agent what not to do. To actually obtain rewards, the agent has to explore and figure out optimal policies on its own. (2) Since constraints are decoupled from rewards and policies, agents trained to understand certain constraints can transfer their understanding to respect these constraints in new tasks, even when the new optimal policy is drastically different. Therefore, we view this work as orthogonal to traditional instruction following–one could of course combine both instructions and textual constraints to simultaneously advise an agent on what to do and what not to do.
Connection to Seldonian algorithms. POLCO can also be interpreted as a Seldonian algorithm [52]. Seldonian algorithms ensure ML safety through three steps: (1) defining a goal, (2) defining an interface for users to provide constraints, and (3) creating an algorithm that satisfies the goal and constraints. Here, we use natural language as the interface for end users and map natural language into optimization constants and vector representations. Thus, POLCO is also a potential step towards widely developing and deploying Seldonian algorithms.
Constraints in natural language. Our notion of ‘constraints’ in this paper differs from prior work that uses instructions to induce planning constraints [33, 53, 39]–these works again provide instructions for the agent on how to perform the task. Perhaps closest to this paper is the work of Misra et al. [5], which proposes datasets to study spatial and temporal reasoning, containing a subset focusing on trajectory constraints (e.g., “go past the house by the right side of the apple”). However, they do not disentangle the rewards from the constraints, which may be orthogonal to each other. Prakash et al. [54] train a constraint checker to identify whether a constraint (specified in text) has been violated in a trajectory. While their motivation is similar, they ultimately convert constraints to negative rewards, whereas we use a modular approach that allows disentangling reward maximization from minimizing constraint violations and is compatible with modern algorithms for safe RL.

heading: 3 Preliminaries
text: Problem formulation. Our learning problem can be viewed as a partially observable constrained Markov decision process [55], which is defined by the tuple < S,O,A, T, Z,X , R, C >. Here S is the set of states, O is the set of observations, A is the set of actions, T is the conditional probability T (s′|s, a) of the next state s′ given the current state s and the action a, and Z is the conditional probability Z(o|s) of the observation o given the state s. In addition, X is the set of textual constraint specifications, R : S ×A → R is the reward function, which encodes the immediate reward provided when the agent takes an action a in state s, and C : S ×A×X → R is the true underlying constraint function described by x ∈ X , which specifies positive penalties for constraint violations due to an action a in a state s. Finally, we assume each x ∈ X corresponds to a specific cost function C.
2In this paper, we consider minimizing constraint violations in expectation [9] and leave stricter notions such as enforcing zero violations [26] to future work.
RL with constraints. The goal of the learning agent is to acquire a good control policy that maximizes rewards, while adhering to the specified constraints as much as possible during the learning process. Thus, the agent learns a policy π : O ×X → P(A), which is a mapping from the observation space O and constraint specification X to the distributions over actions A. Let γ ∈ (0, 1) denote a discount factor, µ(S) denote the initial state distribution, and τ denote a trajectory sequence of observations and actions induced by a policy π, i.e., τ = (o0, a0, o1, · · · ). For any given x, we seek a policy π that maximizes the cumulative discounted reward JR while keeping the cumulative discounted cost JC below a specified cost constraint threshold hC(x):
max π JR(π) . = E τ∼π [ ∞∑ t=0 γtR(st, at) ] s.t. JC(π) . = E τ∼π [ ∞∑ t=0 γtC(st, at, x) ] ≤ hC(x),
where τ ∼ π is shorthand for indicating that the distribution over trajectories depends on π : s0 ∼ µ, ot ∼ Z(·|st), at ∼ π(·|ot, x), st+1 ∼ T (·|st, at). We use C(st, at, x) and hC(x) here to emphasize that both functions depend on the particular constraint specification x.
Task setup. Our goal is to show that constraints specified in natural language allow for generalization to new tasks that require similar constraints during learning. With this in mind, we consider the following safety training and safety evaluation setup:
(1) Safety training: During training, we generate random environment layouts and starting states s0 while keeping the reward function R fixed. For each episode, we randomly generate a constraint function C and limit hC . We then sample a constraint text x that describes C and hC from the training set of texts. The constraint text x is an input to the agent’s policy. Whenever the agent violates a constraint (at any step), it is provided with a scalar cost penalty learned by the model from C(s, a, x). The agent, therefore, sees a variety of different task layouts and constraints, and learns a policy with respect to the constraints for this task as well as how to interpret textual constraints.
(2) Safety evaluation: During evaluation, we place the agent in new environments with randomly generated layouts, with a different reward function R′. The set of possible constraints C is the same as seen in training, but the corresponding constraint texts are from an unseen test set. During this phase, the agent is not provided any cost penalties from the task. This setup allows us to measure two things: (1) how well an agent can learn new tasks while following previously learned textual constraints, and (2) the applicability of our method when using textual constraints unseen in training.

heading: 4 HAZARDWORLD
text: To our knowledge, there do not currently exist datasets for evaluating RL agents that obey textual constraints.3 Thus, we design a new benchmark called HAZARDWORLD in which the agent starts
3Even though there are several instruction following tasks, our task setup is different, as mentioned previously.
each episode at a random location within a procedurally generated environment and receives a textual constraint x, sampled from a pool of available constraints. The agent’s goal is to collect all the reward-providing entities while adhering to the specified constraint. Other than the constraint specified, the agent has complete freedom and is not told about how to reach reward-providing states.
HAZARDWORLD contains three types of constraints–(1) budgetary constraints, which impose a limit on the number of times a set of states can be visited, (2) relational constraints, which define a minimal distance that must be maintained between the agent and a set of entities, and (3) sequential constraints, which are constraints that activate unsafe states when a specific condition has been met. In total, we collect 984 textual constraints for HAZARDWORLD-grid (GridWorld environment) and 2,381 textual constraints for HAZARDWORLD-robot (robotic tasks). Table 1 provides examples.
HAZARDWORLD-grid. We implement HAZARDWORLD-grid (Table 1(a)) atop the 2D GridWorld layout of BabyAI [17, 56]. We randomly place three reward entities on the map: ‘ball,’ ‘box,’ and ‘key,’ with rewards of 1, 2, and 3, respectively. We also randomly place several cost entities on the map: ‘lava,’ ‘water,’ and ‘grass’. We give a cost penalty of 1 when agents step onto any cost entities, which are specified using a textual constraint x. The entire state st is a grid of size 13×13, including the walls, and the agent’s observation ot is a 7×7 grid of its local surroundings. There are 4 actions–for moving up, down, left and right. We use the deterministic transition here.
Train-test split. We generate two disjoint training and evaluation datasets Dtrain and Deval. Dtrain consists of 10,000 randomly generated maps paired with 80% of the textual constraints (787 constraints overall), i.e., on average each constraint is paired with 12.70 different maps. Deval consists of 5,000 randomly generated maps paired with the remaining 20% of the textual constraints (197 constraints), i.e., on average one constraint is paired with 25.38 maps. In Deval we change the rewards for ball, box, and key to 1, 2, and -3, respectively. Therefore, in Deval, the agent has to avoid collecting the key to maximize reward.
HAZARDWORLD-robot. We build HAZARDWORLD-robot (Table. 1(b)) atop the SAFETY GYM environment [18] to show the applicability of our model to tasks involving high-dimensional continuous observations. In this environment, there are five constraint entities paired with textual constraints: hazards (dark blue puddles), vases (stationary but movable teal cubes), pillars (immovable cylinders), buttons (touchable orange spheres), and gremlins (moving purple cubes). This task is more challenging than the 2D case since some obstacles are constantly moving. The agent receives a reward of 4 for reaching a goal position and a cost penalty of 1 for bumping into any constraint entities. The observation ot is a vector of size 109, including coordinate location, velocity of the agent, and observations from lidar rays that detect the distance to entities. The agent has two actions–control signals applied to the actuators to make it move forward or rotate. The transitions are all deterministic.
Train-test split. We follow the same process for obtaining a train-test split as in HAZARDWORLD-grid. Dtrain consists of 10,000 randomly generated maps paired with 80% of textual constrains (1,905 constraints), i.e., on average one constraint is paired with 5.25 maps. Deval consists of 1,000 randomly generated maps paired with the remaining 20% of textual constrains (476 constraints), i.e., on average one constraint is paired with 2.10 maps. In Deval we add four additional goal locations to each map (i.e., the maximum reward is 20). The agent has to learn to navigate to these new locations.
Data collection. For the textual constraints in both environments, we collected free-form text in English using Amazon Mechanical Turk (AMT) [57]. To generate a constraint for HAZARDWORLD, we provided workers with a description and picture of the environment, the cost entity to be avoided, and one of the following: (a) the cost budget (budgetary), (b) the minimum safe distance (relational), or (c) the other cost entity impacted by past events (sequential). We then cleaned the collected text by writing a keyword matching script followed by manual verification to ensure the constraints are valid.

heading: 5 Learning to Interpret Textual Constraints
text: We seek to train agents that can adhere to textual constraints even when learning policies for new tasks with different reward structures. We now describe our model and training and evaluation procedures.

heading: 5.1 Model
text: We design the RL agent as a deep neural network that consists of two parts (Fig. 2)–(1) a constraint interpreter which processes the text into structured safety criteria (a constraint mask and threshold)
and (2) a policy network which uses the output of the interpreter along with observations to produce an action. For simplicity, in the following descriptions, we assume state s and observation o to be 2D matrices, although the model can easily be extended to other input representations.
(1) Constraint interpreter (Fig. 3). We concatenate an observation embedding of size n× n×m from observations o of size n× n with the embedding of the textual constraints x of size l from a long-short-term-memory (LSTM), followed by using a convolutional neural network (CNN) to get an embedding vector. We use this vector to produce a constraint mask M̂C , a binary matrix with the same dimension as o–each cell of the matrix is 0/1 depending on whether the model believes the absence or presence of a constraint-related entity (e.g., ‘lava’) in the corresponding cell of the observation o. In addition, we feed the textual constraints into an LSTM to produce ĥC , a real-valued scalar which predicts the constraint threshold, i.e., the number of times an unsafe state is allowed.
For the case of sequential constraints with long-term dependency of the past events, M̂C will depend on the past states visited by the agent. For example, in Fig. 3(b), after the agent visits ‘water’, M̂C starts to locate the cost entity (‘grass’). Thus, for sequential constraints, we modify the interpreter by adding an LSTM layer before computing M̂C to take the state history into account. Using M̂C and ĥC allows us to embed textual constraints in the policy network.
(2) Policy network. The policy network produces an action using the state observation ot and the safety criteria produced by the constraint interpreter. The environment embedding is concatenated with the constraint mask M̂C (predicted by the constraint interpreter) and a cost budget mask, denoted by M̂B . The cost budget mask is derived from ĥC (also predicted by the constraint interpreter) and keeps track of the number of constraint violations that the agent has made in the past over the threshold. M̂B is an n × n matrix where each element takes the value of ∑t′ t=0 Ĉ(st, at;x) − ĥC (i.e., the value of constraint violations past the budget until t′th step) if there is a cost entity in ot(i, j), or zero otherwise. During the safety evaluation phase, we estimate the cumulative cost ∑t′ t=0 Ĉ(st, at;x) using the predicted M̂C and the agent’s current location at time t. After concatenating both the constraint mask M̂C and cost budget mask M̂B to the observation embedding, we then feed the resulting tensor into CNN to obtain a vector (grey in Fig. 2). This vector is concatenated with a vectorized int(ĥC) (i.e., ĥC rounded down) and fed into an MLP to produce an action.
POLCO in HAZARDWORLD-robot. To apply POLCO in this environment, the constraint interpreter predicts the cost entity given the textual constraints. We then map the cost entity to the pre-defined embedding vector (i.e., one-hot encoding). We then concatenate the embedding vector, the embeddings of the predicted ĥC , and the value of cost budget (rounded down) to the observation vector. Finally, the policy network takes in this concatenated observation and produces a safe action.
Advantages of the design. The design of POLCO tightly incorporates textual constraints into the policy network. Our model factorization–into (1) a constraint interpreter and (2) a policy network–
allows us to design specific constraint interpreters for different types of constraints.4 Furthermore, our approach scales gracefully to handling multiple constraints. While existing safe RL algorithms require retraining the policy for each unique combination of constraints, we can simply add together the M̂C of each constraint to handle multiple constraints imposed simultaneously.

heading: 5.2 Safety training
text: We first train the constraint interpreter using a random policy to collect trajectories, and then we use the trained interpreter to predict constraints while training the policy network.
Stage 1: Interpreter learning. We use a random policy to explore the environment, and obtain trajectories consisting of observations ot, along with the corresponding textual constraint x. Using the constraint violations encountered in the trajectory and the cost specification C, we obtain a target MC for training the constraint interpreter. In addition, we also obtain the ground-truth value of hC for learning the constraint threshold module.
We train the constraint mask module of the constraint interpreter by minimizing the following binary cross-entropy loss over these trajectories: L(Θ1) = −E(ot,x)∼Dtrain [ 1 |MC | ∑n i,j=1 y log ŷ + (1 −
y) log(1 − ŷ) ] , where y is the target MC(i, j; ot, x), which denotes the target (binary) mask label
in ith row and jth column of the n × n observation ot, ŷ is the predicted M̂C(i, j; ot, x), i.e., the probability prediction of constraint mask, and Θ1 are the parameters of the constraint mask module.
For the constraint threshold module, we minimize the following loss: L(Θ2) = E(ot,x)∼Dtrain [ (hC(x)− ĥC(x))2 ] , where Θ2 are the parameters of the constraint threshold module.
This approach ensures cost satisfaction during both policy learning and safety evaluation, an important feature of safe RL. If we train both the policy and the interpreter simultaneously, then we risk optimizing according to inaccurate M̂C and ĥC values, as observed in our experiments.
Stage 2: Policy learning. We use a safe RL algorithm called projection-based constrained policy optimization (PCPO) [9] to train the policy network. During training, the agent interacts with the environment to obtain rewards and penalty costs (M̂C) are provided from the trained constraint interpreter for computing JR(π) and JC(π) (ground-truth C is not used). PCPO is an iterative method that performs two key steps in each iteration5–optimize the policy according to reward and project the policy to a set of policies that satisfy the constraint. During safety evaluation, we evaluate our model in the new task with the new reward function and the textual constraints from Deval.
4M̂B equates to a scaled up version of M̂C since we assume only one constraint specification per episode, but this is not necessary in general since we may have multiple constraints over different cost entities. In that case, M̂B may have different cost budgets for different cells (entities).
5One can use other safe RL algorithms such as Constrained Policy Optimization (CPO) [7]

heading: 5.3 Safety evaluation
text: (1) Transfer to new tasks: We take the policy trained in Dtrain and fine-tune it on tasks having new reward functions with textual constraints from Deval. We do not retrain the constraint interpreter on Deval. The policy is fine-tuned to complete the new tasks without the penalty signals from the cost function C. In HAZARDWORLD-robot, we optimize the policy using CPO [7].
(2) Handling multiple textual constraints: We also test the ability of our model to handle multiple constraints imposed simultaneously (from Deval), by adding the cost constraint masks M̂C of each constraint together when given multiple constraints. During safety training, the policy is still trained with a single constraint. No fine-tuning is performed and the reward function is maintained the same across training and evaluation in this case.

heading: 6 Experiments
text: Our experiments aim to study the following questions: (1) Does the policy network, using representations from the constraint interpreter, achieve fewer constraint violations in new tasks with different reward functions? (2) How does each component in POLCO affect its performance?

heading: 6.1 Setup
text: Baselines. We consider the following baselines: (1) Constraint-Fusion (CF) with PCPO: This model [58] takes a concatenation of the observations and text representations as inputs (without MC , MB and hC ) and produces an action, trained with an end-to-end approach using PCPO. This model jointly processes the observations and the constraints. (2) CF with TRPO: We train CF using trust region policy optimization (TRPO) [59], which ignores all constraints and only optimizes the reward. This is to demonstrate that the agent will have substantial
constraint violations when ignoring constraints. (3) Random Walk (RW): We also include a random walk (RW) baseline, where the agent samples actions uniformly at random.
Evaluation metrics. To evaluate models, we use (1) the average value of the reward JR(π), and (2) the average constraint violations ∆C := max(0, JC(π)−hC). Good models should have a small ∆C (i.e., close to zero) while maximizing JR(π). More details on the implementation, hyper-parameters, and computational resources are included in the Appendix A,B, and C.

heading: 6.2 Results
text: HAZARDWORLD-grid. Fig. 4(1) shows results for all models in the first evaluation setting of transfer to new tasks. POLCO has lower constraint violations in excess of hC while still achieving better reward performance in all cases. In comparison, the high cost values (∆C ) obtained by RW and CF with TRPO indicate the challenges of task. This supports our idea of using the learned constraint interpreter to learn a new task with similar textual constraints while ensuring constraint satisfaction. CF with PCPO has higher constraint violations, and in the most cases, does not optimize the reward, which suggests that it cannot transfer the constraint understanding learned in Dtrain to Deval.
Fig. 4(2) shows our evaluation with multiple textual constraints. We see that POLCO achieves superior reward and cost performance compared to the baselines, while CF with PCPO has worse reward and cost performance. This shows that our approach is flexible enough to impose multiple constraints than that of existing safe RL methods which requires retraining the policy for each unique combination of constraints.
HAZARDWORLD-robot. Fig. 5 shows transfer to new tasks in HAZARDWORLD-robot. The JR(π) and ∆C of RW is relatively small since the agent does not move much because of random force applied to each actuator. For the budgetary constraints, although CF with TRPO achieves the best reward when hC = 0, it has very large constraint violations. POLCO performs better than the baselines–it induces policies with higher reward under fewer constraint violations in most cases. In contrast, CF with CPO has lower reward performance.
Having demonstrated the overall effectiveness of POLCO, our remaining experiments analyze (1) the learned models’ performance evaluated on the same reward function as in Dtrain, and (2) the importance of each component–MB ,MC and hC embedding in POLCO. For compactness, we restrict our consideration in HAZARDWORLD-grid.
Evaluation with reward function from Dtrain. To provide another point of comparison in addition to our main results, we evaluate all models using the same reward function as in Dtrain, but with unseen textual constraints from Deval. ( Fig. 6) We observe POLCO achieves the lowest violations across different choices of hC compared to the baselines. This implies that merely combining the observations and the text is not sufficient to learn an effective representation for parsing the constraints. In addition, POLCO achieves the best reward performance under cost satisfaction for the more complex relational and sequential constraints. For the relational case, although the CF agent trained with PCPO satisfies the constraints, it has a relatively low reward.
Ablation studies. We also examine the importance of each part in POLCO (Fig. 7). To eliminate prediction errors from the constraint interpreter, we use the true MC and hC here. Our full model achieves the best performance in all cases, averaging 5.12% more reward and 2.22% fewer constraint violations. Without MC , the agent cannot recognize cost entities effectively, which causes the agent to incur 66.67% higher ∆C compared with the full model (which has a ∆C close to zero). This shows that hC embedding and the MB mask are useful in enabling constraint satisfaction given textual constraints.

heading: 7 Conclusion
text: Our work provides a view towards machines that can interoperate with humans. As autonomous agents proliferate into our world, they should be able to understand safety constraints set by human agents around them. Accordingly, we proposed the problem of safe RL with natural language constraints, created a new benchmark called HAZARDWORLD to test agents and developed a new algorithm for the task (POLCO) that learns to interpret constraints. Our paper defines and trains machine agents that understand what not to do in natural language, much like instruction following tasks enable agents in understanding what to do.
The thesis of our POLCO approach is that modularity enables reuse. By bootstrapping a modular constraint interpreter through exploration, our model scales easily to multiple constraints and to shifts in the environment’s reward structure, all while exploring new environments safely. We applied POLCO within HAZARDWORLD to train an agent that navigates safely by obeying natural language constraints. This agent is a step towards creating applications like cleaning robots that can obey free form constraints, such as “don’t get too close to the TV” – a relational constraint in our formulation.
No model is without limitations. The absolute scores of POLCO on HAZARDWORLD still leave a lot of room for improvement using better models or training techniques. The current version of HAZARDWORLD is also not all-encompassing – we envision it as a benchmark that evolves over time, with the addition of new types of constraints and new environments. Future work can investigate training without explicit labels for the constraint interpreter, potentially using techniques like Gumbel softmax [60], or extending POLCO to tasks with more realistic visuals.

heading: Acknowledgments
text: The authors would like to thank members of the Princeton NLP Group, the anonymous reviewers, and the area chair for their comments and feedback. Tsung-Yen Yang thanks Siemens Corporation, Corporate Technology for their support.
Funding Transparency Statement
Funding in direct support of this work: scholarship by Siemens Corporation, Corporate Technology, and the Princeton CSML Summer Research Award.

Review summaries:
The paper is proposing an end-to-end reinforced approach of sequential task learning incorporating constraint using the natural language called POLCO standing for Policy Optimization with Language Constraints.   The authors propose a novel grid-world and robotic manipulation environment, called HazardWorld, to illustrate their proposal. Each world is composed of a variety of cells symbolizing a different level of safe reachability, in the case of robotic manipulation the state is discretized for applying the proposedapproach. Then, a set of three different constraint types expressed in natural language are studied: Budgetary, Relational, Sequential and correspond to different types and levels of compositionality.   Constraints are translated into reachability space through a CNN encoder and transfer into the policy and can be learnt in an end-to-end manner.  Reaching an unsafe place would produce a large negative reward. Proposing this setting, natural language is seamlessly incorporated into a non-reachable space of the considered environment.  The paper introduces a two-step process for safety policy training. First, a random policy explores the environment and grasp reward in accordance to language specified set of constraints. A constraint mask over the environment definition is computed using binary cross-entropy. Second, the policy is trained using PCPO which allow optimizing a policy by maximizing the reward function while satisfying the learnt constraint mask through re-projection in the satisfiability policy subspace.  The overall policy architecture is an original composition of state of the art differentiable building blocks suitable for differentiable training.

This paper presents a new multi-task environment for learning the agent with natural language constraints. The author breaks away from existing approaches that specify the constraints in mathematical form and extends it to specify the constraints in natural language. Their environment provides grid-world and robotics environments, and various kinds of constraints for safe RL. Finally, the authors propose a model for constraint policy optimization applicable to language constraint settings and show the generalization effect of language constraints.

This paper contributes to the area of constrained RL by adding machinery to interpret and apply constraints specified in natural language.  They also design and implement a testbed world to evaluate such systems.  In this world, they demonstrate that their approach consistently outperforms other methods, both in cumulative reward and constraint violations.  They also use an ablation study to demonstrate the need for each component of their approach.