Safe Reinforcement Learning with Natural Language Constraints
Authors: Tsung-Yen Yang
Michael Hu
Yinlam Chow
Peter Ramadge
Karthik R Narasimhan
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: Safe reinforcement learning
Language grounding

Abstract: While safe reinforcement learning (RL) holds great promise for many practical applications like robotics or autonomous cars, current approaches require specifying constraints in mathematical form. Such specifications demand domain expertise, limiting the adoption of safe RL. In this paper, we propose learning to interpret natural language constraints for safe RL. To this end, we first introduce HAZARDWORLD, a new multi-task benchmark that requires an agent to optimize reward while not violating constraints specified in free-form text. We then develop an agent with a modular architecture that can interpret and adhere to such textual constraints while learning new tasks. Our model consists of (1) a constraint interpreter that encodes textual constraints into spatial and temporal representations of forbidden states, and (2) a policy network that uses these representations to produce a policy achieving minimal constraint violations during training. Across different domains in HAZARDWORLD, we show that our method achieves higher rewards (up to11x) and fewer constraint violations (by 1.8x) compared to existing approaches. However, in terms of absolute performance, HAZARDWORLD still poses significant challenges for agents to learn efficiently, motivating the need for future work.

Safe Reinforcement Learning with Natural Language Constraints
T'sun-Yeen Yang" Princeton University ty3@princeton.. edu
Michael Hu"
Princeton University
michael hw@yobi..bb... ventures
Yinlam Chow
Google Research
yin 1 Manllamhh@@@@oogle..co com
Peter J. Ramadge
Karthik Narasimhan
Princeton University
Princeton University
ramacce@princceton.ce
karthilk@@rrineeton..c edu
Abstract
While safe reinforcement learning (RL) holds great promise for many practical ap- plications like robotics or autonomous cars, current approaches require specifying constraints in mathematical form. Such specifications demand domain expertise, limiting the adoption of safe RL. In this paper, we propose learning interpret natural language constraints for safe RL. To this end, we first introduce HAZARD- WORLD, new multi-task benchmark that requires an agent to optimize reward while not violating constraints specified in free-form text. We then develop an agent with a modular architecture that can interpret and adhere such textual constraints while learning new tasks. Our model consists of (1) a constraini interpreter that encodes textual constraints into spatial and temporal representations of forbidden states, and (2) policy network that uses these representations to produce policy achieving minimal constraint violations during training. Across different domains in HAZARD WORID, we show that our method achieves higher rewards (up to 11x) and fewer constraint violations (by 1.8x) compared existing approaches. However, in terms of absolute performance, HAZARDWORLD still poses significant challenges for agents to learn efficiently, motivating the need for future work
Introduction
Although reinforcement learning (RL) has shown promise in several simulated domains such as games [ and autonomous navigation 45. deploying RL in real-world scenarios remains challenging In particular, real-world RL requires ensuring the safety of the agent and its sur- roundings, which means accounting for constraints during training that are orthogonal to maximizing rewards. For example, cleaning robot must be careful to not knock the television over, even if the television lies on the optimal path to cleaning the house.
Safe RL tackles these challenges with algorithms that maximize rewards while simultaneously minimizing constraint violations during exploration [7 9 10
11
12
13
15 [6] However,
these algorithms have two key limitations that prevent their widespread use. First, they require us
Equal contribution.
Code and data available wttp:p///gttta...com//rttttttttt
35th Conference on Neural Information Processing Systems (NeurIPS 2021).

---

to provide constraints in mathematical or logical forms, which calls for specific domain expertise. Second, policy trained with a specific set of constraints cannot be transferred easily learn new tasks with the same set of constraints, since current approaches do not maintain an explicit notion of constraints separate from reward maximizing policies. This means one would have to retrain the policy (with constraints) from scratch.
We consider the use of natural language to spec- ify constraints (which are orthogonal to rewards) on learning. Human languages provide an intu- itive and easily- easily-accessible medium for describ- ing constraints not just for machine learning ex- perts or system developers, but also for potential end users interacting with agents such as house- hold robots. Consider the environment in Fig. | for example. Instead of expressing a constraint
Budgetary Constraints: The lova really hot, so jrr will hurt you a lot. Only walk on it times
Relational Constraints: There shoold always be oot least squares between you water
Sequential Constraints: Make sure you don't walk after walking grass
Safety Training
Reward:
Constraint x
Constraint
Policy
Interpreter
Network
3
as Ei Elava Ewater, "E.o,,......1 0, one could simply say "Do not visit the lava be- fore visiting the water". The challenge of course, lies in training the RL agent to accurately inter- pret and adhere to the textual constraints as it learns a policy for the task.
Safety Evaluation in the New Task
Reward:
Constraint X
Constraint Policy
Constraint x Interpreter
Network
Learning to navigate with language constraints. The figure shows (1) third-person view of the environment (red dotted square box), (2) three types of language constraints, (3) items which provide rewards when collected. During safety training, the agent learns to interpret textual constraints while learning the task (i.e., collect re- wards). During safety evaluation, the agent learns a new task with different rewards while following the constraints and minimizing violations.
Figure 1:
To study this problem, we first create HAZ- ARDWORD ORLD, collection of grid-world and roboties environments for safe RL with textual constraints (Fig. [I. WORADDWORLD consists of separate 'safety training' and 'safety evalu- ation' sets, with disjoint sets of reward func- tions and textual constraints between training and evaluation. To do well on HAZARDWORLD, an agent has to learn to interpret textual con- straints during safety training and safely adhere to any provided constraints while picking up new tasks during the safety evaluation phase. Built on existing RL software frameworks [17 [8] HAZ- ARDWORLD consists of navigation and object collection tasks with diverse, crowdsourced, free-form text specifying three kinds of constraints: (1) budgetary constraints that limit the frequency of being in unsafe states, (2) relational constraints that specify unsafe states in relation to surrounding entities, and (3) sequential constraints that activate certain states to be unsafe based on past events (e.g., "Make sure you don't 't walk on water after walking on grass"). >. Our setup differs from instruction following [19 20 [1] 22 [3] 24 in two ways. First, instructions specify what to do, while textual constraints only inform the agent on what not to do, independent maximizing rewards. Second, learning textual constraints is means for ensuring safe exploration while adapting to a new reward function.
In order to demonstrate learning under this setting, we develop Policy Optimization with Language COnstraints (POLCO), where we disentangle the representation learning for textual constraints from policy learning. Our model first uses a constraint interpreter to encode language constraints into representations of forbidden states. Next, a policy network operates these representations and state observations to produce actions. Factorizing the model in this manner allows the agent to retain its constraint comprehension capabilities while modifying its policy network to learn new tasks.
Experiments demonstrate that our approach achieves higher rewards (up to 11x) while maintaining lower constraint violations (up to 1.8x) compared to several baselines on two different domains within HAZARDW WOLLD. Nevertheless, HAZARDWORLD remains far from being solved, especially in tasks with high-dimoon high-dimensional observations, complex textual constraints and those requiring high-level planning or memory-based systems.
2

---

2
Related Work
Safe RL. Safe RL deals with learning constraint-satiifyinn policies [25], or learning to maximize This rewards while minimizing constraint violations This is constrained optimization problem, and thus different from simply assigning negative reward values to unsafe states. Furthermore, large negative reward values for constraint violations destabilize training and lead degenerate behavior, such as the agent refusing to move. In prior work, the agent typically learns policies either by (1) exploring the environment identify forbidden behaviors 77 27
8 9 28], (2) using expert demonstration data to recognize safe trajectories (29 30 31 10 All these works require a human to specify the cost constraints in mathematical or logical form, and the learned constraints cannot be easily reused for new learning tasks. In this work, we design a modular architecture to learn to interpret textual constraints, and demonstrate transfer to new learning tasks.
Instruction following. Our work closely relates to the paradigm of instruction following in RL, R4
which has previously been explored in several environments [2] 32 20
[3 21
65,
36
37
38
39
Prior work has also focused creating realistic vision-language navigation datasets
40
42
and proposed computational models to learn multi-modal representations that fuse images with goal
instructions 43
45
46
47
48
49
50
51 Our work differs from the traditional instruction
following setup in two ways: (1) Instruction following seeks to (roughly) 'translate' an instruction directly into an action policy. This does not apply to our setting since the textual constraints only tell an agent what not to do. To actually obtain rewards, the agent has to explore and figure out optimal policies on its own. (2) Since constraints are decoupled from rewards and policies, agents trained to understand certain constraints can transfer their understanding to respect these constraints in new tasks, even when the new optimal policy is drastically different. Therefore, we view this work as orthogonal to traditional instruction following-on could of course combine both instructions and textual constraints to simultaneously advise an agent on what to do and what not to do.
Connection to Seldonian algorithms. POLCO can also be interpreted as a Seldonian algorithm [52]. Seldonian algorithms ensure ML safety through three steps: (1) defining a goal, (2) defining an interface for users to provide constraints, and (3) creating algorithm that satisfies the goal and constraints. Here, we use natural language as the interface for end users and map natural language into optimization constants and vector representations, Thus, POLCO is also potential step towards widely developing and deploying Seldonian algorithms.
Constraints in natural language. Our notion of 'constraints' in this paper differs from prior work that uses instructions to induce planning constraints [33 53 39 thess works again provide instructions for the agent on how to perform the task. Perhaps closest to this paper is the work of Misra et al. [5], which proposes datasets to study spatial and temporal reasoning, containing a subset focusing on trajectory constraints (e.g.. "go past the house by the right side of the apple"). However, they do not disentangle the rewards from the constraints, which may be orthogonal to each other. Prakash et al. [54) train a constraint checker to identify whether a constraint (specified in text) has been violated in trajectory. While their motivation is similar, they ultimately convert constraints to negative rewards, whereas we use a modular approach that allows disentangling reward maximization from minimizing constraint violations and is compatible with modern algorithms for safe RI..
3 Preliminaries
Problem formulation. Our learning problem can be viewed as a partially observable constrained Markov decision process (55), which is defined by the tuple S, O, A, T, 7, X, R, >.> Here S is the set of states, O is the set of observations, AA is the set of actions, T is the conditional probability T(s'ls, a) of the next state s' given the current state and the action a, and 7 is the conditional probability Z(ola s)) of the observation o given the state s. In addition, X' is the set of textual constraint specifications, R S x A- -> R is the reward function, which encodes the immediate reward provided when the agent takes an action a in state s, and C: : S x A x X - R is the true underlying constraint function described by X, which specifies positive penalties for constraint violations due to an action a in a state s. Finally, we assume each x E X' corresponds to a specific cost function C.
"In this paper, we consider minimizing constraint violations in expectation 9 and leave stricter notions such enforcing zero violations (26) to future work.
3

---

Constraint
Examples
Lava hurts a lot, but you have special shoes that you can use to walk on it, but only up times, remember!
Budgetary
Relational
Water will hurt you if you are two steps or less from them.
Sequential
Make sure you don't walk on water after walking on grass.
(a) HAZARDWORLD-grid
Constraint
Examples
Budgetary
Do not enter the blue square. It is safe to never cross at all.
Relational
Three feet is the minimum distance to all the dark blue circles. Once a purple box gets touched dark blue circles are disallowed.
Sequential
(b) HALLRDWoRLD-robot
Table 1: Examples of textual constraints for HAZARDWORLD
and HAZARDWORLD-OOD (a) An agent (red triangle) seeks to collect the reward entity (ball, box, key) while avoiding the cost entity (lava, water, grass). (b) An agent (red point) aims to reach goal position (green area) while avoiding the obstacles (vases, pillars, cubes, etc.). Please see the supplementary material for more details.
RL with constraints. The goal of the learning agent is to acquire a good control policy that maximizes rewards, while adhering to the specified constraints as much as possible during the learning process. Thus, the agent learns a policy π O X - P(A), which is a mapping from the observation space O and constraint specification X' to the distributions over actions A. Let E (0, 1) denote discount factor, yuS) denote the initial state distribution, and r denote trajectory sequence of observations and actions induced by policy π, i.e., т (og, ao, 01, ............ For any given H, we seek a policy æ that maximizes the cumulative discounted reward JR while keeping the cumulative discounted cost Jc below a specified cost constraint threshold hc()):
max Jn(T) = E Σ R'ss, az)
s.t. Jo(x)
-
E TAM
Σ C'((s,, a)) S ho(#))
where ~ π is shorthand for indicating that the distribution over trajectories depends on w 80 ~ 11 0: Z--(se), ~ (T-lo,, 8141 T(-|s,, at). We use C(st, a,, x) and he(a) here to emphasize that both functions depend on the particular constraint specification x.
Task setup. Our goal is show that constraints specified in natural language allow for generalization to new tasks that require similar constraints during learning. With this in mind, we consider the following safety training and safety evaluation setup:
(1) Safety training: During training, we generate random environment layouts and starting states so while keeping the reward function R fixed. For each episode, we randomly generate a constraint function C' and limit he. We then sample a constraint text x that describes C' and he from the training set of texts. The constraint text x is an input to the agent's 's policy. Whenever the agent violates constraint (at any step), it is provided with scalar cost penalty learned by the model from C(s,, a, .). The agent, therefore, sees a variety of different task layouts and constraints, and learns a policy with respect to the constraints for this task as well how interpret textual constraints.
During evaluation, we place the agent in new environments with randomly generated layouts, with different reward function R'. The set of possible constraints C' is the same as seen in training. but the corresponding constraint texts are from an unseen test set. During this phase, the agent is not provided any cost penalties from the task. This setup allows us to measure two things: (1) how well agent can learn new tasks while following previously learned textual constraints, and (2) the applicability of our method when using textual constraints unseen in training.
(2) Safety evaluation:
4 HAZARDWORLDD
To our knowledge, there do not currently exist datasets for evaluating RL agents that obey textual constraints Thus, we design a new benchmark called HAZARDWORLAS in which the agent starts
Even though there several instruction following tasks, task setup is different, mentioned previously.

---

each episode at a random location within a procedurally generated environment and receives textual constraint z, sampled from a pool of available constraints. The agent's 's goal is collect all the seward-providiie
entities while adhering to the specified constraint. Other than the constraint specified, the agent has complete freedom and is not told about how reach reward -worddprovidiin states.
HAZARDW ORLD contains three types of constraints (1) budgetary constraints, which impose a limit on the number of times a set of states can be visited, (2) relational constraints, which define minimal distance that must be maintained between the agent and a set of entities, and (3) sequential constraints, which are constraints that activate unsafe states when specific condition has been met. In total, we collect 984 textual constraints for HAZARD W WORLD-gri --rid (GridWorld environment) and 2,381 textual constraints for HAZARDW WAZARDWORLD--RE -RLoDrobot (robotic tasks). Table provides examples.
(Table [a)) atop the 2D GridWorld layout of BabyAI [17 [6). We randomly place three reward entities on the map: 'ball,' 'box,' and "key,' with rewards of 1, 2, and 3, respectively. We also randomly place several cost entities on the map: 'lava,' 'water,' and 'grass'. We give a cost penalty of when agents step onto any cost entities, which are specified using textual constraint x. The entire state St is a grid of size 13x 13, including the walls, and the agent's observation o, is a 7x7 grid of its local surroundings. There are 4 actions-for for moving up, down, left and right. We use the deterministic transition here.
HAZA SeZZARDWOLL-ggrin We implement HAZARDWORLDS.
Train-test split. We generate two disjoint training and evaluation datasets Dirain and Deval- Dirain consists of 10,000 randomly generated maps paired with 80% of the textual constraints (787 con- straints overall), i.e., average each constraint is paired with 12.70 different maps. Deval consists of 5,000 randomly generated maps paired with the remaining 20% of the textual constraints (197 constraints), i.e., average one constraint is paired with 25.38 maps. In D Deval we change the rewards for ball, box, and key to 1, 2, and -3, respectively. Therefore, in Deval, the agent has to avoid collecting the key to maximize reward.
HAZARDW ORLD-robot. We build HAZARDWORLD--oll (Table. [b)) atop the SAFETY GYM environment 18 to show the applicability of our model to tasks involving high-dimensional continu- ous observations. In this environment, there five constraint entities paired with textual constraints: hazards (dark blue puddles), vases (stationary but movable teal cubes), pillars (immovable cylinders), buttons (touchable orange spheres), and gremlins (moving purple cubes). This task is more chal- lenging than the 2D case since some obstacles are constantly moving. The agent receives a reward of 4 for reaching goal position and cost penalty of 1 for bumping into any constraint entities. The observation o, is a vector of size 109, including coordinate location, velocity of the agent, and observations from lidar rays that detect the distance to entities. The agent has two actions-control signals applied to the actuators to make move forward or rotate. The transitions are all deterministic.
Train-test split. We follow the same process for obtaining a train-test split as in HAZARDWORLDD-riu Derain consists of 10,000 randomly generated maps paired with 80% of textual constrains (1,905 constraints), i.e., on average one constraint is paired with 5.25 maps. Deval consists of 1,000 randomly generated maps paired with the remaining 20% of textual constrains (476 constraints), i.e., average one constraint is paired with 2.10 maps. In Deval we add four additional goal locations to each map (i.e., the maximum reward is 20). The agent has to learn to navigate to these new locations.
Data collection. For the textual constraints in both environments, we collected free-form text in English using Amazon Mechanical Turk (AMT) [57]. To generate a constraint for HAZARDWORLD we provided workers with a description and picture of the environment, the cost entity to be avoided, and one of the following: (a) the cost budget (budgetary), (b) the minimum safe distance (relational), or (c) the other cost entity impacted by past events (sequential). We then cleaned the collected text by writing a keyword matching script followed by manual verification to ensure the constraints are valid.
5 Learning to Interpret Textual Constraints
We seek to train agents that adhere to textual constraints even when learning policies for new tasks with different reward structures. We now describe our model and training and evaluation procedures.
5.1 Model
We design the RL agent as a deep neural network that consists of two parts (Fig. 2 (1) a constraint interpreter which processes the text into structured safety criteria (a constraint mask and threshold)
5

---

Constraint Mask Mc
Cost Budget Mask M C( a+) - hha)* (or(L Entity)
(oott Entity)
Env.
Policy Network
Env. Emb.
O
n
CNN
MLP
int(fc)
x Travel on grass less than twice
Constraint
Constraint
Mask Module
Threshold Module
Constraint Interpreter
Figure 2: Model overview. Our model consists of two parts: (1) the constraint interpreter produces constraint mask and cost constraint threshold prediction from a textual constraint and an observation, (2) policy network takes in these presentations and produces constraint- peustrint-sattifying policy.
and (2) a policy network which uses the output of the interpreter along with observations to produce an action. For simplicity, in the following descriptions, we assume state and observation o to be 2D matrices, although the model can easily be extended other input representations.
(1) Constraint interpreter (Fig. 3).
We concatenate an observation embedding of size n. X n X m from observations of size X with the embedding of the textual constraints of size from oug-short-term-memoor (LSTM), followed by using a convolutional neural network (CNN) to get an embedding vector. We use this vector to produce a constraint mask Mc. a binary matrix with the same dimension as o-each cell of the matrix is 0/1 depending the absence or presence of a constraint-relater entity (e.g., 'lava') )) in the corresponding cell of the observation o. addition, we feed the textual constraints into an LSTM to produce he. a valluvd scalar which predicts the constraint threshold, i.e., the number of times an unsafe state is allowed.
whether the model believes
For the case of sequential constraints with long-term dependency of the past events, Mc will depend on the past states visited by the agent. For example, in Fig. 3b), after the agent visits 'water', Mc starts to locate the cost entity ('grass"). Thus, for sequential constraints, we modify the interpreter by adding an LSTM layer before computing Mc to take the state history into account. Using Mc and be allows us to embed textual constraints in the policy network.
(2) Policy network. The policy network produces an action using the state observation and the safety criteria produced by the constraint interpreter. The environment embedding concatenated with the constraint mask Mc (predicted by the constraint interpreter) and a cost budget mask, denoted by M.. The cost budget mask is derived from he (also predicted by the constraint interpreter) and keeps track of the number of constraint violations that the agent has made in the past over the threshold.
Mn is an n X n matrix where each element takes the value Ei Ü(st de; x) hhe (i.e., the value of constraint violations past the budget until l'th step) if there is a cost entity in oz (z j), or zero otherwise. During the safety evaluation phase, we estimate the cumulative cost Eizo C( az;x)) using the predicted Mc and the agent's current location at time t. After concatenating both the constraint mask Mc and cost budget mask M to the observation embedding, we then feed the resulting tensor into CNN to obtain vector (grey in Fig. 2 This vector is concatenated with vectorized int (hc) (i.e., be rounded down) and fed into an MLP to produce an action.
POLCO in HAZARDWORLD- preter predicts the cost entity given the textual constraints. We then map the cost entity to the pre-defined embedding vector (i.e., one-hot encoding). We then concatenate the embedding vector, the embeddings of the predicted hc, and the value of cost budget (rounded down) to the observation vector. Finally, the policy network takes in this concatenated observation and produces a safe action.
To apply POLCO in this environment, the constraint inter-
Advantages of the design. The design of POLCO tightly incorporates textual constraints into the policy network. Our model factorization-into (1) constraint interpreter and (2) policy network-

---

Constraint Mask Module
M
LSTM Cell
CNN
CNN
visit water
LSTM
Constraint Module.
Travel an grass less. than ofter visiting water
Travel on grass less than twice
LSTM
LSTM
LSTM
Constraint
Threshold Module
Constralint Threshald Module (b) For sequential constraints
(a) For budgetary and relational constraints
Figure 3: Constraint interpreter. (a) For the budgetary and relational constraints, a constraint mask module takes the environment embedding and text vector representation as inputs and predicts Mc. (b) For the sequential constraints, we use an LSTM to store the information of the past visited states. For these three types of constraints, we use another LSTM given x to predict hc-
allows us to design specific constraint interpreters for different types of constraints Furthermore, our approach scales gracefully to handling multiple constraints. While existing safe RL algorithms require retraining the policy for each unique combination of constraints, we can simply add together the Mc of each constraint to handle multiple constraints imposed simultaneously.
5.2 Safety training
We first train the constraint interpreter using random policy collect trajectories, and then use the trained interpreter to predict constraints while training the policy network.
Stage 1: Interpreter learning. We use a random policy to explore the environment, and obtain trajectories consisting of observations ot, along with the corresponding textual constraint Using the constraint violations encountered in the trajectory and the cost specification C. we obtain a target Mc for training the constraint interpreter. In addition, also obtain the ground-truth value of ha for learning the constraint threshold module.
We train the constraint mask module of the constraint interpreter by minimizing the following binary cross-entropy loss over these trajectories: ((O,) -E Derain
Σ y/ log ÿ + (1
Mc
8) log(1 - 0)} where y is the target (c, 0,, r), which denotes the target (binary) mask label
in ith row and jth column of the observation ot, ý is the predicted Mc j; ot, x), i.e., the probability prediction of constraint mask, and θ1 are the parameters the constraint mask module.
For the constraint threshold module, we minimize the following loss: (z))² where O2 the parameters of the constraint threshold module.
((
((or.)))D...n [hc((z) (bc())
This approach ensures cost satisfaction during both policy learning and safety evaluation, an important feature of safe RL. If we train both the policy and the interpreter simultaneously, then we risk optimizing according to inaccurate Mc and he values, as observed in our experiments.
Stage 2: Policy learning. We use safe RL algorithm called projection- based constrained policy optimization (PCPO) [9] to train the policy network. During training, the agent interacts with the environment to obtain rewards and penalty costs (Mc) are provided from the trained constraint interpreter for computing Ja(T) and Jo(T) (ground-trutt -trth C' is not used). PCPO is an iterative method that performs two key steps in each iterations] iteratiorf-optiiiizee optimize the policy according to reward and project the policy set of policies that satisfy the constraint. During safety evaluation, we evaluate our model in the new task with the new reward function and the textual constraints from Deval-
4 MM equates to scaled up version of Mc since we assume only one constraint specification per episode, but this is not necessary in general since we may have multiple constraints over different cost entities. In that case, MB g may have different cost budgets for different cells (entities). 'One use other safe RL algorithms such Constrained Policy Optimization (CPO) 7
7

---

De and J (T)
6.6
0.5
Ac and (R)
Ac and (
0.2
010
0.4
0.00
0.5
8.5
12
10:2
9.5
12
13.6
12
14:39
16
hec
he
hec = 4
he
hc
(a) Budgetary
(b) Relational
(c) Sequential
and (x)
4,9
1.0
5.0
3,4
2 and (x)
1.2
4c and (
12
16
12
<
20
17.0
16.5
12
13.5
21
20.5
16
13.
16
(2)
hc
hg 2
hc 4
he = 0
hg 0
(d) Budgetary
(e) Relational
(f) Sequential
POLCO (ours)
w/ PCPO
CF w/ TRPO
RW Reward: (*)), Cost violations: Ae max (e)) - he)
Figure 4: Results HAZARDW ORLD -rid over different values of hc. These graphs represent the results of budgetary. relational, and sequential constraints, respectively. The blue bars are the reward performance (J (T)) and the red bars are the constraint violations (Ac)) For JR(T), higher values are better and for Ac. lower values are better. (1) Results for transfer to the new tasks. (2) Results for handling multiple textual constraints. POLCO generalizes unseen reward structure and handle multiple constraints with minimal constraint violations in the new task.
5.3 Safety evaluation
(1) Transfer to new tasks: We take the policy trained in Durain and fine-tun it on tasks having new reward functions with textual constraints from Deval- We do not retrain the constraint interpreter Deval. The policy is sue-ttned to complete the new tasks without the penalty signals from the cost function C. In HAZARDWORLD-TOOn
optimize the policy using CPO 771.
(2) Handling multiple textual constraints: We also the ability of our model handle multiple
constraints imposed simultaneously (from Deval), by adding the cost constraint masks Mc of each constraint together when given multiple constraints. During safety training, the policy is still trained with single constraint, No fine-tuning is performed and the reward function is maintained the same across training and evaluation in this case.
Experiments
Our experiments aim to study the following questions: (1) Does the policy network, using representa- tions from the constraint interpreter, achieve fewer constraint violations in new tasks with different reward functions? (2) How does each component in POLCO affect its performance?
6.1 Setup
Baselines. We consider the following baselines:
(1) Constraint-Fuuse (CF) with PCPO: This model [58) takes concatenation of the observations and text representations as inputs (without Mc, MB and lc)) and produces an action, trained with an end-to-end approach using PCPO. This model jointly processes the observations and the constraints. (2) CF with TRPO: We train CF using trust region policy optimization (TRPO) [59], which ignores all constraints and only optimizes the reward. This is to demonstrate that the agent will have substantial

---

24
Ae and (z)
17.0
Ac and (R(())
Ac aud (R)
0.0 0.0
onk O
0.0
0.2
26
3.6
2.4
010
00
00
03
2:5
16.3
20
hg
hg= = 10
he 20
he
hc 0
(a) Budgetary
(b) Relational
(c) Sequential
POLCO (ours)
CF CPO
CF TRPO
RW Reward: (")s), Cost violations max (x))
Figure 5: Results in HAZARDW ORLD-robot over different values of loc for transfer to the new tasks. POLCO achieves competitive results with higher rewards and lower cost violations.
constraint violations when ignoring constraints. (3) Random Walk (RW): We also include random walk (RW) bascline, where the agent samples actions uniformly at random.
Evaluation metrics. To evaluate models, we use (1) the average value of the reward (R((T), and (2) the average constraint violations Ac == JC(T) (i.e., close to zero) while maximizing JR (T). More details on the implementation, syper-parameters. and computational resources are included in the Appendix A.B, and C.
Good models should have small Ac
6.2 Results
HAZARDWORLD-gridd Fig. 4 () shows results for models in the first evaluation setting of transfer to new tasks. POLCO has lower constraint violations in excess of lic while still achieving better reward performance all cases. In comparison, the high cost values (Ac) obtained by RW and CF with TRPO indicate the challenges of task. This supports our idea of using the learned constraint interpreter to learn a new task with similar textual constraints while ensuring constraint satisfaction. CF with PCPO has higher constraint violations, and in the most cases, does not optimize the reward, which suggests that it cannot transfer the constraint understanding learned in Dursin to Deval-
Fig. 2 shows our evaluation with multiple textual constraints. We see that POLCO achieves superior reward and cost performance compared to the basclines, while CF with PCPO has worse reward and cost performance. This shows that our approach is flexible enough to impose multiple constraints than that of existing safe RL methods which requires retraining the policy for each unique combination of constraints.
Fig. 5 shows transfer to new tasks in HAZARDWORLD-rot and A of RW is relatively small since the agent does not move much because of random force applied to each actuator. For the budgetary constraints, although CF with TRPO achieves the best reward when he 0, it has very large constraint violations. POLCO performs better than the baselines-ie it induces policies with higher reward under fewer constraint violations in most cases. In contrast, CF with CPO has lower reward performance.
HAZARDWORLD
The "
Having demonstrated the overall effectiveness of POLCO, our remaining experiments analyze (1) the learned models' performance evaluated on the same reward function as in Derain- and (2) the importance of each component-MB. Mc and he embedding in POLCO. For compactness, we restrict our consideration in HAZARD WORLLARDWRDD-gniie
Evaluation with reward function from Dtrain. To provide another point of comparison in addition to our main results, we evaluate all models using the same reward function as D but with unseen textual constraints from D ( Fig. 66 We observe POLCO achieves the lowest violations across different choices of he compared
the baselines. This implies that merely combining the observations and the text is not sufficient to learn an effective representation for parsing the constraints. In addition, POLCO achieves the best reward performance under cost satisfaction for the more complex relational and sequential constraints. For the relational case, although the CF agent trained with PCPO satisfies the constraints, has relatively low reward.

---

3.0
1.1 OB
0.9
3U5
Ag and [
0.0
Ac and J
0.5
0.5
0.5
0.5
0.0
12
12
12
8.7
16
8
20
17.8
15.5
20
19.1
12
he =
he 2
hgg
he
hCc
(a) Budgetary
(b) Relational
(c) Sequential
POLCO (ours)
PCPO
CF TRPO
RW
Reward: ("(r), Cost violations: max(0, (s) he)
Figure 6: Results in WINZARDWORLD-OE for the setting of evaluation with the same reward function
as seen training. POLCO achieves higher reward and lower constraint violations over the baselines.
Ablation studies, We also examine the impor- tance of each part in POLCO (Fig. 7 To elim- inate prediction errors from the constraint inter- preter,
Ag and (5)
the true Mc and he here. Our
full model achieves the best performance all
0.0 0.9
0.0
cases, averaging 5. 5.12% more reward and 2.22% fewer constraint violations. Without Mc. the agent cannot recognize cost entities effectively, which causes the agent to incur 66.67% higher Ac compared with the full model (which has Ac close zero). This shows that le embed- ding and the MB mask are useful in enabling constraint satisfaction given textual constraints.
0.6
23
hg=
hg
hC
Full model
Mr
hp emb..
Figure 7: Ablations showing the effect of each com- ponent in POLCO for the budgetary constraint.
7
Conclusion
Our work provides a view towards machines that can interoperate with humans. As autonomous agents proliferate into world, they should be able to understand safety constraints set by human agents around them. Accordingly, we proposed the problem of safe RL with natural language constraints, created algorithm for the task (POLCO) that learns to interpret constraints. Our paper defines and trains machine agents that understand what not to do in natural language, much like instruction following tasks enable agents in understanding what to
benchmark called HAZARDWORLD to test agents and developed new
The thesis of our POLCO approach is that modularity enables reuse. By bootstrapping a modular constraint interpreter through exploration, our model scales easily multiple constraints and shifts in the environment's 's reward structure, all while exploring new environments safely. We applied POLCO within HAZARDW ORLD train agent that navigates safely by obeying natural language constraints. This agent is a step towards creating applications like cleaning robots that can obey free form constraints, such as "don't i get too close the TV relational constraint in our formulation.
No model without limitations. The absolute scores of POLCO on HAZARDWORDE still leave a lot of room for improvement using better models or training techniques. The current version of HAZARDWORLL
also all-encompassing- we envision with the addition of new types of constraints and new environments. Future work can investigate training without explicit labels for the constraint interpreter, potentially using techniques like Gumbel softmax (60] or extending POLCO to tasks with more realistic visuals.
benchmark that evolves over time,
Acknowledgments
The authors would like to thank members of the Princeton NLP Group, the anonymous reviewers, and the area chair for their comments and feedback. Tsung-Yer Yen Yang thanks Siemens Corporation, Corporate Technology for their support.
10

---

Funding Transparency Statement
Funding in direct support of this work: scholarship by Siemens Corporation, Corporate Technology, and the Princeton CSML Summer Research Award.
References
[1] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Pe- tersen, Charles Beattie, Amir Sadik, loannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 5 518(7540):529 533, 2015.
[2] David Silver, Richard S. Sutton, and Martin Müller. Reinforcement learning of local shape in the game of go. Proceedings of International Joint Conferences on Artificial Intelligence, volume 7, pages 1053-1058 2007.
[3] S.R.K. Branavan, David Silver, and Regina Barzilay. Learning to win by reading manuals in a Monte-Carlo framework. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 268-277, Portland, Oregon, USA, 2011. Association for Computational Linguistics.
[4] Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian D. Reid, Stephen Gould, and Anton den Hengel. Vision-and-language navigation: Interpreting visually-grounde navigation instructions real environments. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018. Salt Lake City, UT. USA, June 18-22, 2018, pages 3674-3683 IEEE Computer Society, 2018.
[5] Dipendra Misra, Andrew Bennett, Valts Blukis, Eyvind Niklasson, Max Shatkhin, and Yoav Artzi. Mapping instructions to actions in 3D environments with visual goal prediction. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2667-2678, Brussels, Belgium, 2018. Association for Computational Linguistics.
[6] Gabriel Dulac-Arnold, Daniel Mankowitz, and Todd Hester. Challenges real-world reinforce- ment learning. arXiv preprint arXiv: 1904: 2019.
[7] Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In Doina Precup and Yee whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pages 22-31. PMLR, 2017.
[8] Yinlam Chow, Ofir Nachum, Aleksandra Faust, Mohammad Ghavamzadeh, and Edgar Duenez- Guzman. Lyapunov-baut based safe policy optimization for continuous control. arXiv preprint 2019.
[9] Tsung- Yen Yang. Justinian Rosca, Karthik Narasimhan, and Peter J. Ramadge. Projection- constrained policy optimization. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
[10] Tsung- Yen Yang, Justinian Rosca, Karthik Narasimhan, and Peter J. Ramadge. Accelerating safe reinforcement learning with constraint-mismatched policies. arXiv preprint arXiv:2006.11645. 2020.
[11] Joshua Achiam and Dario Amodei. Benchmarking safe exploration in deep reinforcement learning. 2019.
[12] Yinlam Chow, Ofir Nachum, Edgar Duenez-Guzmant and Mohammad Ghavamzadeh. A Syapunov-basee approach to safe reinforcement learning. In Proceedings of the 32nd Interna- tional Conference on Neural Information Processing Systems, NIPS' 18, page 8103-81 12, Red Hook. NY, USA. 2018. Curran Associates Inc.

---

[13] Felix Berkenkamp, Matteo Turchetta, Angela Schoellig, and Andreas Krause. Safe model-based reinforcement learning with stability guarantees. In I. Guyon, U. V. Luxburg, S. Bengio. Wal- lach, R. Fergus, S. Vishwanathan, and R. Garnett, editors. Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[14] Mahmoud El Chamie, Yue Yu, and Behget Agıkmeşe. Convex symthesis of randomized policies for controlled markov chains with density safety upper bound constraints. In 2016 American Control Conference (ACC), pages 2290-6295, 2016.
[15] Matteo Turchetta, Andrey Kolobov. Shital Shah, Andreas Krause, and Alekh Agarwal. Safe reinforcement learning via curriculum induction. arXiv preprint arXi::2006. 12136, 2020.
[16] Brijen Thananjeyan, Ashwin Balakrishna, Suraj Nair, Michael Luo, Krishnan Srinivasan, Minho Hwang, Joseph E. Gonzalez, Julian Ibarz, Chelsea Finn, and Ken Goldberg. Recovery rl: Safe reinforcement learning with learned recovery zones. IEEE Robotics and Automation Letters, 6(3):4915-4922, 2021.
[17] Maxime Snevvaler-BBisveer Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Sa- haria, Thien Huu Nguyen, and Yoshua Bengio. Babyai: A platform to study the sample efficiency of grounded language learning. In 7th International Conference Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
[18] Alex Ray, Joshua Achiam, and Dario Amodei. Benchmarking Safe Exploration in Deep Reinforcement Learning. arXiv preprint 1rXiv:/910.0/70000 2019.
[19] Matt MacMahon, Brian Stankiewicz, and Benjamin Kuipers. Walk the talk: Connecting language, knowledge, and action in route instructions. Proceedings of the Conference on Artificial Intelligence, pages 1475-1482, 2006.
[20] David L. Chen and Raymond J. Mooney. Learning to interpret natural language navigation instructions from observations. In Wolfram Burgard and Dan Roth, editors, Proceedings of the Twenty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2011, San Francisco, California, USA, August 7-11, 2011. AAAI Press, 2011.
[21] Yoav Artzi and Luke Zettlemoyer. Weakly supervised learning of semantic parsers for mapping instructions to actions. Transactions of the Association for Computational Linguistics, 1:49-62. 2013.
[22] Dipendra Misra, John Langford, and Yoav Artzi Mapping instructions and visual observations to actions with reinforcement learning. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1015-1015, Copenhagen, Denmark, 2017.
Association for Computational Linguistics.
[23] Karl Moritz Hermann, Mateusz Malinowski, Piotr Mirowski, Andras -orvith Keith Anderson, and Raia Hadsell. Learning to follow directions in street view. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 11773-11781 2020.
[24] Weituo Hao, Chunyuan Li, Xiujun Li, Lawrence Carin, and Jianfeng Gao. Towards learning generic agent for vision-and lan-ann--lggggee navigation via pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13137-13146. 2020.
[25] Javier Garcia and Fernando Fernandez. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research, 16(1):1437-1480, 2015.
[26] Thiago D Simão, Nils Jansen, and Matthijs TJ Spaan. Alwayssafe: Reinforcement learning without safety constraint violations during training. In Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems, pages 1226-1235, 2021.
[27] Chen Tessler, Daniel J. Mankowitz, and Shie Mannor. Reward constrained policy optimization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
12

---

[28] Adam Stooke, Joshua Achiam, and Pieter Abbeel. Responsive safety in reinforcement learning by PID lagrangian methods. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020. 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 9133-9143, PMLR, 2020.
[29] Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of International Conference on Artificial Intelligence and Statistics, pages 627-635, 2011.
[30] Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. In Proceedings of Robotics: Science and Systems. 2018.
[31] Yang Gao, Ji Lin, Fisher Yu, Sergey Levine, and Trevor Darrell. Reinforcement learning from imperfect demonstrations. Proceedings of the 35th International Conference on Machine Learning, 2018
[32] Adam Vogel and Daniel Jurafsky. Learning to follow navigational directions. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 806-81 14, Uppsala, Sweden, 2010. Association for Computational Linguistics.
[33] Stefanie Tellex, Thomas Kollar, Steven Dickerson, Matthew Walter, Ashis Banerjee, Seth Teller, and Nicholas Roy. Understanding natural language commands for robotic navigation and mobile manipulation. In Proceedings of the AAAI Conference on Artificial Intelligecce, volume 25, 2011.
[34] Joohyun Kim and Raymond Mooney. Adapting discriminative reranking to grounded language learning. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 218 227, Sofia, Bulgaria, 2013. Association for Computational Linguistics.
[35] Jacob Andreas and Dan Klein. Alignment-based compositional semantics for instruction following. In Proceedings of the 2015 Conference Empirical Methods in Natural Lan- guage Processing, pages 1165-1174, Lisbon, Portugal, 2015. Association for Computational Linguistics.
[36] Jesse Thomason, Michael Murray, Maya Cakmak, and Luke Zettlemoyer. Vision-and-dialun navigation. In Proceedings of Conference on Robot Learning, pages 394-406, 2020.
[37] Jelena Luketina, Nantas Nardelli, Gregory Farquhar, Jakob N. Foerster, Jacob Andreas, Edward Grefenstette, Shimon Whiteson, and Tim Rocktäschel. A survey of reinforcement learning informed by natural language. In Sarit Kraus, editor, Proceedings of the Twenty-Eighth Interna- tional Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019, pages 3309-6317. 17. ijcai.org, 2019.
[38] Stefanie Tellex, Nakul Gopalan, Hadas Kress-Gazee and Cynthia Matuszek. Robots that use language. Annual Review of Control, Robotics, and Autonomous Systems, 3:25-55, 2020.
[39] Christopher Wang, Candace Ross, Yen-Ling Kuo, Boris Katz, and Andrei Barbu. Learning a language Itl executable semantic parser for grounded robotics. arXiv preprint NXiv:2008.032777 2020.
[40] Yonatan Bisk, Kevin J. Shih, Yejin Choi, and Daniel Marcu. Learning interpretable spatial operations in a rich 3d blocks world. In Sheila A. Mcllraith and Kilian Q. Weinberger, editors, P Proceedings of the Thirlo-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pages 5028-5033 AAAI Press, 2018
[41] Howard Chen, Alane Suhr, Dipendra Misra, Noah Snavely, and Yoav Artzi TOUCHDOWN: natural language navigation and spatial reasoning in visual street environments. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 1 Computer Vision Foundation/ / IEEE, 2019.
13

---

[42] Harm de Vries, Kurt Shuster, Dhruv Batra, Devi Parikh, Jason Weston, and Douwe Kiela. Talk the walk: Navigating new york city through grounded dialogue. arXiv preprint 1rXiv:1807.033000 2018.
[43] Michael Janner, Karthik Narasimhan, and Regina Barzilay, Representation learning for grounded spatial reasoning. Transactions of the Association for Computational Linguistics, 6:49-61, 2018.
[44] Valts Blukis, Dipendra Misra, Ross A. Knepper, and Yoav Artzi. Mapping navigation instruc- tions to continuous control actions with cosition-visitatii prediction. In Proceedings of the Conference on Robot Learning, pages 505-518, 2018.
[45] Daniel Fried, Ronghang Hu, Volkan Cirik, Anna Rohrbach, Jacob Andreas, Louis-Philippe Morency, Taylor Berg-Kirkpatrick Kate Saenko, Dan Klein, and Trevor Darrell. Speaker- follower models for iision-and-language navigation. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa- Aasa-Bianchi, and Roman Garnett, editors, Advances in Neural Information Processing Systems 31: Annual Conference Neural Information Processing Systems 2018, NeurlPS 2018, December 3-8, 2018, Montréal, Canada, pages 3318 11118--329,22 2018.
[46] Ninghao Liu, Mengnan Du, and Xia Hu, Representation interpretation with spatial encoding and multimodal analytics. In J. Shane Culpepper, Alistair Moffat, Paul N. Bennett, and Kristina Lerman, editors, Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining, WSDM 2019, Melbourne, VIC, Australia, February 11-15, 2019, pages 60-68. ACM, 2019.
[47] Vihan Jain, Gabriel Magalhaes, Alexander Ku, Ashish Vaswani, Eugene le, and Jason Baldridge. Stay on the path: Instruction fidelity in vision-and-land and-language navigation. In Proceedings of the 57th Anmual Meeting of the Association for Computational Linguistics, pages 1872, Florence, Italy, 2019. Association for Computational Linguistics.
[48] David Gaddy and Dan Klein. Pre-learning environment representations for data-efficient neural instruction following. In Proceedings of the 57th Annual Meeting of the Association for Compu- tational Linguistics, pages 1946-195 1956, Florence, Italy, 2019. Association for Computational Linguisties.
[49] Yordan Hristov, Daniel Angelov, Michael Burke, Alex Lascarides, and Subramanian Ramamoor- thy. Disentangled relational representations for explaining and learning from demonstration. In Proceedings of the Conference on Robot Learning, 2019.
[50] Justin Fu, Anoop Korattikara, Sergey Levine, and Sergio Guadarrama. From language to goals: Inverse reinforcement learning for vision-based instruction following. In 7th International Conference Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
[51] Sagar Gubbi Venkatesh, Anirban Biswas, Raviteja Upadrashta, Vikram Srinivasan, Partha Talukdar, and Bharadwaj Amrutur. Spatial reasoning from natural language instructions for robot manipulation. arXiv preprint 1rXiv:20/2.13669 2020.
[52] Philip S. Thomas, Bruno Castro da Silva, Andrew G. Barto, Stephen Giguere, Yuriy Brun, and Emma Brunskill. Preventing undesirable behavior of intelligent machines. Science, 366(6468):999-1000) 2019.
[53] Thomas M Howard, Stefanie Tellex, and Nicholas Roy. A natural language planner interface for mobile manipulators. In 2014 IEEE International Conference on Robotics and Automation (ICRA), pages 6652-6659, IEEE, 2014.
[54] Bharat Prakash, Nicholas Waytowich, Ashwinkumar Ganesan, Tim Oates, and Tinoosh Mohs- enin. Guiding safe reinforcement learning policies using structured language constraints. UMBC Student Collection, 2020.
[55] Eitan Altman. Constrained Markov decision processes, volume 7. CRC Press, 1999.
14

---

[56] Maxime Chevalier-Boisee Boisvert, Lucas Willems, and Suman Pal. Minimalistic gridworld environ- ment for openai gym. /ttps://github. com. max me wttpcc//gggggggnnneeeeee 2018.
[57] Michael Buhrmester, Tracy Kwang, and Samuel Gosling. Amazon's mechanical turk: new source of inexpensive, yet high-quality data? 2016.
(58) Aaron Walsman, Yonatan Bisk, Saadia Gabriel, Dipendra Misra, Yoav Artzi, Yejin Choi, and Dieter Fox. Early fusion for goal directed robotic vision. In Proceedings of IEEE/RSI International Conference on Intelligent Robots and Systems, pages 1025-1031. IEEE, 2019.
[59] John Schulman, Sergey Levine, Pieter Abbeel, Michael I. Jordan, and Philipp Moritz. Trust region policy optimization. In Francis R. Bach and David M. Blei, editors, Proceedings of the 32nd International Conference on Machine Learning. ICML 2015, Lille, France, 6-11 July 2015, volume 37 of JMLR Workshop and Conference Proceedings, pages 1889-1897. JMLR.org. 2015.
[60] Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmn volume abs/1611.00 1.4,144, 2017.
[61] John Schulman, Philipp Moritz, Sergey Levine, Michael I. Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. In Yoshua Bengio and Yann LeCun, editors, 4th International Conference on Learning Representations, ICLR 2016, San Juan Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016.
[62] Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep reinforcement learning for continuous control. In Maria-Florina Balcan and Kilian Q Weinberger, editors, Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016, volume 48 of JMLR Workshop and Conference Proceedings, pages 1329-1338. JMLR. 2016.
15


Reviews:
[{"review_id": "wAfyZkFLepe", "reviewer": "Reviewer_8Sns", "summary": "This paper contributes to the area of constrained RL by adding machinery to interpret and apply constraints specified in natural language.  They also design and implement a testbed world to evaluate such systems.  In this world, they demonstrate that their approach consistently outperforms other methods, both in cumulative reward and constraint violations.  They also use an ablation study to demonstrate the need for each component of their approach. \n", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is a clear contribution to the field and merits publication.\n\nOne thing that I would like to see, however, is a comparison with other constrained RL approaches that are given the constraints in their native formats, e.g., logic rules, finite automata, etc.  It's reasonable that this paper's approach might not always win in these tests, but they still maintain the advantage of NL-formatted constraints.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors briefly describe limitations of their work on page 9. \n\nThey state in the Checklist that they do not see potential negative impacts, but do not address it in the main body of the manuscript.\n", "review_text": "", "extra_scores": {"ethical_concerns": "I don't see any ethical concerns in this work.\n", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}, {"review_id": "jnPri4AyHY6", "reviewer": "Reviewer_YDss", "summary": "This paper presents a new multi-task environment for learning the agent with natural language constraints. The author breaks away from existing approaches that specify the constraints in mathematical form and extends it to specify the constraints in natural language. Their environment provides grid-world and robotics environments, and various kinds of constraints for safe RL. Finally, the authors propose a model for constraint policy optimization applicable to language constraint settings and show the generalization effect of language constraints.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is well written and addresses an important problem of language constraints for safe RL. I agree that language constraints are intuitive and easy-accessible, and I think specifying the constraints in natural language can provide various advantages when applying the constraint to a real-world problem. The detailed comments and questions are as follows:\n\n1. In the result of experiments (Figure 4, 5, 6, 7) what is the maximum performance that can be achieved while satisfying the cost constraint? It would be helpful to understand how well the proposed algorithm performs. \n\n2. The authors define three types of constraint and are modeling two types to consider different types of constraint. Is there any reason to model this way? (Is it not possible with a single model to consider all constraints as the same?)\n\n3. In safety training, the authors mention that PCPO was used for policy learning (page 7 lines 260-261), and in safety evaluation, CPO was used for HazardWorld-robot (page 7 line 271). Is there any reason to use different methods?\n\n4. (Page 9, lines 326-337) In the ablation studies, it is necessary to add a detailed explanation, such as exactly which domain was tested.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, the authors have adequately addressed it.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10"}, "extra_text": {}}, {"review_id": "X1rbRy9G1E", "reviewer": "Reviewer_v8rs", "summary": "The paper is proposing an end-to-end reinforced approach of sequential task learning incorporating constraint using the natural language called POLCO standing for Policy Optimization with Language Constraints. \n\nThe authors propose a novel grid-world and robotic manipulation environment, called HazardWorld, to illustrate their proposal. Each world is composed of a variety of cells symbolizing a different level of safe reachability, in the case of robotic manipulation the state is discretized for applying the proposedapproach. Then, a set of three different constraint types expressed in natural language are studied: Budgetary, Relational, Sequential and correspond to different types and levels of compositionality. \n\nConstraints are translated into reachability space through a CNN encoder and transfer into the policy and can be learnt in an end-to-end manner.  Reaching an unsafe place would produce a large negative reward. Proposing this setting, natural language is seamlessly incorporated into a non-reachable space of the considered environment.\n\nThe paper introduces a two-step process for safety policy training. First, a random policy explores the environment and grasp reward in accordance to language specified set of constraints. A constraint mask over the environment definition is computed using binary cross-entropy. Second, the policy is trained using PCPO which allow optimizing a policy by maximizing the reward function while satisfying the learnt constraint mask through re-projection in the satisfiability policy subspace.\n\nThe overall policy architecture is an original composition of state of the art differentiable building blocks suitable for differentiable training.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper addresses an important problem of sequential decision learning which is safety at training and test time and its seamless specification.\nThe proposed method proposes to use an interesting modality which is the natural language for this matter.\nThe model is trained in a differentiable manner and is tested on a large set of possible constraints expressing various levels of compositional in language.\nThe two-step learning process is novel and intuitive and the policy architecture is leveraging the know components of differentiable programming.\nThe experiments show convincing results of the capability of the method to handle the various types of constraint, however at the cost of an initial random exploration, which may be considered as contradictory to the overall purpose of the work.\nThe main concern is about the generalization of such an approach including stochastic and/or partially observable environments.\nAt the current stage, the paper hardly allows appreciating the possible impact of the approach, especially the constraint mask module, on such a more realistic scenario.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No section is explicitly addressing these concerns. However, as the question of the safety of sequential decision algorithms is the subject of this paper, it seems largely related to this subject of concern.", "review_text": "", "extra_scores": {"ethical_concerns": "No Ethical issue to mention.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a framework wherein constraints (particularly safety constraints) on a reinforcement learning agent can be specified using natural language. The paper is well written, clear, and was well-received by the reviewers, all of whom recommend acceptance. The AC finds the idea of natural language constraints particularly compelling, and strongly supports acceptance.

One additional related work is "Preventing undesirable of intelligent machines", which proposes an "interface" that the users of ML/RL algorithms can use to define safety constraints. It seems like the method proposed here would plug-in directly as such an "interface", making this paper a valuable contribution to Seldonian methods (perhaps beyond RL) as well.
Decision: Accept (Spotlight)