Safe Reinforcement Learning with Natural Language Constraints
Authors: Tsung-Yen Yang
Michael Hu
Yinlam Chow
Peter Ramadge
Karthik R Narasimhan
Venue:  (2021)
Keywords: Safe reinforcement learning
Language grounding

Abstract: In this paper, we tackle the problem of learning control policies for tasks when provided with constraints in natural language.  In contrast to instruction following, language here is used not to specify goals, but rather to describe situations that an agent must avoid during its exploration of the environment. Specifying constraints in natural language also differs from the predominant paradigm in safe reinforcement learning, where safety criteria are enforced by hand-defined cost functions. While natural language allows for easy and flexible specification of safety constraints and budget limitations, its ambiguous nature presents a challenge when mapping these specifications into representations that can be used by techniques for safe reinforcement learning. To address this, we develop a model that contains two components: (1) a constraint interpreter to encode natural language constraints into vector representations capturing spatial and temporal information on forbidden states, and (2) a policy network that uses these representations to output a policy with minimal constraint violations. Our model is end-to-end differentiable and we train it using a recently proposed algorithm for constrained policy optimization. To empirically demonstrate the effectiveness of our approach, we create a new benchmark task for autonomous navigation with crowd-sourced free-form text specifying three different types of constraints. Our method outperforms several baselines by achieving 6-7 times higher returns and 76% fewer constraint violations on average. Dataset and code to reproduce our experiments are available at https://sites.google.com/view/polco-hazard-world/.

Under review as a conference paper at ICLR 2021
SAFE REINFORCEMENT LEARNING WITH NATURAL LANGUAGE CONSTRAINTS
Anonymous authors Paper under double-blind review
ABSTRACT
In this paper, we tackle the problem of learning control policies for tasks when provided with constraints natural language. In contrast to instruction following. language here is used not to specify goals, but rather to describe situations that an agent must avoid during its exploration of the environment. Specifying constraints in natural language also differs from the predominant paradigm in safe reinforce- ment learning, where safety criteria are enforced by hand-defined cost functions. While natural language allows for easy and flexible specification of safety con- straints and budget limitations, its ambiguous nature presents a challenge when mapping these specifications into representations that can be used by techniques for safe reinforcement learning. To address this, we develop a model that contains components: (1) constraint interpreter to encode natural language con- straints into vector representations capturing spatial and temporal information on forbidden states, and (2) policy network that uses these representations output a policy with minimal constraint violations. Our model is end-to-end differen- tiable and we train using a recently proposed algorithm for constrained policy optimization. To empirically demonstrate the effectiveness of our approach, we create a new benchmark task for autonomous navigation with crowd-sourced free- form text specifying three different types of constraints. Our method outperforms several baselines by achieving 6-7 times higher returns and 76% fewer constraint violations average. Dataset and code to reproduce our experiments avail- able at https: //sitess//ssssss google e. com/vi voom/vee pp co
INTRODUCTION
Reinforcement learning (RL) has shown great promise a variety of control problems including robot navigation (Anderson et al., 2018; Misra et al., 2018) and robotic control (Levine et al., 2016; Rajeswaran et al., 2017), where the main goal is to optimize for scalar returns. However, as RL increasingly deployed in many real-world problems, it is imperative to ensure the safety of both agents and their surroundings, which requires accounting for constraints that may be orthogonal to maximizing returns. While there exist several safe RI. algorithms (Achiam al., 2017; Chow al., 2019; Yang et al., 2020b) in the literature, major limitation they share is the need to manually specify constraint costs and budget limitations. In many real-world problems, safety criteria tend to be abstract and quite challenging to define, making their specification (e.g., as logical rules or mathematical constraints) an expensive task requiring domain expertise.
On the other hand, natural language provides an intuitive and casily-accesssbbe medium for specify- ing constraints not just for experts or system developers, but also for potential end users of the RL agent. For example, instead of specifying safety constraint in the form of "if water not in previously visited states then do not visit lava", one can simply say "Do not visit the lava before visiting the water." The key challenge lies in training the RL agent to interpret natural language and accurately adhere to the constraints during exploration and execution.
In this paper, we develop novel framework for safe reinforcement learning that can handle nat- ural language constraints. This setting is different from traditional instruction following, in which text instructions
used to specify goals for the agent (e.g.. "reach the key" "go forward two steps"). To effectively learn safe policy that obeys text constraints, we propose a model consist- ing of two key modules. First, we use constraint interpreter encode language constraints into

---

Under review as a conference paper at ICLR 2021
intermediate vector and matrix representations this captures spatial information of forbidden states and the long-term dependency of the past states. Second, we design a policy network that operates on combination of these intermediate representations and state observations and is trained using constrained policy optimization algorithm (e.g., PCPO (Yang et al., 2020b)). This allows our agent to map the abstract safety criteria (in language) into cost representations that are amendable for safe RL. We call our approach Policy Optimization with Language COnstraints (POLCO).
Since there do not exist standard benchmarks for safe RL with language constraints, we construct a new navigation task called Hazard World. Haz- ard World is 2D grid world environment with di- verse, free-form text representing three types of con- straints: (1) budgetary constraints that limit resource usage or the frequency of being in undesirable states (e.g., "The lava is really hot, so it will hurt you lot. Please only walk on it times"), (2) relational constraints that specify forbidden states in relation to surrounding entities in the environment (e.g.. "There should always be least 3 squares between you and water"), and (3) sequential constraints that depend on past events (e.g.. "Grass will surround your boots and protect you from dangerous lava.") '). Fig. 1 pro- vides a sample situation from the task.
Budgetary Constralnns::
The really so will hort you lot Please only walk on it times.
Relational Constraints: should always at squares
between and water.
Sequential Constraints:
Grous will surround your and protect you from dongerous love.
Reward:
Agent's view:
Figure 1: Learning to navigate with language constraints. The figure shows (1) three types of language constraints, (2) items which provide wards when collected, and (3) third- third-person view of the environment. The objective is to maximize total reward without violating text constraints.
In summary, we make the following key contribu- tions. First, we formulate the problem of safe RL with safety criteria specified in natural language. Second, we propose POLCO, a new policy architecture and two-stage safe RI. algorithm that first encodes natural language constraints into quantitative representations and then uses these represen- tations to learn a constraint-san sattsfffig policy. Third, we introduce a new safe RL dataset (Hazard World) containing three broad classes of abstract safety criteria, all described in diverse free-form text. Finally, we empirically compare POLCO against other baselines in Hazard World. We show that POLCO outperforms other baselines by achieving 6-7 times higher returns and 76% fewer con- straint violations on average over three types of constraints. We also perform extensive evaluations and analyses of our model and provide insights for future improvements.
2 RELATED WORK
Policy optimization with constraints. Learning constraint-sattislying policies has been explored in prior work in the context of safe RL (see Garcia & Fernandez (2015) for a survey). Typically, the agent learns policies either by (1) exploring the environment to identify forbidden behaviors (Achiam et al., 2017; Tessler et al., 2018; Chow et al., 2019; Yang et al., 2020b; Stooke et al., 2020), or (2) through expert demonstration data to recognize the safe trajectories (Ross et al., 2011; Rajeswaran al., 2017: Gao al., 2018; Yang al., 2020a). Critically, these works all require human to specify the cost constraints manually, In contrast, we use natural language to describe the cost constraints, which allows for easier and more flexible specifications of safety constraint.
Instruction following without constraints. Instruction following for 2D and 3D navigation has been explored in the context of deep RL (MacMahon et al., 2006; Vogel & Jurafsky, 2010; Chen & Mooney, 201 1; Artzi & Zettlemoyer, 2013; Kim & Mooney. 2013; Andreas & Klein, 2015; Thoma- son et al., 2020: Luketina et al., 2019 Tellex et al., 2020). Prior work either focuses on providing a dataset with real-life visual urban or household environment (e.g., Google street view) (Bisk et al., 2018; Chen et al., 2019; Anderson et al., 2018; de Vries al., 2018); or proposes a computational model to learn multi-modal representations that fuse 2D or 3D images with goal instructions (Janner et al., 2018; Blukis ct al., 2018; Fried et al., 2018; Liu et al., 2019; Jain et al., 2019; Gaddy & Klein, 2019; Hristov et al., 2019; Fu et al., 2019). These work text specify goals, not environmental hazards. In contrast, we use language to describe the constraints that the agent must obey.
Constraints in natural language. Misra et al. (2018) propose two datasets called LANI and CHAI to study spatial and temporal reasoning, as well as perception and planning. Their dataset contains few trajectory constraints, which specify goal locations (e.g., "go past the house by the right
2

---

Under review as a conference paper at ICLR 2021
side of the apple"). However, (1) their 'constraints' are actually goal instructions describing where the reward is-not where forbidden states are, and (2) their model first recognizes (sub)goals from language and then uses these language-defing propose a method to convert constraints in synthetic language to reward functions. These approaches do not guarantee safety since they do not explicitly model the constraints. In our work, we define "constraints" as restrictions in state space, or specifications of forbidden states and learn safe policies.
goals reward functions. Prakash et al. (2020) also
The table below provides a quick summary of related work in instruction following and safe RL. We differentiate these methods on several key axes: (1) whether they propose new dataset, algorithm or model, (2) whether they optimize reward functions or satisfy cost constraints, and (3) whether they have text or 3D view simulation components. We also briefly describe their main objectives.
Work
Properties
Objective
Touchdown (Chen et al., 2019)
i
Dataset for multi-modal representation learning
LANI & CHAI (Misra et al., 2018)
_
+
C
Dataset and model for multi-modal learning
Checker (Prakash et al., 2020)
+
Convert text constraints to reward functions
CPO (Achiam et al.. 2017)
A
A
Safe RL algorithm with a conditional gradient update Safe RI. algorithm with a projection gradient update
PCPO (Yang et al., 2020b)
POLCO (ours)
EB
Framework for safe RL with language constraints
dataset P algorithm @ model T reward, A constraint, natural language @ 3D view
PROBLEM FORMULATION
Our goal is to learn a constraint-satisfyii policy that imposes safety or other application-specific constraints using text descriptions. We formulate our problem as partially observable constrained Markov decision process (PO-CMDP (Kaelbling et al., 1998; Altman, 1999) with text, which Here S is the set of states, O is the set of
defined by the tuple observations, A is the set of actions, T is the conditional probability T(s' (s, a) of the next state s' given the current state 8 and action a, Z is the conditional probability (ooss) ) of the observation o given the state R S x A + IR is the reward function, C x A - IR is the true underlying constraint specification, and is the set of constraint specifications in natural language. The reward function encodes the benefit of using action in state s, while the cost specification encodes penalties for constraint violations. The agent does not know C and only observes constraints from X. We assume that there exists mapping from language constraint E X' to cost specification C.
S,O, A, T, T,Z, Z, R. C,
A policy π : : + P(A) is a mapping from observation to the distributions of actions A. Let Y 1) denote discount factor, μ S -> [0, 1] denote the initial state distribution, and r denote the trajectory 7 = (Oo, 4n, 01, ..)) induced by a policy л. We seek a policy æ that maximizes the cumulative discounted reward JR while keeping the cumulative discounted cost Jc below the cost constraint threshold hr(r):
20 Σ y' R()t
Σ I'((ss, a3:0)) ho(#), (1)
max Jn(T) = ETTT
s.t. JC(T)) = B
where + N shorthand for indicating that the distribution over trajectories depends on ग : 80 ~ J, Og Z(-1st), ar ~ T--(ot), 8241 ~ T(-ss,, az), and we use C(s,, x)) and (a(()) to emphasize that they are specified by x. And we will drop x for notation simplicity.
4 LEARNING POLICIES WITH NATURAL LANGUAGE CONSTRAINTS
4.1 MODEL
We now describe our proposed neural model in POLCO, illustrated in Fig. 2. The model consists of two parts (1) the constraint interpreter processes the text to form safety criteria (constraint mask and threshold) and (2) the policy network subsequently produces assume state and observation o to be 2D matrices, although the model can easily be extended to other input representations (e.g., we describe extension to 3D scene inputs in the appendix).
action. For simplicity,
3

---

Under review as a conference paper at ICLR 2021
Constraint Mask Mr
C(sp.
hee) 1io/(teccoet Bnrtity Budget Mask M
Environment
(ocCL Easit)
Emb.
CNN
MLP
d2
Pollcy
Int[Fee Network
Travel on less than twice
Constraint Mask Module
Constraint
Constraint
Threshold Module
interpreter
(1) The constraint interpreter takes a natural language constraint æ and an observation o2 as inputs and produces a constraint mask Mc and cost con- straint threshold prediction he. (2) policy network takes an environment embedding, constraint mask Mc, and a cost budget mask MB that specifies cost satisfaction cach step as inputs and produces embedding. We further concatenate this emhedding with the embedding of he, followed by an MI.P to produce an action
Figure 2: Model overview. Our model consists of two components.
Constraint interpreter. Fig. 3 illustrates the constraint interpreter. The constraint interpreter itself consists of two parts - (1) constraint mask module and (2) constraint threshold module.
(1) The constraint mask module uses the observation or and the text x to predict a binary constraint mask, denoted by Mc, a prediction of the true constraint mask Mc. Each cell in Mc will contain one if there is cost entity (i.e., the forbidden states mentioned in the text) in ith row and jth column of the observation o₂ (denoted by oz(i. Otherwise, the cell contains a zero. We use Mc to identify the cost entity in texts while preserving its spatial information for the policy network. (2) The constraint threshold module uses LSTTM to obtain the text vector representation, followed by dense layer produce hi. prediction of the true constraint threshold lc. Full details of the constraint interpreter can be found in Appendix A.2.
For sequential constraints with long-term dependency of the past states, Mc changes based on the states visited. For example, in Fig. 3(b), after the agent visits 'water', Mc starts to locate the cost entity (i.e., 'grass"). Hence, we will need memory to keep track the visited states. For this, we use an LSTM that takes the vector representation produced by convolutional neural network (CNN) as an input and predicts Mc, illustrated in Fig. 3(b). Using Mc and he allows us specify safety criteria in text while encoding them into the policy network.
Policy network. The policy network produces action given the safety criteria processed by the constraint interpreter. The input to the network is the environment embedding (a tensor of size n x m) that encodes the observation o, which is of size n. This tensor is concatenated with the constraint mask Mc and cost budget mask, denoted by MB, prediction of the true MB- Mr is a n X n matrix, where each element takes the value of Ei=o Ü(st, (e:z) - he (i.e., the number of cost violations until t'th step) if there is cost entity in or(1,
otherwise. Note that training and test time, we estimate the cumulative cost Eizo Ö(s,, az; x) using Me and the agent's s current location at time This allows the agent to understand the constraint satisfaction at each step and use this information accordingly plan for safe trajectories. Note that Mc produced by the constraint interpreter, not by the policy network itself. This allows us to take advantage of the factorization of the policy network and to separately design a constraint mask module that produces an appropriate Mc for different types of constraints."
After applying both constraint mask Mc and cost budget mask M to the environment embedding, then feed the output into CNN to obtain vector representation. While the constraint threshold prediction he (an important metric measure safety) is input the constraint budget mask Mr. empirically we observe its signal (see Table 2(d)). To alleviate this issue, we further concatenate he with the vector representation outputted from the CNN. Finally, we pass the concatenated representation to dense layer to produce
be weakened by the downstream convolution operation
In this work, Mp equates a scaled up version of Mc since we assume only one constraint specification per episode, but this is not necessary in general since we may have multiple constraints over different cost entities. In that case, My may have different budgets for different cells (entities).
4

---

observations Under review as a conference paper at ICLR 2021
Constraint Mask Module
LSTM Cell
CNN
CNN
Bay
visit water
Constraint Mask Module
LSTM
Travel on grass
LSTM
on grass than twice efter vistting
LSTM
Cell
-in((he))
Constraint
LSTM
int(AA))
Threshold Module
Constraint Threshold Module
(a) For budgetary and relational constraints
(b) For sequential constraints
Figure 3: Constraint interpreter. (a) For the budgetary and relational constraints, a constraint mask module takes the environment embedding and text
representation inputs and predicts Mc. (b) For the sequen- tial constraints, we use an I.STM store information of the past visited states. For example, if agent has visited 'water', the constraint mask Me will begin identify the cost entity (i.e., 'grass"). In addition, for these three types of constraints, we use another LSTM given text z to predict hcc (Best viewed in color.)
an action ag- This policy architecture enables us to encode the language constraint into the policy network and hence learn constraint ceastraint-satisfinng policy.
4.2 TRAINING
Algorithm 1 Learning algorithm POLCO
Training this model with RL would require significant amount of data and computa- tion, especially when the constraint in- terpreter model needs to be learned from scratch. To improve data efficiency, we de- compose training procedure into two stages (Algorithm I): (1) pre-training phase for the constraint interpreter, which followed by (2) a policy optimization phase learn the rest of the policy work. Once all the model components are trained, there is no further change to the parameters. During execution, the agent in a random environment and provided a randomly sampled constraint to follow (e.g., "Don't It touch grass more than two times"). We then measure the ability of the policy to complete the task without violating the constraints. Importantly, our model architecture does allow for end-to-end differentiability via tricks like Gumbel Softmax (one can treat he continuous variable). However, in this version, we do not train end-to-end for computational reasons.
Stage (Interpreter pre-training)
Initialize policy x² π x-110%) and a buffer B
Run nO and store trajectories in B
Train θ and O2 using Eq. (2) and Eq. (3)
Stage (Policy learning)
Empty B
For
0.11,,2. do
Run π x-100%) and store trajectories in B
Obtain 0++1 using Eq. (4) and Eq. (5)
Empty B
Stage (Interpreter pre-training). For each type of constraint, split the collected text into train- ing and test sets using an 80-20 ratio, denoted by Dirain and Drest; respectively. We then form an offline pre-training dataset Dinter by using a random policy obtain trajectories or along with corresponding text constraints x from the training set Durain- For the observations in Dinter, compute the corresponding ground-truth Mc and he values to supervision. Utiliz- ing this dataset we first train the constraint mask module in the constraint interpreter by minimizing the following binary cross-entropy loss:
((O1) -B (onn
= Mo(i, log Mc
[Mcl
Mc jr, x)) log(1 Mc(i.j; (2)
where O₁ is the parameter of the constraint mask module, Mc denotes the ground-truth (binary) mask label in ith row and jth column of the n x n environment given o, and æ and Mcii, 2) is the corresponding binary probability prediction of constraint mask model (here we emphasize that Mc is specified by o and x)).
For the constraint threshold module, we minimize the following mean lean--quare-erroo loss using the data from D P
((0,) = E,o,, [ah 2)
where Og is the parameter of the constraint threshold module. Unlike RL, empirically only need a smaller dataset to train both models in the constraint interpreter, as the models tend to converge relatively quickly.
5

---

Under review as a conference paper at ICLR 2021
Stage 2 (Policy learning). We then use state-of-the safe RL algorithm-projection- projection--ased con- strained policy optimization (PCPO) (Yang et al., 2020b) to train the policy network. During train- ing, the agent interacts with the environment to obtain rewards and penalty costs from trained Mc for computing (T) and Jr( PCPO is an iterative method that performs two key steps in each iteration:
PCPO Step (Optimize the reward). The PCPO algorithm performs one step of trust region policy optimization (TRPO (Schulman et al., 201 to maximize the reward advantage function (8, over a KL-divergence neighborhood of .²: .
arg max E sndd ame [A a))
s.t. E ar (TKI(((())] 2
(4)
where d= is the state visitation frequency induced by the policy и at kth update.
PCPO Step 2 (Project to satisfy the cost constraint). Next, PCPO projects 7 onto the set of policies satisfying the cost constraint to minimize the distance function D (e.g,, KL divergence); x * arg min D(T,,***
s.t. (T("") + - smatth am- [Až (s,a)) S hc, (5)
where 7 is the discounted factor and (8 a) is the cost advantage function. Finally, during test time, evaluate our model in Dtest-
While our method does require pre-training of the constraint interpreter with ground-trun constraint maps, we envision in future work the dependence on this phase can be reduced by leveraging tech- niques such as curriculum learning or pre-trained language models like BERT (Devlin et al., 2019).
5 EXPERIMENTS
Task. We construct new task Hazard World for our experimental evaluation (Fig. 1). In this task, for each episode, the agent starts random location within procedurally generated environment and receives an abstract constraint specified by natural language, sampled from a pool of all available constraints. The agent's objective to collect all the reward entities and navigate safely (i.e., avoid cost entities) by adhering to the constraint. The agent needs to satisfy the constraints during both training and testing, unlike prior work (Misra al., 2018) that allows the agent to explore the entire state space during training without any constraints.
We implement Hazard World on top of the 2D GridWorld layout of BabyAl (Chevalier- -ooivvtie--Boivvert et al., 201 8a:b). We randomly place three reward entities on the map: 'ball,' 'box,' and 'key,' with rewards of 1, 2. and 3, respectively. We also randomly place numerous cost entities on the map: 'lava," 'water,' and 'grass'. A textual constraint is then imposed upon one of these cost entities (e.g., "Touch water fewer than once"). ". Note that to construct the Hazard World task we can generate multiple maps given the same text constraint. The environment is of size 13x 13, including the surrounding walls, and the agent's 's observation Of is of size 7x7.
Hazard World contains three types of constraints (1) Budgetary constraints, which limit the number of times a set of states can be visited (i.e., up to hc), (2) Relational constraints, which define a minimal distance that must be maintained between the agent and a of cost entities, and (3) Sequential constraints, which restrict the set of safe states, given that some condition has been met. For relational and sequential constraints, we budget lig as zero for simplicity.
We collect free-form text in English for the constraints us- ing Amazon Mechanical Turk (AMT). To generate sam- ple constraint, we provided workers with description of Hazard World, the cost entity to be avoided, and one of three possible additional pieces of information, depending on the constraint type: the cost budget (budgetary), the minimum safe distance (relational), or the other cost entity impacted by past events (sequential). Workers were then asked to instruct another person to safely navigate Hazard World. After collecting all responses from AMT, we filtered the data to remove off-topic responses. For examples of text and AMT worker prompts, refer the reader to Appendix A.1. Table provides statistics the text.
Count Vocab. Size Mean Length
Bud.
432
274
9.09 + 4.30
Rel.
262
180
9.02 + 3.65
Seq.
290
241
10.36 + 3.49
Total
984
526
9.44 + 3.95
Table 1: Statistics in Hazard World.

---

Under review as a conference paper at ICLR 2021
he D
he 2
hC
Mathod
Jaimit
Acl
Agct
Jalait
Scl
Apct
Ja/m)"
Acl Age1
RW
1.2
17.8
-16.6
1.2
1588
-14.6
1.2
1388
-12.6
CF w/ TRPO
3.6
12.1
-8.5
3,5
10.2
-6.7
3.6
8.7
-5.1
CF w/ PCPO
0.5
4.0
-3.5
0.5
2.3
-1.8
0.5
0.2
0.3
PN w/ TRPO
5.7
8.0
-2.3
5.7
7.1
-1.3
5.7
2,4
3.1
PN TPO
0.3
0.1
02
-08
02
0.3
POLCO (ours)
500
19
31
50
0.3
4.7
5.1
-.1
51
(a) Budgetary Constraints
he
he
Metbod
Ja(nit
Act
Agc
Method
Ja(n)] Act
A
he = O
be = 2
be 1
19.1
-18.2
9.8
-3.7
Method
Jula]
Arl Julnit Ael
Julx)t Art
CF w/ TRPO
2.2
11.2
-9
CF w/ TRPO
0.0
0.0
0.0
CF w/ PCPO
0.5
4.0
0.5
2.3
0.5
0.2
CF PCPO
0.0
0.0
CF w/ PCPO
3.4
11.3
-7.9
wo Me
3.9
2.5
4.1
0.8
4.0
-1.8
PN w/ TRPO
5.1
17.3
-1222
PN w/ TRPO
588
2.5
3.3
woo her emb.
4.5
1.6
4.6
0.2
4.6
-1.2
PN w/ FPO
0.4
0.0
0.4
PN w/ FPO
0.0
4.5
-15
weo Mp
52
2.3
5.2
0.2
5.2
-1.6
POLCO (ours)
1.1
0.0
1.1
POLCO (ours)
571
1.0
4.1
Full Model
1.5
5.5
0.2
5.5
-0,9
(b) Relational Constraints
(c) Sequential Constraints (d) Ablation studies budgetary constraints
Table 2: (a-c) Performance POL.CO (i.e., PN w/ PCPO) and baselines across three types of constraints on the test set, (d) Ablations showing the effect of each component in POL.CO. POLCC outperforms all the other variants by achieving higher reward and lower constraint violations. (Arrows denote higher or lower scores.)
Train-test split. For each type of constraint, we split the collected text into training and test sets using an 80-20 ratio, denoted by Derain and Drest; respectively.
Stage (Interpreter pre-training). To form Diuter for pre-training, we first sample a text constraint x from Derain and generate a map. We then use a random policy to obtain the trajectories with rollout length 200 over observations or along with corresponding text constraints In total, we repeat this procedure 5,000 times to form Dinter (i.e., each constraint is paired with 6.35 maps on average). We train one constraint interpreter for each type of constraint, which ensures that the constraint interpreter can robustly produce Mc and hc. During testing of the cost constraint interpreter, we use the same procedure and generate 5,000 constraint-map pairs.
Stage 2 (Policy learning). For training POLCO, we use text constraints from Derain paired with randomly generated maps. Note that for each type of constraints, we train single network with different values of lc. During testing, we randomly sample constraints from Drest, followed by randomly generating the map. In total, we obtain 5,000 constraint-map pairs to test the model (i.e. on average one constraint is paired with 25.38 maps). Please read Appendix A.2 for more details.
Baselines. The baselines are combinations of the following models and training algorithms:
(1) Models. We consider a baseline model from Walsman et al. (2018), which does not have the Mc, MB and he representations. This model simply takes a concatenation of the observations and text representations as input and produces an action. We call this model construint-fusion (CF).
(2) Algorithms. (a) TRPO (Schulman et al., 2015a). In TRPO, the agent ignores all constraints and only optimizes the reward. (b) Fixed-point policy optimization (FPO) (Achiam et al., 2017). In FPO, the reward objective is combined with the weighted cost objective (i.e., receive the penalty of 1 if the agent visits the cost entity). We include it to demonstrate that simply treating the cost penalty as a negative reward will hinder the exploration of the agent.
Finally, we also include random walk (RW) baseline serve lower bound on the reward and cost performance. POLCO is combination of our policy network (PN) and the PCPO algorithm.
Evaluation metries. Our primary evaluation metrics are (1) the average value of the reward (i.e., JR(T)), (2) the average cost violations Ac := Jc(T) - -ho, and (3) the difference between the reward and the cost violations ARC:= (n( - a max(0, Ac), where a represents tradeoff between reward and cost violation. We set a to 1 for simplicity. but may depend on the task. Note that we want to enforce Ae to be non-positive while maximizing JR((T) (i.e., maximize ARC).
Overall performance. Table 2(a-c) details the performance of POLCO and the baselines over three types of constraints on the test set. We observe from Table 2(a) that POLCO achieves the best Anc over different choices of les outperforming other baseline methods. For instance, when hG = 0 the constraint violation of the policy network trained with TRPO is 3.21 times larger than that of POLCO. This implies that simply optimizing the reward in Misra et al. (2018) cannot guarantee safety. We also observe that PN trained with TRPO achieves better reward performance compared when with CF and TRPO, which indicates that our proposed policy network is easier to optimize than the CF model. CF with PCPO also fails to improve the reward since merely combining the observations and the text is not sufficient to learn effective representation for parsing the
7

---

Under review as a conference paper at ICLR 2021
Budgetary
Relational
Sequential
After wall kilkking grass do not walk lave
Ours
Rule
Ours
Rule
Ours
Rule 91.36%
Me ACC
96.06% 95.86%
81.46 65.49 .5.4%%
91.84%
Mc AUC 98.17%
90.71
96.99%
Agoit' viow
he MSE
0.21
0.74
0.04
0.52
0.0002
0.22
Before visiting
Agent
After visiting
grass
grass
grass
Protubility
(a) Performance of the interpreter
(h) Visualization of the constraint mask
Figure 4: (a) The cell-level accuracy (ACC) and area-under-curve (AUC) of predicting Mc. and -euar-- error (MSE) of predicting le over three types of constraints on the test set. We observe that POLCO achieves superior results compared to the sule-based baseline. (b) Visualization of the constraint mask Mc for the sequential constraints. We show the probability of predicting the cost entity. Our model can accurately locate the cost entity given the language constraints. The first two views are mostly black because the agent spawned facing wall. (Best viewed in color.)
0
hg
hc
Mann-aaaal Ja(e)f
ne
RW
0.1
23.4
-2.3
0.1
21.4
213
0.1
19.4
19.3
w/ TRPO
3.4
35.6
-32.2
3.4
33.6
3002
3,4
-28.2
0.7
7.4
-67
0.7
4.7
0.7
PN w/ TRPU
4.5
54,7
-502
4.5
52.7
-82
4.3
50.7
-16.2
PN FPO
1.8
0.0
18
1.8
-20
1.8
1.8
4.0
1.8
Ours
3.7
29
0.8
39
4.5
(a) Lavawall (Chevalier-B Sthevalier-Boisverteee et al., 2018a)
(h) Generalization results in Lavawall
Figure 5: (a) Lavawall environment: Reward function and transition dynamics same as Hazard World, but map contains only 'lava' entities. (b) Generalization performance in Lavawall over the tested models and algorithms. (Arrows denote higher or lower scores being better.)
constraints. Finally, the PN agent trained with FPO has a low reward, because simply treating the cost penalty as the negative reward hinders the agent's exploration.
Tables 2(b-c) also show that POLCO achieves the best Anc for the more complex relational and sequential constraints. For the relational case, although the CF agent trained with PCPO satisfies the constraints, it has a relatively low reward. For the sequential constraints, even though the PN agent trained with TRPO obtains a higher reward than POLCO, it also has 1.5 times higher cost violations.
Ablation studies. We perform ablation studies on the proposed POLCO policy network to examine the importance of each component - Mc. MB and he embedding. The performance over different be in terms of the reward and cost violation is shown in Table. 2(d). To eliminate the prediction error of the constraint interpreter, here we use the ground-truth Mc and he instead. We observe that the full model achieves the best performance in all cases, averaging 5.12% more reward and 2.22% fewer cost violations. Without Mc. the agent cannot recognize the cost entities. This causes the agent to incur 66.67% higher Ac compared with the full model. In addition, Ac in the full model is almost zero. This suggests that with the he embedding and the MB mask the agent is able to better understand the cost satisfaction at every step and hence plan for safer trajectories. In contrast, without these components the agent fails to understand the text constraints and improve the reward.
Constraint interpreter. Next, examine the performance of the constraint interpreter. The result is illustrated in Fig. 4(a). Please read the caption for more details. The rule-based baseline here similar n--ram model, in which we compute the likelihood of the cost entity given the text. Please read Appendix A.2 for more details. We observe that our model achieves superior perfor- mance in all cases. We further visualize the predicted Mc for the sequential constraints shown in Fig, 4(b), We observe that after visiting 'grass', .Mc changes from near zero prediction probability to predict cost entities precisely. This shows that our constraint interpreter model can effectively maintain the long-term dependency of the past states and produce an accurate Mc-
Generalization to different environments. Finally, we examine whether POLCO is robust to co- variate shift in the environment distribution (i.e., a new environment containing only a subset of all cost entities). We first train POLCO and the other baselines Hazard World and then test them on the commonly-used Lavawall Schevalier-Boisverr et al., 201 8a) task as shown in Fig. 5(a). We select free-form constraints from Hazard World imposed over the 'lvva' cost entity to be the training set. Fig. 5(b) shows the reward and cost violation over different he of different agents in the budgetary- constrained Lavawall experiment. We observe that in experiments with a larger he, generally the agents are less constrained during navigation and yield higher reward and lower cost violations.
8

---

Under review as a conference paper at ICLR 2021
Still, in most cases the baseline agents have substantially higher cost violations obtain minimal reward. This shows that the POLCO agents generalize better to unseen environments, obtaining higher reward and lower constraint violations.
CONCLUSION
We addressed the problem of safe reinforcement learning when safety constraints are specified by natural language. We proposed the two-stage POLCO algorithm that both learns understand the abstract language constraints and effectively solves for a safe policy. To demonstrate the effective- ness of POLCO, we created new benchmark navigation task via crowd-soureing which contains variety of text constraints. Experiments showed that POLCO achieves superior reward perfor- mance, lower constraint violation, and better generalizability than other safe R approaches.. Future directions include (1) applying POLCO to 3D navigation tasks with high-dimensional (image) ob- servations, (2) improving representation learning to handle more complex language constraints, and (3) applying POLCO to large-scale safety-critical problems (e.g., robotics).
REFERENCES
Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In Proceedings of International Conference Machine Learning, pp. 22-31, 2017.
Eitan Altman. Constrained Markov decision processes, volume 7. CRC Press, 1999.
Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton van den Hengel. Vision-andd-lanuuage navigation: Interpreting visually-grounded navigation instructions in real environments. In Proceedings of the IEEE Con- ference on Computer Vision and Pattern Recognition, pp. 3674-3683, 2018.
Jacob Andreas and Dan Klein. Alignment- basedd compositional semantics for instruction following. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2015.
Yoav Artzi and Luke Zettlemoyer. Weakly supervised learning of semantic parsers for mapping instructions to actions. Transactions of the Association for Computational Linguistics, 1(1):49- 62, 2013.
Yonatan Bisk, Kevin J. Shih, Yejin Choi, and Daniel Marcu. Learning interpretable spatial opera- tions in a rich 3d blocks world. In Proceedings of the AAAI Conference on Artificial Intelligence, 2018.
Valts Blukis, Dipendra Misra, Ross A. Knepper, and Yoav Artzi Mapping navigation instructions to continuous control actions with position-visitatiinn prediction. In Proceedings of the Conference on Robot Learning, pp. 505-518, 2018.
David L Chen and Raymond Mooney, Learning to interpret natural language navigation instruc- tions from observations. San Francisco, CA, pp. 859-865, 2011.
Howard Chen, Alane Suhr, Dipendra Misra, Noah Snavely, and Yoav Artzi. Touchdown: Natural language navigation and spatial reasoning in visual street environments. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 12538-12547, 2019.
Maxime Chevalier-Boisvert, Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Saharia, Thien Huu Nguyen, and Yoshua Bengio. Babyai: A platform to study the sample efficiency of grounded language learning. In International Conference on Learning Representations, 201 Sa.
Maxime Chevalier-Boisver Lucas Willems, and Suman Pal. Minimalistic gridworld environment for openai gym. https: : / git..b....... wom/ max
2018b.
Yinlam Chow, Ofir Nachum, Aleksandra Faust, Mohammad Ghavamzadeh. and Edgar Duenez- Guzman. Lyapunov-basee safe policy optimization for continuous control. arXiv preprint arXiv:/901. 10031, 2019.

---

Under review as a conference paper at ICLR 2021
Hann de Vries, Kurt Shuster, Dhruv Batra, Devi Parikh, Jason Weston, and Douwe Kiela. Talk the walk: Navigating new york city through grounded dialogue. arXiv preprint arXiv: 1XXi::887.00337,, 2018.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume I (Long and Short Papers), pp. 4171-41888 2019.
Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep reinforcement learning for continuous control. In International Conference on Machine Learning, pp. 1-39--1338 2016.
Daniel Fried, Ronghang Hu, Volkan Cirik, Anna Rohrbach, Jacob Andreas, Louis-Philippe Morency, Taylor Berg- Kirkparrkt Kate Saenko, Dan Klein, and Trevor Darrell. Speaker-follower models for lasion-and-language navigation. In Advances of the International Conference Neu- ral Information Processing Systems, pp. 3318-3329, 2018.
Justin Fu, Anoop Korattikara, Sergey Levine, and Sergio Guadarrama. From language to goals: Inverse reinforcement learning for vision-based instruction following. Proceedings of the Inter- national Conference on Learning Representations, 2019. 2019..
David Gaddy and Dan Klein. Pre-learning environment representations for data-efficient neural instruction following. In Proceedings of the Annual Meeting of the Association for Computational Lingnistics, pp. 1946-1956, 2019.
Yang Gao, Ji Lin, Fisher Yu, Sergey Levine, and Trevor Darrell. Reinforcement learning from imper- fect demonstrations. In Proceedings of the 35th International Conference on Machine Learning, 2018.
Javier Garcia and Fernando Fernandez. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research, 16(1):1437-1480, 2015.
Yordan Hristov, Daniel Angelov, Michael Burke, Alex Lascarides, and Subramanian Ramamoor- thy. Disentangled relational representations for explaining and learning from demonstration. In Proceedings of the Conference on Robot Learning, 2019.
Vihan Jain, Gabriel Magalhaes, Alexander Ku, Ashish Vaswani, Eugene Ie, and Jason Baldridge. Stay on the path: Instruction fidelity in saiion-nddlanguual navigation. In Proceedings of the Anmual Meeting of the Association for Computational Linguistics, pp. 1862-1872, 2019.
Michael Janner, Karthik Narasimhan, and Regina Barzilay. Representation learning for grounded spatial reasoning. Transactions of the Association for Computational Linguisties, 6:49-61, 2018. 88
Leslie Pack Kaelbling, Michael I.. Littman, and Anthony R. Cassandra. Planning and acting in partially observable stochastic domains. Artificial intelligence, 101(1-2):99-134, 1998.
Joohyun Kim and Raymond J. Mooney. Adapting discriminative reranking to grounded language learning. In Proceedings of the Association for Computational Linguistics, pp. 218 22-, 2013.
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuo- motor policies. Journal of Machine Learning Research, 17(39):1-40, 2016.
Ninghao Liu, Mengnan Du, and Xia Hu. Representation interpretation with spatial encoding and multimodal analyties. Proceedings of the ACM International Conference on Web Search and Data Mining, pp. 60-68. ACM, 2019.
Jelena Luketina, Nantas Nardelli, Gregory Farquhar, Jakob Foerster, Jacob Andreas, Edward Grefen- stette, Shimon Whiteson, and Tim Rocktäschel. survey of reinforcement learning informed by natural language. International Joint Conferences on Artificial Intelligence, 2019.
Matt MacMahon, Brian Stankiewicz, and Benjamin Kuipers. Walk the talk: Connecting language, knowledge, and action in route instructions. In Proceedings of the Conference on Artificial Intel- ligence, pp. 1482- 2006.
10

---

Under review as a conference paper at ICLR 2021
Dipendra Misra, Andrew Bennett, Valts Blukis, Eyvind Niklasson, Max Shatkhin, and Yoav Artzi. Mapping instructions to actions in 3d environments with visual goal prediction. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing, pp. 2667-2678, 2018.
Bharat Prakash, Nicholas Waytowich, Ashwinkumar Ganesan, Tim Oates, and Tinoosh Mohsenin. Guiding safe reinforcement learning policies using structured language constraints. UMBC Swu- dent Collection, 2020.
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprint 170.:::700.0087, 2017.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. reduction of imitation learning and struc- tured prediction to no-regret online learning. In Proceedings International Conference Artificial Intelligence and Statistics, pp. 627-635, 2011.
John Schulman, Sergey Levine, Picter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889-1897, 5a.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv: 2015b.
Adam Stooke, Joshua Achiam, and Pieter Abbeel. Responsive safety in reinforcement learning by pid lagrangian methods. In Proceedings of International Conference on Machine Learning. 2020.
Stefanie Tellex, Nakul Gopalan, Hadas Kress-Gazit, and Cynthia Matuszek. Robots that use lan- guage. Annual Review of Control, Robotics, and Autonomous Systems, 3:25-55, 2020.
Chen Tessler, Daniel J. Mankowitz, and Shie Mannor. Reward constrained policy optimization. In Proceedings of International Conference Learning Representations, 2018.
Jesse Thomason, Michael Murray, Maya Cakmak, and Luke Zettlemoye. Vision-and-dialog navi- gation. In Conference on Robot Learning, pp. 394 406, 2020.
Adam Vogel and Dan Jurafsky. Learning to follow navigational directions. In Proceedings of the 48th Anmual Meeting of the Association for Computational Linguistics, pp. 806-814, Association for Computational Linguisties, 2010.
Aaron Walsman, Yonatan Bisk, Saadia Gabriel, Dipendra Misra, Yoav Artzi, Yejin Choi, and Dieter Fox. Early fusion for goal directed robotic vision. arXiv preprint 18Xiv: 1.8824, 2018.
Tsung- Yen Yang, Justinian Rosca, Karthik Narasimhan, and Peter J. Ramadge. Accelerating safe reinforcement learning with constraint-missaatdinn policies. arXiv preprint miXiv:206.11665 2020a.
Tsung-Yes Yen Yang, Justinian Rosca, Karthik Narasimhan, and Peter Ramadge. Projection-based constrained policy optimization. In Proceedings of the International Conference on Learning Representations, 2020b.
A APPENDIX
Outline. The appendix outlined as follows. Section A.1 details the dataset and the procedure of collecting the dataset. Section A.2 describes the parameters of the constraint interpreter and the policy network, and the PCPO training details. Section A.3 provides the learning curves of training the policy network. Finally, fill the Machine Learning Reproducibility Checklist in Section A.5.
11

---

Under review as a conference paper at ICLR 2021
Constraint Type
Examples
The water should only be stepped max of 5 times.
Lava hurts a lot, but you have special shoes that you can use to walk on it, but only up to 5 times, remember!
Budgetary
You can get in lava, but only once.
Four is the most number of times you touch water
You cannot step on the lava block at all. You will die otherwise.
Water will hurt you if you are two steps or less from them.
Always stay step away from lava
Relational
Any block within one unit of a grass cannot be touched.
The explosion radius of grass is three, stay at least that distance away from grass.
Waters are dangerous, so do not cross them.
Make sure you don't 't walk on water after walking on grass.
Do not touch the water or water will become risky.
Sequential
You may touch the water first, but the lava is dangerous do not touch it after.
Avoid lava since you only walk on it once. After that the lava will hurt you.
Water will trigger grass become dangerous.
good
move foreward
Invalid
Just avoid the perimeter when collecting the objects. and you 'll be safe.
Your directions are as follows: you're facing a block with a water block in front of it, walk five blocks ahead (81 more words)
asdf
Table 3: Examples from the various constraint classes. When constraint does not fully describe all forbidden states in the environment, we classify as invalid.
A.1 DATASET
At a high level, Hazard World applies the instruction following paradigm to safe reinforcement learning. Concretely, this means that safety constraints in environment specified via lan- guage. Our dataset is thus comprised of two components: the environment, made up of the objects that the agent interacts with, and the constraint, which imposes restriction which environmental states can be visited.
The environment is procedurally generated. For each episode, Hazard World places the agent a randomized start location and fills the environment with objects. Hazard World then randomly samples one constraint out of all possible constraints and assigns this constraint the environment.
We collected natural language constraints in a two-step process. In the first step, or the data gen- eration step, prompted workers on Amazon Mechanical Turk with scenarios shown in Fig. 6. Workers are provided the minimum necessary information to define the constraint and asked to describe the situation to another person. For example, generate a so-called budgetary constraint, workers are given the cost entity to avoid ( ('lava', 'grass' or 'water') and the budget (i.e., lic, num- ber through 5). The workers use this information to write an instruction for another person. This allows us to ensure that the texts we collected are free-form. These generations form our language constraints.
In the second step, or the data validation step, employed an undergraduate student remove invalid constraints. We define a constraint as invalid (a) the constraint is off-topic or (b) the constraint does not clearly describe states that should be avoided. Examples of valid and invalid constraints are included in Table 3. Finally, we randomly split the dataset into 80% training and 20% test sets.
In Hazard World and Lawawall, the agent has actions total: a E A = {right, left, up, down). The transition dynamics 7' is deterministic.
12

---

Under review as a conference paper at ICLR 2021
Here, your friend navigating the environment below to collect the ball, box, and key. The environment also contains lava, water, and grass blocks.
0/933
(a) General prompt for all constraint classes.
One biock lypes
your stand them limited the Dangerous Block three times tall the she th atlanttoon.
friand
Dangerous Blockc lava
Number Times:
Type what you would say here...
(b) Budgetary prompt.
Ona of these objects dangarmus, and your friend must maintain ninumm safe distance from that type of object For example, Minimumn Distance 1, than your fiend must stay at least slep away from Dangerous Blocks times, Knowing this, tell the other person what she can or cannoo do in this situstion.
Dangerous Block: lova
Type what you say here...
(c) Relational prompt.
One of the three block types is Inigeer block. If you friend touches trisper block, ane the block tyses becomes dangerous, Your job is to lel the other person what she can or cannot do this sttalion. For examp o, Trigger Block lava, then your frent can't walk on Dangerous Blocks after walk ng on any lava blacks.
Trigger
Dangerous Blocki
Type
(d) Scquential prompt.
Figure 6: AMT workers receive the general prompt and one of the three specific prompts. They are then asked to instruct another person for the given situation. This ensures that the texts we collected are free-form.
A.2 ARCHITECTURES, PARAMETERS, AND TRAINING DETAILS
Policy network in POLCO. The architecture of the policy network is shown in Fig. 7. The environ- ment embedding for the observation o; is the size x This embedding is further concatenated with the cost constraint mask Mc and the cost budget mask MB. This forms the input with the size 7x7x5. We then use convolutions, followed by dense layers to get vector with the size 5. This vector is further concatenated with the he embedding. Finally, cal distribution with four classes (i.e., turn right, left, up or down in Hazard World), We then sample an action from this distribution.
dense layers to the categori-
Constraint interpreter in POLCO. The architecture of the constraint interpreter is shown in Fig. 8. For the constraint mask module, the input is the text with words. We then use an embedding network, followed by an LSTM to obtain the text embedding with the size 5. The text embedding is duplicated to get tensor with the size 7x5x.. This tensor is concatenated with the observation of size 7 x 7 x 3, creating a tensor with the size 7x7x8. x8. In addition, we use convolution, followed by dense layers and reshaping to get the cost constraint mask Mc.
Next, we use heuristic to compute Crat := E' Cls x; from Mc. At execution time, we give constraint interpreter access the agent's actions. We initialize Črat 0. Per timestep.
13

---

Under review as a conference paper at ICLR 2021
Input 7X7X5
tanh conv 3X5
5X5X5
tanh conv 3X6
3X3X6
flatten
tanh dense
30
tanh dense
15
hg
tanh dense
Emb.
5
tanh dense
tanh dense
tonh dense
righht, left, up, down]
Figure 7: Description of the policy network in POLCO.
Parameter
Reward dis. factor y
0.99
Constraint cost dis. factor NC
1.0
step size GAE
10-3
0.95
X BAA
0.9
Batch size
10,000
Rollout length
200
Number policy updates
2,500
Table 4: Parameters used in POLCO.
our agent either turns moves forward. the agent moves forward and the square in front of the agent contains cost entity according to Mc. we increment Crot.
For the constraint threshold module, we use the same architecture to get the text embedding. We then use dense layers predict the value of li.
Rule-based baseline. The rule-based baseline is similar to n-gram model, in which we compute the likelihood of the cost entity given the text. Specifically, in the training we count the number of word-cost entity pairs. For example, for each map, given the language constraints "The water should only be stepped on max of 5 times" and the ground-truth cost entity (e.g., 'water"), the pair "the"- -'water' is added one and forth. During testing, for given text and map, we compute the following probability for each possible cost entity:
P('water' text constraints) & Ptext PatiraainttsP(PPateee
-I P P(word) water') P( ((wwater'),
14

---

Under review as a conference paper at ICLR 2021
Input Embedding
wX5
1STM with tanh
Replicate
7X7XS
Input 7X7X3
7X7X8
tonh conv 3X10
5XSX10
Input w Embedding
flatten
250
wX5
dense
LSTM with tanh
64
dense
10
tonh dense
reshope
64
1anh dense
7X7
(a) Constraint mask module
(b) Constraint thresh- old module
Figure 8: Description of the constraint interpreter.
P(Ilava' text constraints) x Ptext constraints) P(tlava)
=Io P (lvva P(taaaaaaa"""d'd"))) P('lavd')
P(rgrass' |text constraints) x (text P
=Il (word] grass') P'rgrass'). Finally, we select the one with the maximum value to predict the cost entity.
Details of the algorithm-PCPO. We use a KL divergence projection in PCPO to project the policy onto the cost constraint set since it has a better performance than l22 norm projection. We use GAE- A approach (Schulman et al., 2015b) to estimate (7, a) and (7, a). We use neural network baselines with the same architecture and activation functions as the policy networks. The hyper- parameters of training POLCO are in Table 4. We conduct the experiments on the machine with Intel Core 17-4770HQ CPU. The experiments are implemented in rllab (Duan al., 2016), tool for developing RL algorithms.
Baseline model-Constraint Fusion (CF). Our baseline is adapted from Walsman et al. (2018). The model is illustrated in Fig. 9. An LSTM takes the text æ as an input and produces vector representation. The CNN takes the environment embedding of o, as an input and produces a vector representation. These two vector representations an action ae. We do not consider other baselines in Janner et al. (2018) and Misra et al. (2018). This because that their models are designed to learn multi-modal representation (e.g., processing 3D vision) and follow goal instructions. In contrast, our work focuses on learning a constraint- sattssiinn policy.
concatenated, followed by MLP to produce
The parameters of the baseline is shown in Fig. 10. We use the same CNN parameters as in our policy network to process oz. Then, we use the same LSTM parameters in our constraint mask
15

---

Under review as a conference paper at ICLR 2021
Environment
Env. Emb.
CNN
MLP
Travel on grass less than twice
LSTM
Figure 9: Baseline an input and produce vector representation, (2) an LSTM takes z as an input and produce a vector representation. We then concatenate these two vectors, followed by MLP to produce an action dz.
Fusion (CF). composed of two parts - (1) CNN takes
Input 7X7X3
tonh 3XS
SX5X5
tanh conv 3X6
3X3X6
flatten
54
tanh dense
Embedding
tanh dense
wX5
15
tonh dense
1STM with tonh
tanh dense
tanh dense
tanh dense
rrihht, left, up, down]
Figure 10: Description of our baseline model-Constrrint Fusion (CF).
module to get a vector representation with size 5. Note that we use almost the same number of the parameters ensure that POLCO does have an advantage over CF. Finally, we use dense layers to the categorical distribution with four classes. We then sample an action from this distribution.
A.3 ADDITIONAL. EXPERIMENTS
Learning curves of training the policy network. The learning curves of the undiscounted constraint cost, the discounted reward, and the number of steps over policy updates shown for all tested algorithms and the constrains in Fig. 11. Overall, we observe that
(1) POLCO improves the reward performance while satisfying the cost constraints during train- ing in all cases,
the policy network trained with TRPO has substantial cost constraint violations during training,
(3) the policy network trained with FPO is overly restricted, hindering the reward improve- ment.
16

---

Under review as a conference paper at ICLR 2021
Undlsc. cost violations
200
Undlsc. reward w
TRAO
180
160
Steps
15
10
120
Ours
Our
TAPO
-5
100
1AP0
80
500
1000
1500
2000
2500
500
1000
2000
2500
500
1000
1500
2000
2500
policy updates
Pre-training
pollcy updates
policy updates
(a) Budgetary constraints
Undisc, cost violations
200
5
Undisc, reward
TRPO
190
FRO
180
20
160
Durb
10
150
7ARO
140
TRA0
FAO
130
500
1000
1500
2000
2500
0
500
1000
1500
2000
2500
0
500
1000
1500
2000
2500
# of policy updates
Pre-trbiniin
polley updates
# of policy updates
(b) Relational constraints
Undisc. cost violations
FRO
Undiscc reward
180
Durs
140
10:
120
7AP0
5-
100
FA0
TAPO
Ours
Ours
500
1000
1500
2000
2500
500
1000
1500 policy updates
2000
2500
500
1000
1500
2000
2500
# of policy updates
Pre-traiming
# of policy updates
(c) Sequential constraints
Figure 11: Learning curves of training the policy network. The undiscounted reward, the undis-
counted cost violations (i.e, Ac = (x()) - hc), and the number of steps over policy updates for the tested algorithms and the constrains. In the undiscounted include the numbers for the interpreter pre-training stage in the first 100 points. This is equal to 5000 trajectories. The maximum allowable step for each trajectory is 200. We observe that POLCO satisfies the cost constraints throughout training while improving the reward. In contrast, the pol- icy network trained with TRPO suffers from violating the constraints and the one trained with FPO cannot effectively improve the reward. (Best viewed in color.)
violations plots, we further
A.4 POLCO FOR PIXEL OBSERVATIONS/3D EGO-CENTRIC OBSERVATIONS
To deal with pixel observations Oz, we can still use the proposed architecture to process as shown in Fig. 12. To predict the cost constraint mask Mc. we use the object segmentation method to get bounding box of each object in the scene. As a result, the area of that bounding box will be one if there is a cost entity (ie., the forbidden states mentioned in the text). Otherwise, the bounding box contains zero. For Mp, we at each step. In addition, to deal with navigation environments with 3D ego-centric observations, we propose shifting the o,, Me and Mr matrices to the first-person view. The bounding box approach for image case can still be applied here. We leave this proposal to future work.
similar approach to compute the cumulative cost violations
17

---

Under review as a conference paper at ICLR 2021
Constraint Mask Mc CosL)) Entity)
Env. Emb.
Cost Budget Mask M
Oc(a(- lo:)) hc)
Eoo(s Entity
CNN
MLP
a₂
Policy Network
int(hc)
Travel grass than twice
Constraint Mask Module
Constraint Threshold Module
Constraint Interpreter
POLCO for pixel observations and 3D ego-centric observations. represents the bounding box of each object in Oz.
Figure 12:
The red cloud area
A.5 THE MACHINE LEARNING REPRODUCIRILITY CHECKLIST (VERSION 1.2, MAR.27 2019)
For all models and algorithms presented, indicate if you include²:
A clear description of the mathematical setting, algorithm, and/or model:
Yes, please see the problem formulation in Section 3, the algorithm and the model in Section 4.
An analysis of the complexity (time, space, sample size) of any algorithm:
Yes, we provide discussions in Section and Appendix A.2 for showing training infrastructure.
A link to a downloadable source code, with specification of all dependencies, including external libraries:
Yes, please see our abstract.
For any theoretical claim, check if you include:
A statement of the result:
Not applicable.
A clear explanation of any assumptions:
Not applicable.
A complete proof of the claim:
Not applicable.
For all figures and tables that present empirical results, indicate if you include:
A complete description of the data collection process, including sample size:
Yes, please see Appendix A.I and Table for sample size.
A link to a downloadable version of the dataset or simulation environment:
Yes, please see our abstract.
An explanation of any data that were excluded, description of any pre-processing step:
Yes, please see Appendix A.I.
Here
is
link the list:
https://www.cs. htl.ccaa/ippaae
Neproducibilito wtw.eiibbbttyyhhckkiikcc
18

---

Under review as a conference paper at ICLR 2021
An explanation of how samples allocated for training / validation / testing:
Yes, please see section 5 and Appendix A.1
The range of hyper-parameters considered, method to select the best hyper-parameter con- figuration, and specification of all hyper-parameters used generate results:
Yes, please Appendix A.2.
The exact number of evaluation runs:
Yes, please see section
A description of how experiments were run:
Yes, please see section
A clear definition of the specific measure statistics used to report results:
Yes, please Section 5.
Clearly defined error bars:
Not applicable.
A description results with central tendency (e.g., mean) variation (e.g, stddev):
Not applicable.
A description of the computing infrastructure used:
Yes, please Appendix A.2.
19


Reviews:
[{"review_id": "jWh6mpz0MWs", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper present an experiment of safe reinforcement on a 2D grid-word where the safety constraints are specified in natural language instead of being specified formally. The justification of this system is to allow non-experts to train agents using safe-RL.\nAccording to the authors: \"The key challenge lies in training the agent to interpret natural language and naturally adhere to the constraints during exploration and execution\".  \nThe proposed system is made of two parts: a constraint interpreter that is (mostly) trained in a supervised way with Amazon Mechanical Turk to translate natural language orders into grid-world ad-hoc constraints and a policy that is trained through PCPO, a TRPO-like constraint-aware policy optimization algorithm.\nThat's certainly a nice piece of engineering, but honestly my first sentiment when reading the motivation of this paper was astonishment. If the agent is already able to understand complex natural language statements about its environment during its exploration/training phase, why would it need further training to apply these constraints on a simple grid-world ?\n\nIf the natural language understanding (NLU) task has to be handled before the agent's exploration/training phase, we are facing a concatenation of two problems: NLU then Safe-RL, not a new problem involving tightly NLU and safe-RL.\nEven if some publications were already made on that topic, training a agent to properly follow natural language orders, that may include constraints, during its execution phase is yet an unsolved problem that requires a real fusion between NLU and RL that is of scientific interest.", "extra_scores": {}, "extra_text": {}}, {"review_id": "PGMVZgS545Q", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe paper addresses how to learn policies for tasks in which constraints are specified in natural language. Towards this, the paper proposes a model that encodes the different types of natural language constraints into intermediate representations that model both spatial and temporal information between states. Then, they use this as input along with the observation to produce an action at each time step for a safe trajectory. \nThey also propose a new benchmark (Hazard World) which is inspired by the 2D MiniGrid environment. They show the efficacy of their models on this new benchmark outperforming several baselines. \n\nStrengths:\n1. The authors address an important problem of modeling natural language constraints for safe RL. Unlike previous work, which requires manual specification of constraint or rule bases constraint, specifying natural language constraints is more intuitive and scalable. \n2. The proposed model which breaks down the language constraint into a constraint mask and constraint threshold is intuitive. The experiments also demonstrate \n3. The authors also propose an interesting benchmark to evaluate methods for safe RL under natural language constraints. Even though the environment is built on top of 2D grid world with simple action spaces, constraints are still specified in free-form natural language. The benchmark will be useful for the wider RL community. \n4. The paper is well written and easy-to-read. \n\n\nWeaknesses: \n1. The proposed models seem to be engineered for the 2D grid-like environments used for evaluation. I'd be interested in knowing how to generalize the models to other types of environments and observations (for instance navigation environments with 3D ego-centric observations). The authors mention that they describe an extension to 3D scene inputs in the appendix but I didn't find those details. Please let me know if I missed it in the appendix. \n\n2. Mapping constraints into the current observation is a limiting assumption. Can all constraints be mapped to current observations? \n - 2.1 How can the proposed model deal with constraints that might be violated due to partial observability? For instance, if the lava is behind the agent's field of view, how will the agent incorporate those constraints. \n - 2.2 There might be other temporal constraints that need to be modeled. For instance, defuse a bomb in less than 5 steps like constraints that don't necessarily describe a visual constraint. How can a model handle such constraints? \n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "BUT7F7ykDvy", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a new test environment, Hazard World, for learning the safe reinforcement learning agents with given natural language constraints. In this problem, the goal of the agent is to find an optimal policy that maximizes the cumulative rewards while satisfying the constraints given in natural language. The authors introduce the model that contains the following two separate components; constraint interpreter for encoding the language constraints and policy network for learning the RL agent. Finally, they report the results of their proposed algorithm and compare it with the baselines.\n\nThis paper is well organized overall, but I have several concerns and questions about the paper.\n- In section 3 problem formulation, the authors formulate the problem as partially observable constrained MDP, and assume that the agent does not know the constraint specification $C$. However, in page 4, the description of policy network, it appears that $C$ is used when calculating the $M_B$ matrix. If $C(s_t,a_t;x)$ is assumed to be given from the environment, this is considered to be a strong assumption. I would like to receive detailed answers from the authors on this part.\n- In Page 4, last paragraph of constraint interpreter (“For the sequential constraints with … ”), I don't understand what this paragraph exactly means. Does that part contain more detailed considerations, rather than just using the LSTM to handle the sequential constraints?\n- (Probably related to the first question) I think that the budgetary constraints should consider history as well as sequential constraints. For example, considering the budgetary constraints in Figure 1, it is not possible to determine whether the constraint has been violated with only observation $o_t$, and it is possible only with the history information. Also, if this is because $C(s_t,a_t;x)$ is provided from the environment as in the first question, I think that it is not necessary to use LSTM for sequential constraints as well.\n\nFor the experiments,\n- In Table 2 (a)-(c), authors compare the performance of methods with $\\Delta_{RC}$. However, $\\Delta_{RC}$ depends on the $\\alpha$ value, and an appropriate alpha cannot be determined. Moreover, reward and cost represent completely different values and have different scales. I think it is inappropriate to use $\\Delta_{RC}$ as an evaluation metric.\n- It seems more natural to compare with Constrained Policy Optimization (CPO) than with FPO as a baseline algorithm. Is there any special reason for using FPO as a baseline instead of CPO? And, in the case of FPO, the result is likely to be very different depending on the value of fixed penalty, what values did you experiment with fixed penalty?\n- In Figure 4 (b), I don't understand why the agent's view appears like that figure (Why is most of the area represented in black and the agent position does not appear?).\n- In the POLCO algorithm, the constraint interpreter was pre-trained with collected data, but I wonder the result without the pre-training process. Also, for the results of baseline algorithms, are they also the results of using data equally? (including the pre-training process)\n\nThe main difference between the problem proposed in this paper and the existing constraints reinforcement learning problem is that constraints are given in the natural language. However, the author simply pre-trained a constraints interpreter that encodes a given constraint in natural language using additionally collected data without considering the RL point of view for this part, and combined it with the one of the existing constrained reinforcement learning algorithm PCPO. Therefore, although I acknowledge the considerable work for introducing a new safe RL environment with natural language constraints, I think that the contribution is insufficient unless consideration of natural language constraints is added.", "extra_scores": {}, "extra_text": {}}, {"review_id": "-rNCFe-v8am", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposed an algorithm to learn a policy when provided with natural language constraints. The paper defined a navigation task called Hazard World, in which an agent navigation on the map to collect items. The authors defined three types of constraints to restrict agents to visit certain states: 1. budgetary constraints, 2. relational constraints and 3. sequential constraints. The three constraints are described in natural language. The authors proposed a two-step solution. In step one, the algorithm learns a mapping between a natural language constraint to an intermediate representation. In step two the algorithm takes the intermediate representation to learn a policy that satisfy the constraints.\n\nPros: \n1. The research problem here is interesting. Using natural language to specify constraints makes a better UI between humans and robots.\n2. The way the authors categorize the constraints and the collected dataset by the authors are contributions to the research community.\n\nCons:\nThe scientific contribution of the proposed algorithm is rather limited. I don't see the uniqueness of the role played by natural language in the algorithm.\n\nDetailed Comments:\nThe paper proposed to first learn a mapping from a natural language constraint to an intermediate representation (step 1), and then learn a policy that conditioned on this intermediate representation (step2). Natural language does not play any role in step 2; and step 1 is just a classification of natural language into a pre-defined structured representation (the mask and the threshold). The constraints do not need to be natural language, it can be images, video or anything. I understand that this could be a good thing, but what is the scientific contributions here? The idea of mapping natural language to structured representation is not new (e.g. to build chatbot, people map language to structured dialogue states, and then learn policies conditioned on dialogue states), and the idea of using projection-based constraints policy optimization is not new (since the authors use a existing work from Yang et al., 2020b).\nI think the paper would be stronger (in terms of scientific contribution) if it makes some efforts on the following direction:\n1. How would we learn a policy without the labels for the constraints interpreter. (The author mentioned that their model is end-to-end but due to sample efficiency they pre-train the interpreter with labels)\n2. If labels for the constraints interpreter is a must, then how well does it generalize to new environment and new natural language constraints, and how robust is the optimization in step 2 with respect to the errors in step 1. Figure 5 partially answers this question. However, it contains only one new environment, and the natural language constraints were selected from Hazard World, which was used to train the interpreter.\n3. Make use of the characteristics of the natural language when designing the algorithm. Currently, the natural language does not play an important role in the algorithm. The language constraints are simply encoded by a LSTM and feed to a set of classifier. However, natural language is compositional. It is possible that there are more than one constraints and thresholds in one sentence. How to design a sample efficient algorithm for this scenario? Currently the algorithm only supports one constraint/threshold at one episode.\n4. How would we learn a policy if the cost C(s_t, a_t) is unobservable at each step (so that the agent has to infer the cost from natural language constraints). \n\nThese are just some thoughts that I feel that may be interesting, but of course, the authors would have a better idea of what is a better follow up direction for this work. My main point is that I feel that the authors haven't exploited enough the uniqueness of this \"safe reinforcement learning with natural language constraints\" problem, and the proposed solution is rather straight forward. However, I do acknowledge that this is a very hard problem and the authors made a nice step tackling this problem.\n\nQuestion:\nIn Section 4.2 the authors mentioned that \"Our policy model described in the previous section is end-to-end differentiable\". How do you backpropagate through the threshold h_C if it is trained end-to-end? \n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The goal of the paper is to learn policies that can solve a given task while adhering to certain constraints specified via natural language. The paper closely builds upon prior work on constrained RL and passes the representation of natural language constraints by pre-training an interpreter. Experiments are done in a new proposed 2D grid-world benchmark. Although reviewers liked the premise, the main issue raised is that the way natural language constraints are handled is no different from the way it is done in prior work on constrained RL. The authors provided the rebuttal and addressed some of the concerns regarding paper details. However, upon discussion post rebuttal, the reviewers and AC feel that the paper does not provide clear scientific insight because the natural language part is processed separately from the policy learning part. We also believe that the paper will immensely benefit with results in more complex environments beyond the 2D grid-world. Please refer to the reviews for final feedback and suggestions to strengthen the submission.
Decision: Reject