Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: Summary:
The paper addresses how to learn policies for tasks in which constraints are specified in natural language. Towards this, the paper proposes a model that encodes the different types of natural language constraints into intermediate representations that model both spatial and temporal information between states. Then, they use this as input along with the observation to produce an action at each time step for a safe trajectory. 
They also propose a new benchmark (Hazard World) which is inspired by the 2D MiniGrid environment. They show the efficacy of their models on this new benchmark outperforming several baselines. 

Strengths:
1. The authors address an important problem of modeling natural language constraints for safe RL. Unlike previous work, which requires manual specification of constraint or rule bases constraint, specifying natural language constraints is more intuitive and scalable. 
2. The proposed model which breaks down the language constraint into a constraint mask and constraint threshold is intuitive. The experiments also demonstrate 
3. The authors also propose an interesting benchmark to evaluate methods for safe RL under natural language constraints. Even though the environment is built on top of 2D grid world with simple action spaces, constraints are still specified in free-form natural language. The benchmark will be useful for the wider RL community. 
4. The paper is well written and easy-to-read. 


Weaknesses: 
1. The proposed models seem to be engineered for the 2D grid-like environments used for evaluation. I'd be interested in knowing how to generalize the models to other types of environments and observations (for instance navigation environments with 3D ego-centric observations). The authors mention that they describe an extension to 3D scene inputs in the appendix but I didn't find those details. Please let me know if I missed it in the appendix. 

2. Mapping constraints into the current observation is a limiting assumption. Can all constraints be mapped to current observations? 
 - 2.1 How can the proposed model deal with constraints that might be violated due to partial observability? For instance, if the lava is behind the agent's field of view, how will the agent incorporate those constraints. 
 - 2.2 There might be other temporal constraints that need to be modeled. For instance, defuse a bomb in less than 5 steps like constraints that don't necessarily describe a visual constraint. How can a model handle such constraints? 



review_title: Important problem; Novel formulation

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper presents a new test environment, Hazard World, for learning the safe reinforcement learning agents with given natural language constraints. In this problem, the goal of the agent is to find an optimal policy that maximizes the cumulative rewards while satisfying the constraints given in natural language. The authors introduce the model that contains the following two separate components; constraint interpreter for encoding the language constraints and policy network for learning the RL agent. Finally, they report the results of their proposed algorithm and compare it with the baselines.

This paper is well organized overall, but I have several concerns and questions about the paper.
- In section 3 problem formulation, the authors formulate the problem as partially observable constrained MDP, and assume that the agent does not know the constraint specification $C$. However, in page 4, the description of policy network, it appears that $C$ is used when calculating the $M_B$ matrix. If $C(s_t,a_t;x)$ is assumed to be given from the environment, this is considered to be a strong assumption. I would like to receive detailed answers from the authors on this part.
- In Page 4, last paragraph of constraint interpreter (“For the sequential constraints with … ”), I don't understand what this paragraph exactly means. Does that part contain more detailed considerations, rather than just using the LSTM to handle the sequential constraints?
- (Probably related to the first question) I think that the budgetary constraints should consider history as well as sequential constraints. For example, considering the budgetary constraints in Figure 1, it is not possible to determine whether the constraint has been violated with only observation $o_t$, and it is possible only with the history information. Also, if this is because $C(s_t,a_t;x)$ is provided from the environment as in the first question, I think that it is not necessary to use LSTM for sequential constraints as well.

For the experiments,
- In Table 2 (a)-(c), authors compare the performance of methods with $\Delta_{RC}$. However, $\Delta_{RC}$ depends on the $\alpha$ value, and an appropriate alpha cannot be determined. Moreover, reward and cost represent completely different values and have different scales. I think it is inappropriate to use $\Delta_{RC}$ as an evaluation metric.
- It seems more natural to compare with Constrained Policy Optimization (CPO) than with FPO as a baseline algorithm. Is there any special reason for using FPO as a baseline instead of CPO? And, in the case of FPO, the result is likely to be very different depending on the value of fixed penalty, what values did you experiment with fixed penalty?
- In Figure 4 (b), I don't understand why the agent's view appears like that figure (Why is most of the area represented in black and the agent position does not appear?).
- In the POLCO algorithm, the constraint interpreter was pre-trained with collected data, but I wonder the result without the pre-training process. Also, for the results of baseline algorithms, are they also the results of using data equally? (including the pre-training process)

The main difference between the problem proposed in this paper and the existing constraints reinforcement learning problem is that constraints are given in the natural language. However, the author simply pre-trained a constraints interpreter that encodes a given constraint in natural language using additionally collected data without considering the RL point of view for this part, and combined it with the one of the existing constrained reinforcement learning algorithm PCPO. Therefore, although I acknowledge the considerable work for introducing a new safe RL environment with natural language constraints, I think that the contribution is insufficient unless consideration of natural language constraints is added.
review_title: Official Blind Review #3

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper present an experiment of safe reinforcement on a 2D grid-word where the safety constraints are specified in natural language instead of being specified formally. The justification of this system is to allow non-experts to train agents using safe-RL.
According to the authors: "The key challenge lies in training the agent to interpret natural language and naturally adhere to the constraints during exploration and execution".  
The proposed system is made of two parts: a constraint interpreter that is (mostly) trained in a supervised way with Amazon Mechanical Turk to translate natural language orders into grid-world ad-hoc constraints and a policy that is trained through PCPO, a TRPO-like constraint-aware policy optimization algorithm.
That's certainly a nice piece of engineering, but honestly my first sentiment when reading the motivation of this paper was astonishment. If the agent is already able to understand complex natural language statements about its environment during its exploration/training phase, why would it need further training to apply these constraints on a simple grid-world ?

If the natural language understanding (NLU) task has to be handled before the agent's exploration/training phase, we are facing a concatenation of two problems: NLU then Safe-RL, not a new problem involving tightly NLU and safe-RL.
Even if some publications were already made on that topic, training a agent to properly follow natural language orders, that may include constraints, during its execution phase is yet an unsolved problem that requires a real fusion between NLU and RL that is of scientific interest.
review_title: Hybridization of two problems (NLU and Safe RL) is not clearly a new problem

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper proposed an algorithm to learn a policy when provided with natural language constraints. The paper defined a navigation task called Hazard World, in which an agent navigation on the map to collect items. The authors defined three types of constraints to restrict agents to visit certain states: 1. budgetary constraints, 2. relational constraints and 3. sequential constraints. The three constraints are described in natural language. The authors proposed a two-step solution. In step one, the algorithm learns a mapping between a natural language constraint to an intermediate representation. In step two the algorithm takes the intermediate representation to learn a policy that satisfy the constraints.

Pros: 
1. The research problem here is interesting. Using natural language to specify constraints makes a better UI between humans and robots.
2. The way the authors categorize the constraints and the collected dataset by the authors are contributions to the research community.

Cons:
The scientific contribution of the proposed algorithm is rather limited. I don't see the uniqueness of the role played by natural language in the algorithm.

Detailed Comments:
The paper proposed to first learn a mapping from a natural language constraint to an intermediate representation (step 1), and then learn a policy that conditioned on this intermediate representation (step2). Natural language does not play any role in step 2; and step 1 is just a classification of natural language into a pre-defined structured representation (the mask and the threshold). The constraints do not need to be natural language, it can be images, video or anything. I understand that this could be a good thing, but what is the scientific contributions here? The idea of mapping natural language to structured representation is not new (e.g. to build chatbot, people map language to structured dialogue states, and then learn policies conditioned on dialogue states), and the idea of using projection-based constraints policy optimization is not new (since the authors use a existing work from Yang et al., 2020b).
I think the paper would be stronger (in terms of scientific contribution) if it makes some efforts on the following direction:
1. How would we learn a policy without the labels for the constraints interpreter. (The author mentioned that their model is end-to-end but due to sample efficiency they pre-train the interpreter with labels)
2. If labels for the constraints interpreter is a must, then how well does it generalize to new environment and new natural language constraints, and how robust is the optimization in step 2 with respect to the errors in step 1. Figure 5 partially answers this question. However, it contains only one new environment, and the natural language constraints were selected from Hazard World, which was used to train the interpreter.
3. Make use of the characteristics of the natural language when designing the algorithm. Currently, the natural language does not play an important role in the algorithm. The language constraints are simply encoded by a LSTM and feed to a set of classifier. However, natural language is compositional. It is possible that there are more than one constraints and thresholds in one sentence. How to design a sample efficient algorithm for this scenario? Currently the algorithm only supports one constraint/threshold at one episode.
4. How would we learn a policy if the cost C(s_t, a_t) is unobservable at each step (so that the agent has to infer the cost from natural language constraints). 

These are just some thoughts that I feel that may be interesting, but of course, the authors would have a better idea of what is a better follow up direction for this work. My main point is that I feel that the authors haven't exploited enough the uniqueness of this "safe reinforcement learning with natural language constraints" problem, and the proposed solution is rather straight forward. However, I do acknowledge that this is a very hard problem and the authors made a nice step tackling this problem.

Question:
In Section 4.2 the authors mentioned that "Our policy model described in the previous section is end-to-end differentiable". How do you backpropagate through the threshold h_C if it is trained end-to-end? 


review_title: The research problem is interesting, but the scientific contribution of this paper is limited.

Meta-review: The goal of the paper is to learn policies that can solve a given task while adhering to certain constraints specified via natural language. The paper closely builds upon prior work on constrained RL and passes the representation of natural language constraints by pre-training an interpreter. Experiments are done in a new proposed 2D grid-world benchmark. Although reviewers liked the premise, the main issue raised is that the way natural language constraints are handled is no different from the way it is done in prior work on constrained RL. The authors provided the rebuttal and addressed some of the concerns regarding paper details. However, upon discussion post rebuttal, the reviewers and AC feel that the paper does not provide clear scientific insight because the natural language part is processed separately from the policy learning part. We also believe that the paper will immensely benefit with results in more complex environments beyond the 2D grid-world. Please refer to the reviews for final feedback and suggestions to strengthen the submission.
Decision: Reject