Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper is proposing an end-to-end reinforced approach of sequential task learning incorporating constraint using the natural language called POLCO standing for Policy Optimization with Language Constraints. 

The authors propose a novel grid-world and robotic manipulation environment, called HazardWorld, to illustrate their proposal. Each world is composed of a variety of cells symbolizing a different level of safe reachability, in the case of robotic manipulation the state is discretized for applying the proposedapproach. Then, a set of three different constraint types expressed in natural language are studied: Budgetary, Relational, Sequential and correspond to different types and levels of compositionality. 

Constraints are translated into reachability space through a CNN encoder and transfer into the policy and can be learnt in an end-to-end manner.  Reaching an unsafe place would produce a large negative reward. Proposing this setting, natural language is seamlessly incorporated into a non-reachable space of the considered environment.

The paper introduces a two-step process for safety policy training. First, a random policy explores the environment and grasp reward in accordance to language specified set of constraints. A constraint mask over the environment definition is computed using binary cross-entropy. Second, the policy is trained using PCPO which allow optimizing a policy by maximizing the reward function while satisfying the learnt constraint mask through re-projection in the satisfiability policy subspace.

The overall policy architecture is an original composition of state of the art differentiable building blocks suitable for differentiable training.
main_review: The paper addresses an important problem of sequential decision learning which is safety at training and test time and its seamless specification.
The proposed method proposes to use an interesting modality which is the natural language for this matter.
The model is trained in a differentiable manner and is tested on a large set of possible constraints expressing various levels of compositional in language.
The two-step learning process is novel and intuitive and the policy architecture is leveraging the know components of differentiable programming.
The experiments show convincing results of the capability of the method to handle the various types of constraint, however at the cost of an initial random exploration, which may be considered as contradictory to the overall purpose of the work.
The main concern is about the generalization of such an approach including stochastic and/or partially observable environments.
At the current stage, the paper hardly allows appreciating the possible impact of the approach, especially the constraint mask module, on such a more realistic scenario.

limitations_and_societal_impact: No section is explicitly addressing these concerns. However, as the question of the safety of sequential decision algorithms is the subject of this paper, it seems largely related to this subject of concern.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper presents a new multi-task environment for learning the agent with natural language constraints. The author breaks away from existing approaches that specify the constraints in mathematical form and extends it to specify the constraints in natural language. Their environment provides grid-world and robotics environments, and various kinds of constraints for safe RL. Finally, the authors propose a model for constraint policy optimization applicable to language constraint settings and show the generalization effect of language constraints.
main_review: This paper is well written and addresses an important problem of language constraints for safe RL. I agree that language constraints are intuitive and easy-accessible, and I think specifying the constraints in natural language can provide various advantages when applying the constraint to a real-world problem. The detailed comments and questions are as follows:

1. In the result of experiments (Figure 4, 5, 6, 7) what is the maximum performance that can be achieved while satisfying the cost constraint? It would be helpful to understand how well the proposed algorithm performs. 

2. The authors define three types of constraint and are modeling two types to consider different types of constraint. Is there any reason to model this way? (Is it not possible with a single model to consider all constraints as the same?)

3. In safety training, the authors mention that PCPO was used for policy learning (page 7 lines 260-261), and in safety evaluation, CPO was used for HazardWorld-robot (page 7 line 271). Is there any reason to use different methods?

4. (Page 9, lines 326-337) In the ablation studies, it is necessary to add a detailed explanation, such as exactly which domain was tested.

limitations_and_societal_impact: Yes, the authors have adequately addressed it.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper contributes to the area of constrained RL by adding machinery to interpret and apply constraints specified in natural language.  They also design and implement a testbed world to evaluate such systems.  In this world, they demonstrate that their approach consistently outperforms other methods, both in cumulative reward and constraint violations.  They also use an ablation study to demonstrate the need for each component of their approach. 

main_review: This paper is a clear contribution to the field and merits publication.

One thing that I would like to see, however, is a comparison with other constrained RL approaches that are given the constraints in their native formats, e.g., logic rules, finite automata, etc.  It's reasonable that this paper's approach might not always win in these tests, but they still maintain the advantage of NL-formatted constraints.

limitations_and_societal_impact: The authors briefly describe limitations of their work on page 9. 

They state in the Checklist that they do not see potential negative impacts, but do not address it in the main body of the manuscript.
review_title: null

Meta-review: This paper presents a framework wherein constraints (particularly safety constraints) on a reinforcement learning agent can be specified using natural language. The paper is well written, clear, and was well-received by the reviewers, all of whom recommend acceptance. The AC finds the idea of natural language constraints particularly compelling, and strongly supports acceptance.

One additional related work is "Preventing undesirable of intelligent machines", which proposes an "interface" that the users of ML/RL algorithms can use to define safety constraints. It seems like the method proposed here would plug-in directly as such an "interface", making this paper a valuable contribution to Seldonian methods (perhaps beyond RL) as well.
Decision: Accept (Spotlight)