Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: 
Summary:
 
This paper introduced a general framework that incorporates multi-object reinforcement learning(MORL) perspective for constrained reinforcement learning to a policy set, called Pareto front, that meets the constrained. The author has instantiated a method based on the previous method MO-MPO. Compared to previous Lagrangian-based approaches, the proposed method has advantages in solution quality, stability, and sample-efficiency in few empirical environments.

##########################################################################
Reasons for score: 
 
Overall, I'd vote for rejection for this paper. My major concerns lie in two aspects: 1) the innovation is minor compared to the previous method(MO-MPO). 2) the empirical evidence for the proposed method is marginal to me. Hopefully, the authors can address my concern in the rebuttal period.  

Pros:
	1. The paper is straightforward and clear to read.

Cons:
	1. The motivation of the proposed framework is not strong. Besides that, the improvement of constrained MO-MPO is minor.
	2. The domains ( runs & walk) didn't illustrate the idea well. It is not clear to me what the constrained is. The advantage of constrained MO-MPO over lagrangian-based approaches is marginal. The advantages could vary from sed choice. 
##########################################################################
review_title: Recommendation to Reject

final_score: rating: 7: Good paper, accept
review_content: The paper is interested in reinforcement learning where one needs to satisfy constraints (for instance energy spent) in addition to maximizing rewards. The proposed approach proposes to extend any method able to approximate the Pareto front of optimal policies by also learning portions of the front that satisfy user constraints (given that only small portion of the front may be valid). In particular, a previous approach MO-MPO is used to approximate the Pareto front in a first step and a second step learns another policy find preference vectors satisfying given constrains. Experiments are carried in two environments and show that the proposed method outperforms Lagrangian baselines (e.g. optimizing alternatively for reward and constrain violation) in particular in settings where constraints are harder to satisfy.

The main strength of this paper are its sound experimental results and overall improvements compared to the baselines studied. In addition, the paper is well-written and tackle a highly relevant problem. The weakness of the paper is that the method is a bit incremental compared to MO-MPO and that it may be hard to reproduce without code.

For these reasons, I recommend accepting the paper given its improvement over Lagrangian methods and its clarity. It would clearly help if the code allowing to rerun comparison could be provided so that future work can compare with the proposed approach.

# Additional feedback and suggestions

- The problem of concave Pareto front is indeed problematic for linear scalarization but recent work in scalarization may allow to bypass the issue [1]
- Figure 1: typo "contrianed"
- Figure 1: Is there a reason or an intuition on why outliers appear in MPO Lagrangian (bottom-left)? Is it because of the unstable behavior explained in Fig 2? 
- "but as the constraint threshold becomes more difficult to satisfy, our approach dominates", except for "point push" this could be precised. Also is it because the front is closer to a convex?
- Figure 4: I suggest drawing a vertical line with the constraint at -3000, -2000 and -1500. Also it may be clearer to use a normalized action norm cost (e.g. -2.0 instead of -2000) to make it easier to compare with the cost constrain.

[1] Random Hypervolume Scalarizations for Provable Multi-Objective Black Box Optimization. Golovin et al. ICML2020

review_title: Recommendation to Accept

final_score: rating: 6: Marginally above acceptance threshold
review_content: In Reinforcement Learning (RL) it is common that one searches for policies that optimize the reward and that meet certain constraints. A common approach for handling the constraints is Lagrangian relaxation, i.e., to incorporate the constraints in some way into the objective. In this submission, a different approach is suggested, in which the constraints are treated as additional objectives and then the Pareto front of policies is computed. This is reasonable as the optimal policy for a single objective that satisfies certain constraints must be Pareto optimal when the constraints are interpreted as additional objectives. It is then explained how the new approach can be implemented and several experiments are conducted on well-known data sets. It turns out that it is indeed beneficial for some data set to compute the set of Pareto-optimal policies instead of using Lagrangian relaxation because in some cases Lagrangian relaxation finds only the extreme policies while the Pareto front also contains intermediate policies that balance the criteria better.

I find the suggested approach very natural. The connection between constrained problems and the Pareto front is not very surprising and it is well-known and used extensively in the field of multi-objective optimization (for different problems). The writeup is rather condensed. I found it hard to follow Section 4 in detail (but I am not an expert in the field of RL). The experimental results look convincing. It is hard for me as as a non-expert in RL to give a clear recommendation for this submission. It is not clear to me if there is a novel contribution except for the idea to look at the Pareto front instead of a Lagrangian relaxation. It does not become clear to me if there is any non-trivial contribution in Section 4 or if the implementation follows more or less along the lines of previous RL implementations.

I got the impression (on page 3 this is said more or less explicitly) that the submission makes the assumption that for any Pareto-optimal solution there is a preference vector for which this particular solution is optimal. I don't think that this is true. This way one can characterize the convex hull of solutions but in general there can be Pareto-optimal solution that do not lie on the convex hull and for which there does not exist such a preference vector.
review_title: Contribution of the submission not clear enough

Meta-review: Considering reviewers' comments and comparing with similar papers recently published or submitted, this is a good paper but hasn't reached the bar of ICLR.  We believe that the paper is not ready for publication yet, and strongly encourage the authors to use the reviewers' feedback to improve the work and resubmit to one of the upcoming conferences.

Decision: Reject