Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The authors present a more generalized framework for handling multi-objective RL problems which merges concepts of both constrained MDPs and multi-objective MDPs with in a circular two-phase optimization where the RL policy is optimized for a set of preferences followed by the preferences being optimized for a set of constraints and repeating. The authors also claim that the resulting LP3 is more sample efficient, able to find sets of viable policies and able to handle soft constraints.
strengths_and_weaknesses: ### Strengths:

S1) The paper has generally decent flow and communicates most of its ideas clearly with sufficient background

S2) I always appreciate the process of starting demonstrations with toy problems and working up to more complex domains as it does better to establish the viability of an approach in more intuitive terms.

S3) I like the idea of addressing soft constraints or constraint bounds as opposed to strict constraints as this can relate to more realistic scenarios.

S4) The website has intuitive visualizations that help drive the key points of the paper home.

### Weaknesses:

W1) It appears to me that, despite talking about a 'multi-objective' setting, many of the experiments relate mainly to just a single reward and a single policy regularization constraint - and even that seems mostly related to actuation costs. How would this method extend to other types of objectives, and how well would the algorithm handle a more than two soft constraints - what are the associated performance and computational tradeoffs? I feel like these are important questions to address that are not addressed in the paper.

W2) I am not convinced that treating MO-MPO as a special module really constitutes a novel instantiation of LP3 in a significantly meaningful sense (at least not in a way that I'd regard a novel module) - unless I am mistaken, the extension proposed in this paper is mostly just input-state augmentation, am I wrong?

W3) It would help if the figure captions were more self-contained and descriptive, with important takeaways summarized. The inconsistency of the formatting for the caption for Figure 2 also interrupted the flow of reading.

### General notes:

Overall, I think the methods proposed here have promise, but need more thorough evaluation to demonstrate broader value. One possible exploration could be in looking at how the different reward components for the various tasks could be satisfied (for example, the gym mujoco control tasks have rewards associated with joint limits, energy consumption, task progress, etc. - so considering them all separately as constraints could make for interesting test cases)
summary_of_recommendation: While I think the work shows promise, I am as yet unconvinced of broader utility by the experimental results.

(Updated after rebuttal):
I am more convinced, given the additional results presented and additional contextualization provided. I am still not convinced on the issue of the novelty of MO-MPO, but I am willing to let it slide given that no other major concerns are raised
issues: Experiments with more than 2 objectives would be the major thing I'd hope to see before I can accept this as a 'multi' objective solution
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This work explores two problems in RL, learning a policy with constraints and multi-objective learning. Currently, known methods exist to solve each of these problems.  The authors merge solutions to these two problems so that constraints can be placed on a multi-objective MDP.  To do this, the authors define an MDP with multiple objectives, and each objective can have a constraint on it, which they title a "constrained multi-objective MDP".  The method is then derived which combines known work on multi-objective policy optimization (MO-MPO) and learning with constraints to solve this class of MDPs.  The main contribution of this work is that the approach is able to approximate the Pareto front of optimal policies by learning a segment of the front that satisfies the given constraints.  This presents the benefit of finding Pareto optimal policies that balance between satisfying constraints and task reward.
strengths_and_weaknesses: The approach of this work is quite intuitive, and the writing is clear and convincing. This work derives intuitive connections between the constrained optimization and the Pareto front.  The problem of constrained multi-objective learning is highly relevant to robot learning and could make a good contribution to CoRL.

A key strength of this paper is that the results achieved appear to be significant over baselines and the experimental evaluation is quite extensive.  I feel that this work will provide a useful contribution to the set of RL tools and will likely be used.  The clarity of this paper is also good considering the wide range and complexity of methods discussed.  The writing is well polished.

My main issue is that the method in this paper feels incremental compared to MO-MPO. The authors apply well known methods from constrained policy optimization to the multi objective framework.   The connections found between constrained optimization and the Pareto front are well known in other multi-objective problems but have not been explored in RL, to my knowledge.  Overall, this does help improve the novelty.
summary_of_recommendation: I am learning towards recommending acceptance of this work and think that it will be useful to the audience of CoRL.  The problem is highly relevant, and the connections made are intuitive.  The evaluation of the method is convincing and the writing is clear.
issues: Mainly concerning the novelty over MO-MPO.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper proposes a high-level framework called Learning Preferences and Policies in Parallel (LP3), which can derive new approaches basing on existing methods. Based on the idea of learning two modules separately, this paper solves the problem of constrained multi-objective reinforcement learning with better performance. The article has some innovative significance, and the experimental scheme is basically reasonable. What’s more, the experimental results and data in this paper are relatively good and reliable. In the open simulation environment, the RL system is verified. From the perspective of the algorithm, the verification is obtained, which makes a certain contribution to the development of Reinforcement Learning.
strengths_and_weaknesses: This paper proposes the LP3 framework, considering the RL algorithm from the top-level perspective, which improves the efficiency of algorithm promotion, and at the same time has a positive significance for the development of RL. However，for the mathematical concepts and derivation of the traditional RL algorithm, the description in this paper is slightly cumbersome. What's more, the definition of the framework in the title of this paper is not reasonable, because the framework is not universally validated.
summary_of_recommendation: This article proposes a new RL insight (introducing learning preference), and tries to explain the idea with a general framework. However, the transferability of this framework to other Constrained Multi-Objective Reinforcement Learning problems remains to be verified. Therefore, it is recommended to reconsider the definition of the framework.
issues: 1.Lines 105-106, for the traditional definition of non-dominated policy, the meaning of these two sentences in this position and the author's intention are not clear enough. 

2.Lines 214-215, the reason for using uniform-discrete distribution is not clear enough. 

3.The length of the basic knowledge in RL should be reduced, and the algorithm implementation and experimental results should be analyzed more. For those who are new to this professional field, this paper can also cite classic representative documents to convey basic theories.   
4. From Figure 1 left, the framework composed of module 1 and module 2 gets a loop iteration, but in fact, only a limited number of combinations are made in the paper. Strictly speaking, this cannot be called a framework, in other words, the description of the framework in the paper is a little exaggerated.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This work proposes a general framework to leverage multi-objective learning for constraint RL, for which Lagrangian relaxation is one possible option. Further, the authors propose two advances to this setting. First, instead of linear scalarization they use Multi-Objective Maximum-a-Posteriori Optimization (MO-MPO) to allow for a distributional approach that combines objectives. Second, they introduce the use of a preference distribution in place of a single preference. This additionally enables to find a set of constraint-satisfying policies that offer different levels of trade-off between constraints and task performance. 
strengths_and_weaknesses: Using multi-objective optimization for constraint RL is an intuitive direction. While the idea in itself is not novel, this work is still introducing the idea to a new area and is at the same time approaching an important field in deep RL and Robotics.
Overall, this work is well motivated, structured excellently, and provides sufficient background information to understand all aspects presented throughout every chapter. I very much enjoyed reading this paper and the general approach itself was apparent. 
 
An important weakness with the current approach is the applicability to physical robotics systems, which the authors state in their conclusion as interest for future work. However, given the currently required sample complexity, it might be challenging to directly apply the algorithm in its current form. Other than that, the work does not put much emphasis on demonstrating the limitations of the framework. Only the appendix contains additional discussion, e.g. for the `point` task the constraints are consistently violated due to the Q-function approximation. 
summary_of_recommendation: First and foremost, the technical quality as well as the clarity of this work is well done and there is little to complain about. 
The novelty of the proposed framework is mainly given by transferring the existing idea of multi-objective learning for constraint optimization to the (deep) RL space. While this is not a groundbreaking idea, I think it has its place in the given setting.

Despite the good writing, I am still left with some open points: 
- For the preference-conditioned MO-MPO, the replay buffer transitions are relabeled based on the current preference distribution $p(\epsilon)$. Do you have any results that demonstrate that this indeed leads to improved stability and sample efficiency?
- While the approach can learn a set of constraint-satisfying policies, how would one choose an appropriate trade-off. In simulation one option would compare multiple policies, albeit, this is assuming a human expert that can search the preference distribution. Considering a physical system, this might not even be an option, especially when trying to avoid potentially system-harming/-damaging trade-offs. 
- For easier comparison LP3 [MO-MPO-D] uses equality constraints to find a single solution. However, wouldn't that lead to a different optimization problem in the first place? 
- Figure 4 shows how the learned set performs, when using inequality constraints, albeit to me it seems the coverage of the Pareto front is quite seed dependent. Exploring this more would also be in the interest of determining the limitations of the framework.
- For the toy domain I did not find any details about the initialization or seed choice. It might as well be that LP3 [LS] is performing poor simply because of the chosen seed and not necessarily due to the concave Pareto front. 
- For the reward shaping baseline Calian, et al. consider quite large steps for the chosen weights. I expect when tuning those weights for the task the performance for this method will increase and therefore present a better baseline.
- The appendix also touches slight upon using multiple constraints. How would you expect your method to perform when introducing a larger number of constraints, which for some tasks can be easily possible.    
issues: ### Minor Issues 
- One of the main references is cited incorrectly: D. A. Calian, D. J. Mankowitz, T. Zahavy, Z. Xu, J. Oh, N. Levine, and T. Mann. Balancing
constraints and rewards with meta-gradient d4pg. arXiv preprint arXiv:2010.06324, 2020. --> ICLR 2021
- Table 1: LP3 [LS] Humanoid walk is neither bold nor marked with a $p\leq0.05$ for Welch's t-test
- Comparing the numbers reported in this work and the numbers reported in table 7 by Calian, et al. I cannot find a direct match, most of the results are close but not identical.
review_title: null

Meta-review: This paper proposes one general framework which can deal with the constrained multi-objective RL problems. Reviewers own the consistent comments that this framework is promising and useful in some sense. Experimental results are also convincing. However, there are still some concerns raised by reviewers, which need to be well answered to make this work much more qualified. For example, multi-objective setting in the experiments sounds like not so complex and general. Some empirical tuning and algorithmic design should be presented more clearly. 

Although there are still some minor issues in this paper, most of reviewers think authors have addressed the majority of the concerns during the rebuttal period and it has met the quality of CoRL paper. 
Decision: Accept (Poster)