Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper presents a multi-objective method for linear preferences. The key concept used for the design of the approach is that there always exists a common ascent direction for all the objectives. The authors leverage this property for introducing a novel loss function that can be integrated into different DeepRL methods. The proposed approach is simple and proved to be effective in the proposed experiments.
main_review: While the idea is interesting, my main concern is about the novelty and the appropriate reference of previous work. As far as I know, the first-order necessary stationary condition was already used together with REINFORCE in previous work (see [31]). You mentioned [31] in the intro but I think a more detailed comparison is due. In [31], the authors already discussed the existence of a common ascent direction and provided the associated quadratic programming approach for the computation. They also presented gradient-based approaches for the computation of first-order stationary policies. I think a deeper discussion of the differences between your and their work is necessary. Given this, I think it is also necessary to add their algorithms to the empirical comparison. 

The domains considered in the experiments are standard but I would have appreciated a more extended comparison. For example, why do you focus on domains with very few objectives (3 domains with 2 objectives and one with 5)?

Finally, there are a few typos in the paper that makes the reading complicated, especially in section 3.
- The definition of the value function $v : |S| \to \mathbb{R}^|S|$ is not precise. You should use the notation $v \in \mathbb{R}^{|S|}$ or $v : S \to \mathbb{R}$. Same for the multi-objective case.
- Why are you considering finite state space and continuous actions?
- Eq. 3 is not clear. In particular, the first equality is not correct. There is an expectation w.r.t. the initial distribution missing and probably an additional $w^T$. Please clarify it.
- In the proof of lemma 1, you refer to definition 2. It should be Def. 1
- Based on what you mentioned at the end of section 2.2, there is a constraint missing in definition 2 ($w$ should belong to the simplex).
summary_of_the_review: My score is justified by the doubts I have about the novelty, comparison with previous work, and clarity of the writing.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper introduces a policy gradient approach to multi objective RL. The authors learn show that an optimal solution for a specific reward vector can be found by minimizing a constrained quadratic optimization problem. They propose to solve this problem using projected gradient descent. Then, the authors offer an efficient solution for optimizing the Pareto front through "Pareto Policy Adaptation" loss function, which trades off learning the Pareto front and the policy. The authors conduct experiments and ablations studies on four benchmarks, validating their method with improvement over current sota approaches for MORL.
main_review: This work is written well and easy to follow. I didn't find major drawbacks in the paper, and overall the theoretical results seem correct (yet not very novel or fundamental). Overall the work seems solid to me. A few items I would like the authors to address:
1. In the proof of Lemma 1, $d^\pi(s,a)$ can be 0. This would make the lemma incorrect I believe.
2. In Thm 1, $E_{\pi_\theta}$ should be $E_{d^{\pi_\theta}}$
3. In Thm 3, $g_m$ and not defined. I'm assuming these are $g_m(\theta) = \nabla_\theta E_{d^{\pi_\theta}} r_m(s,a)$.
4. The authors wrote: "We start by interpreting the optimal solution of Problem PA as a function of the policy parameters $\omega^* = \omega^*(\theta)$". Can you elaborate on this? Why should the policy and preference parameters have the same parameters?
summary_of_the_review: See review
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper presents a new method for solving MORL problems. The main contributions are: a generic gradient descent MORL algorithm for finding multiple solutions, and another algorithm to retrieves preferences from the given solutions. The authors also present theoretical proofs of their method.
main_review: The paper is well written and the method is well presented. The introduction covers well prior work. However, I think that the authors do not properly compare their method to the existing ones discussed in the introduction, and I find it hard to distinguish between what is really novel and what has already been done. Furthermore, there are some other issues that should be clarified.

1) The idea of convex coverage set (CCS) is well known in MORL since years. Roijers et al. have published several papers on the subject, e.g., "A Survey of Multi-Objective Sequential Decision-Making" (2013) and "Computing Convex Coverage Sets for Faster Multi-objective Coordination" (2015). Therefore everything up to line 150 is already known (correct me if I am wrong). The novel part should be the gradient descent leveraging on the fist-order condition, resulting in a method that can be applied to a large variety of RL algorithms. I think it would be better to put the CCS part in "Preliminaries", and keep the novel part in Section 3 to highlight it.

2) The first-order condition is a necessary but not sufficient condition for Pareto optimality. You discuss this but only at the very end, as well as the other limitation (PPA can find only convex frontiers). It would be better to write a short paragraph at the end of Section 4 to highlight this. Similarly, you should mention that the Deep Sea Treasure version used for the experiments is not the original version [1]. The original has a convex frontier, which PPA cannot solve. 

[1] Vamplew et al., "Empirical evaluation methods for multiobjective reinforcement learning algorithms" (2011)

3) Third, the adaptation part is highly related to Envelope Q-Learning, which you use as baseline in the experiments. You should discuss similarities and differences more in-depth, given that both PPA and EQL retrieve unknown preferences given a solution, and their losses are very similar. For instance, the first term of your Eq. 8 is the same of EQL Eq. 6, while the second term differs from EQL Eq. 7 (I am referring to EQL equations in its paper).

4) It is a bit misleading to claim in the abstract that the final PPA method learns under “unknown preferences” since the objective in (8) does require $\omega$ to evaluate the loss.

5) In Figure 2(b), why PPA is guaranteed to converge to in this example?

6) For the MuJoCo experiments, the paper only considers the more basic baselines (nA-PPA and fixed-preference agents). It would be helpful to compare PPA with other stronger baselines, such as PGMORL (Xu et al., ICML 2020).

7) You should discuss PPA limitation more in details. E.g., PPA directly maximizes the expected return for a given preference or a preference distribution, which means when the environmental preference changes, one has to finetune the model with some extra cost. However, other value-based baselines align the optimal policies with preferences during a single training procedure and can respond with the corresponding optimal policy to any given preference without fine-tuning.

Finally, some questions regarding the experiments.

* Why did you not use the hypervolume as evaluation criterion? It is probably the most common and accurate criterion in MORL literature and could easily replace UT. UT is computed over some samples preferences (how many?), and you could use the same preferences to compute the hypervolume of the corresponding solutions. For MOSM you could use a Monte-Carlo approximation of the hypervolume.
* Why did you not compute PD and UT against baselines for MuJoCo experiments as well?
* Why did you choose to reward the agent for moving in circular paths as second objective for MuJoCo? Usually, the first reward is the distance from a goal, while the second is the energy consumption (see Abdomaleki et al., "A Distributional View on Multi-Objective Policy Optimization" (2020)).
* I am a bit confused by the results. In Table 1, PPA does well under PD and UT criterion. In the plots, results are mixed. PPA does well in MOGW, but is not the best in DST. Then in MuJoCo experiments plots are very noisy, and performance curves sometimes increase and decrease multiple times. For instance, why does w=0.5 increase and then decrease in Hopper? The same curve then performs terribly in Cheetah, why? And why w=0.25 show opposite behavior (decent on Cheetah, terrible in Hopper and Ant)? I am not convinced by your intuition that "given the complex inter-dependence between the two rewards, simply hand-crafting a preference vector is far from optimal". The rewards are the same for all domains, why would Cheetah be different?

summary_of_the_review: The idea of the paper is interesting, but there are many things that the authors should clarify, both in regards to limitations of the methods and evaluation (metrics, domains, comparison against other baselines).
For these reasons, I vote for its rejection.
review_title: null

Meta-review: While this paper has divergent reviews, reviewer hSRE has by far the most detailed review, seems clearly the most informed on the subject, and is the least supportive.  The main issues with the paper seems to be the degree of novelty and reviewer hSRE's feeling that the results on MojuCo are unclear and not explained in the paper.  But this alone does not seem like an adequate reason for rejection and hSRE seems happy with the other aspects of the paper.  Some of hSRE's original complaints do not concern me, such as the fact that first-order stationarity does not imply Pareto-optimality.  I am recommending a poster.
Decision: Accept (Poster)