Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: In this paper, the authors propose a new multi-objective RL algorithm named Preference-Driven MORL (PD-MORL). PD-MORL uses the preference to align the Q function updates. Theoretical analysis shows that the new preference-driven optimality operator ensures convergence to the optimal Q function. Experiments on classical benchmark environments and multi-objective variants of MuJoCo environments are provided.
strength_and_weaknesses: Strength: 
The idea to align the preference vector better with the Q function is interesting and relevant to the multi-objective RL community. 

The writing is mostly clear and easy to follow.

Weakness:
There are important baselines missing in the empirical evaluations. The envelope Q-learning algorithm by Yang et al. is only compared in DST and FTN but missing from MuJoCo environments. 

Another important baseline is the conditional network with diverse experience replay algorithm in Abels et al. (Dynamic Weights in Multi-Objective Deep Reinforcement Learning). I believe adding both baselines would make the comparisons more informative. 

There seems to be a mismatch between the practical version of the algorithm and the theoretical version. Algorithm 1 uses the interpolated preference rather than the raw preference. Can the authors comment on this choice and how the theoretical result extends to this different update rule?

There are some details missing in the description of the algorithm. The preference interpolation is an important part of the algorithm but its description is lacking. I suggest having a more detailed exposition on this, including how the key preference points are selected, how the normalization procedure is done, and a mathematical expression of the interpolation function.

clarity,_quality,_novelty_and_reproducibility: Most of my clarity comments are in the above section. 

I have one question about using the cosine similarity in the preference-driven optimality operator. The cosine similarity is closely related to the inner product of the preference vector and the Q value vector, which is also in the operator itself. So is it really important to have the $\omega^t Q(s’, a’, \omega)$ term in the update rule, or is the cosine similarity itself sufficient?

In theorem 4, the authors should define what Q* is as it is not clear in the multi-objective setting.

summary_of_the_review: Because of the important baseline results missing, I am not convinced that the paper in its current form is ready to be published.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes PD-MORL, which extends Envelope (Yang et al., 2019). The major difference is illustrated in Equation 5 in the paper: an extra cosine similarity term is added, and the supremum is only taken over actions, not preferences (compared to Envelope). The authors evaluated their method with Envelope on Deep Sea Treasure and Fruit Tree Navigation, and with PG-MORL and META on some MuJoCo continuous control tasks.
strength_and_weaknesses: - Multi-objective reinforcement learning is an important task, and learning a single network for multiple preferences is an efficient approach to it.
- Theoretical analysis is provided. The authors proved (in the appendix) that their optimality operator is a contraction and the optimal value function is a fixed point for the optimality operator.
- The effectiveness of the cosine similarity term in equation 5. In equation 5, we have cos(w, Q)•w^T•Q in the supremum operator. However, w^T•Q=cos(w, Q)•|w|•|Q|, which already includes the cosine similarity term. Is there a specific reason we need an extra cosine similarity term? A good way to verify its effectiveness is to perform an ablation study. Note that simply removing the cosine similarity term is not equivalent to Envelope as the supremum in the Envelope algorithm takes over different preferences.

----Rebuttal----
Dear authors,

Thank you for your detailed response. Unfortunately, I'm afraid I disagree with your explanation of the cosine similarity term. In your explanation, if we simply change the range of latency to [1, 100] and set Q2 to {0.1, 100}, even with the cosine similarity term, we still will prefer action 2 (you can verify as 0.68 < 1.12). Therefore, if we have to think of w as a preference vector, the cosine similarity term is an inappropriate approach to fix the issue you mentioned. Instead, we can rescale and normalize the magnitude of all the objectives, which easily fixes the issue you mentioned.

As I can't agree with your explanation of the cosine similarity term, I have to lower my rating of the paper.
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written. The major novelty of the paper is the cosine similarity term in the optimality operator. This work has good reproducibility as the authors provide source code in the submission.
summary_of_the_review: This paper proposes a new method of MORL based on Envelope, where they include a cosine similarity term in their optimality operator. They provided a theoretical analysis of their method, as well as performance evaluations with several baselines on several different tasks. My major concern in this work is the effectiveness of the cosine similarity term, and I believe an ablation study can demonstrate it.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper presents PD-MORL as a fascinating new method to learn a single policy that can integrate task preferences and cover more of the Pareto front than any baseline algorithm. This differentiates it from previous methods that must learn a new policy for any particular preference. It demonstrates marginally better results in existing multi-objective baseline problems, with excellent analysis to explain the improvement.
strength_and_weaknesses: The paper excels at explaining the problem, related works, algorithm, and results. The clarity of most sections is very good and the design of the algorithm, choice of experiments, metrics, figures, etc are all well justified and effective. It also very effectively highlights the differences of PD-MORL from previous works, and justifies why these differences make this approach a promising direction for multi-objective reinforcement learning. Figures 4 and 5 are also excellent at visualizing the algorithm and key ingredients to its success.

There are a few weaknesses in the paper too, however. First, MORL benchmarks used are not very well explained and the visualizations for the tasks and policies are not very clear. I have minimal understanding of what the challenges in each benchmark are and why baselines failed in these tasks -- this paper does not need an exhaustive description of these benchmarks but could certainly explain them and the learned policies better through some textual revisions and more figures (e.g., fig 3 is not very well explained/useful in its current state). Next, the experimental results are not sufficiently explored. For example, the paper does not address why the contributed PD-MORL performs significantly worse than PG-MORL in the hopper experiment, or why sparsity is sometimes worse than PG-MORL. Including these results is essential, but so is some analysis insight into the weaknesses of PD-MORL relative to PG-MORL in these test cases, and perhaps also some ideas on how these issues in PD-MORL might be overcome. It also seems worth including some discussion on the limitations of these metrics. On a related note, the results in Table 1 should have error values representing. Thirdly, the usage of the cosine similarity in eq. 5 is not well enough explained -- the inner product of omega and Q should already reflect the cosine similarity of omega and Q(s,a, omega), so why is the scalar Sc(omega, Q) also multiplied by the inner product of omega and Q? I also don't fully understand the preference alignment procedure, including how the normalized solutions in Fig 2b are formed and what purpose there is to interpolating them, e.g. rather than interpolating in a different space. Finally, the related works and experiments sections are rather monolithic and would benefit greatly from splitting the large paragraphs into individual ideas or techniques, possibly using other methods (e.g. figures/tables and subsections) to better organize and discuss the ideas in them. 

A few more small notes: the vector nature of Q is not well explained (why is the dimensionality of Q(s, a, w) equal to L and what does it represent before/after taking the inner product with omega?), the walker results in fig 5 need y-axis labels, the paper should justify why fewer parameters is a big deal, the limitations/future work discussion isn't very interesting/compelling and should be more specific and try to offer some deeper insights. Also, the use of L2 distance in sparsity is possibly worth discussing if there's space -- it seems like a rather arbitrary choice of metric so I am curious if there is a reason for it.


clarity,_quality,_novelty_and_reproducibility: The paper is mostly well written and clear, aside from a few monolithic paragraphs that should be better split up and organized (particularly in sections 2 & 4). The quality of the experiments and results is good, and a skim of the appendix makes me think it has solid coverage of the details. The main challenge here is demonstrating a significant and consistent improvement over the baselines, which was discussed as a weakness earlier. The main strength of the paper is probably the novelty and significance of the approach -- the paper makes a compelling case for why this type of approach should be used going forward (order of magnitude fewer parameters, single policy to cover any/all preferences), which I think makes the paper interesting and relevant to the community even though the empirical results are more marginal. The reproducibility statement reassures that the results will be reproducible, and the usage of standard benchmarks is well appreciated.
summary_of_the_review: This is a good paper that takes a clever, novel, and not overly complex approach to multi-objective reinforcement learning. While the empirical results aren't outstanding and the description of the algorithm itself is a little unclear, the rest of the analysis is solid and overall everything seems sufficiently suitable and solid for this conference. I think this is sure to inspire good follow-up work, further justifying why it should be shared with the wider community.

Update: The author responses have addressed all of my earlier concerns and the few new concerns are very minor. I think the changes address most of my fellow reviewers concerns and improved the paper overall, so I have decided to maintain my score of 8.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This work proposes an algorithm for solving multi-objective reinforcement learning (MORL) problems with a single network, termed preference-driven MORL (PD-MORL). The main insight to do so (instead of computing a policy for each potential preference)
strength_and_weaknesses: The main strengths of this paper:
- Novel approach to solve MORL with a single network
- Extensive experiments both in discrete and continuous state-action spaces
- Theoretical support of optimality

The main weakness of this paper:
- Some details are lacking in Section 4.2, specifically concerning the division of the preference space, the preference alignment using normalization and more importantly the extension of the algorithm to continuous action space. While it is true that these details are provided in the supplementary material, it would be helpful to have some hints in the main text. Space could be saved by condensing the introduction and related work section, which are rather detailed, thus leaving more space to describe the main novel components of this work. 
clarity,_quality,_novelty_and_reproducibility: Besides the main weakness described above, the work is clearly presented and provides a novel contribution.

Small comments and typos:
- Section 2: the last sentence is repeated almost identically twice
- Section 3, definition 1: "For the same P be a Pareto front" - what does for the same P means here? (it seems there is a grammar issue in this sentence)
- Section 4, paragraph 1: optimalioty (typo)
- Section 4.1, below eq. (3): identitity (typo)
- Section 4.1, theorems 3 and 4: "Let (Q,d) is" -> "Let (Q,d) be"
- Section 4.1, below theorem 3: "evaluation and optimality operators are contraction" -> "contractions"
- Section 4.2, first paragraph: "to obtain a single parameterized function represents" -> representing? that represents?
- Section 5: "Let the first objective is the treasure value and the second objective is" -> be and be
summary_of_the_review: This work provides a strong contribution, back up with theoretical results as well as empirical experiments.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes an approach to multi-objective reinforcement learning (MORL) using a single (universal) network that is intended to generalize across the space of preference vectors. The Preference-Driven MORL (PD-MORL) algorithm trains a neural network that takes the preference vector as input (in addition to state) and predict Q-values. A novel contribution of this work is an update to the Bellman update operator that includes the cosine similarity between the preference vector and the Q-value vector. Another is the use of hindsight experience replay (HER) as a means of improving sample efficiency. The paper evaluates PD-MORL on a series of discrete and continuous control domains and shows that it outperforms the Envelop algorithm on discrete tasks while performing comparably to continuous control baselines that must learn separate policies for each preference vector.

The paper received four reviews with overall ratings that range from one reject (3) to two accepts (8). Several of the reviewers agreed on the relevance of MORL to the community and the merits of learning a single network over the space of preference vectors. They also agree that the paper is well written and that the results are readily reproducible due in large part to the inclusion of source code. As two reviewers (Mx6C and sFJy) point out, an additional strength of the paper is the theoretical analysis that proves that the proposed optimality operator is a contraction and that the optimal value function is a fixed point.

The two more negative leaning reviews (QTgN and Mx6C, who gives score of "Marginally Above") are based on concerns that the authors have largely addressed through their response and updates to the paper. Among them, Reviewer QTgN states that the paper should compare against Envelope on the continuous control domains as well as another baseline, however the authors point out that the Envelope implementation only supports discrete environments and that Envelope, which PD-MORL outperforms, has been shown to yield better results than the referenced baseline. Reviewers Mx6C an scaX question the role of the additional cosine similarity term, and Reviewer Mx6C adds that an ablation is necessary to demonstrate its utility. Reviewer Mx6C identifies these as being their major concern with the paper. The authors provide justification for the inclusion of this additional term and point to an existing ablation in the paper.

The authors and the AC made several attempts to engage Reviewers QTgN and Mx6C in discussion, and the AC asked that they update their review to indicate that they had read the author response. Unfortunately. these attempts were unsuccessful. As such the AC is placing more weight on their own reading of the paper and the reviews and discussion provided by Reviewers scaX and sFJy. The AC notes the concerns of these reviewers regarding the clarity of the experimental evaluation related to what they see as only marginal improvements over baselines as well as the need for more insights into settings that exhibit poor performance. The authors are encouraged to revisit the discussion of the experimental results to make sure that the analysis is clearer.
justification_for_why_not_higher_score: None. AC does not have an objection to this being a spotlight.
justification_for_why_not_lower_score: While one reviewer recommended rejecting the paper, their criticisms are either unfounded (e.g., the need to compare to Envelope on the continuous control tasks) as clarified by the author response, or addressed by the authors.
Decision: Accept: poster