Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: Thank the authors for the response. I agree with R2 that the paper lacks comparisons with previous works. I will stick to my previous decision.
----------------------------------------
Summary
This paper presents a new approach for single-objective reinforcement learning by preferencing multi-objective reinforcement learning. The general idea is to first figure out a few important objectives, add some helper-objectives to the original problem, and learn the weights for each individual objective by trying to keep the same order as Pareto dominance. This paper has potential, but I lean to vote for rejecting this paper now, since it is still not ready. I might change my score based on the reviews from other reviewers.
Strengths
- The idea is novel. Learning weights for each objective by keeping the order as Pareto dominance is an interesting idea to me.
Weaknesses
- The lack of experiments. The authors tested their method in only one scenario, which makes me feel unsafe. Only testing on one simple scenario does not demonstrate the effectiveness. The authors are supposed to test their method on more (complex) scenarios to show the effectiveness of their method.
Possible Improvements
As mentioned before, the proposed method can be tested on more scenarios (e.g., Deep Sea Treasure, SuperMario, etc.).
review_title: Official Blind Review #3

final_score: rating: 1: Reject
review_content: After Responses:
I understand the differences that authors pointed to the relevant literature. However, it is still lacking comparisons to these relevant methods. The proposed method has not been compared with any of the existing literature. Hence, we do not have any idea how does it stand against the existing approaches. Hence, I believe the empirical study is still significantly lacking. I will stick to my decision. Main reason is as follows; I believe the idea is interesting but it needs a significant empirical work to be published. I recommend authors to improve empirical study and re-submit.
-------
The submission is proposing a method for multi-objective RL such that the preference of tasks learned on the fly with the policy learning. The main idea is converting the multi-objective problem into single objective by scalar weighting. The weights are learned in a structured learning fashion by enforcing them to approximate the Pareto dominance relations.

The submission is interesting; however, its novelty is not even clear since authors did not discuss majority of the existing related work. 

Authors can consult the AAMAS 2018 tutorial "Multi-Objective Planning and Reinforcement Learning" by Whiteson&Roijers for relevant papers. It is also important to note that there are other methods which learn weighting. Optimistic linear support is one of such methods. Hence, this is not the first of such approaches. Beyond RL, it is also studied extensively in supervised learning. For example, authors can see "Multi-Task Learning as Multi-Objective Optimization" from NeurIPS 2018.

The manuscript is also very hard to parse and understand. For example, Definition 2 uses but not define "p" in condition (2). Similarly, Lemma 1 states sth is "far greater" than something else. However, "far greater" is not really defined. I am also puzzled to understand the relevance of Theorem 1. It is beyond the scope of the manuscript, and also not really new.

Authors suggest a method to solve multi-objective optimization. However, there is no correctness proof. We do not know would the algorithm result in Pareto optimal solution even asymptotically. Arbitrary weights do not result in Pareto optimality.

Proposing a new toy problem is well-received. However, not providing any experiment beyond the proposed problem is problematic. Authors motivate their method using DOOM example. Why not provide experimental results on a challenging problem like DOOM?

In summary, I definitely appreciate the idea. However, it needs better literature search. Authors should position their paper properly with respect to existing literature. The theory should be revised and extended with convergence to Pareto optimality. Finally, more extensive experiments on existing problems comparing with existing baselines is needed.
review_title: Official Blind Review #2

Meta-review: The paper considers planning through the lenses both of a single and multiple objectives. The paper then discusses the pareto frontiers of this optimization. While this is an interesting direction, the reviewers feel a more careful comparison to related work is needed.
Decision: Reject