Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies some theoretical questions about multi-objective reinforcement learning. The focus is on policy-induced value functions. Section 3 develops several tools to characterize changes to value functions when one changes policies. Section 4 uses those tools to show the value function space is convex and provides insights into why linear scalarization methods fail to find Pareto-efficient policies. The authors present a concave-augmented Pareto Q-learning algorithm (CAPQL) and demonstrate its effectiveness on several multi-objective variants of the MuJoCo environments.
strength_and_weaknesses: Strength: overall the theoretical contribution is quite interesting and insightful. As the authors pointed out, recent multi-objective RL research has not presented a coherent view on this topic. This paper is a timely investigation of some fundamental properties in this space.

Weakness: 
The empirical evaluation section, in comparison, is weak. As the new algorithm proposes adding a strongly concave term to the reward function, it would be better to explain the choice of selecting the entropy operator in the implementation. Can one use any other operator in its place?

While the results on the 4 presented environments look promising, I noticed the sample size is relatively small (10k steps). Are there longer runs so we can see result comparison at convergence?

The authors should also discuss a relevant paper on introducing an improved scalarization method beyond linear scalarization. 

Kristof Van Moffaert; Madalina M. Drugan; Ann Nowé: Scalarized multi-objective reinforcement learning: Novel design techniques

clarity,_quality,_novelty_and_reproducibility: The paper is well-written and relatively easy to follow. The theoretical novelty is high and relevant to the multi-objective RL community.
summary_of_the_review: This a well-written paper with a good theoretical contribution. The empirical evaluation section could be strengthened to make the whole package stronger.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper discusses the impact of multiple objectives. Pareto optimality guarantees are discussed, and actor-critic formulation is provided. 
strength_and_weaknesses: The paper analyses the dynamics of the induced value functions resultant from policy alterations in MORL problems. The analysis seems good. 

The proposed algorithm mainly adds a concave function of the policy, which can help with determinism and numerical instability. The algorithms could have some sample complexity type results which will be helpful, with comparisons to the existing works. This is especially since many algorithms like policy gradient, actor critic, etc. have sample complexity guarantees. Even though this paper is different from CURL frameworks, it is not clear why we cannot use the model like there to model the multi-objective problem - and what benefits this formulation gives or what problems this could solve which the other cannot.
clarity,_quality,_novelty_and_reproducibility: The algorithm table and the corresponding novelty there could be in main text. 
summary_of_the_review: The paper provides some results on Pareto optimality of MORL with concave regularization of the policy. Sample complexity results would have been nice for the paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies the fundamental properties of the learning spaces in multi-objective reinforcement learning (MORL). The authors give a theoretical analysis of policy induced value functions and discuss three metrics of Pareto optimality. Their results imply the convexity of the induced value function, and show that training a policy based on linear scalarization (LS) can achieve any point of the Pareto front. The authors propose a new vector reward-based Q-learning algorithm, called CAPQL, which adds a strongly concave term to the immediate reward and solves the problems of existing LS-based algorithms.
strength_and_weaknesses: Strengths:
1.	This paper is very well-written and easy to follow. The authors use an example (Example 1) throughout the paper to illustrate their insights, which facilitate well readers’ understanding.
2.	The investigation on the relationship between three types of Pareto optimality is novel and interesting, which is supported by theoretical analysis.

Weaknesses:
1.	According to the results in Section 4, i.e., adding a strongly concave term can fix the problems of existing LS-based algorithms, it seems that the proposed algorithm CAPQL can add an arbitrary strongly concave term to the immediate reward. Why does CAPQL use the entropy operator $\mathcal{H}(q)$? Can this entropy operator be changed to other strongly concave terms? It would be better if the authors can include more discussion on the strongly concave terms used for CAPQL.
2.	The theoretical analysis for algorithm CAPQL is not sufficient. In the current version of this paper, the theoretical guarantee for CAPQL fully replies on the analysis for the limitations of existing LS-based algorithms and three types of Pareto optimality. It would be better if the authors can provide a more thorough theoretical analysis for CAPQL, e.g., a convergence or sample complexity guarantee.
3.	Can the authors provide the experimental results for the comparison to existing LS-based algorithms? The results in Section 4 will be more convincing if the authors can show some experimental phenomena that support the theoretical results.

clarity,_quality,_novelty_and_reproducibility: This paper is clearly written. The analysis for three types of Pareto optimality for multi-objective reinforcement learning (MORL) is novel. The authors illustrate the implementation details for the experiments, but do not provide the code.
summary_of_the_review: I think that the theoretical analysis for the problems of existing LS-based algorithms and the relationship among three types of Pareto optimality for multi-objective reinforcement learning (MORL) is novel and interesting. But the proposed algorithm CAPQL lacks a sufficient discussion on the used concave term and a more in-depth theoretical analysis, e.g., a convergence or sample complexity guarantee. The experiments can be further enhanced by adding a comparison to existing LS-based algorithms and a discussion on how the empirical phenomena support the theoretical fundings in Section 4. Overall, I give borderline acceptance.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper conducts a rigorous analysis on the fundamental properties of the policy induced value functions in multi-objective reinforcement learning (MORL). Based on the analysis, the authors further distinguish and consolidate three existing definitions of Pareto optimal policies and identify issues of training policies via linear scalarization (LS). The authors fix this issue by adding strongly concave terms to the immediate rewards and verify the effectiveness of the approach via experiments on simulated control tasks. 
strength_and_weaknesses: I am not an expert in MORL. But after reading this paper, I envision that the contributions of this paper will have a large impact on the MORL community. They can accelerate the research in this field. Furthermore, as many practical RL problems can be better formulated using multiple objectives, the contributions are very important to the RL community in general. 

The authors offer several novel theoretical results, including

- The induced value function’s range for stationary policies is concave. This has been thought hard to characterize and of irregular shapes. 
- There are at least three existing definitions of Pareto optimal policies. The authors rigorously distinguish and consolidate them. 

I really like the new theoretical results. However, I have some questions regarding the Problems of the existing LS-based algorithms paragraph and the proposed algorithm. 

- The solutions to linear programs are almost always on the vertices. If this does not make solving them less numerically stable, why would LS-based algorithms be less numerically stable? 
- The motivation of identifying all SPE policies is not very clear to me. When is this required in practice? 
- Maybe derived from a different perspective, based on Eq (9), the proposed algorithm CAPQL is almost the same as entropy-regulated RL algorithms. Is this correct? 

clarity,_quality,_novelty_and_reproducibility: The paper is very pleasant to read and has high clarity and quality. Using a running toy example to provide intuition for the theoretical results is very helpful.

The theoretical results seem novel and significant, although the algorithm is not very novel. 

Typo:
- at the bottom of page 6, g_s is not defined. 


summary_of_the_review: Due to the strong theoretical contributions, I recommend acceptance.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper studies the fundamental properties of the policy-induced value functions in multi-objective reinforcement learning (MORL). 
The authors identify issues in training a policy via linear scalarization of the objectives and propose to fix them by adding strongly concave terms to the immediate rewards. The proposed approach has shown to be more stable in solving multiple MuJoCo tasks.
After reading each others' reviews and the authors' feedback, the reviewers start to discuss the strong and weak points of this paper.
They appreciated the theoretical contributions, while there were some concerns about the empirical validation, such as the realization of ablation studies.
Overall, during the discussion, the reviewers agreed that the positive aspects outweighed the negative ones.
The authors must follow the reviewers' suggestions while preparing their camera ready.
justification_for_why_not_higher_score: The paper has some limitations on the empirical analysis.
justification_for_why_not_lower_score: The paper addresses a relevant topic and introduces interesting contributions.
The limitations of the paper are not so significant to prevent publication.
Decision: Accept: poster