Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 3: Clear rejection
review_content: The paper proposes a novel approach for solving MORL problems while considering uncertainty in the Pareto frontier.
The contributions are interesting and novel, but the paper has several flaws which make it not ready for acceptance, especially regarding experiments.

First, the authors overlook many Bayesian MORL algorithms which also consider the Pareto frontier uncertainty. For instance, 

Calandra et al, "Pareto Front Modeling for Sensitivity Analysis in Multi-Objective Bayesian Optimization"

Calandra et al, "Bayesian Multiobjective Optimisation With Mixed Analytical and Black-Box Functions: Application to Tissue Engineering")

Hernandez-Lobato et al, "Predictive entropy search for multi-objective Bayesian optimization"

Olofsson et al, "Bayesian multi-objective optimisation of neotissue growth in a perfusion bioreactor set-up"

In the introduction the authors say "In addition, most approaches still only work in domains with low-dimensional and discrete action spaces." This is not true. Simply, all cited algorithms have just been tested on low-dimensional problems. Since they were not evaluated on larger problems, we do not know how they behave. The authors do not even include any of them in the evaluation to show that they actually fail.
Furthermore, the authors claim that their experiments have large action spaces. How big are they? This is not mentioned in the paper. For instance, the Mujoco environments tested in the paper do not have that large action spaces (eg, Swimmer has 15 actions).

The writing can be also improved. The sections feel a bit disconnected, and sometimes it is not easy to highlight the contributions. For instance, Section 4.3.1 takes quite some space and seem to be part of the novelty contributions, but the losses are taken from Yang et al.

My biggest issue is with the experiments. First, the authors should say right away on which environment they are testing the algorithms, and not just "Mujoco" and "two provided by Xu et al". Furthermore, these two environments are never actually tested, since the experiments are only on SUMO, Swimmer, Walker, and HalfCheetah.
And why are results for the Swimmer shown as figure, while Walker and Cheetah have tables? And why these environments out of all Mujoco ones? (these three are known to be the easiest).
Moreover, Mujoco environments are single-objective. How did you turn them into multi-objective? This is crucial and not mentioned.
Finally, there is no evaluation against any of the MORL algorithms mentioned in related work. The paper makes the point that these should fail with large action spaces and uncertainty, but this is not shown.

Overall, the experiments section feels rushed and incomplete, and the paper is not ready for publication.

** EDIT **
The authors added many experiments, tables, figures, sections, and an appendix. The changes are too substantial and the paper looks like a completely new one. 
The purpose of author rebuttals is to address issues like a reviewer’s uncertainty about a point, an incorrect assumption, a misconception, or a misunderstanding of a part of the paper, not to revamp the paper completely.
The paper was clearly incomplete at the time of its submission, and I still vote for its rejection. 
review_title: Interesting idea, but not ready for publication

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary:
This paper seeks to train multi-objective RL policies that are robust to environmental uncertainties. There are two main contributions: a novel approach to solve this problem, and a novel metric to evaluate Pareto fronts. The metric combines the typical hypervolume metric (that captures the quality/performance of a Pareto front) with a novel "evenness" metric, that captures how well solutions are spread out across the space of preferences. The proposed approach, called BRMORL, consists of training a protagonist policy that maximizes utility alongside an adversarial policy that seeks to minimize utility (motivated by zero-sum game theory), while using Bayesian optimization to select preferences to train on, in order to optimize the hypervolume-and-evennesss metric. Both the protagonist and adversarial policy are conditioned on preferences.

Recommendation:
This paper connects two seemingly orthogonal problems, multi-objective RL and robustness. This is an interesting topic, but there are several issues regarding clarity and the motivation (as detailed in the cons list below). I think this paper could be a valuable contribution for MORL, but _not_ for MORL that is robust to environmental uncertainty, which is what the claim is. Thus I recommend rejection.

Pros:
* Training policies that are robust _and_ flexibly trade off between preferences is an interesting and relevant problem.
* The empirical evaluation shows that the approach outperforms ablations and an existing state-of-the-art MORL approach (Xu et al. 2020) on continuous control tasks.

Cons:
* Clarity: the introduction should clearly define what _robustness_ means. Currently it's unclear what problem this paper is trying to solve. Does the approach try to achieve robustness to environment dynamics / perturbations, or robustness across preferences, or both? My interpretation is that robustness refers to both kinds. I can understand how BRMORL would improve robustness across preferences, and perhaps also perturbations, but am skeptical about whether it improves robustness to environment dynamics (see next point).
* The motivation behind this approach is questionable: I'm not convinced that BRMORL actually leads to training policies that are more robust, with respect to environment dynamics or perturbations. This is not shown clearly in the empirical evaluation, and also is not obvious from the approach itself. I don't see the connection between having an adversarial policy and being robust to the dynamics of the system (e.g., masses of limbs). Figure 6 shows that BRMORL has better robustness to environmental uncertainty than SMORL, but that could just be because SMORL is the worst-performing ablation, and just doesn't find particularly high-performing policies (as shown in Figure 5c). How does BRMORL compare to RMORL or SRMORL?
* It would help to have an algorithm box for BRMORL, that clarifies how the adversary policy, protagonist policy, and Bayesian optimization are used to gather data and for training.
* The proposed metric is questionable. The goal is to capture both diversity and quality of solutions, but in Figure 3, I would argue that Pareto front 1 is indeed better, because these points dominate _all_ of the points on Pareto fronts 2 and 3, and the purpose of MORL is to find non-dominated policies.
* The chosen scalar utility function $U$ is not properly justified. In particular, does $M$ (in Equation 2) still make sense when the objectives have significantly different reward scales (e.g., if one objective's return is typically from 0 to 10, and the other's is from 10 to 100)? Even after normalizing, the Q-value term will only be in a portion of the first quadrant, whereas the $w$ term can cover the entire first quadrant.
* Unjustified hyperparameters for trading off between terms in the losses: $k$ in the scalar utility function, $\beta$ for the two terms in the Q-function loss, and $\lambda$ for the comprehensive metric that combines hypervolume and evenness. How should these be chosen?
* The Related Work doesn't give enough credit to existing MORL approaches. First, Xu et al. (2020) is actually able to find a well-distributed set of Pareto-optimal solutions. In addition, existing methods are stated to only be able to find solutions on the convex portions of a Pareto front. Bringing up this point implies that BRMORL does better (i.e., is able to find solutions on concave portions of the Pareto front), but this is not shown empirically. Finally, the related work states that most existing approaches are only applied to domains with discrete action spaces. It should acknowledge that both Abdolmaleki et al. (2020) and Xu et al. (2020) are applied to high-dimensional continuous control tasks.
* Lack of experimental details for reproducibility, e.g., network architetures and DDPG hyperparameters.

Other comments:
* There are quite a few grammatical errors and typos throughout the paper.
* Definition 3 is imprecise. First, is $a$ a policy or an action? It seems like it should be a policy because it's a member of the policy set, but it's used to denote actions in the previous section, Section 3.1. Also, why are $I$ and $II$ included in the game definition, when they are already represented by the policy sets?
* There is not enough explanation given for Figure 1. Where do the uniformly-sampled preferences come from (the gray dashed lines)? What is the "optimal guess point"? Does Bayesian optimization only suggest one preference at a time (in red)? What is the acquisition function? (This is defined too late in the paper, and only in the caption for Figure 4.)
* It would be more accurate to make the $k$ explicit in equations 8 and 9, because it's different in $M(\cdot)$ for the two equations, but the current notation implies it's the same.
* In the empirical evaluation, SRMORL, an ablation of BRMORL, finds policies that dominate those found by BRMORL (Figure 5c). How can this be interpreted / explained?
* Table 3 needs an accompanying explanation of the different MORL methods.
review_title: Interesting problem, but lacking clarity and motivation

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper proposes a robust multi-objective RL approach and a non-linear utility metric to enforce an accurate and evenly distributed representation of the Pareto frontier. Robustness is obtained by formulating the problem as a two-player zero-sum game. The goal of the main agent is thus to learn the policies on the Pareto frontier under attacks from the adversary. This is achieved by training a single network to generate approximate Pareto optimal policies for any provided preference. To train this network, they introduce a new metric for Pareto frontier evaluation based on hypervolume and entropy (to force evenly distributed solutions). The resulting algorithm has the classical structure of an actor-critic algorithm where the critic provides an estimate of the Q-function and the actor updates the policies of the protagonist and adversary through alternate optimization.

Could you give more motivations why a robust approach is needed for MORL? The motivation now seems simply to be that the literature didn't take into account robustness. 

I think the idea of the paper is quite interesting but it is not well written/explained. I think a few details are missing.

For example, in section 4.3.1 it is not clear to me how the loss is constructed. You mentioned that the objective is to learn the Q-function of the protagonist but the target value $y$ is built using the mix policy and Q-function. Could you clarify the meaning of this loss?
There are also two different definitions of $y$ ($s'$ vs $s$ and $\omega'$ vs $\omega$).

What is the meaning of $\mathbb{E}^{\pi^{mix}}$? Does it mean expectation wrt to the stationary distribution induced by the policy?
It is important for understanding equations 8 and 9. Why there is an approximation in the definition of the gradients? Shouldn't be $\nabla \mathbb{E}^{\pi}[]$?

Concerning section 4.4, you compare with the metric proposed in (Xu et al. 2020) without explaining it. It is hard for the reader to understand why it is not a good metric without knowing the metric. In general, as you acknowledged, a good approximate Pareto frontier should be accurate, evenly distributed and have a covering similar to the one of the true Pareto frontier. These properties are not equally important. This is to say that I think the example in figure 3 has a major drawback. Frontier 2 and 3 are not Pareto frontier since are dominated by 3. Not sure that everyone will agree that 2 is better than 3. I suggest you change the example.

To overcome this, you introduced an entropic measure based on a partitioning of the Pareto frontier. There is no mention of how to do that in practice. Could you explain it? Are you partitioning the space of preferences?
A standard measure to enforce spread solutions is the crowding distance (eg in your reference Parisi et al. 2017 and many more), could you apply the same idea of interval partition on this?
Could you explain why you introduced evenness as a multiplicative factor rather than an additive one in $I(P)$?


Experiments. There is no information about the implementation and parameters/configurations used. It is thus very difficult to parse the results. For example, how are preferences selected for standard methods (ie all expect BRMORL)?
Why is the comparison with (Yang et al. 2019) missing? I think this is a relevant algorithm for the setting.
You should add an explicit reference to the papers introducing the methods in Table 3, ie RA, PFA, MOEA/D and META. 

I think the paper contains an interesting idea but, given the mentioned concerns, I think this is a borderline paper. I'm looking forward to the authors' feedback.

Minor issues
-I think it's more precise to define \mathcal{A}^{mix} as the space of possible combinations of actions. The probability $\alpha$ should be associated with $\pi^{mix}$ rather than with the action space.
-Reference missing to SUMO
-You use both $\mathbb{E}_\pi$ and $\mathbb{E}^{\pi, \pi'}$ but these symbols are never explained
-Figure 2:  "OA can not parallel to OB"
-You use often the term convergence in the description of the algorithm. How do you evaluate convergence?
-In figure 4 what is $\alpha(\Omega)$? Is it related to $\alpha$ used in the definition of the mix policy?

review_title: Interesting idea but many details are missing

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes a framework to tackle uncertainty in multi-objective optimization of reinforcement learning problems. Uncertainty is represented as an adversary over preferences. Fitness is measured by a multi-objective quality indicators while Bayesian optimization is used to bring improvements. The proposed method is evaluated on four benchmark problems.

While the page limit makes it difficult to detail all the different aspect of the contribution, the clarity still needs to be improved. For instance, it is worth adding a detailed algorithmic description of the approach. At various stages there are many different options, so the specific choices should be better introduced and limits discussed.

Detailed comments and questions:

a) There are many parameters to set (alpha, beta, omega, lambda), how sensitive are they in practice? They are not mentioned in the experimental part, hence the results cannot be reproduced.

b) Nash games are defined over partitions of the design variables, but it does not seem to be the case here?

c) Robustness can be defined in many different ways (expectation/variance trade-off, quantiles, worst case, chance constraints). How does it differ from optimizing the worst case here? 

d) Pareto front quality indicators are widely studied in the multi-objective optimization literature, existing ones should be reviewed first. See e.g.,:
- Charles Audet, J Bigeon, D Cartier, Sébastien Le Digabel, Ludovic Salomon. Performance indicators in multiobjective optimization. 2020. ⟨hal-02464750⟩
- Cheng S., Shi Y., Qin Q. (2012) On the Performance Metrics of Multiobjective Optimization. In: Tan Y., Shi Y., Ji Z. (eds) Advances in Swarm Intelligence. ICSI 2012. Lecture Notes in Computer Science, vol 7331. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-30976-2_61

e) Pareto based acquisition functions are generally preferred in Bayesian MOO compared to scalar ones, such as the one proposed here on a GP fit of the defined utility. See for instance:
- Emmerich, M. T.; Deutz, A. H. & Klinkenberg, J. W. Hypervolume-based expected improvement: Monotonicity properties and exact computation. Evolutionary Computation (CEC), 2011 IEEE Congress on, 2011, 2147-2154
- Picheny, V. Multiobjective optimization using Gaussian process emulators via stepwise uncertainty reduction Statistics and Computing, Springer, 2015, 25, 1265-1280
- Predictive Entropy Search for Multi-objective Bayesian Optimization. Daniel Hernandez-Lobato,  Jose Hernandez-Lobato,  Amar Shah,  Ryan Adams ; PMLR 48:1492-1501.

Additional related references of interest from the literature that could be discussed:
- Lepird, J. R.; Owen, M. P. & Kochenderfer, M. J., Bayesian preference elicitation for multiobjective engineering design optimization, Journal of Aerospace Information Systems, American Institute of Aeronautics and Astronautics, 2015, 12, 634-645
- Paria, B.; Kandasamy, K. & Póczos, B. A flexible framework for multi-objective bayesian optimization using random scalarizations Proceedings of The 35th Uncertainty in Artificial Intelligence Conference, PMLR 115:766-776, 2020.
- Multi-attribute Bayesian optimization with interactive preference learning. R Astudillo, P Frazier
International Conference on Artificial Intelligence and Statistics, 4496-4507


f) Fig. 3: either a point is Pareto optimal, or it is not. All blue and violet solutions are dominated by the green ones, so the green Pareto front is better.

g) Fig. 1 lacks a detailed description. Are the different panel successive iterations for instance?

h) Only bi-objective examples are presented, how does it scale with more objectives?

Typos:
P1: algorithms have demonstrated its worth → their
Fig. 2: can not parallel

## Post rebuttal comments
The authors largely modified the paper according to the comments, with a lot of additional content. While this is quite beneficial, the paper raised many questions, some of which may need further treatment (for instance, increasing the number of objectives has an effect on the number of Pareto optimal solutions that is is not trivial).
review_title: Review of Approximating Pareto Frontier through Bayesian-optimization-directed Robust Multi-objective Reinforcement Learning

Meta-review: The paper studied multi-objective reinforcement learning (MORL), and provided a Bayesian optimization approach for challenging MORL scenarios in several simulation environments. The reviewers generally find it interesting to account for robustness in a MORL setup, and all appreciate the algorithmic contributions. However, there were shared critical concerns among \ reviewers in the technical clarity and positioning of the work. 

The paper has gone through substantial changes during the rebuttal period, which addressed some concerns regarding the experimental details; however, the major revision raised further issues that affects the clarity of the work. The reviewers are hence unconvinced that the paper is ready for publication. In addition to addressing the existing comments on clarifying the experimental details and properly positioning the work against prior art, a reorganization and optimization of the main content would be beneficial for future submission.

Decision: Reject