Approximating Pareto Frontier through Bayesian-optimization-directed Robust Multi-objective Reinforcement Learning
Authors: Xiangkun He
Jianye HAO
Dong Li
Bin Wang
Wulong Liu
Venue:  (2021)
Keywords: Reinforcement Learning
Multi–objective Optimization
Adversarial Machine Learning
Bayesian Optimization

Abstract: Many real-word decision or control problems involve multiple conflicting objectives and uncertainties, which requires learned policies are not only Pareto optimal but also robust. In this paper, we proposed a novel algorithm to approximate a representation for robust Pareto frontier through Bayesian-optimization-directed robust multi-objective reinforcement learning (BRMORL). Firstly, environmental uncertainty is modeled as an adversarial agent over the entire space of preferences by incorporating zero-sum game into multi-objective reinforcement learning (MORL). Secondly, a comprehensive metric based on hypervolume and information entropy is presented to evaluate convergence, diversity and evenness of the distribution for Pareto solutions. Thirdly, the agent’s learning process is regarded as a black-box, and the comprehensive metric we proposed is computed after each episode of training, then a Bayesian optimization (BO) algorithm is adopted to guide the agent to evolve towards improving the quality of the approximated Pareto frontier. Finally, we demonstrate the effectiveness of proposed approach on challenging multi-objective tasks across four environments, and show our scheme can produce robust policies under environmental uncertainty.

Under review as a conference paper at ICLR 2021
APPROXIMATING PARETO BANESIAN-OPPIMIAATIONDDDEEEE ROBUST MULTI- OBJECTIVE REINFORCEMENT LEARNING
FRONTIER THROUGH
Anonymous authors Paper under double-blind review
ABSTRACT
Many real-word decision or control problems involve multiple conflicting objec- tives and uncertainties, which requires learned policies are not only Pareto optimal but also robust. In this paper, we proposed novel algorithm approximate repre- sentation for robust Pareto frontier through ayvesian-optimizatin direeced robust multi-objective reinforcement learning (BRMORL). Firstly, environmental uncer- tainty is modeled as an adversarial agent over the entire space of preferences by incorporating zero-sum game into multi-objective reinforcement learning (MORL). Secondly, a comprehensive metric based on hypervolume and information entropy is presented evaluate convergence, diversity and evenness of the distribution for Pareto solutions. Thirdly, the agent's 's learning process is regarded as a black-box -bx, and the comprehensive metric we proposed computed after each episode of training, then a Bayesian optimization (BO) algorithm is adopted to guide the agent to evolve towards improving the quality of the approximated Pareto frontier. Finally, we demonstrate the effectiveness of proposed approach on challenging multi-objective tasks across four environments, and show our scheme can produce robust policies under environmental uncertainty.
INTRODUCTION
Reinforcement learning (RL) algorithm has demonstrated its worth in a series of challenging se- quential decision making and control tasks, which train policies to optimize a single scalar reward function (Mnih al. 2015 Silver et al.. 2016, Haarnoja et al. 2018 (wangho et al. 2019). However, many real-world tasks are characterized by multiple competing objectives whose relative impor- tance (preferences) ambiguous in most cases. Moreover, uncertainty or perturbation caused by environment dynamic change, is inevitable in real-world scenarios, which may result in lowered agent performance (Pinto et al., 2017 Ji et al., 2018). For instance, autonomous electric vehicle requires trading off transport efficiency and electricity consumption while considering environmental uncertainty (c.g., vehicle mass, tire pressure and road conditions might vary over time). Consider a dacision-making problem for traffic mode, shown in Figure I A practitioner for picking the appropriate preference among time and cost, and the agent need to determine different policies depending on the chosen trade-off between these two metrics. Whereas, the environment contain uncertainty factors related to actions of other agents or to dynamic changes of Nature, which may lead more randomness these two metrics, and makes multi-objective decision-makee maninn or control more challenging. If weather factors are taken into account, e.g., heavy rain may cause traffic congestion, which can increase the time and cost of the plan-A, A. but not have significant impact on the two metrics of the plan-B. From this perspective, selecting plan-B is more robust, i.e., a policy is said to be robust if its capability to obtain utility is relatively stable under environmental changes. Therefore, preference and uncertainty jointly affect the decision- making behavior of the agent.
rule is responsible
In traditional multi-objective reinforcement learning (MORL), one popular way is scalarization, which is to convert the multi-objective reward vector into single scalar reward through various techniques (e.g., by taking a convex combination). and then adopt standard RL algorithms to op- timize this scalar reward (Vamplew al. 2011). Unfortunately, it is very tricky to determine an appropriate scalarization, because often common approach only learn an 'aaverge'' policy over the space of preferences (Yang et al. 2019), or though the obtained policies be relatively quickly

---

Under review as a conference paper at ICLR 2021
Taxi
Plan-A
Walk
Plan-A
Plan-Bi B
Time
Short
Long
Cost
High
Low
Figure 1: Diagram of decision-making problem for traffic mode. If time is crucial, the agent tend to choose plan-A that takes less time, but it costs more. On the other hand, if cost is more important matters, the agent will be inclined to select plan-B that requires less cost, but it takes more time.
adapted to different preferences between performance objectives but are not necessarily optimal. Furthermore, these methods almost did not take into account the robustness of the policies under different preferences, which means the agent cannot Iearn robust Pareto optimal policies.
In this work, we propose a novel approach to approximate well-distributed robust Pareto frontier through BRMORL. This allows our trained single network model to produce the robust Pareto optimal policy for any specified preference, i.e., the learned policy is not only robust to uncertainty (e.g.. random disturbance and environmental change) but also Pareto optimal under different preference conditions. Our algorithm is based on three key ideas, which are also the main contributions of this paper: (1) present generalized robust MORL framework through modelling uncertainty as an adversarial agent; (2) inspired by Shannon-Wiener diversity index, a novel metric is presented to evaluate diversity and evenness of distribution for Pareto solutions. In addition, combined with hypervolume indicator, comprehensive metric is designed, which can evaluate the convergence, diversity and evenness for the solutions on the approximated Pareto frontier; (3) regard agent's learning process in each episode black-box, and BO algorithm is used to guide agent to evolve towards improving the quality of the Pareto set. Finally, we demonstrate our proposed algorithm outperform competitive baselines on multi-objective tasks across several MuJoCo (Todorov et al. 2012) environments and SUMO (Simulation of Urban Mobility) (Lopez et al., 2018 and show our approach can produce robust policies under environmental uncertainty.
RELATED WORK
2.1 MULTI--BBJCTTVV REINFORCEMEN LEARNING
MORL algorithms can be roughly classified into two main categories: single-policy approaches and multiple-policy approaches (Roijers et al. 2013 Liu et al. 2014). Single-policy methods seek to find the optimal policy for a given preference among multiple competing objectives. These approaches convert the multi-objective problem into a singlc-objective problem through different forms of scalarization, including linear and non non-linear ones (Mannor & Shimkin 2002 Tesauro et al.) 2008). The main advantage of scalarization is its simplicity, which can be integrated into single-policy scheme with very little modification. However, the main drawback of these approaches is that the preference among the objectives must be set in advance.
Multi-policy methods aim to learn set of policies that approximate Pareto frontier under different preference conditions. The most common approaches repeatedly call single-policy scheme with different preferences (Natarajan & Tadepalli 2005 Van Moffaert et al. 2013) Zuluaga et al.) 2016). Other methods learn a set of policies simultancously via using a mulli-objecttvv extended version of value-based RL (Barrett & Narayanan] 2008; Castelletti et al., 2012 Van Moffaert & Nowe 2014, Mossalam al. [016]] Nottingham et al.) 20.9) or via modifying policy-based RL. as a MORL variant (Pirotta et al. 20155) Parisi et al. [2017] [ et al.) 2020 Xu et al. 2020). Nevertheless, most of these methods are offen constrained to convex regions of the Pareto front and explicitly maintain sets of policies, which may prevent these schemes from finding the sets of well-distributed Pareto solutions which can represent different preferences. There are also meta- meta-policy methods, which can be relatively quickly adapted different preferences (Chen et al.) 2018 Abels et al. 2019 Yang et al. (2019). Although the above works were successful to some extent, these approaches share the same shortcomings that attention is paid to the robustness of Pareto-optimal policy over the
2

---

Under review as a conference paper at ICLR 2021
entire space of preferences. In addition, most approaches still focus on the domains with discrete action space. In contrast, our scheme can guarantee the learned policies is approximately robust Pareto-optimal on continuous control tasks.
2.2 ROBUST REINFORCEMENT LEARNING
Robust reinforcement learning (RRL) algorithms can be broadly grouped into three distinct meth- ods (Derman et al. 2020). The first approach focuses solving robust Markov decision process (MDP) with rectangular uncertainty sets. Some researches proposed RRL algorithms for learning optimal policies using coupled uncertainty sets (Mannor et al.) 2012). Other works modeled ambiguous linear function of a factor matrix as a selection setting from an uncertainty set (Goyal Grand-Clement 2018). The second RRL approach considered distribution over the uncertainty set to mitigate the conservativeness. Yu & Xu (2015) presented the distributional RRL method by supposing the uncertain parameters are random variables following an unknown distribution. Tirinzoi et al. (2018) proposed a RRL scheme using conditioned probability distribution that de- fines uncertainty sets. third RRL method mostly concerns adversarial setting RI.. Pinto et al.) (2017) developed a robust adversarial reinforcement learning (RARL) scheme through modeling uncertainties via adversarial agent which applies disturbances to the system. Tessler et al. (2019) proposed an adversarial RRL framework through structuring probabilistic action robust MDP and noisy action robust MDP. Nonetheless, these researches do not take into account the connection between Pareto-optimal policy and robust policy, which leaves room for improving the performance of them in practical applications. In contrast, our scheme can learn robust Pareto-optimal policies through modeling uncertainty adversary over the entire space of preferences.
3 BACKGROUND
3.1 MULTI-OBIECTIN MARKOV DECISION PROCESS
In this work, we consider a MORL problem defined by multi-objective Markov decision process (MOMDP), which represented by the tuple (S, A, P, R. 72 Un) with state space S, action space A, state transition probability P(s'| (s a), vector reward function R(s, a) = [ri,.... rx]" rA] the space of preferences Ω and preference functions, e.g., U. R) which produces an utility function using preference E Ω and discount factor 7 [1. 0). In MOMDP, policy is associated with a vector of expected returns (=, = .Q...... QTIT" TJ where the action-value function of n for objective & can be represented as QT a) E EE >'rx(Se, as) do aJ. For MOMDP, set of non-dominated policies called as the Pareto frontier.
Definition 1. A policy कi Pareto dominates another policy т2, i.e. कi > क2 when
Bi (s, (F(..)))v : > (s.
Definition 2. A policy π is Pareto optimal if and only if it is non-dominated by any other policies.
3.2 TWO-PERSON ZERO-SUM GAMES
zero-sum games, players have opposite goals the payoff of a player equals the loss of the opponent (Mazalov 2014), i.e., V + > O, where and V are payoff of player and the opponent, respectively.
In standard
For two player discounted zero-sum Markov game, assuming protagonist is playing policy w and adversary is playing the policy म. transition kernel P(s'|s, [s, a, depend on both players. In the game, the value function based on я and म can be represented as प- (s) = =E R "r((st oe, az) so =] Vs S. Each player chooses his policy regardless of the opponent. Protagonist attempts to maximize the value function (i.e., total expected discounted reward), and adversary seeks to minimize this function.
Nash equilibrium is a key role in game theory, which one kind of game solution concept. A Nash equilibrium (T²", #*) in zero-sum Markov game exists when the following relation holds (Shapley
3

---

Under review as a conference paper at ICLR 2021
1953
Başar & Olsder 1998):
00 (8)
max min B
( 01, (t)) |80 =
(1)
t=
00
min max T
(rr(s, 0e, 0z) 80 = 8),
T
t=
where x² and H are the optimal policies of protagonist and adversary respectively, is optimal equilibrium value of the game. In such situation, neither player can improve their respective returns, and there is an important relation., i.c., Va,, H. u
S e S 0
4
BAYESIAN-OPTIMIAATION DRRR
ROBUST MORL
4.1 OVERVIEW
We propose a generalized robust MORL framework to learn a single parametric representation for robust Pareto optimal policy over the space of preferences (see Algorithm [ for implementation scheme based on DDPG). The optimization process of our proposed approach is illustrated in Figure 2 Bayesian model based on Gaussian process is adopted to predict the Pareto quality and estimate the model uncertainty. Then, using the Bayesian model, acquisition function (Frazier 2018) can determine optimal guess point, which the suggested preference in our task. In order to prevent the policy from falling into local optimum, some preferences is randomly sampled from replay buffer, which guide the training of the agent together with the preferences from BO. addition, the policy of the adversary evolves in the opposite direction the policy of the protagonist in each preference.
In Sections 4.2 and through incorporating zero-sum game into MORL, environmental uncertainty is modeled as adversarial agent. This means that the protagonist needs learn Pareto optimal policy under attack from the adversary. In Section 4.4, inspired by Shannon- Wiener diversity index, novel metric for Pareto quality is presented to evaluate the distribution of Pareto solutions from diversity and evenness. Moreover, combined with hypervolume index, a comprehensive metric is designed, which evaluate the convergence, diversity and evenness for solutions in Pareto set. In Section 4455 regard agent's learning process as a black-box, and the comprehensive metric for the approximated Pareto frontier is computed after each episode of training, then BO algorithm is adopted to guide the protagonist to evolve towards improving the Pareto quality (i.e., maximizing the comprehensive metric).
Objective-2
Objective-2
Objestiv-22
Objcctive-2
Objective-l
Objective-
Objective-]
Objocrive-]
Suggreeed Friterncc from Dbyyndes Optimisttion
Sampled Profenacce for Prongreis frrm Replay Bufler
Sempled Prifonnoo for Advonary from Replay Buller
Confidene ineeval
Apprssimat Pantn fnesier
Aegustion Fundim
Pursto Ogtinal Solation on Sugasited Profecace
Parsto Oyiarll Saluin on Sampled Prefeencc
Optnall Guew Point
Figure 2: Illustration for process to approximate well-distributed robust Pareto frontier through the proposed algorithm.
4.2 ROBUST MULTI-OBIECTIVE MDP
In this section, propose robust multi-objective MDP (RMO-MDP), which considers both the Pareto optimality and robustness for the learned policies. Probabilistic action robust MDP (PR-MDP) TTssler al. 2019) is adopted to improve the robustness of the policies, which can be
4

---

Under review as a conference paper at ICLR 2021
Poreoo Froet Pureio Froes Parcoo Froet
Objective-2
Objective-2
Objective-1
Objective-1
Figure 3: Illustration of the mismatch between the Pareto optimal solution and the correspond- ing preference. Suppose the point A represents a Pareto optimal solution, which and the origin form the vector O'A. The corresponding prefer- ence vector can be represented by the vector OB, In most cases, O'A is not parallel to OB.
Figure Quality analysis of Pareto frontiers. The Pareto frontiers 1, 2 and 3 are approximated by different approaches. The green, blue and purple points represent the solutions frontiers 1, and 3 respectively. The hypervol- ume formed by the solutions on Pareto front and the reference point is the blue shaded region.
Pareto
regarded special zero-sum game between protagonist and an adversary. We refer to the optimal policies of the protagonist as robust Pareto-optimal policies in RMO-MDP, which the difference from the MOMDP is that the action space here includes not only the actions of the protagonist, but also the actions of the adversary with a certain probability.
Definition 3. A RMO-MDP can be defined by the tuple AS,A P, P,R. Y, A is the mixed action space. The mixed policy π mix (I, T) is defined as xmix ma 8, s,w) = (1 a))
w) a w. vs S and E [0, 1]. I and ā policies the players take, and mix ~ (m)),
In this work, in order to improve the quality of the approximated Pareto frontier, the scalar utility function Un is designed non- non-linear combinations of objectives:
Un1 mix w) = TTQ mi (I, (s, (s,a mix w) kM(s, mix ,w),
Q mix w)
M(s, mix ww = Q((8, amix w)ll
| l|
Qaa mix (x,5t) (xx,))
mix w)
az) Q() w) (s(s,,
where (s, mix w) metric, which can evaluate the mismatch between the Pareto optimal solution and the corresponding preference. Figure illustrates metric function M (s, mix w) in more detail. The distribution of solutions on the Pareto front can be more well-distributed through optimizing function (s,, a mix w). & is coefficient can adjust the role of M(, mix w) in the utility function 3 For a protagonist, A is negative, and & is positive for an adversary. This means that the policy with higher preference is more likely to be violently attacked by adversary, which can makes the policy with higher preference stronger robust.
Under the condition of adversary attack, the utility value of protagonist's policy can be defined as miix w)]. Therefore, the robust Pareto optimal policy optimal policy in RMO-MDP, which can be represent as:
e ming E= """"""" (x, [Un(s, (x,")
T E arg max ET (T.,) (Ux(s, mix amix, w)]
(6)
The complexity of greedy solution to finding the Nash equilibria policies is exponential in the cardinality of the action space, which makes it unworkable in most cases (Schulman al. 2015). In addition, most two player discounted zero-sum Markov game methods require solving for the equilibrium policy of minimax action-value function at each iteration. This is typically intractable optimization problem (Pinto et al. 2017). Instead, we focus on approximating equilibrium solution to avoid this tricky optimization.
5

---

Under review as a conference paper at ICLR 2021
4.3 POLICY ITERATION FOR RMO-MDP
In this section, we present a policy iteration (PI) approach for solving RMO-MDP called robust multi-objective PI (RMO-PI). RMO-PI algorithm can decompose the RMO-MDP problem into two sub-problems (policy evaluation and policy improvement) and iterate until convergence.
4.3.1 ROBUST MULTI- OBIECTIVEEE POLICY EVALUATION
In this stage, the vectorized Q-function learned to evaluate the policy и of the protagonist. With Equation we define the target vectorized Q-function
E
R + -Q= (T.F) mix wi o 6]
Eg ja jamis
{R + 7/(1 a Q a w o + aQ(s's (0;00"0,;;:00) w
Then, we minimize the following loss function at each step:
(=) = E"a mix [1
(s, a, Wro: 2)12 + Ilyws. - ((s,, a, who: x)
where @ and o are the parameters of the Q-function network and the target Q-function network, wrb and W/o are obtained from replay buffer and Bayesian-optimizaton Yors and ywu.. represent y(s', anixx, wrb) and y(s', a Who), respectively. In order to improve the smoothness of the landscape of loss function, the auxiliary loss setting is used (Yang et al. 2019):
L2 (Ø)
E min
Q(s, w Φ) |
Iww.oWw.. Hur. who Qss, a, Whot 2)1
The final loss function can be written as: L(Ø) (1- ()-8)) + 3L2()), where 8 is a weighting coefficient to trade off between losses In (Ø) and (2()).
4.3.2 ROBUST MULTI-OBIECTIVE POLICY IMPROVEMENT
In RMO-PI, policy improvement refers to optimizing and updating the policies of a protagonist and adversary for the given utility function. RMO-PI optimizes both of the agents through the following alternating process. In the first stage, the policy of protagonist is learned while holding the adversary's policy fixed. the second stage, the policy of protagonist is held constant and the adversary's ' policy is learned. This learning sequence is repeated until convergence.
The protagonist seeks to maximize the utility function Un. and then the policy gradient can be represented as: Vo = Vol + VoL where
VoLw,, ~ = [1 (1-a)V Q Whbi o) VoT whoi W VoT wro
0)
(10)
VOL ~ = mix
[l-a)V aw Q(, Who:
wbo 1)
01]
(1I)
0 is the model parameters of the protagonist.
Next, the adversary tries to minimize the utility function Ug. and the policy gradient can be written as: VaL" = VoLū + ViLū... where
8 wib se R [aV
wr:; 0) + A M( a. wrb V Wrb: 0)
12)
V
VgL wro se aV w bo Q( a Who V Who: 0) + AV M a, Who) V who: 0)
(13)
Ø is the model parameters of the adversary. The derivation details of the policy gradients are available Appendix A.1.2
4.4 METRICS FOR PARETO REPRESENTATION
Since the true Pareto set is intractable to obtain in complex problems. the goal of MORL is to find the set of policies that best approximates the optimal Pareto front. Many researchers have reported the works for quality metrics of Pareto front (Cheng et al.) 2012 Parisi et al.) 2017 Audet et al.) (018).

---

Under review as a conference paper at ICLR 2021
Hypervolume indicator is widely adopted to evaluate the quality of an approximated Pareto frontier. which can measure the convergence and uniformity for the distribution of Pareto solutions (Zitzler & Thicle 1999. Xu et al.) 2020). From our perspective, this indicator may be difficult to accurately measure the uniformity of the Pareto solution distribution.
As shown in Figure suppose the Pareto frontiers 2 and 3 are obtained by different algorithms, and compared with the Pareto frontiers 2 and 3, although the hypervolume metric formed by the solutions on Pareto frontier and the reference point O is optimal, the distribution of solutions on the frontier is not well-distributed, which makes the valid preferences of the practitioner or the agent to choose is very limited. Moreover, imagine the solutions on Pareto frontier are very close to each other or even overlap into one solution. At this time, if we adopt the metric (integrated hypervolume metric and sparsity metrie) proposed in the paper (Xu et al., 2020) to measure the quality of Pareto frontier 1, the result to have high hypervolume and low sparsity is very ideal. However, such Pareto frontier 1 might not satisfy the needs of the practitioner or the agent.
word, the high quality of the approximated Pareto frontier is expected to have high hypervolume, and the distribution of solutions is well-distributed. Therefore, in this section, we proposed a novel metric for quality of the approximated Pareto frontier through combining hypervolume metric and evenness metric.
Inspired by Shannon- Wiener diversity index, the diversity metric for the solutions of the Pareto Σ [p; in (p:)], where P represents the solutions of the Pareto
frontier can be expressed as D(P) frontier, and p; is the proportion of the number of non-dominated solutions in the corresponding solution interval the total number of the solutions of Pareto set Dmaz can be defined as In(Sx), and Sn is the number of solution intervals, Then, our evenness metric E(P) can be represented as D(P)/Dmaz- For example, in Figure 4 Sn 6, and the evenness metrics for the distribution of the solutions on the Pareto frontiers 1,2 and are approximately equal to 0.37, and 0.56, respectively. Hence, we can get the following two inferences.
Pareto frontier. The expected diversity
Proposition 1. As E(P) and Sn increases, the distribution of solutions in Pareto set becomes denser and more uniform, and the Pareto frontier becomes more continuous.
Proposition 2. The Pareto frontier is continuous as E(P) = 1 and S, + co.
Combined with the hypervolume indicator (P)), we propose a comprehensive metric I (P) that can measure the convergence, diversity and evenness of the solutions:
((P) = H
(14)
where À is weight coefficient.
4.5
PARETO REPRESENTATION IMPROVEMENT
In this Section, in order to further improve the representation of the approximated Pareto frontier, the agent's 's learning process is regarded as a black-box, and the comprehensive metric (P) is computed after each episode of training, then BO algorithm is adopted to guide the protagonist to evolve towards maximizing the proposed metric I(P). As shown in Figure 5, the Pareto representation improvement scheme based on di-directed is illustrated. The value of the objective function f(22) equals the value of the comprehensive metric T(P), which is obtaind after each episode of training. In addition, suggested preferences from BO algorithm and sampled preferences from replay buffer are simultancously used to guide the learning process, which is to avoid the algorithm into local optimum. The scheme to guide the learning process with BO has high universality for Pareto quality improvement, and does not require much expert experience in the selection of prediction models.
5 EXPERIMENTS
In order to benchmark our proposed scheme, we develop two MORL environments with continuous action space based on SUMO and Swimmer-v2. Moreover, we also adopt HalfCheetah-v2 and Walker2d-v2. which are two MORL domains provided by Xu et al. (2020). The goal of all tasks is to try to optimize the speed of the agent while minimizing energy consumption. The observation and action space settings are shown in Table I The more details can be found in Appendix A.2
7

---

Under review as a conference paper at ICLR 2021
D Training Data
Surrogate Model
Posterior
Acquisition Function
(2( Uniliy Candidatc Set
Q Candidate Set
Gaussian Process
P((())
Expocted Improvement
Figure 5: Pareto representation improvement scheme based BO algorithm. The surrogate model for the objective function f(22) is typically Gaussian Process. Posteriors represent the confidence model has about the function values at a point or set of points. Acquisition function employed to evaluate the usefulness of optimal guess point corresponding to posterior distribution over ([2). The expected improvement method chosen to design the acquisition function our scheme.
Table 1: Observation space and action space of the experiment environments.
SUMO
Swimmer.v2
HaliCheetah.22
Walker2d
Observation Space 8 R20
SER
SER
S RET
Action Space
AER'
A
R
A
Our algorithm is implemented based on Deep Deterministic Policy Gradient (DDPG) (Lillicrap et al. 2015) framework. In principle, our scheme can be combined with any RL method, regardless of whether it is off-policy or on-policy. Moreover, we implement three baseline methods for comparison and ablation analysis: SMORL represents MO-DDPG method based on linear scalarization function, which is a linear combination of rewards in the form of preference; SRMORL RMO-DDPG approach using linear scalarization function; RMORL represents R.M-- approach with the utility function Un. BRMORL is RMO-DDPG scheme combined with the utility function Un and BO algorithm. More details about the algorithms are described in Appendix A.1.1
Figure 6 and show the learning curves and Pareto frontiers comparison results on SUMO and Swimmer-v2 respectively. Moreover, the results in Table 2 and 3 demonstrate that our proposed BRMORL scheme outperforms
the baseline methods on SUMO and Swimmer-v2 environments in hypervolume and evenness. It can also be found from Figure ([c) the BRMORL mothod is not only able to find solutions on the convex portions the Pareto frontier, but also the concave portions.
30
290
Energy Conservation
(b)
Figure 6: The learning curves and the Pareto frontiers obtained by different algorithms on SUMO.
Table 2: Training results on SUMO.
Table 3: Training results on Swimmer-v2
Ilypervolume
Evenness
Hlpeevolume
Evenness
SMORI
4547.43 +: 1919.14 1919.14
0.45 = 0.30
SMORL 3045 63 2546.65
0.34
SRMORL
4904.25 1 2480.33
0.42 0.37
SRMORL 4164 1: 1487.67
0.20 1 0.18
RMORL 5900.67
+: 2443.58
0.67 0.36
RMORI. 7682
4844.27
0.325
BRMORL
6219.57 2164.15
0.81 = 0.24
BRMORL
8118.58 4344.80
0.74
Figure illustrates that the robustness different policy models under the preference=(0.55005], on
Swimmer-v v2 domain. We test with jointly varying both mass and disturbance probability. Obviously.
the capability to obtain return based on BRMORL approach is less affected by environmental changes than other schemes. Moreover, the standard deviation based the utility of the policy is adopted to
8

---

Under review as a conference paper at ICLR 2021
quantify the robustness. This means that the stronger the robustness of policy is, then the smaller its standard deviation is. Table 4 shows the quantitative analysis results of robustness under different Swimmer-v2. For more results and implementation and and A.4.1
preferences and environmental changes, details, please refer to Appendix A.3
2080
1080
390
Edlsode 64)
Episode
Energy Conservation
(b)
(C)
Figure 7: The learning curves and the Pareto frontiers obtained by different methods Swimmer-v2 v2
Preferunce [0.S 8.5
65
21 0.3 o4
14 ar 24 6.1
53 DJ O. 81
(4)
(9)
(6)
(0)
Figure 8: Robustness to environmental uncertainty. Disturbance probability represents the probability of random disturbance being played instead of the selected action. Relative mass denotes the ratio of the current agent's 's mass to its original mass.
In Table 5 and 7 we compare our BRMORL scheme with state-of-tte-arr baseline (PG- MORL) provided by Xu et al. (2020). Although our method not superior in hy- pervolume, it outperforms the baseline in evenness, robustness and utility. In this sec- tion, the utility is defined as the expectation return based on a policy under envi- ronmental changes. More details and results be found in Appendix A.3 and A.4.2
Table 4: Quantitative analysis results robustness.
Table 5: Test results on Walker2d-v2
[0.1.0.9]
[0.3,0.7]
[0.5.0.5]
[0.7,033]
[0.9.0.1]
PGMORL
BRMORL
SMORL.
8.96
9.55
4.61
9.58
16.48
Hypervolume
57132.70
30737.01
SRMORI.
0.44
1.01
3.07
7.88
15.86
Evenness
RMORL
1.99
0.59
3.11
8.45
14.36
Robustness
34.91
14,53
BRMORL
Utility
-193.86
-11.63
CONCLUSION AND DISCUSSION
In this paper, we proposed a generalized robust MORL framework to approximate representation for robust Pareto frontier, which allows our trained single model to produce the robust Pareto optimal policy for any specified preference.
Our experiments across four different domains demonstrate that scheme is effective and advanced. Most importantly, we note that training with appropriate adversarial setting can not only result in robust policies. but also improve the performance even. Moreover, both solutions convex and concave portions of the Pareto frontier can be found through our approach. Although our scheme cannot guarantee the learned policy is optimal, it is approximately robust Pareto optimal.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Abbas Abdolmaleki, Sandy H Huang, Leonard Hasenclever, Michael Neunert, H Francis Song, Martina Zambelli, Murilo F Martins, Nicolas Heess, Raia Hadsell, and Martin Riedmiller. A distributional view on multi-objective policy optimization. arXiv preprint arXiv:2005.07355 2020.
Axel Abels, Diederik Roijers, Tom Lenaerts, Ann Nowé, and Denis Steckelmacher. Dynamic weights in multi-objective deep reinforcement learning. In International Conference on Machine Learning. pp. 11-20. PMLR, 2019.
Charles Audet, J Bigeon, D Cartier, Sébastien Le Digabel, and Ludovic Salomon. Performance indicators multiobjective optimization. Optimization Online, 2018.
Leon Barrett and Srini Narayanan. Learning all optimal policies with multiple criteria. In Proceedings of the 25th international conference Machine learning, pp. 41-47, 2008.
Tamer Başar and Geert Jan Olsder. Dynamic noncooperative game theory. SIAM, 1998.
Andrea Castelletti, Francesca Pianosi, and Marcello Restelli. Tree-based fitted q-iteration for multi- objective markov decision problems. The 2012 International Joint Conference on Neural Networks (IJCNV), pp. 1-8. IEEE, 2012.
Xi Chen, Ali Ghadirzadeh, Marten Björkman, and Patric Jensfelt. Meta-learning for multi-objective reinforcement learning. arXiv preprint arXiv:181/ 1. 1.03376, 2018.
Shi Cheng, Yuhui Shi, and Quande Qin. On the performance metrics of multiobjective optimization.
In International Conference in Swarm Intelligence, pp. 504-512. Springer, 2012.
Esther Derman, Daniel Mankowitz, Timothy Mann, and Shie Mannor. A bayesian approach robust reinforcement learning. In Uncertainty in Artificial Intelligence, pp. 648-658. PMLR, 2020.
Peter Frazier. A tutorial on bayesian optimization. arXiv preprint
2018.
Vineet Goyal and Julien Grand-Clement Robust markov decision process: Beyond rectangularity. arXiv preprint 181.0025, 2018
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and 2018.
applications. arXiv preprint
Jemin Hwangbo, Joonho Lee, Alexey Dosovitskiy, Dario Bellicoso, Vassilios Tsounis, Vladlen Koltun, and Marco Hutter. Learning agile and dynamic motor skills for legged robots. Science Robotics, 4(26), 2019.
Xuewu Ji, Xiangkun He, Chen Lv, Yahui Liu, and Jian Wu. Addapive-neurral-eewworkk--ss robust lateral motion control for autonomous vehicle at driving limits. Control Engineering Practice, 76: 41-53, 2018
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02977 2015.
Chunming Liu, Xin Xu, and Dewen Hu. Multiobjective reinforcement learning: A comprehensive overview. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 45(3):385-333 2014.
Pablo Alvarez. Lopez, Michael Behrisch, Laura Bieker-Wan Jakob Erdmann, Yun-Pang Flötteröd, Robert Hilbrich, Leonhard Lücken, Johannes Rummel, Peter Wagner, and Evamarie Wießner. Mi- croscopic traffic simulation using sumo. In The 21st IEEE International Conference on Intelligent Transportation Systems. IEEE, 2018 https: http://liii.. alr http://lib..lrr.eee/22/44999999
Shie Mannor and Nahum Shimkin. The steering approach for multi-criteria reinforcement learning. In Advances in Neural Information Processing Systems, pp. 1563-1570, 2002.
Shie Mannor, Ofir Mebel, and Huan Xu. Lightning does not strike twice: Robust mdps with coupled uncertainty. arXiv preprint / 2012.
10

---

Under review as a conference paper at ICLR 2021
Vladimir Mazalov. Mathematical game theory and applications. John Wiley & Sons, 2014.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 51 518(7540):529-533 2015.
Hossam Mossalam, Yannis M Assael, Diederik M Roijers, and Shimon Whiteson. Multi-objective deep reinforcement learning. arXiv preprint 161vv06610002007. 2016.
Sriraam Natarajan and Prasad Tadepalli. Dynamic preferences in multi-criteria reinforcement learning. In Proceedings of the 22nd international conference Machine learning, pp- 601-608, 2005.
Kolby Nottingham, Anand Balakrishnan, Jyotirmoy Deshmukh, Connor Christopherson, and David Wingate. Using logical specifications of objectives in multi-objective reinforcement learning. arXiv preprint arXiv: 19100.1130.. 2019.
Simone Parisi, Matteo Pirotta, and Jan Peters. Manifold multi-objective policy search with sample reuse. Neurocomputing, 263:3-14, 2017.
Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial reinforce- ment learning. arXiv preprint arXiv: /tf:::770002000 2017.
Matteo Pirotta, Simone Parisi, and Marcello Restelli. Multi-objective reinforcement learning with continuous pareto frontier approximation. In 29th AAAI Conference on Artificial Intelligence, AAAI 2015 and the 27th Innovative Applications of Artificial Intelligence Conference, IAAI 2015, pp. 2928-2924 AAAI Press, 2015.
Diederik M Roijers, Peter Vamplew, Shimon Whiteson, and Richard Dazeley. survey of multi- objective sequential decision-making Journal of Artificial Intelligence Research, 48:67-113, 2013.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889-1897, 2015.
Lloyd S Shapley. Stochastic games. Proceedings of the national academy of sciences, 39(10):
1095-1100, 1953.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schritwiessrr, loannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484 489, 2016.
Gerald Tesauro, Rajarshi Das, Hoi Chan, Jeffrey Kephart, David Levine, Freeman Rawson, and Charles Lefurgy. Managing power consumption and performance of computing systems using reinforcement learning. In Advances in Neural Information Processing Systems, pp. 1497-1504, 2008
Chen Tessler, Yonathan Efroni, and Shie Mannor. Action robust reinforcement learning and applica- 2019.
tions in continuous control. arXiv preprint
Andrea Tirinzoni, Marek Petrik, Xiangli Chen, and Brian Ziebart. Policy-conditioned uncertainty sets for robust markov decision processes. In Advances in Neural Information Processing Systems, pp. 1939-8949, 2018.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-basee control. In 2012 IEFF/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012.
Peter Vamplew, Richard Dazeley, Adam Berry, Rustam Issabekov, and Evan Dekker. Empirical evaluation methods for multiobjective reinforcement learning algorithms. Machine learning. 84 (1-2):51-80, 2011.
Kristof Van Moffaert and Ann Nowé. Multi-objective reinforcement learning using sets of pareto dominating policies. The Journal of Machine Learning Research, 15():334333351, 2014.

---

Under review as a conference paper at ICLR 2021
Kristof Van Moffaert, Madalina M Drugan, and Ann Nowé. Scalarized multi-objective reinforcement learning: Novel design techniques. In 2013 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL), pp. 191-199. IEEE, 2013.
Jie Xu, Yunsheng Tian, Pingchuan Ma. Daniela Rus, Shinjiro Sueda, and Wojeiech Matusik. Prediction predictionguuded guided multi-objective reinforcement learning for continuous robot control. In Proceed- ings of the 37th International Conference on Machine Learning. 2020.
Runzhe Yang, Xingyuan Sun, and Karthik Narasimhan, A generalized algorithm for multi-objective reinforcement learning and policy adaptation. In Advances in Neural Information Processing Systems, pp. 14636-14647, 2019. 2019.
Pengqian Yu and Huan Xu. Distributionally robust counterpart in markov decision processes, IEEE Transactions on Automatic Control, 61(9)22538-2433 2015.
Eckart Zitzler and Lothar Thiele. Multiobjective evolutionary algorithms: a comparative case study and the strength pareto approach. IEEE transactions on Evolutionary Computation, 3(4):257-271, 1999.
Marcela Zuluaga, Andreas Krause, and Markus Püschel. E-pal: active learning approach to the multi-objective optimization problem. The Journal of Machine Learning Research, 17(1): 2016.
A APPENDIX
A.I ALGORITHM
A.I.I ALGORITHM OVERVIEW
The details of the BRMORL, RMORL, SRMORL and SMORL schemes are provided Algorithm a respectively.
12

---

T) Under review as a conference paper at ICLR 2021
Algorithm 1 Bayeesarropiimiatttoonddirredd robust multi-objective DDPG
Input: probability a, weighting coefficients B3, X, T, Randomly initialize actor (s, w; 0), adversary T ((s, w; 0) and critic network a(;;), Initialize target networks with weights A_ A and Φ
Initialize replay buffer B and comprehensive metric I for episode = O...M
Receive initial state
Predict preference who through Bayesian-optimization Who [T)
for = 0...7 do
if f & 2 then
W: - Who
else
Sample preference wad based on uniform-distribution::
W: Wad
end if
T (8t, wx; 0)
w.p. (1)
Sample action a,
(se,, w;; 0)
otherwise
a, exploration noise
Execute action 0e and observe reward and new state St-1 Store transition (st, we, de, re, S in B
Extend the dimension of preference who to batch size for O...N do
Sample batch from replay buffer B
Update actor according to equations I0 and
0 VoLw,, VaL
Update critic according to equations and
ó - -1 V
BV L2(
end for
Sample batch from replay buffer B
Update adversary according to equations 12 and
0 VoLE.. + VVLā...
Update the target networks:
A c TB +
θ - 70 + 7
+ + T + (1-1)
end for
Compute comprehensive metric (PP) using equation 14 and Pareto solution set end for
13

---

Under review as a conference paper at ICLR 2021
Algorithm 2 Robust multi-objective DDPG with the utility function Un
Input: probability a, weighting coefficients B, T, Randomly initialize actor (s,, w; 0), adversary ((s,, w; Initialize target networks with weights A_, A and Φ Initialize replay buffer B and comprehensive metric I for episode O...M
0) and critic network a(;
Receive initial state
for t = O...' do
Sample preference Wad based on uniform-diitriiution:
W, < Wid
x (51, wi:0) w.p. - a)
Sample action a;
(8e, w;; 0) ootherwise
à = a, + exploration noise Execute action âe and observe reward and new state 5c-1 Store transition (3t, Wi, des Te, 8 in for = O...N do
Sample batch from replay buffer
Update actor according to equations I0
0 + Val
U Update critic network:
(1 B) V B mix
|1 -Q w )|
BV VVBT" Thte [Iw.t wh, Q(s a, wro: ||
end for
Sample batch from replay buffer B
Update adversary according to equations 12
8 + VoLw...
Update the target networks:
0 - 70 + (1 - T
i- - 70 + (1 - -
Φ - TΦ (1 T)
end for
end for
14

---

Under review as a conference paper at ICLR 2021
Algorithm 3 Robust multi-objective DDPG with linear scalarization function
Input: probability a, weighting coefficients T Randomly initialize actor (s,, w; 0), adversary ((ss w; 0) and critic network a(;,), Initialize target networks with weights A_ A- and Φ
Initialize replay buffer B and comprehensive metric I for episode O...M
Receive initial state
for t = O.. do
Sample preference Wad based on uniform-diitriiution:
W, < Wid
x (51, wx: w.p. - a)
Sample action a;
(8e, w;; ootherwise
à = a, + exploration noise Execute action âe and observe reward and new state 5c-1 Store transition (3t, Wi, des Te, 8 in for = O...N do
Sample batch from replay buffer Update actor network:
A E [[1 a) Va)aa, Q(, (s, wrii (VV 0)]
Update critic network:
(1 8) V
[law... Q(s wrs 1)
VV [Wwhhww.. - w Q(s, a, wi:: ))
end for
Sample batch from replay buffer B
Update adversary network:
0 - ET
(a aw (s, Wyoi (87
Update the target networks:
θ t + + (1 -
t 70 + -
Φ - TΦ (1 -T)
end for
end for
15

---

Under review as a conference paper at ICLR 2021
Algorithm 4 multi-objective DDPG with linear scalarization function
Input: weighting coefficients 3 and
Randomly initialize actor (ss, w; 0), adversary ((s, w; 0) and critic network Q(s, a, w; Initialize target networks with weights A_, A- and Φ Initialize replay buffer B and comprehensive metric I for episode O...M
Receive initial state
for t = O...' do
Sample preference Wad based on uniform-distriiutiin::
W; <- Wid
Sample action a, ([st:We;;)
ãr = + exploration noise
Execute action a, and observe reward r; and new state 8c-1 Store transition (st, wi, 0e. Tes S1+1) in B
for O...N do
Sample batch from replay buffer B
Update actor network:
0 + E" Naw Q(, a, wirs Φ) (o, (s, 0)]
Update critic network:
(1 B) V n
|lww. wri:))
VV Bo WTQ 2)
end for
Update the target networks:
0 - 70 + (1 0
θ t 7 + T)
+ TΦ (1 =)Ø
end for end for
A.1.2 THEORETICAL DERIVATION
this part, we provide the derivation details of some formulas.
The policy gradient of the protagonist based on wis can be derived:
VOL wa
~ E mix
[VUUn(s, a mix wwri)l
= E
mix
Tow 16 Q
mix (5.9)) (s, mix w Φ)+ + kV,M a mix wr+b]
[1 a) V Q
w.:: + &V Do w,6: f)),
(15)
The policy gradient of the protagonist based who be written
VOL
x =
[oUn mix Who)l
= E Now bo Q miin (,,
mix Whoi Φ) + AV a mix who)
- [11 a) Vaw
Q 60
Whoi VOT Who 0) + a a, Who) VoT() Who: Wbo:))]
(16)
The policy gradient of the adversary based on wyb can derived:
V ViLa
[VgUn(s, mix wro)]
i (T.,5) mix wi:: + &V &V mix wrb))
aV
Q
W Of 0) + &V AV.M M a, (.r w,: 0)),
(17)
16

---

Under review as a conference paper at ICLR 2021
The policy gradient of the adversary based who can be described as:
V wro F [VUUn(, a mix Who)]
(awl Q 2 (1,,5 a mix Whoi Φ) + &VVMM(s (s, mix whol
aV Q Whoi (0 V8 Whoi 0) + AV a MM, 0, Who) V (8, Who: 0)] (18)
A.2 DOMAIN
In this section, give more details about training environment. We demonstrate our approach in four challenging continuous control domains
(a)
(b)
(c)
(d)
Figure 9: (a) Velocity control for autonomous electric vehicle in SUMO; (b) Swimmer environment
Mujoco; (c) Walker environment in Mujoco; (d) HalfCheetah environment Mujoco.
A.2.1 SUMO
SUMO is free and open source traffic simulation suite, which provide intelligent driving function verification environment. Transport efficiency and energy consumption are two conflicting objectives for autonomous electric vehicle. In this paper, robust multi-objective longitudinal decision-making problem is focused to verify the effectiveness and advancedness of our proposed algorithm.
Observation and action space dimension: The first objective is vehicle velocity:
1 II R.
R1
Umax
The second objective energy conservation:
Ux
R2 = PV 1
Umax
where is vehicle longitudinal velocity, Umaz is maximum speed, p 1 is scaling factor.
A.2.2 SWIMMER-V2
Observation and action space dimension: E R2, A E R
The first objective is running velocity:
R1 = R
The second objective is energy conservation:
R2 = max( /1 min Ur, 1)2 - 0.15 Σ af.00 0),
where u, is longitudinal velocity, a; is the action of each actuator.
17

---

Under review as a conference paper at ICLR 2021
A.2.3 WALKER2D-V2
Observation and action space dimension: S R. A R The first objective is running velocity:
R1 Ux + 1,
The second objective is energy conservation:
R₂ = 5 Σ
2 4
where U2 is longitudinal velocity, a, is the action of each actuator.
A.2.4 HALFCHEETAII-V
Observation and action space dimension: & R²7 A R
The first objective is running velocity:
= min min 4) + 1,
The second objective is energy conservation:
R₂ = 5 Σ 2
where U2 is longitudinal velocity, a; is the action of each actuator.
A.3 IMPLEMENTATION
We implement the actor, adversary and critic neural networks by 2 fully connected hidden layers, which layer sizes are {256, 256} and (512, 256} in SUMO and Mujoco respectively.
Our scheme contains three important hyperparameters a, and k. mainly affects the robustness of the policy model. If a is set to a smaller value, the robustness of the model will be reduced, otherwise, the performance of the model will be reduced. and & are responsible for balancing hypervolume, diversity and evenness metrics. if À and k are set to larger values, then the diversity and evenness of the learned policies will be better, otherwise, the hypervolume will converge to a larger value. In order to select appropriate values, these three parameters need to be adjusted in the experiment.
The number of solution intervals Sn is set 9, and the main parameters of our BRMO- DDDG algorithm are reported in the Table 6
Table 6: BRMO-DDPG Parameters.
Parameter Name
Value
a
0.1
0.2
7
0.99
X
0.01
k
100
Sa
batch size
128
replay size
l
noise scale
0.1
learning rate for actor
1
learning rate for critic
1
learning rate for adversary
1
18

---

Under review as a conference paper at ICLR 2021
In the robustness and comparing the PGMORL tests, the utility is designed as follows:
= wT
|rl
3 l12
We test the utility of the policy ten times under each mass and disturbance changes, and calculate the cumulative value of the utility. The mean and standard deviation of each policy's utility obtained from the whole test process are used to evaluate the performance of the policy under the environment uncertainty. The smaller standard deviation is, the stronger the robustness of the policy is. Moreover, for policy the higher the mean is, the stronger the capability to obtain utility under the environment uncertainty is. Hence, the robustness and utility in Table and refer the standard deviation and the mean here.
In the experiment comparing the PGMORL scheme, we evenly selected ten trained policy models. Because we use one model for testing, we evenly selected ten preferences to input the tested model. For the PGMORL method, we tested each model one hundred times and calculated the cumulative value of the utility. For BRMORL scheme, we tested each preference one hundred times, and then calculated the cumulative value of the utility. Therefore, the hypervolume and evenness Table and 7 are caleulated by using the returns.
A.4 RESULTS
In this section, we give more experimental results.
A.4.1 SWIMMER-Y V2
Preference
SMCAL
3RHOOL
1.5
6-5 00
03 04 Dh
Doterbence (c1
64)
Figure 10: Robustness to environmental uncertainty on Swimmer-v2 domain. Disturbance probability represents the probability of a random disturbance being played instead of the selected action. Relative mass denotes the ratio of the current agent's mass its original mass.
GHOOL
SEMOAL
Preference [0.3 0.7)
RMOAL
SAHOOL
Canurbance Procas By
D Ditubberc Fabsbillty by
Desurcance Pobebiity
Desurtancs Probus By
Figure 11: Robustness environmental uncertainty on Swimmer-v2 domain.
A.4.2 HALFCHEETAII-2 AND WALKER2D-V2
The comparison results of the haseline (PGMORI)) are provided here.
19

---

Under review as a conference paper at ICLR 2021
Profereace [87,0.3]
Didusturoe PotAbeile
D'lustinne Pububellyy
Ditusteroo Prtubbi
Ditanturso Pobubbell
(C)
Figure 12: Robustness to environmental uncertainty on Swimmer-v2 domain.
1.5
Telatve Mart
Diturbarce Poistbiity
INturtatcco Noboba by
Disuraance
PHIS P9IS30 By
Figure 13: Robustness environmental uncertainty on Swimmer-v2 domain.
Table Test results on HalfCheetah--2.
PUMORL
BRMORL
Hypervolume 89984,28
21620.47
Evenness
0.31
Robustness
30.80
20.84
Utility -198.23
-22. 12
BeaihnorMManu Preference (0.5, 0.5)
35
0.1
6.2
63
03.
32
8.
64
60
(h)
Figure 14: Robustness to environmental uncertainty on Walker2d-v2 domain. Disturbance probability represents the probability of random disturbance being played instead of the selected action. Relative mass denotes the ratio of the current agent's 's mass to its original mass.
Preference (8.5, [.5)
63 Deturbence Proastd
Distuteece Prbuuiilty (3]
(b)
Figure 15: Robustness to environmental uncertainty on HalfCheetah-v2 domain.
20


Reviews:
[{"review_id": "X-7-qOHSTSi", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a framework to tackle uncertainty in multi-objective optimization of reinforcement learning problems. Uncertainty is represented as an adversary over preferences. Fitness is measured by a multi-objective quality indicators while Bayesian optimization is used to bring improvements. The proposed method is evaluated on four benchmark problems.\n\nWhile the page limit makes it difficult to detail all the different aspect of the contribution, the clarity still needs to be improved. For instance, it is worth adding a detailed algorithmic description of the approach. At various stages there are many different options, so the specific choices should be better introduced and limits discussed.\n\nDetailed comments and questions:\n\na) There are many parameters to set (alpha, beta, omega, lambda), how sensitive are they in practice? They are not mentioned in the experimental part, hence the results cannot be reproduced.\n\nb) Nash games are defined over partitions of the design variables, but it does not seem to be the case here?\n\nc) Robustness can be defined in many different ways (expectation/variance trade-off, quantiles, worst case, chance constraints). How does it differ from optimizing the worst case here? \n\nd) Pareto front quality indicators are widely studied in the multi-objective optimization literature, existing ones should be reviewed first. See e.g.,:\n- Charles Audet, J Bigeon, D Cartier, Sébastien Le Digabel, Ludovic Salomon. Performance indicators in multiobjective optimization. 2020. ⟨hal-02464750⟩\n- Cheng S., Shi Y., Qin Q. (2012) On the Performance Metrics of Multiobjective Optimization. In: Tan Y., Shi Y., Ji Z. (eds) Advances in Swarm Intelligence. ICSI 2012. Lecture Notes in Computer Science, vol 7331. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-30976-2_61\n\ne) Pareto based acquisition functions are generally preferred in Bayesian MOO compared to scalar ones, such as the one proposed here on a GP fit of the defined utility. See for instance:\n- Emmerich, M. T.; Deutz, A. H. & Klinkenberg, J. W. Hypervolume-based expected improvement: Monotonicity properties and exact computation. Evolutionary Computation (CEC), 2011 IEEE Congress on, 2011, 2147-2154\n- Picheny, V. Multiobjective optimization using Gaussian process emulators via stepwise uncertainty reduction Statistics and Computing, Springer, 2015, 25, 1265-1280\n- Predictive Entropy Search for Multi-objective Bayesian Optimization. Daniel Hernandez-Lobato,  Jose Hernandez-Lobato,  Amar Shah,  Ryan Adams ; PMLR 48:1492-1501.\n\nAdditional related references of interest from the literature that could be discussed:\n- Lepird, J. R.; Owen, M. P. & Kochenderfer, M. J., Bayesian preference elicitation for multiobjective engineering design optimization, Journal of Aerospace Information Systems, American Institute of Aeronautics and Astronautics, 2015, 12, 634-645\n- Paria, B.; Kandasamy, K. & Póczos, B. A flexible framework for multi-objective bayesian optimization using random scalarizations Proceedings of The 35th Uncertainty in Artificial Intelligence Conference, PMLR 115:766-776, 2020.\n- Multi-attribute Bayesian optimization with interactive preference learning. R Astudillo, P Frazier\nInternational Conference on Artificial Intelligence and Statistics, 4496-4507\n\n\nf) Fig. 3: either a point is Pareto optimal, or it is not. All blue and violet solutions are dominated by the green ones, so the green Pareto front is better.\n\ng) Fig. 1 lacks a detailed description. Are the different panel successive iterations for instance?\n\nh) Only bi-objective examples are presented, how does it scale with more objectives?\n\nTypos:\nP1: algorithms have demonstrated its worth → their\nFig. 2: can not parallel\n\n## Post rebuttal comments\nThe authors largely modified the paper according to the comments, with a lot of additional content. While this is quite beneficial, the paper raised many questions, some of which may need further treatment (for instance, increasing the number of objectives has an effect on the number of Pareto optimal solutions that is is not trivial).", "extra_scores": {}, "extra_text": {}}, {"review_id": "QdaHHLQmoTJ", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a robust multi-objective RL approach and a non-linear utility metric to enforce an accurate and evenly distributed representation of the Pareto frontier. Robustness is obtained by formulating the problem as a two-player zero-sum game. The goal of the main agent is thus to learn the policies on the Pareto frontier under attacks from the adversary. This is achieved by training a single network to generate approximate Pareto optimal policies for any provided preference. To train this network, they introduce a new metric for Pareto frontier evaluation based on hypervolume and entropy (to force evenly distributed solutions). The resulting algorithm has the classical structure of an actor-critic algorithm where the critic provides an estimate of the Q-function and the actor updates the policies of the protagonist and adversary through alternate optimization.\n\nCould you give more motivations why a robust approach is needed for MORL? The motivation now seems simply to be that the literature didn't take into account robustness. \n\nI think the idea of the paper is quite interesting but it is not well written/explained. I think a few details are missing.\n\nFor example, in section 4.3.1 it is not clear to me how the loss is constructed. You mentioned that the objective is to learn the Q-function of the protagonist but the target value $y$ is built using the mix policy and Q-function. Could you clarify the meaning of this loss?\nThere are also two different definitions of $y$ ($s'$ vs $s$ and $\\omega'$ vs $\\omega$).\n\nWhat is the meaning of $\\mathbb{E}^{\\pi^{mix}}$? Does it mean expectation wrt to the stationary distribution induced by the policy?\nIt is important for understanding equations 8 and 9. Why there is an approximation in the definition of the gradients? Shouldn't be $\\nabla \\mathbb{E}^{\\pi}[]$?\n\nConcerning section 4.4, you compare with the metric proposed in (Xu et al. 2020) without explaining it. It is hard for the reader to understand why it is not a good metric without knowing the metric. In general, as you acknowledged, a good approximate Pareto frontier should be accurate, evenly distributed and have a covering similar to the one of the true Pareto frontier. These properties are not equally important. This is to say that I think the example in figure 3 has a major drawback. Frontier 2 and 3 are not Pareto frontier since are dominated by 3. Not sure that everyone will agree that 2 is better than 3. I suggest you change the example.\n\nTo overcome this, you introduced an entropic measure based on a partitioning of the Pareto frontier. There is no mention of how to do that in practice. Could you explain it? Are you partitioning the space of preferences?\nA standard measure to enforce spread solutions is the crowding distance (eg in your reference Parisi et al. 2017 and many more), could you apply the same idea of interval partition on this?\nCould you explain why you introduced evenness as a multiplicative factor rather than an additive one in $I(P)$?\n\n\nExperiments. There is no information about the implementation and parameters/configurations used. It is thus very difficult to parse the results. For example, how are preferences selected for standard methods (ie all expect BRMORL)?\nWhy is the comparison with (Yang et al. 2019) missing? I think this is a relevant algorithm for the setting.\nYou should add an explicit reference to the papers introducing the methods in Table 3, ie RA, PFA, MOEA/D and META. \n\nI think the paper contains an interesting idea but, given the mentioned concerns, I think this is a borderline paper. I'm looking forward to the authors' feedback.\n\nMinor issues\n-I think it's more precise to define \\mathcal{A}^{mix} as the space of possible combinations of actions. The probability $\\alpha$ should be associated with $\\pi^{mix}$ rather than with the action space.\n-Reference missing to SUMO\n-You use both $\\mathbb{E}_\\pi$ and $\\mathbb{E}^{\\pi, \\pi'}$ but these symbols are never explained\n-Figure 2:  \"OA can not parallel to OB\"\n-You use often the term convergence in the description of the algorithm. How do you evaluate convergence?\n-In figure 4 what is $\\alpha(\\Omega)$? Is it related to $\\alpha$ used in the definition of the mix policy?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "IId0S4t0fKr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a novel approach for solving MORL problems while considering uncertainty in the Pareto frontier.\nThe contributions are interesting and novel, but the paper has several flaws which make it not ready for acceptance, especially regarding experiments.\n\nFirst, the authors overlook many Bayesian MORL algorithms which also consider the Pareto frontier uncertainty. For instance, \n\nCalandra et al, \"Pareto Front Modeling for Sensitivity Analysis in Multi-Objective Bayesian Optimization\"\n\nCalandra et al, \"Bayesian Multiobjective Optimisation With Mixed Analytical and Black-Box Functions: Application to Tissue Engineering\")\n\nHernandez-Lobato et al, \"Predictive entropy search for multi-objective Bayesian optimization\"\n\nOlofsson et al, \"Bayesian multi-objective optimisation of neotissue growth in a perfusion bioreactor set-up\"\n\nIn the introduction the authors say \"In addition, most approaches still only work in domains with low-dimensional and discrete action spaces.\" This is not true. Simply, all cited algorithms have just been tested on low-dimensional problems. Since they were not evaluated on larger problems, we do not know how they behave. The authors do not even include any of them in the evaluation to show that they actually fail.\nFurthermore, the authors claim that their experiments have large action spaces. How big are they? This is not mentioned in the paper. For instance, the Mujoco environments tested in the paper do not have that large action spaces (eg, Swimmer has 15 actions).\n\nThe writing can be also improved. The sections feel a bit disconnected, and sometimes it is not easy to highlight the contributions. For instance, Section 4.3.1 takes quite some space and seem to be part of the novelty contributions, but the losses are taken from Yang et al.\n\nMy biggest issue is with the experiments. First, the authors should say right away on which environment they are testing the algorithms, and not just \"Mujoco\" and \"two provided by Xu et al\". Furthermore, these two environments are never actually tested, since the experiments are only on SUMO, Swimmer, Walker, and HalfCheetah.\nAnd why are results for the Swimmer shown as figure, while Walker and Cheetah have tables? And why these environments out of all Mujoco ones? (these three are known to be the easiest).\nMoreover, Mujoco environments are single-objective. How did you turn them into multi-objective? This is crucial and not mentioned.\nFinally, there is no evaluation against any of the MORL algorithms mentioned in related work. The paper makes the point that these should fail with large action spaces and uncertainty, but this is not shown.\n\nOverall, the experiments section feels rushed and incomplete, and the paper is not ready for publication.\n\n** EDIT **\nThe authors added many experiments, tables, figures, sections, and an appendix. The changes are too substantial and the paper looks like a completely new one. \nThe purpose of author rebuttals is to address issues like a reviewer’s uncertainty about a point, an incorrect assumption, a misconception, or a misunderstanding of a part of the paper, not to revamp the paper completely.\nThe paper was clearly incomplete at the time of its submission, and I still vote for its rejection. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "6isphtcv46i", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper seeks to train multi-objective RL policies that are robust to environmental uncertainties. There are two main contributions: a novel approach to solve this problem, and a novel metric to evaluate Pareto fronts. The metric combines the typical hypervolume metric (that captures the quality/performance of a Pareto front) with a novel \"evenness\" metric, that captures how well solutions are spread out across the space of preferences. The proposed approach, called BRMORL, consists of training a protagonist policy that maximizes utility alongside an adversarial policy that seeks to minimize utility (motivated by zero-sum game theory), while using Bayesian optimization to select preferences to train on, in order to optimize the hypervolume-and-evennesss metric. Both the protagonist and adversarial policy are conditioned on preferences.\n\nRecommendation:\nThis paper connects two seemingly orthogonal problems, multi-objective RL and robustness. This is an interesting topic, but there are several issues regarding clarity and the motivation (as detailed in the cons list below). I think this paper could be a valuable contribution for MORL, but _not_ for MORL that is robust to environmental uncertainty, which is what the claim is. Thus I recommend rejection.\n\nPros:\n* Training policies that are robust _and_ flexibly trade off between preferences is an interesting and relevant problem.\n* The empirical evaluation shows that the approach outperforms ablations and an existing state-of-the-art MORL approach (Xu et al. 2020) on continuous control tasks.\n\nCons:\n* Clarity: the introduction should clearly define what _robustness_ means. Currently it's unclear what problem this paper is trying to solve. Does the approach try to achieve robustness to environment dynamics / perturbations, or robustness across preferences, or both? My interpretation is that robustness refers to both kinds. I can understand how BRMORL would improve robustness across preferences, and perhaps also perturbations, but am skeptical about whether it improves robustness to environment dynamics (see next point).\n* The motivation behind this approach is questionable: I'm not convinced that BRMORL actually leads to training policies that are more robust, with respect to environment dynamics or perturbations. This is not shown clearly in the empirical evaluation, and also is not obvious from the approach itself. I don't see the connection between having an adversarial policy and being robust to the dynamics of the system (e.g., masses of limbs). Figure 6 shows that BRMORL has better robustness to environmental uncertainty than SMORL, but that could just be because SMORL is the worst-performing ablation, and just doesn't find particularly high-performing policies (as shown in Figure 5c). How does BRMORL compare to RMORL or SRMORL?\n* It would help to have an algorithm box for BRMORL, that clarifies how the adversary policy, protagonist policy, and Bayesian optimization are used to gather data and for training.\n* The proposed metric is questionable. The goal is to capture both diversity and quality of solutions, but in Figure 3, I would argue that Pareto front 1 is indeed better, because these points dominate _all_ of the points on Pareto fronts 2 and 3, and the purpose of MORL is to find non-dominated policies.\n* The chosen scalar utility function $U$ is not properly justified. In particular, does $M$ (in Equation 2) still make sense when the objectives have significantly different reward scales (e.g., if one objective's return is typically from 0 to 10, and the other's is from 10 to 100)? Even after normalizing, the Q-value term will only be in a portion of the first quadrant, whereas the $w$ term can cover the entire first quadrant.\n* Unjustified hyperparameters for trading off between terms in the losses: $k$ in the scalar utility function, $\\beta$ for the two terms in the Q-function loss, and $\\lambda$ for the comprehensive metric that combines hypervolume and evenness. How should these be chosen?\n* The Related Work doesn't give enough credit to existing MORL approaches. First, Xu et al. (2020) is actually able to find a well-distributed set of Pareto-optimal solutions. In addition, existing methods are stated to only be able to find solutions on the convex portions of a Pareto front. Bringing up this point implies that BRMORL does better (i.e., is able to find solutions on concave portions of the Pareto front), but this is not shown empirically. Finally, the related work states that most existing approaches are only applied to domains with discrete action spaces. It should acknowledge that both Abdolmaleki et al. (2020) and Xu et al. (2020) are applied to high-dimensional continuous control tasks.\n* Lack of experimental details for reproducibility, e.g., network architetures and DDPG hyperparameters.\n\nOther comments:\n* There are quite a few grammatical errors and typos throughout the paper.\n* Definition 3 is imprecise. First, is $a$ a policy or an action? It seems like it should be a policy because it's a member of the policy set, but it's used to denote actions in the previous section, Section 3.1. Also, why are $I$ and $II$ included in the game definition, when they are already represented by the policy sets?\n* There is not enough explanation given for Figure 1. Where do the uniformly-sampled preferences come from (the gray dashed lines)? What is the \"optimal guess point\"? Does Bayesian optimization only suggest one preference at a time (in red)? What is the acquisition function? (This is defined too late in the paper, and only in the caption for Figure 4.)\n* It would be more accurate to make the $k$ explicit in equations 8 and 9, because it's different in $M(\\cdot)$ for the two equations, but the current notation implies it's the same.\n* In the empirical evaluation, SRMORL, an ablation of BRMORL, finds policies that dominate those found by BRMORL (Figure 5c). How can this be interpreted / explained?\n* Table 3 needs an accompanying explanation of the different MORL methods.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper studied multi-objective reinforcement learning (MORL), and provided a Bayesian optimization approach for challenging MORL scenarios in several simulation environments. The reviewers generally find it interesting to account for robustness in a MORL setup, and all appreciate the algorithmic contributions. However, there were shared critical concerns among \ reviewers in the technical clarity and positioning of the work. 

The paper has gone through substantial changes during the rebuttal period, which addressed some concerns regarding the experimental details; however, the major revision raised further issues that affects the clarity of the work. The reviewers are hence unconvinced that the paper is ready for publication. In addition to addressing the existing comments on clarifying the experimental details and properly positioning the work against prior art, a reorganization and optimization of the main content would be beneficial for future submission.

Decision: Reject