Approximating Pareto Frontier through Bayesian-optimization-directed Robust Multi-objective Reinforcement Learning

1 INTRODUCTION . Reinforcement learning ( RL ) algorithm has demonstrated its worth in a series of challenging sequential decision making and control tasks , which train policies to optimize a single scalar reward function ( Mnih et al. , 2015 ; Silver et al. , 2016 ; Haarnoja et al. , 2018 ; Hwangbo et al. , 2019 ) . However , many real-world tasks are characterized by multiple competing objectives whose relative importance ( preferences ) is ambiguous in most cases . Moreover , uncertainty or perturbation caused by environment dynamic change , is inevitable in real-world scenarios , which may result in lowered agent performance ( Pinto et al. , 2017 ; Ji et al. , 2018 ) . For instance , autonomous electric vehicle requires trading off transport efficiency and electricity consumption while considering environmental uncertainty ( e.g. , vehicle mass , tire pressure and road conditions might vary over time ) . Consider a decision-making problem for traffic mode , as shown in Figure 1 . A practitioner or a rule is responsible for picking the appropriate preference among time and cost , and the agent need to determine different policies depending on the chosen trade-off between these two metrics . Whereas , the environment contain uncertainty factors related to actions of other agents or to dynamic changes of Nature , which may lead to more randomness in these two metrics , and makes multi-objective decision-making or control more challenging . If weather factors are taken into account , e.g. , heavy rain may cause traffic congestion , which can increase the time and cost of the plan-A , but it not have a significant impact on the two metrics of the plan-B . From this perspective , selecting plan-B is more robust , i.e. , a policy is said to be robust if its capability to obtain utility is relatively stable under environmental changes . Therefore , preference and uncertainty jointly affect the decision-making behavior of the agent . In traditional multi-objective reinforcement learning ( MORL ) , one popular way is scalarization , which is to convert the multi-objective reward vector into a single scalar reward through various techniques ( e.g. , by taking a convex combination ) , and then adopt standard RL algorithms to optimize this scalar reward ( Vamplew et al. , 2011 ) . Unfortunately , it is very tricky to determine an appropriate scalarization , because often common approach only learn an ’ average ’ policy over the space of preferences ( Yang et al. , 2019 ) , or though the obtained policies can be relatively quickly adapted to different preferences between performance objectives but are not necessarily optimal . Furthermore , these methods almost did not take into account the robustness of the policies under different preferences , which means the agent can not learn robust Pareto optimal policies . In this work , we propose a novel approach to approximate well-distributed robust Pareto frontier through BRMORL . This allows our trained single network model to produce the robust Pareto optimal policy for any specified preference , i.e. , the learned policy is not only robust to uncertainty ( e.g. , random disturbance and environmental change ) but also Pareto optimal under different preference conditions . Our algorithm is based on three key ideas , which are also the main contributions of this paper : ( 1 ) present a generalized robust MORL framework through modelling uncertainty as an adversarial agent ; ( 2 ) inspired by Shannon-Wiener diversity index , a novel metric is presented to evaluate diversity and evenness of distribution for Pareto solutions . In addition , combined with hypervolume indicator , a comprehensive metric is designed , which can evaluate the convergence , diversity and evenness for the solutions on the approximated Pareto frontier ; ( 3 ) regard agent ’ s learning process in each episode as a black-box , and BO algorithm is used to guide agent to evolve towards improving the quality of the Pareto set . Finally , we demonstrate our proposed algorithm outperform competitive baselines on multi-objective tasks across several MuJoCo ( Todorov et al. , 2012 ) environments and SUMO ( Simulation of Urban Mobility ) ( Lopez et al. , 2018 ) , and show our approach can produce robust policies under environmental uncertainty . 2 RELATED WORK . 2.1 MULTI-OBJECTIVE REINFORCEMENT LEARNING . MORL algorithms can be roughly classified into two main categories : single-policy approaches and multiple-policy approaches ( Roijers et al. , 2013 ; Liu et al. , 2014 ) . Single-policy methods seek to find the optimal policy for a given preference among multiple competing objectives . These approaches convert the multi-objective problem into a single-objective problem through different forms of scalarization , including linear and non-linear ones ( Mannor & Shimkin , 2002 ; Tesauro et al. , 2008 ) . The main advantage of scalarization is its simplicity , which can be integrated into single-policy scheme with very little modification . However , the main drawback of these approaches is that the preference among the objectives must be set in advance . Multi-policy methods aim to learn a set of policies that approximate Pareto frontier under different preference conditions . The most common approaches repeatedly call a single-policy scheme with different preferences ( Natarajan & Tadepalli , 2005 ; Van Moffaert et al. , 2013 ; Zuluaga et al. , 2016 ) . Other methods learn a set of policies simultaneously via using a multi-objective extended version of value-based RL ( Barrett & Narayanan , 2008 ; Castelletti et al. , 2012 ; Van Moffaert & Nowé , 2014 ; Mossalam et al. , 2016 ; Nottingham et al. , 2019 ) or via modifying policy-based RL as a MORL variant ( Pirotta et al. , 2015 ; Parisi et al. , 2017 ; Abdolmaleki et al. , 2020 ; Xu et al. , 2020 ) . Nevertheless , most of these methods are offen constrained to convex regions of the Pareto front and explicitly maintain sets of policies , which may prevent these schemes from finding the sets of well-distributed Pareto solutions which can represent different preferences . There are also meta-policy methods , which can be relatively quickly adapted to different preferences ( Chen et al. , 2018 ; Abels et al. , 2019 ; Yang et al. , 2019 ) . Although the above works were successful to some extent , these approaches share the same shortcomings that no attention is paid to the robustness of Pareto-optimal policy over the entire space of preferences . In addition , most approaches still focus on the domains with discrete action space . In contrast , our scheme can guarantee the learned policies is approximately robust Pareto-optimal on continuous control tasks . 2.2 ROBUST REINFORCEMENT LEARNING . Robust reinforcement learning ( RRL ) algorithms can be broadly grouped into three distinct methods ( Derman et al. , 2020 ) . The first approach focuses on solving robust Markov decision process ( MDP ) with rectangular uncertainty sets . Some researches proposed RRL algorithms for learning optimal policies using coupled uncertainty sets ( Mannor et al. , 2012 ) . Other works modeled an ambiguous linear function of a factor matrix as a selection setting from an uncertainty set ( Goyal & Grand-Clement , 2018 ) . The second RRL approach considered a distribution over the uncertainty set to mitigate the conservativeness . Yu & Xu ( 2015 ) presented the distributional RRL method by supposing the uncertain parameters are random variables following an unknown distribution . Tirinzoni et al . ( 2018 ) proposed a RRL scheme using conditioned probability distribution that defines uncertainty sets . A third RRL method mostly concerns adversarial setting in RL . Pinto et al . ( 2017 ) developed a robust adversarial reinforcement learning ( RARL ) scheme through modeling uncertainties via adversarial agent which applies disturbances to the system . Tessler et al . ( 2019 ) proposed an adversarial RRL framework through structuring probabilistic action robust MDP and noisy action robust MDP . Nonetheless , these researches do not take into account the connection between Pareto-optimal policy and robust policy , which leaves room for improving the performance of them in practical applications . In contrast , our scheme can learn robust Pareto-optimal policies through modeling uncertainty as an adversary over the entire space of preferences . 3 BACKGROUND . 3.1 MULTI-OBJECTIVE MARKOV DECISION PROCESS . In this work , we consider a MORL problem defined by a multi-objective Markov decision process ( MOMDP ) , which is represented by the tuple 〈S , A , P , R , γ , Ω , UΩ〉 with state space S , action space A , state transition probability P ( s′|s , a ) , vector reward function R ( s , a ) = [ r1 , ... , rk ] T , the space of preferences Ω , and preference functions , e.g. , Uω ( R ) which produces an utility function using preference ω ∈ Ω , and a discount factor γ ∈ [ 1 , 0 ) . In MOMDP , a policy π is associated with a vector of expected returns Qπ ( s , a ) = [ Qπ1 , ... , Q π k ] T , where the action-value function of π for objective k can be represented as Qπk ( s , a ) = Eπ [ ∑ t γ trk ( st , at ) |s0 = s , a0 = a ] . For MOMDP , a set of non-dominated policies is called as the Pareto frontier . Definition 1 . A policy π1 Pareto dominates another policy π2 , i.e. , π1 π2 when ∃i : Qπ1i ( s , a ) > Q π2 i ( s , a ) ∧ ∀j 6= i : Q π1 j ( s , a ) > Q π2 j ( s , a ) . Definition 2 . A policy π is Pareto optimal if and only if it is non-dominated by any other policies . 3.2 TWO-PERSON ZERO-SUM GAMES . In standard two-person zero-sum games , players have opposite goals—the payoff of a player equals the loss of the opponent ( Mazalov , 2014 ) , i.e. , V + V̄ = 0 , where V and V̄ are payoff of a player and the opponent , respectively . For two player discounted zero-sum Markov game , assuming protagonist is playing policy π and adversary is playing the policy π̄ , transition kernel P ( s′|s , a , ā ) depend on both players . In the game , the value function based on π and π̄ can be represented as vπ , π̄ ( s ) ≡ Eπ , π̄ [ ∑∞ t=0 γ tr ( st , at , āt ) | s0 = s ] , ∀s ∈ S . Each player chooses his policy regardless of the opponent . Protagonist attempts to maximize the value function ( i.e. , total expected discounted reward ) , and adversary seeks to minimize this function . Nash equilibrium is a key role in game theory , which is one kind of game solution concept . A Nash equilibrium ( π∗ , π̄∗ ) in zero-sum Markov game exists when the following relation holds ( Shapley , 1953 ; Başar & Olsder , 1998 ) : v∗ ( s ) = max π min π̄ Eπ , π̄ [ ∞∑ t=0 γtr ( st , at , āt ) | s0 = s ] ( 1 ) = min π̄ max π Eπ , π̄ [ ∞∑ t=0 γtr ( st , at , āt ) | s0 = s ] , ( 2 ) where π∗ and π̄∗ are the optimal policies of protagonist and adversary respectively , v∗ is optimal equilibrium value of the game . In such a situation , neither player can improve their respective returns , and there is an important relation. , i.e. , ∀π , π̄ , vπ , π̄∗ ≤ v∗ ≤ vπ∗ , π̄ .

Summary: This paper seeks to train multi-objective RL policies that are robust to environmental uncertainties. There are two main contributions: a novel approach to solve this problem, and a novel metric to evaluate Pareto fronts. The metric combines the typical hypervolume metric (that captures the quality/performance of a Pareto front) with a novel "evenness" metric, that captures how well solutions are spread out across the space of preferences. The proposed approach, called BRMORL, consists of training a protagonist policy that maximizes utility alongside an adversarial policy that seeks to minimize utility (motivated by zero-sum game theory), while using Bayesian optimization to select preferences to train on, in order to optimize the hypervolume-and-evennesss metric. Both the protagonist and adversarial policy are conditioned on preferences.