Accommodating Picky Customers: Regret Bound and Exploration Complexity for Multi-Objective Reinforcement Learning

min { d , S } ·H2SAK ) , where d is the number of objectives , S is the number of states , A is the number of actions , H is the length of the horizon , and K is the number of episodes . Furthermore , we consider preference-free exploration , i.e. , the agent first interacts with the environment without specifying any preference and then is able to accommodate arbitrary preference vector up to error . Our proposed algorithm is provably efficient with a nearly optimal trajectory complexity Õ ( min { d , S } ·H3SA/ 2 ) . This result partly resolves an open problem raised by Jin et al . [ 2020 ] . 1 Introduction . In single-objective reinforcement learning ( RL ) , a scalar reward is pre-specified and an agent learns a policy to maximize the long-term cumulative reward [ Sutton and Barto , 2018 ] . However , in many real-world applications , we need to optimize multiple objectives for the same ( unknown ) environment , even when these objectives are possibly contradicting [ Roijers et al. , 2013 ] . For example , in an autonomous driving application , each passenger may have a different preference of driving styles : some of the passengers prefer a very steady riding experience while other passengers enjoy the fast acceleration of the car . Therefore , traditional single-objective RL approach may fail to be applied in such scenarios . One way to tackle this issue is the multi-objective reinforcement learning ( MORL ) [ Roijers et al. , 2013 , Yang et al. , 2019 , Natarajan and Tadepalli , 2005 , Abels et al. , 2018 ] 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . method , which models the multiple objectives by a vectorized reward , and an additional preference vector to specify the relative importance of each objective . The agent of MORL needs to find policies to optimize the cumulative preference-weighted rewards . If the preference vector is fixed or drawn from a fixed distribution , MORL is no more challenging than single-objective RL , as we can predict the objective to be optimized and apply ( variants of ) single-objective RL algorithms ( e.g. , [ Azar et al. , 2017 ] ) . However , more often in practice , the preference vector ( under which the weighted objective needs to be optimized ) is : ( i ) adversarially provided , ( ii ) or even not available in the learning phase . Once more taking the autonomous driving application as example : ( i ) the intelligent system needs to adapt driving configurations to accommodate every customer , even though the next customer can have picky preference that is unpredictable ; ( ii ) when the intelligent system is under development , the future customer can not be known in advance , nonetheless , when the system is deployed , it has to be capable to accommodate any potential customer who rides the car . Such MORL examples are common , to name a few beyond the autonomous driving one : medical treatment must take care of every patient even in very rare health conditions ; an education system should accommodate every student according to his/her own characteristics ; and emergency response systems have to be responsible in all extreme cases . Due to these exclusive challenges that have not appeared in single-objective RL , new sample-efficient algorithms for MORL need to be developed , as well as their theoretical grounds need to be established . In this work , we study provable sample-efficient algorithms for MORL that resolve the aforementioned issues . Specifically , we consider MORL on a finite-horizon Markov decision process ( MDP ) with an unknown transition kernel , S states , A actions , H steps , and d reward functions that represent the d difference objectives . We investigate MORL problems in two paradigms : ( i ) online MORL ( where the preferences are adversarially presented ) , and ( ii ) preference-free exploration ( where the preferences are not available in the learning/exploration phase ) . The two settings and our contributions are explained respectively in the following . Setting ( i ) : Online MORL . We first consider an online learning problem to capture the challenge that preference vectors could be adversarially provided in MORL . In the beginning of each episode , the MORL agent is provided ( potentially adversarially ) with a preference vector , and the agent interacts with the unknown environment to collect data and rewards ( that is specified by the provided preference ) . The performance of an algorithm is measured by the regret , i.e. , the difference between the rewards collected by the agent and those would be collected by a theoretically optimal agent ( who could use varying policies that adapt to the preferences ) . This setting generalizes the classical online single-objective RL problem [ Azar et al. , 2017 ] ( where d = 1 ) . Contribution ( i ) . For online MORL , we provide a provably efficient algorithm with a regret upper bound Õ ( √ min { d , S } ·H2SAK ) 1 , where K is the number of episodes for interacting with the en- vironment . Furthermore , we show an information-theoretic lower bound Ω ( √ min { d , S } ·H2SAK ) for online MORL . These bounds together show that , ignoring logarithmic factors , our algorithm resolves the online MORL problems optimally . Setting ( ii ) : Preference-Free Exploration . We further consider an unsupervised MORL problem to capture the issue that preferences could be hard to obtain in the training phase . The MORL agent first interacts with the unknown environment in the absence of preference vectors ; afterwards , the agent is required to use the collected data to compute near-optimal policies for an arbitrarily specified preference vector . The performance is measured by the sample complexity , i.e. , the minimum amount of trajectories that an MORL agent needs to collect during exploration in order to be near-optimal during planning . This setting extends the recent proposed reward-free exploration problem [ Jin et al. , 2020 ] to MORL . Contribution ( ii ) . For preference-free exploration , we show that a simple variant of the proposed online algorithm can achieve nearly optimal sample complexity . In particular , the algorithm achieves a sample complexity upper bound Õ ( min { d , S } ·H3SA/ 2 ) where is the tolerance of the planning error ; and we also show a sample complexity lower bound , Ω ( min { d , S } ·H2SA/ 2 ) , for any 1We use Õ ( · ) to hide ( potential ) polylogarithmic factors in O ( · ) , i.e. , Õ ( n ) : = O ( n logk n ) for sufficiently large n and some absolute constant k > 0. algorithm . These bounds suggest that our algorithm is optimal in terms of d , S , A , up to logarithmic factors . It is also worth noting that our results for preference-free exploration partly answer an open question raised by Jin et al . [ 2020 ] : reward-free RL is easier when the unknown reward functions enjoy low-dimensional representations ( as in MORL ) . Paper Layout . The remaining paper is organized as follows : the preliminaries are summarized in Section 2 ; then in Section 3 , we formally introduce the problem of online MORL , our algorithm and its regret upper and lower bounds ; then in Section 4 , we turn to study the preference-free exploration problem in MORL , where we present an exploration algorithm with sample complexity analysis ( with both upper bound and lower bound ) , and compare our results with existing results for related problems ; finally , the related literature is reviewed in Section 5 and the paper is concluded by Section 6 . 2 Preliminaries . We specify a finite-horizon Markov decision process ( MDP ) by a tuple of ( S , A , H , P , r , W ) . S is a finite state set where |S| = S. A is a finite action set where |A| = A. H is the length of the horizon . P ( · | x , a ) is a stationary , unknown transition probability to a new state for taking action a at state x. r = { r1 , . . . , rH } , where rh : S ×A → [ 0 , 1 ] d represents a d-dimensional vector rewards function that captures the d objectives2 . W : = { w ∈ [ 0 , 1 ] d , ‖w‖1 = 1 } specifies the set of all possible preference vectors3 , where each preference vector w ∈ W induces a scalar reward function by4 rh ( · , · ) = 〈w , rh ( · , · ) 〉 for h = 1 , . . . , H . A policy is represented by π : = { π1 , . . . , πH } , where each πh ( · ) maps a state to a distribution over the action set . Fixing a policy π , we will consider the following generalized Q-value function and generalized value function : Qπh ( x , a ; w ) : = E [ ∑H j=h〈w , rj ( xj , aj ) 〉 ∣∣xh = x , ah = a ] , V πh ( x ; w ) : = Qπh ( x , πh ( x ) ; w ) , where xj ∼ P ( · | xj−1 , aj−1 ) and aj ∼ πj ( xj ) for j > h. Note that compared with the typical Q-value function ( or the value function ) used in single-objective RL , here the generalized Q-value function ( or the generalized value function ) takes the preference vector as an additional input , besides the state-action pair . Fixing a preference w ∈ W , the optimal policy under w is defined as π∗w : = arg maxπ V π 1 ( x1 ; w ) . For simplicity , we denote Q∗h ( x , a ; w ) : = Q π∗w h ( x , a ; w ) , V ∗ h ( x ; w ) : = V π∗w h ( x ; w ) = maxa Q∗h ( x , a ; w ) . The following abbreviation is adopted for simplicity : PV πh |x , a , w : = ∑ y∈SP ( y|x , a ) V πh ( y ; w ) , and similar abbreviations will be adopted for variants of probability transition ( e.g. , the empirical transition probability ) and variants of value function ( e.g. , the estimated value function ) . Finally , we remark that the following Bellman equations hold for the generalized Q-value functions { Qπh ( x , a ; w ) = 〈w , rh ( x , a ) 〉+ PV πh+1 ∣∣ x , a , w , V πh ( x ; w ) = Q π h ( x , πh ( x ) ; w ) ; { Q∗h ( x , a ; w ) = 〈w , rh ( x , a ) 〉+ PV ∗h+1 ∣∣ x , a , w , V ∗h ( x ; w ) = maxaQ ∗ h ( x , a ; w ) . With the above preparations , we are ready to discuss our algorithms and theory for online MORL ( Section 3 ) and preference-free exploration ( Section 4 ) .

Summary: This paper tackles the reinforcement learning problem in a multi-objective and online setting. Two different cases are considered: (i) when preferences over the multiple objectives are given at the start of each episode and (ii) when preferences are unknown and the environment needs to be explored in preparation for any arbitrary preferences. For both cases, the paper proposes optimal solutions (up to logarithmic factors, and a factor of $H$ in the second case).