Accommodating Picky Customers: Regret Bound and Exploration Complexity for Multi-Objective Reinforcement Learning

min { d , S } ·H2SAK ) , where d is the number of objectives , S is the number of states , A is the number of actions , H is the length of the horizon , and K is the number of episodes . Furthermore , we consider preference-free exploration , i.e. , the agent first interacts with the environment without specifying any preference and then is able to accommodate arbitrary preference vector up to error . Our proposed algorithm is provably efficient with a nearly optimal trajectory complexity Õ ( min { d , S } ·H3SA/ 2 ) . This result partly resolves an open problem raised by Jin et al . [ 2020 ] . 1 Introduction . In single-objective reinforcement learning ( RL ) , a scalar reward is pre-specified and an agent learns a policy to maximize the long-term cumulative reward [ Sutton and Barto , 2018 ] . However , in many real-world applications , we need to optimize multiple objectives for the same ( unknown ) environment , even when these objectives are possibly contradicting [ Roijers et al. , 2013 ] . For example , in an autonomous driving application , each passenger may have a different preference of driving styles : some of the passengers prefer a very steady riding experience while other passengers enjoy the fast acceleration of the car . Therefore , traditional single-objective RL approach may fail to be applied in such scenarios . One way to tackle this issue is the multi-objective reinforcement learning ( MORL ) [ Roijers et al. , 2013 , Yang et al. , 2019 , Natarajan and Tadepalli , 2005 , Abels et al. , 2018 ] 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . method , which models the multiple objectives by a vectorized reward , and an additional preference vector to specify the relative importance of each objective . The agent of MORL needs to find policies to optimize the cumulative preference-weighted rewards . If the preference vector is fixed or drawn from a fixed distribution , MORL is no more challenging than single-objective RL , as we can predict the objective to be optimized and apply ( variants of ) single-objective RL algorithms ( e.g. , [ Azar et al. , 2017 ] ) . However , more often in practice , the preference vector ( under which the weighted objective needs to be optimized ) is : ( i ) adversarially provided , ( ii ) or even not available in the learning phase . Once more taking the autonomous driving application as example : ( i ) the intelligent system needs to adapt driving configurations to accommodate every customer , even though the next customer can have picky preference that is unpredictable ; ( ii ) when the intelligent system is under development , the future customer can not be known in advance , nonetheless , when the system is deployed , it has to be capable to accommodate any potential customer who rides the car . Such MORL examples are common , to name a few beyond the autonomous driving one : medical treatment must take care of every patient even in very rare health conditions ; an education system should accommodate every student according to his/her own characteristics ; and emergency response systems have to be responsible in all extreme cases . Due to these exclusive challenges that have not appeared in single-objective RL , new sample-efficient algorithms for MORL need to be developed , as well as their theoretical grounds need to be established . In this work , we study provable sample-efficient algorithms for MORL that resolve the aforementioned issues . Specifically , we consider MORL on a finite-horizon Markov decision process ( MDP ) with an unknown transition kernel , S states , A actions , H steps , and d reward functions that represent the d difference objectives . We investigate MORL problems in two paradigms : ( i ) online MORL ( where the preferences are adversarially presented ) , and ( ii ) preference-free exploration ( where the preferences are not available in the learning/exploration phase ) . The two settings and our contributions are explained respectively in the following . Setting ( i ) : Online MORL . We first consider an online learning problem to capture the challenge that preference vectors could be adversarially provided in MORL . In the beginning of each episode , the MORL agent is provided ( potentially adversarially ) with a preference vector , and the agent interacts with the unknown environment to collect data and rewards ( that is specified by the provided preference ) . The performance of an algorithm is measured by the regret , i.e. , the difference between the rewards collected by the agent and those would be collected by a theoretically optimal agent ( who could use varying policies that adapt to the preferences ) . This setting generalizes the classical online single-objective RL problem [ Azar et al. , 2017 ] ( where d = 1 ) . Contribution ( i ) . For online MORL , we provide a provably efficient algorithm with a regret upper bound Õ ( √ min { d , S } ·H2SAK ) 1 , where K is the number of episodes for interacting with the en- vironment . Furthermore , we show an information-theoretic lower bound Ω ( √ min { d , S } ·H2SAK ) for online MORL . These bounds together show that , ignoring logarithmic factors , our algorithm resolves the online MORL problems optimally . Setting ( ii ) : Preference-Free Exploration . We further consider an unsupervised MORL problem to capture the issue that preferences could be hard to obtain in the training phase . The MORL agent first interacts with the unknown environment in the absence of preference vectors ; afterwards , the agent is required to use the collected data to compute near-optimal policies for an arbitrarily specified preference vector . The performance is measured by the sample complexity , i.e. , the minimum amount of trajectories that an MORL agent needs to collect during exploration in order to be near-optimal during planning . This setting extends the recent proposed reward-free exploration problem [ Jin et al. , 2020 ] to MORL . Contribution ( ii ) . For preference-free exploration , we show that a simple variant of the proposed online algorithm can achieve nearly optimal sample complexity . In particular , the algorithm achieves a sample complexity upper bound Õ ( min { d , S } ·H3SA/ 2 ) where is the tolerance of the planning error ; and we also show a sample complexity lower bound , Ω ( min { d , S } ·H2SA/ 2 ) , for any 1We use Õ ( · ) to hide ( potential ) polylogarithmic factors in O ( · ) , i.e. , Õ ( n ) : = O ( n logk n ) for sufficiently large n and some absolute constant k > 0. algorithm . These bounds suggest that our algorithm is optimal in terms of d , S , A , up to logarithmic factors . It is also worth noting that our results for preference-free exploration partly answer an open question raised by Jin et al . [ 2020 ] : reward-free RL is easier when the unknown reward functions enjoy low-dimensional representations ( as in MORL ) . Paper Layout . The remaining paper is organized as follows : the preliminaries are summarized in Section 2 ; then in Section 3 , we formally introduce the problem of online MORL , our algorithm and its regret upper and lower bounds ; then in Section 4 , we turn to study the preference-free exploration problem in MORL , where we present an exploration algorithm with sample complexity analysis ( with both upper bound and lower bound ) , and compare our results with existing results for related problems ; finally , the related literature is reviewed in Section 5 and the paper is concluded by Section 6 . 2 Preliminaries . We specify a finite-horizon Markov decision process ( MDP ) by a tuple of ( S , A , H , P , r , W ) . S is a finite state set where |S| = S. A is a finite action set where |A| = A. H is the length of the horizon . P ( · | x , a ) is a stationary , unknown transition probability to a new state for taking action a at state x. r = { r1 , . . . , rH } , where rh : S ×A → [ 0 , 1 ] d represents a d-dimensional vector rewards function that captures the d objectives2 . W : = { w ∈ [ 0 , 1 ] d , ‖w‖1 = 1 } specifies the set of all possible preference vectors3 , where each preference vector w ∈ W induces a scalar reward function by4 rh ( · , · ) = 〈w , rh ( · , · ) 〉 for h = 1 , . . . , H . A policy is represented by π : = { π1 , . . . , πH } , where each πh ( · ) maps a state to a distribution over the action set . Fixing a policy π , we will consider the following generalized Q-value function and generalized value function : Qπh ( x , a ; w ) : = E [ ∑H j=h〈w , rj ( xj , aj ) 〉 ∣∣xh = x , ah = a ] , V πh ( x ; w ) : = Qπh ( x , πh ( x ) ; w ) , where xj ∼ P ( · | xj−1 , aj−1 ) and aj ∼ πj ( xj ) for j > h. Note that compared with the typical Q-value function ( or the value function ) used in single-objective RL , here the generalized Q-value function ( or the generalized value function ) takes the preference vector as an additional input , besides the state-action pair . Fixing a preference w ∈ W , the optimal policy under w is defined as π∗w : = arg maxπ V π 1 ( x1 ; w ) . For simplicity , we denote Q∗h ( x , a ; w ) : = Q π∗w h ( x , a ; w ) , V ∗ h ( x ; w ) : = V π∗w h ( x ; w ) = maxa Q∗h ( x , a ; w ) . The following abbreviation is adopted for simplicity : PV πh |x , a , w : = ∑ y∈SP ( y|x , a ) V πh ( y ; w ) , and similar abbreviations will be adopted for variants of probability transition ( e.g. , the empirical transition probability ) and variants of value function ( e.g. , the estimated value function ) . Finally , we remark that the following Bellman equations hold for the generalized Q-value functions { Qπh ( x , a ; w ) = 〈w , rh ( x , a ) 〉+ PV πh+1 ∣∣ x , a , w , V πh ( x ; w ) = Q π h ( x , πh ( x ) ; w ) ; { Q∗h ( x , a ; w ) = 〈w , rh ( x , a ) 〉+ PV ∗h+1 ∣∣ x , a , w , V ∗h ( x ; w ) = maxaQ ∗ h ( x , a ; w ) . With the above preparations , we are ready to discuss our algorithms and theory for online MORL ( Section 3 ) and preference-free exploration ( Section 4 ) .

Summary: This work studies the multi-objective RL (MORL) with unknown transition, where the reward function can be parameterized with d-dimensional preference vectors. In the beginning of each episode, an adversary selects the weight vector and reveals it to the learner, the learner then selects corresponding policy and suffers a regret comparing with the optimal policies with respect to the given preference vector.   Besides this online MORL setting, the authors further study a preference-free exploration (PFE) setting where the learner manually separates the learning and planning phases and her performance is measured by the number of samples that are required to ensure PAC-bound. Then, the authors introduce a lower bound for PFE, which shows there is only an H factor loose.