Offline Constrained Multi-Objective Reinforcement Learning via Pessimistic Dual Value Iteration

In constrained multi-objective RL , the goal is to learn a policy that achieves the best performance specified by a multi-objective preference function under a constraint . We focus on the offline setting where the RL agent aims to learn the optimal policy from a given dataset . This scenario is common in real-world applications where interactions with the environment are expensive and the constraint violation is dangerous . For such a setting , we transform the original constrained problem into a primal-dual formulation , which is solved via dual gradient ascent . Moreover , we propose to combine such an approach with pessimism to overcome the uncertainty in offline data , which leads to our Pessimistic Dual Iteration ( PEDI ) . We establish upper bounds on both the suboptimality and constraint violation for the policy learned by PEDI based on an arbitrary dataset , which proves that PEDI is provably sample efficient . We also specialize PEDI to the setting with linear function approximation . To the best of our knowledge , we propose the first provably efficient constrained multi-objective RL algorithm with offline data without any assumption on the coverage of the dataset . 1 Introduction . There has been increased interest in multi-objective RL in recent years . Compared with traditional single-objective RL , the goal of the multi-objective one depends on a preference function , which takes the multiple objectives as input and outputs a scalar . The multi-objective optimization problems are usually constrained , as otherwise , it may cause danger or malfunction in applications . For example , consider a home automation system that helps humans monitor and control home attributes which can be regarded as multi-objectives . Users at different times may value different aspects of its services . Some may think highly of lighting at night , while others may concern the climate . We can formulate the users ’ preference as a preference function on the multiple objectives . Therefore , the system is dealing with a multi-objective optimization problem . However , the system can not optimize this problem without any constraints , which might go against human ’ s will , such as generating extreme climate inside a house or performing unacceptably energy-consuming operations . We formulate the constrained multi-objective Markov decision process ( CMOMDP ) , which is similar to the constrained Markov decision process ( CMDP ) ( Altman , 1999 ) . The difference is that the objectives are multiple and constraints can be nonlinear . We aim to minimize the value of a preference function , which takes multiple objectives as input and outputs a scalar . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Most existing RL methods assume full accessibility of the environment for the agent , which tends to be impractical as the exploration could be expensive ( Gottesman et al. , 2019 ) and dangerous ( ShalevShwartz et al. , 2016 ) . Hence , we consider the offline case where the agent has only a historical dataset collected a priori and has no further interactions with the environment . This setting is common and embodied in various scenarios such as healthcare ( Chakraborty and Murphy , 2014 ) and auto-driving ( Sun et al. , 2020 ) . However , offline RL is less understood theoretically ( Levine et al. , 2020 ) than online RL , and how to maximally exploit the dataset remains unclear . In this paper , we study the offline CMOMDP . The challenges are threefold : ( i ) Different from CMDPs , the nonlinearity of the preference function and constraints of CMOMDPs makes the analysis challenging . Moreover , constraints can be neither convex nor concave in the policy , which means the optimization problems are usually non-convex . It brings great difficulty to designing a provably efficient algorithm . ( ii ) Since further interaction with the environment is prohibited , the dataset ’ s quality is not guaranteed . The data collected by the experimenter may not sufficiently cover the trajectories induced by the optimal policy . Therefore , the information of the optimal policy could be limited , which probably makes the suboptimality and constraint violation arbitrarily large . ( iii ) As the CMOMDP can be considered a generalization of CMDP ( see the reduction in Appendix C ) , any difficulties emerging in the CMDP will arise here too . For example , due to constraints , the Bellman optimality equation may not hold anymore . Therefore , most existing offline RL algorithms based on dynamic programming are inapplicable . Challenge ( i ) is inherent in CMOMDPs , and certain requirements on the preference function and constraints are inevitable . Hence , We suppose they satisfy some conditions . For instance , a geometric analog of Slater ’ s condition , which is usually assumed in CMDPs , is imposed . To tackle challenge ( ii ) , existing works have put a great effort . One possible principle is pessimism , which applies penalties to ensure pessimistic estimation . This method is successful in ordinary RL , and we extend it to CMOMDPs . We also note that we only impose a minimal assumption on the offline dataset ’ s compliance in our analysis . For challenge ( iii ) , we apply the convex conjugate and Fenchel ’ s duality to transform the problem into a primal-dual formulation . In summary , our work answers the following question : Is it possible to develop a provably efficient offline algorithm for constrained multi-objective reinforcement learning with minimal assumptions on the dataset ? We propose the Pessimistic Dual Iteration ( PEDI ) algorithm . Theoretical contributions are as follows : ( i ) By transforming the original constrained optimization problem of CMOMDPs into a primal-dual formulation via convex conjugate and duality , we develop the algorithm for general CMOMDPs with offline dataset , which iterates in a dual gradient ascent manner . Then , we instantiate the algorithm for linear kernel CMOMDPs , which is a large class of CMOMDPs that includes the tabular case . ( ii ) We show in Appendix F the significance of pessimism in offline CMOMDPs . To summarize , we decompose the discrepancy between a value function and the optimal one into spurious correlation , intrinsic uncertainty , and optimization error , among which the spurious correlation is the most difficult to control . However , by maintaining pessimistic estimates of the value functions , PEDI eliminates the spurious correlation . ( iii ) We establish theoretical guarantees for PEDI . Specifically , we demonstrate that two metrics , the suboptimality and the constraint violation , can be bounded from above by the optimization error of O ( 1/ √ K ) and the intrinsic uncertainty , where K is the number of iterations . When specialized to linear kernel CMOMDPs , the upper bound of suboptimality matches the information-theoretic lower bound up to the optimization error and multiplicative factors of constants related to the CMOMDP , which suggests the near-optimality of PEDI . ( iv ) We show that the error of PEDI is data-dependent , i.e. , it depends on how well the dataset covers the trajectories induced by the optimal policy . When the trajectories are assumed further to be sufficiently covered , the suboptimality and constraint violation are Õ ( 1/ √ N ) where N is the number of trajectories in the dataset . To the best of our knowledge , we are the first to propose a provably efficient offline algorithm that considers the constrained multi-objective RL without any assumptions on the coverage of the dataset . As a by-product , our method can be viewed as a highly generalized one as it can easily reduce to certain simpler cases such as CMDPs , which is discussed in Appendix C . 1.1 Related works . Constrained RL . Our work is closely related to CMDPs ( Altman , 1999 ) . Several recent works ( Efroni et al. , 2020 ; Ding et al. , 2021 ; Qiu et al. , 2020 ; Brantley et al. , 2020 ; Tessler et al. , 2018 ) have studied the MDP with linear constraints and others ( Bhatnagar and Lakshmanan , 2012 ; Chow et al. , 2017 ; Paternain et al. , 2019 ) have considered RL with more complex constraints . Most of them have provided algorithms with both regret and total constraint violation guarantees . However , while these approaches are successful in CMDPs , they can not be applied to CMOMDPs where objectives are multiple and constraints can be nonlinear . The difference between their methods and ours are ( i ) they usually assume the Slater ’ s condition , while we consider its geometric analog , and ( ii ) most works utilize the Lagrangian multiplier to handle constraints , while in our method , we obtain a primal-dual formulation via conjugate . Multi-objective RL . Existing studies on multi-objective RL can be divided into two groups : the first maintains a set of Pareto-optimal policies ( Barrett and Narayanan , 2008 ; Castelletti et al. , 2011 , 2012 ; Wang and Sebag , 2013 ) , and the second collapses multi-objective vector into a scalar and then apply a standard RL method ( Barrett and Narayanan , 2008 ; Natarajan and Tadepalli , 2005 ; Van Moffaert et al. , 2013 ; Cheung , 2019 ) . However , these methods are available in unconstrained multi-objective RL but unsuitable for constrained cases . The difficulty is that the constraints invalidate the Bellman optimality principle , which is the cornerstone of these works . Offline RL . Recent successful offline RL ( Lange et al. , 2012 ; Levine et al. , 2020 ) methods ( Fujimoto et al. , 2019 ; Laroche et al. , 2019 ; Jaques et al. , 2019 ; Wu et al. , 2019 ; Kumar et al. , 2019 , 2020 ; Agarwal et al. , 2020 ; Yu et al. , 2020 ; Kidambi et al. , 2020 ; Siegel et al. , 2020 ; Nair et al. , 2020 ; Liu et al. , 2020 ; Wang et al. , 2020a , b ; Tosatto et al. , 2017 ; Farahmand et al. , 2016 , 2010 ) fall into two categories ( Buckman et al. , 2020 ) : ( i ) uncertainty-aware pessimistic algorithms and ( ii ) proximal pessimistic algorithms . ( i ) gives a lower estimate for states and actions less covered by the dataset , while ( ii ) avoids visiting these less visited states and actions by regularization . However , existing methods are largely based on approximate dynamic programming and do not take constraints into account , thus failing to apply . Our proposed method generalizes the ordinary pessimistic algorithm so that it is compatible with multiple objectives and constraints . In the analysis , we only assume the compliance of the dataset . In comparison with existing works , which assume the sufficient coverage of the dataset such as the finite concentrability coefficients ( Antos et al. , 2008 ; Farahmand et al. , 2010 ; Scherrer et al. , 2015 ; Le et al. , 2019 ; Chen and Jiang , 2019 ; Fu et al. , 2020 ) and uniformly lower bounded densities of visitation measures ( Yin et al. , 2021 ) , we require no assumptions as they often fail to hold in practice . As Wang et al . ( 2020a ) have studied the influence of insufficient coverage of the dataset , their conclusion is also reflected in our main results . Moreover , we do not impose any constraints on the affinity of the behavior policy and the learned policy ( Liu et al. , 2020 ) . In addition , our general algorithm uses convex conjugate , a technique used similarly in Yu et al . ( 2021 ) ; Miryoosefi and Jin ( 2021 ) but their method can not apply to offline situations and Yu et al . ( 2021 ) do not address the function approximation setting .

Summary: The paper proposes and analyzes the setting of off-policy constrained multi-objective Markov decision process.  This is a generalized of the constrained MDP setting.  An algorithm to learn the solution is proposed based on a minimax formulation with a pessimistic penalty.  Theoretical analysis defines upper bounds on the suboptimality of the objective and constraint violation. Efficient learning is also demonstrated, under the assumption that the off-policy data sufficiently covers the trajectories induced by the optimal policy.