Offline Constrained Multi-Objective Reinforcement Learning via Pessimistic Dual Value Iteration

In constrained multi-objective RL , the goal is to learn a policy that achieves the best performance specified by a multi-objective preference function under a constraint . We focus on the offline setting where the RL agent aims to learn the optimal policy from a given dataset . This scenario is common in real-world applications where interactions with the environment are expensive and the constraint violation is dangerous . For such a setting , we transform the original constrained problem into a primal-dual formulation , which is solved via dual gradient ascent . Moreover , we propose to combine such an approach with pessimism to overcome the uncertainty in offline data , which leads to our Pessimistic Dual Iteration ( PEDI ) . We establish upper bounds on both the suboptimality and constraint violation for the policy learned by PEDI based on an arbitrary dataset , which proves that PEDI is provably sample efficient . We also specialize PEDI to the setting with linear function approximation . To the best of our knowledge , we propose the first provably efficient constrained multi-objective RL algorithm with offline data without any assumption on the coverage of the dataset . 1 Introduction . There has been increased interest in multi-objective RL in recent years . Compared with traditional single-objective RL , the goal of the multi-objective one depends on a preference function , which takes the multiple objectives as input and outputs a scalar . The multi-objective optimization problems are usually constrained , as otherwise , it may cause danger or malfunction in applications . For example , consider a home automation system that helps humans monitor and control home attributes which can be regarded as multi-objectives . Users at different times may value different aspects of its services . Some may think highly of lighting at night , while others may concern the climate . We can formulate the users ’ preference as a preference function on the multiple objectives . Therefore , the system is dealing with a multi-objective optimization problem . However , the system can not optimize this problem without any constraints , which might go against human ’ s will , such as generating extreme climate inside a house or performing unacceptably energy-consuming operations . We formulate the constrained multi-objective Markov decision process ( CMOMDP ) , which is similar to the constrained Markov decision process ( CMDP ) ( Altman , 1999 ) . The difference is that the objectives are multiple and constraints can be nonlinear . We aim to minimize the value of a preference function , which takes multiple objectives as input and outputs a scalar . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Most existing RL methods assume full accessibility of the environment for the agent , which tends to be impractical as the exploration could be expensive ( Gottesman et al. , 2019 ) and dangerous ( ShalevShwartz et al. , 2016 ) . Hence , we consider the offline case where the agent has only a historical dataset collected a priori and has no further interactions with the environment . This setting is common and embodied in various scenarios such as healthcare ( Chakraborty and Murphy , 2014 ) and auto-driving ( Sun et al. , 2020 ) . However , offline RL is less understood theoretically ( Levine et al. , 2020 ) than online RL , and how to maximally exploit the dataset remains unclear . In this paper , we study the offline CMOMDP . The challenges are threefold : ( i ) Different from CMDPs , the nonlinearity of the preference function and constraints of CMOMDPs makes the analysis challenging . Moreover , constraints can be neither convex nor concave in the policy , which means the optimization problems are usually non-convex . It brings great difficulty to designing a provably efficient algorithm . ( ii ) Since further interaction with the environment is prohibited , the dataset ’ s quality is not guaranteed . The data collected by the experimenter may not sufficiently cover the trajectories induced by the optimal policy . Therefore , the information of the optimal policy could be limited , which probably makes the suboptimality and constraint violation arbitrarily large . ( iii ) As the CMOMDP can be considered a generalization of CMDP ( see the reduction in Appendix C ) , any difficulties emerging in the CMDP will arise here too . For example , due to constraints , the Bellman optimality equation may not hold anymore . Therefore , most existing offline RL algorithms based on dynamic programming are inapplicable . Challenge ( i ) is inherent in CMOMDPs , and certain requirements on the preference function and constraints are inevitable . Hence , We suppose they satisfy some conditions . For instance , a geometric analog of Slater ’ s condition , which is usually assumed in CMDPs , is imposed . To tackle challenge ( ii ) , existing works have put a great effort . One possible principle is pessimism , which applies penalties to ensure pessimistic estimation . This method is successful in ordinary RL , and we extend it to CMOMDPs . We also note that we only impose a minimal assumption on the offline dataset ’ s compliance in our analysis . For challenge ( iii ) , we apply the convex conjugate and Fenchel ’ s duality to transform the problem into a primal-dual formulation . In summary , our work answers the following question : Is it possible to develop a provably efficient offline algorithm for constrained multi-objective reinforcement learning with minimal assumptions on the dataset ? We propose the Pessimistic Dual Iteration ( PEDI ) algorithm . Theoretical contributions are as follows : ( i ) By transforming the original constrained optimization problem of CMOMDPs into a primal-dual formulation via convex conjugate and duality , we develop the algorithm for general CMOMDPs with offline dataset , which iterates in a dual gradient ascent manner . Then , we instantiate the algorithm for linear kernel CMOMDPs , which is a large class of CMOMDPs that includes the tabular case . ( ii ) We show in Appendix F the significance of pessimism in offline CMOMDPs . To summarize , we decompose the discrepancy between a value function and the optimal one into spurious correlation , intrinsic uncertainty , and optimization error , among which the spurious correlation is the most difficult to control . However , by maintaining pessimistic estimates of the value functions , PEDI eliminates the spurious correlation . ( iii ) We establish theoretical guarantees for PEDI . Specifically , we demonstrate that two metrics , the suboptimality and the constraint violation , can be bounded from above by the optimization error of O ( 1/ √ K ) and the intrinsic uncertainty , where K is the number of iterations . When specialized to linear kernel CMOMDPs , the upper bound of suboptimality matches the information-theoretic lower bound up to the optimization error and multiplicative factors of constants related to the CMOMDP , which suggests the near-optimality of PEDI . ( iv ) We show that the error of PEDI is data-dependent , i.e. , it depends on how well the dataset covers the trajectories induced by the optimal policy . When the trajectories are assumed further to be sufficiently covered , the suboptimality and constraint violation are Õ ( 1/ √ N ) where N is the number of trajectories in the dataset . To the best of our knowledge , we are the first to propose a provably efficient offline algorithm that considers the constrained multi-objective RL without any assumptions on the coverage of the dataset . As a by-product , our method can be viewed as a highly generalized one as it can easily reduce to certain simpler cases such as CMDPs , which is discussed in Appendix C . 1.1 Related works . Constrained RL . Our work is closely related to CMDPs ( Altman , 1999 ) . Several recent works ( Efroni et al. , 2020 ; Ding et al. , 2021 ; Qiu et al. , 2020 ; Brantley et al. , 2020 ; Tessler et al. , 2018 ) have studied the MDP with linear constraints and others ( Bhatnagar and Lakshmanan , 2012 ; Chow et al. , 2017 ; Paternain et al. , 2019 ) have considered RL with more complex constraints . Most of them have provided algorithms with both regret and total constraint violation guarantees . However , while these approaches are successful in CMDPs , they can not be applied to CMOMDPs where objectives are multiple and constraints can be nonlinear . The difference between their methods and ours are ( i ) they usually assume the Slater ’ s condition , while we consider its geometric analog , and ( ii ) most works utilize the Lagrangian multiplier to handle constraints , while in our method , we obtain a primal-dual formulation via conjugate . Multi-objective RL . Existing studies on multi-objective RL can be divided into two groups : the first maintains a set of Pareto-optimal policies ( Barrett and Narayanan , 2008 ; Castelletti et al. , 2011 , 2012 ; Wang and Sebag , 2013 ) , and the second collapses multi-objective vector into a scalar and then apply a standard RL method ( Barrett and Narayanan , 2008 ; Natarajan and Tadepalli , 2005 ; Van Moffaert et al. , 2013 ; Cheung , 2019 ) . However , these methods are available in unconstrained multi-objective RL but unsuitable for constrained cases . The difficulty is that the constraints invalidate the Bellman optimality principle , which is the cornerstone of these works . Offline RL . Recent successful offline RL ( Lange et al. , 2012 ; Levine et al. , 2020 ) methods ( Fujimoto et al. , 2019 ; Laroche et al. , 2019 ; Jaques et al. , 2019 ; Wu et al. , 2019 ; Kumar et al. , 2019 , 2020 ; Agarwal et al. , 2020 ; Yu et al. , 2020 ; Kidambi et al. , 2020 ; Siegel et al. , 2020 ; Nair et al. , 2020 ; Liu et al. , 2020 ; Wang et al. , 2020a , b ; Tosatto et al. , 2017 ; Farahmand et al. , 2016 , 2010 ) fall into two categories ( Buckman et al. , 2020 ) : ( i ) uncertainty-aware pessimistic algorithms and ( ii ) proximal pessimistic algorithms . ( i ) gives a lower estimate for states and actions less covered by the dataset , while ( ii ) avoids visiting these less visited states and actions by regularization . However , existing methods are largely based on approximate dynamic programming and do not take constraints into account , thus failing to apply . Our proposed method generalizes the ordinary pessimistic algorithm so that it is compatible with multiple objectives and constraints . In the analysis , we only assume the compliance of the dataset . In comparison with existing works , which assume the sufficient coverage of the dataset such as the finite concentrability coefficients ( Antos et al. , 2008 ; Farahmand et al. , 2010 ; Scherrer et al. , 2015 ; Le et al. , 2019 ; Chen and Jiang , 2019 ; Fu et al. , 2020 ) and uniformly lower bounded densities of visitation measures ( Yin et al. , 2021 ) , we require no assumptions as they often fail to hold in practice . As Wang et al . ( 2020a ) have studied the influence of insufficient coverage of the dataset , their conclusion is also reflected in our main results . Moreover , we do not impose any constraints on the affinity of the behavior policy and the learned policy ( Liu et al. , 2020 ) . In addition , our general algorithm uses convex conjugate , a technique used similarly in Yu et al . ( 2021 ) ; Miryoosefi and Jin ( 2021 ) but their method can not apply to offline situations and Yu et al . ( 2021 ) do not address the function approximation setting .

Summary: The paper considers policy optimization in a difficult setting involving multiple challenges: (1) multi-objective -- rather than simply optimizing a scalar reward, the policy must optimize a convex function g on vectors of cumulative costs; (2) constrained -- while optimizing g the policy must also satisfy a constraint on the cumulative costs; (3) offline -- the policy learning algorithm cannot actively query the environment, and only has offline access to experience.  For this setting, the paper presents an algorithm with corresponding theoretical analysis. The algorithm follows a primal-dual optimization to handle (1) and (2) above while proposing pessimistic value iteration to handle (3) above. Theoretical analysis provides guarantees on the convergence of the algorithm to an optimal policy under idealized conditions in both general MDPs and kernelized MDPs.