Multi-Objective SPIBB: Seldonian Offline Policy Improvement with Safety Constraints in Finite MDPs

1 Introduction . Reinforcement Learning ( RL ) as a paradigm for sequential decision-making ( Sutton , 1988 ) has shown tremendous success in a variety of simulated domains ( Mnih et al. , 2015 ; Silver et al. , 2017 ; OpenAI , 2018 ) . However , there are still quite a few challenges between the traditional RL research and real-world tasks . Most of these challenges stem from assumptions that are rarely satisfied in practice ( Dulac-Arnold et al. , 2019 ) , or the inability of the algorithm ’ s user to specify the desired behavior of the agent without being a domain expert ( Thomas et al. , 2019 ) . We focus on the real-world application point of view and posit the following requirements : • Multiple reward functions : Traditional RL methods assume a single scalar reward is present in the environment . However , most real-world tasks , have multiple ( possibly conflicting ) objectives or constraints that need to be taken into consideration together , such as the signals related to the safety ( physical well-being of the agent or the environment ) , budget utilization ( energy or maintenance costs ) , etc . • Stakeholder control of the trade-off : The ML practitioners should have the ability to control the different trade-offs the agent is making and choose the one they consider best for the task at hand . • Offline setting : In many real-world domains ( e.g. , healthcare , finance or autonomous vehicles ) , there is an abundance of data , collected under a sub-optimal policy , but training the agent directly via interactions with the environment is expensive and risky . We assume that we only have access to a dataset of past trajectories that can be used for training ( Lange et al. , 2012 ) . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . • Preventing unintended behavior : We want the agent to be robust to both extrapolation errors from offline RL and misaligned objectives that are poor proxy of the user ’ s intentions and algorithm ’ s actual performance ( Ng et al. , 1999 ; Amodei et al. , 2016 ) . We consider the case where the user can specify undesirable behavior in the context of the performance observed in the batch . • Practical guarantees : We want guarantees about the undesirable behavior that might be caused by the agent in the real-world . We care about the results that can be obtained using the finite amount of samples we have in the batch , and aim to provide some measure of confidence in deploying the agents in the environment . To achieve this set of properties , we adopt the Seldonian framework ( Thomas et al. , 2019 ) , which is a general algorithm design framework that allows high-confidence guarantees for constraint satisfaction in a multi-objective setting . Based on the above specifications , we seek to answer the question : if we are given a batch of data collected under some ( suboptimal ) behavioral policy and some user preference , can we build a policy improvement algorithm that returns a policy with practical high-confidence guarantees on the performance of the policy w.r.t . the behavioral policy ? We acknowledge that there are other important challenges in RL , such as partial observability , safe exploration , non-stationary environments and function approximation in high-dimensional spaces , that also stand in the way of making RL a more applicable paradigm . These challenges are beyond the scope of this work , which should rather be thought of as taking a step towards this broader goal . In Section 2 , we present our contribution positioned with respect to other related work . In Section 3 , we formalize the setting and then extend traditional SPI algorithms to this setting . We then show it is possible to extend the previous work on Safe Policy Iteration ( SPI ) , particularly Safe Policy Iteration with Baseline Bootstrapping ( SPIBB , Laroche et al. , 2019 ) , for the design of agents that satisfy the above requirements . We show that the resulting algorithm is theoretically-grounded and provides practical high-probability guarantees . We extensively test our approach on a synthetic safetygridworld task in Section 4 and show that the proposed algorithm achieves better data efficiency than the existing approaches . Finally , we show its benefits on a critical-care task in Section 5 . The accompanying codebase is available at https : //github.com/hercky/mo-spibb-codebase . 2 Related work . Multi-Objective RL ( MORL ) : Traditional multi-objective approaches ( Mannor and Shimkin , 2004 ; Roijers et al. , 2013 ; Liu et al. , 2014 ) focus on finding the Pareto-frontier of optimal reward functions that gives all the possible trade-offs between different objectives . The user can then select a policy from the solution set according to their arbitrary preferences . In practice , an alternate trial and error based approach of scalarization is used to transform the multiple reward functions into a scalar reward based on preferences across objectives ( usually , by taking a linear combination ) . Most traditional MORL approaches have focused on the online , interactive settings where the agent has access to the environment . While some recent approaches are based on off-policy learning methods ( Lizotte et al. , 2012 ; Van Moffaert and Nowé , 2014 ; Yang et al. , 2019 ; Abdolmaleki et al. , 2020 ) , they lack guarantees . In contrast , our work focuses exclusively on learning in the offline setting and gives high-probability guarantees on the performance in the environment . Constrained-RL : RL under constraints frameworks , such as Constrained MDPs ( CMDPs , Altman , 1999 ) , present an alternative way to define preferences in the form of constraints over policy ’ s returns . Here , the user assigns a single reward function to be the primary objective ( to maximize ) and hard constraints are specified for the others . The major limitation of this setting is that it assumes the thresholds for the constraints are known a priori . Le et al . ( 2019 ) study offline policy learning under constraints and provide performance guarantees w.r.t . the optimal policy , but their work relies on the concentrability assumption ( Munos , 2003 ) . Concentrability is a strong assumption that upper bounds the ratio between the future state-action distributions of any non-stationary policy and the baseline policy under which the dataset was generated by some constant . From a practical perspective , it is unclear how to get a tractable estimate of this constant , as the space of future state-action distributions of non-stationary policies is vast . Thus , this constant can be arbitrarily huge , potentially even infinite when the baseline policy fails to cover the support of the space of all non-stationary policies ( such as in the low-data regime ) , leading to the performance bounds given by these methods to blow up ( and even be unbounded ) . Additionally , the guarantees in Le et al . ( 2019 ) are only valid with respect to the performance of the optimal policy . In this work , we instead focus on the performance guarantees based on returns observed in the dataset , as it does not require making any of the above assumptions . Reward design : Reward-design ( Sorg et al. , 2010 ) and reward-modelling approaches ( Christiano et al. , 2017 ; Littman et al. , 2017 ; Leike et al. , 2018 ) focus on designing suitable reward functions that are consistent with the user ’ s intentions . These approaches rely heavily on the human or simulator feedback , and thus do not carry over easily to the offline setting . Seldonian-RL ( and Safe Policy Improvement ) : The Seldonian framework ( Thomas et al. , 2019 ) is a general algorithm design framework that allows the user to design ML algorithms that can avoid undesirable behavior with high-probability guarantees . In the context of RL , the Seldonian framework allows to design policy optimization problems with multiple constraints , where the solution policies satisfy the constraints with high-probability . In the offline-RL setting , SPI refers to the objective of guaranteeing a performance improvement over the baseline with high-probability guarantees ( Thomas et al. , 2015a ; Petrik et al. , 2016 ; Laroche et al. , 2019 ) . Therefore , SPI algorithms are a specific setting that falls in the general category of Seldonian-RL algorithms . We focus on two categories of SPI algorithms that provide practical error bounds on safety : SPIBB ( Laroche et al. , 2019 ) that provides Bayesian bounds and HCPI ( Thomas et al. , 2015a , b ) that provides frequentist bounds . SPIBB methods constrain the change in the policy according to the local model uncertainty . SPIBB has been formulated in the context of a single reward function , and as such does not handle multiple rewards and by extension also lacks the ability for the user to specify preferences . Our primary focus is to provide a construction for extending the SPIBB methodology to the multi-objective setting that handles user preferences and provides high-probability guarantees . Instead of relying on model uncertainty , HCPI methods utilize the high-confidence lower bounds on the Importance Sampling ( IS ) estimates of a target policy ’ s performance to ensure safety guarantees . HCPI has been applied to solve Seldonian optimization problems for constrained-RL setting using an enumerable policy class . Thomas et al . ( 2019 ) suggested using HCPI for the MORL setting , and we build on that idea . Particularly , we show how HCPI can be implemented with stochastic policies in the context of our setting with user preferences and baseline constraints . 3 Methodology . 3.1 Setting . We consider the setting where the agent ’ s interactions with the environment can be modelled as a Markov Decision Process ( MDP , Bellman , 1957 ) . Let X and A respectively be the ( finite ) state and action spaces . Let p⋆ : X ×A → P ( X ) denote the true ( unknown ) transition probability function , where P ( X ) denotes the set of probability distributions on X . Without loss of generality , we assume that the process deterministically begins in the state x0 . We define [ N ] to be the set { 0 , 1 , . . . , N −1 } for any positive integer N . Let there be d different reward signals and r⋆ = { rk } k∈ [ d ] : X ×A → [ −r⊤ , r⊤ ] d be the true ( unknown ) stochastic multi-reward signal.1 Finally , γ = { γk } k∈ [ d ] ∈ [ 0 , 1 ) d is the multi-discount-factor . The MDP , m⋆ , can now be defined with the tuple ( X , A , p⋆ , r⋆ , γ , x0 ) . A policy π : X → P ( A ) maps a state to a distribution over actions . We denote by Π the set of stochastic policies . We consider the infinite horizon discounted return setting . For any k ∈ [ d ] , the kth reward value function vπm , k ( x ) : X → R denotes the expected discounted sum of rewards when when following policy π in an MDP m starting from state x. Analogously , we define the state-action value functions for performing action a in state x in MDP m under π for rewards as qπm , k ( x , a ) . Let Adv π m , k ( x , a ) = q π m , k ( x , a ) − vπm , k ( x ) denote the corresponding advantage function . The expected return of policy π w.r.t . the kth reward in the true MDP m⋆ is denoted by J πm⋆ , k = vπm⋆ , k ( x0 ) = Eπ , m⋆ [ ∑∞ t=0 γ t kRk , t | X0 = x0 ] , where action At ∼ π ( · | Xt ) , immediate reward Rk , t ∼ r⋆k ( · | Xt , At ) , and state Xt+1 ∼ p⋆ ( · | Xt , At ) . We consider the offline setting , where instead of having access to the environment we have a precollected dataset of trajectories denoted by D = { τi } i∈ [ |D| ] , where |D| denotes the number of trajectories in the dataset . A trajectory τ of length T is an ordered set of transition tuples of the form τ = { xi , ai , x′i , ri } i∈ [ T ] , where x′i denotes the state at the next time-step . We denote the Maximum 1Costs , which are meant to be minimized , can be expressed as negative rewards . Likelihood Estimation ( MLE ) of the MDP with m̂ = ( X , A , p̂ , r̂ , γ , x0 ) , where p̂ and r̂ denote the transition and reward models estimated from the dataset ’ s statistics . Assumption 3.1 ( Baseline policy ) . We assume that we have access to the policy that generated the dataset . We call such policy the baseline policy and denote it by πb . 2

Summary: The authors study the safe policy improvement problem in the offline RL settings where the known baseline policy is available. In this problem settings, the dataset has been already collected by the baseline policy and contains multiple reward signals from the environment. Under this problem setting, the authors aim to achieve a policy that balances the different rewards while guaranteeing that the new policy is better than the baseline policy. The authors proposed a new algorithm based on SPIBB and show its effectiveness in a synthetic environment and the one with sepsis management.