Multi-Objective SPIBB: Seldonian Offline Policy Improvement with Safety Constraints in Finite MDPs

1 Introduction . Reinforcement Learning ( RL ) as a paradigm for sequential decision-making ( Sutton , 1988 ) has shown tremendous success in a variety of simulated domains ( Mnih et al. , 2015 ; Silver et al. , 2017 ; OpenAI , 2018 ) . However , there are still quite a few challenges between the traditional RL research and real-world tasks . Most of these challenges stem from assumptions that are rarely satisfied in practice ( Dulac-Arnold et al. , 2019 ) , or the inability of the algorithm ’ s user to specify the desired behavior of the agent without being a domain expert ( Thomas et al. , 2019 ) . We focus on the real-world application point of view and posit the following requirements : • Multiple reward functions : Traditional RL methods assume a single scalar reward is present in the environment . However , most real-world tasks , have multiple ( possibly conflicting ) objectives or constraints that need to be taken into consideration together , such as the signals related to the safety ( physical well-being of the agent or the environment ) , budget utilization ( energy or maintenance costs ) , etc . • Stakeholder control of the trade-off : The ML practitioners should have the ability to control the different trade-offs the agent is making and choose the one they consider best for the task at hand . • Offline setting : In many real-world domains ( e.g. , healthcare , finance or autonomous vehicles ) , there is an abundance of data , collected under a sub-optimal policy , but training the agent directly via interactions with the environment is expensive and risky . We assume that we only have access to a dataset of past trajectories that can be used for training ( Lange et al. , 2012 ) . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . • Preventing unintended behavior : We want the agent to be robust to both extrapolation errors from offline RL and misaligned objectives that are poor proxy of the user ’ s intentions and algorithm ’ s actual performance ( Ng et al. , 1999 ; Amodei et al. , 2016 ) . We consider the case where the user can specify undesirable behavior in the context of the performance observed in the batch . • Practical guarantees : We want guarantees about the undesirable behavior that might be caused by the agent in the real-world . We care about the results that can be obtained using the finite amount of samples we have in the batch , and aim to provide some measure of confidence in deploying the agents in the environment . To achieve this set of properties , we adopt the Seldonian framework ( Thomas et al. , 2019 ) , which is a general algorithm design framework that allows high-confidence guarantees for constraint satisfaction in a multi-objective setting . Based on the above specifications , we seek to answer the question : if we are given a batch of data collected under some ( suboptimal ) behavioral policy and some user preference , can we build a policy improvement algorithm that returns a policy with practical high-confidence guarantees on the performance of the policy w.r.t . the behavioral policy ? We acknowledge that there are other important challenges in RL , such as partial observability , safe exploration , non-stationary environments and function approximation in high-dimensional spaces , that also stand in the way of making RL a more applicable paradigm . These challenges are beyond the scope of this work , which should rather be thought of as taking a step towards this broader goal . In Section 2 , we present our contribution positioned with respect to other related work . In Section 3 , we formalize the setting and then extend traditional SPI algorithms to this setting . We then show it is possible to extend the previous work on Safe Policy Iteration ( SPI ) , particularly Safe Policy Iteration with Baseline Bootstrapping ( SPIBB , Laroche et al. , 2019 ) , for the design of agents that satisfy the above requirements . We show that the resulting algorithm is theoretically-grounded and provides practical high-probability guarantees . We extensively test our approach on a synthetic safetygridworld task in Section 4 and show that the proposed algorithm achieves better data efficiency than the existing approaches . Finally , we show its benefits on a critical-care task in Section 5 . The accompanying codebase is available at https : //github.com/hercky/mo-spibb-codebase . 2 Related work . Multi-Objective RL ( MORL ) : Traditional multi-objective approaches ( Mannor and Shimkin , 2004 ; Roijers et al. , 2013 ; Liu et al. , 2014 ) focus on finding the Pareto-frontier of optimal reward functions that gives all the possible trade-offs between different objectives . The user can then select a policy from the solution set according to their arbitrary preferences . In practice , an alternate trial and error based approach of scalarization is used to transform the multiple reward functions into a scalar reward based on preferences across objectives ( usually , by taking a linear combination ) . Most traditional MORL approaches have focused on the online , interactive settings where the agent has access to the environment . While some recent approaches are based on off-policy learning methods ( Lizotte et al. , 2012 ; Van Moffaert and Nowé , 2014 ; Yang et al. , 2019 ; Abdolmaleki et al. , 2020 ) , they lack guarantees . In contrast , our work focuses exclusively on learning in the offline setting and gives high-probability guarantees on the performance in the environment . Constrained-RL : RL under constraints frameworks , such as Constrained MDPs ( CMDPs , Altman , 1999 ) , present an alternative way to define preferences in the form of constraints over policy ’ s returns . Here , the user assigns a single reward function to be the primary objective ( to maximize ) and hard constraints are specified for the others . The major limitation of this setting is that it assumes the thresholds for the constraints are known a priori . Le et al . ( 2019 ) study offline policy learning under constraints and provide performance guarantees w.r.t . the optimal policy , but their work relies on the concentrability assumption ( Munos , 2003 ) . Concentrability is a strong assumption that upper bounds the ratio between the future state-action distributions of any non-stationary policy and the baseline policy under which the dataset was generated by some constant . From a practical perspective , it is unclear how to get a tractable estimate of this constant , as the space of future state-action distributions of non-stationary policies is vast . Thus , this constant can be arbitrarily huge , potentially even infinite when the baseline policy fails to cover the support of the space of all non-stationary policies ( such as in the low-data regime ) , leading to the performance bounds given by these methods to blow up ( and even be unbounded ) . Additionally , the guarantees in Le et al . ( 2019 ) are only valid with respect to the performance of the optimal policy . In this work , we instead focus on the performance guarantees based on returns observed in the dataset , as it does not require making any of the above assumptions . Reward design : Reward-design ( Sorg et al. , 2010 ) and reward-modelling approaches ( Christiano et al. , 2017 ; Littman et al. , 2017 ; Leike et al. , 2018 ) focus on designing suitable reward functions that are consistent with the user ’ s intentions . These approaches rely heavily on the human or simulator feedback , and thus do not carry over easily to the offline setting . Seldonian-RL ( and Safe Policy Improvement ) : The Seldonian framework ( Thomas et al. , 2019 ) is a general algorithm design framework that allows the user to design ML algorithms that can avoid undesirable behavior with high-probability guarantees . In the context of RL , the Seldonian framework allows to design policy optimization problems with multiple constraints , where the solution policies satisfy the constraints with high-probability . In the offline-RL setting , SPI refers to the objective of guaranteeing a performance improvement over the baseline with high-probability guarantees ( Thomas et al. , 2015a ; Petrik et al. , 2016 ; Laroche et al. , 2019 ) . Therefore , SPI algorithms are a specific setting that falls in the general category of Seldonian-RL algorithms . We focus on two categories of SPI algorithms that provide practical error bounds on safety : SPIBB ( Laroche et al. , 2019 ) that provides Bayesian bounds and HCPI ( Thomas et al. , 2015a , b ) that provides frequentist bounds . SPIBB methods constrain the change in the policy according to the local model uncertainty . SPIBB has been formulated in the context of a single reward function , and as such does not handle multiple rewards and by extension also lacks the ability for the user to specify preferences . Our primary focus is to provide a construction for extending the SPIBB methodology to the multi-objective setting that handles user preferences and provides high-probability guarantees . Instead of relying on model uncertainty , HCPI methods utilize the high-confidence lower bounds on the Importance Sampling ( IS ) estimates of a target policy ’ s performance to ensure safety guarantees . HCPI has been applied to solve Seldonian optimization problems for constrained-RL setting using an enumerable policy class . Thomas et al . ( 2019 ) suggested using HCPI for the MORL setting , and we build on that idea . Particularly , we show how HCPI can be implemented with stochastic policies in the context of our setting with user preferences and baseline constraints . 3 Methodology . 3.1 Setting . We consider the setting where the agent ’ s interactions with the environment can be modelled as a Markov Decision Process ( MDP , Bellman , 1957 ) . Let X and A respectively be the ( finite ) state and action spaces . Let p⋆ : X ×A → P ( X ) denote the true ( unknown ) transition probability function , where P ( X ) denotes the set of probability distributions on X . Without loss of generality , we assume that the process deterministically begins in the state x0 . We define [ N ] to be the set { 0 , 1 , . . . , N −1 } for any positive integer N . Let there be d different reward signals and r⋆ = { rk } k∈ [ d ] : X ×A → [ −r⊤ , r⊤ ] d be the true ( unknown ) stochastic multi-reward signal.1 Finally , γ = { γk } k∈ [ d ] ∈ [ 0 , 1 ) d is the multi-discount-factor . The MDP , m⋆ , can now be defined with the tuple ( X , A , p⋆ , r⋆ , γ , x0 ) . A policy π : X → P ( A ) maps a state to a distribution over actions . We denote by Π the set of stochastic policies . We consider the infinite horizon discounted return setting . For any k ∈ [ d ] , the kth reward value function vπm , k ( x ) : X → R denotes the expected discounted sum of rewards when when following policy π in an MDP m starting from state x. Analogously , we define the state-action value functions for performing action a in state x in MDP m under π for rewards as qπm , k ( x , a ) . Let Adv π m , k ( x , a ) = q π m , k ( x , a ) − vπm , k ( x ) denote the corresponding advantage function . The expected return of policy π w.r.t . the kth reward in the true MDP m⋆ is denoted by J πm⋆ , k = vπm⋆ , k ( x0 ) = Eπ , m⋆ [ ∑∞ t=0 γ t kRk , t | X0 = x0 ] , where action At ∼ π ( · | Xt ) , immediate reward Rk , t ∼ r⋆k ( · | Xt , At ) , and state Xt+1 ∼ p⋆ ( · | Xt , At ) . We consider the offline setting , where instead of having access to the environment we have a precollected dataset of trajectories denoted by D = { τi } i∈ [ |D| ] , where |D| denotes the number of trajectories in the dataset . A trajectory τ of length T is an ordered set of transition tuples of the form τ = { xi , ai , x′i , ri } i∈ [ T ] , where x′i denotes the state at the next time-step . We denote the Maximum 1Costs , which are meant to be minimized , can be expressed as negative rewards . Likelihood Estimation ( MLE ) of the MDP with m̂ = ( X , A , p̂ , r̂ , γ , x0 ) , where p̂ and r̂ denote the transition and reward models estimated from the dataset ’ s statistics . Assumption 3.1 ( Baseline policy ) . We assume that we have access to the policy that generated the dataset . We call such policy the baseline policy and denote it by πb . 2

Summary: The paper proposes a new Safe Policy Improvement problem formulation to deal with multi-objective offline settings.   Authors consider a problem where a batch of data, collected by following a known baseline policy, is given and the goal is to return a new policy that, achieves (with high confidence) better results than the baseline, considering also user’s preferences.   Taking inspiration from the Seldonian approach [Thomas et al., 2019] and offline SPI algorithms (SPIBB [Laroche et al., 2019], Soft-SPIBB [Nadjahi et al., 2019], HCPI [Thomas et al., 2015a,b]), the authors then show that this can be formulated as particular optimization problems which can be solved, by using common techniques with a computational penalty that scales with the number of objectives, to obtain a policy with the required safety guarantees.   Finally, they compare their methods (MO-SPIBB and MO-HCPI) against two baselines on two experimental settings, a synthetic problem and real-world scenario, showing that their approaches are, when optimized, very competitive but also safe, obtaining results that are consistently better than the baseline up to the accuracy desired, especially in a small sample regime.