Multi-Objective SPIBB: Seldonian Offline Policy Improvement with Safety Constraints in Finite MDPs
Authors: Harsh Satija
Philip S. Thomas
Joelle Pineau
Romain Laroche
Venue: NeurIPS 2021 Poster (2021)
Keywords: Reinforcement Learning
Multi-Objective RL
RL Safety
Safe Policy Improvement
Seldonian Algorithms

Abstract: We study the problem of Safe Policy Improvement (SPI) under constraints in the offline Reinforcement Learning (RL) setting. We consider the scenario where: (i) we have a dataset collected under a known baseline policy, (ii) multiple reward signals are received from the environment inducing as many objectives to optimize. We present an SPI formulation for this RL setting that takes into account the preferences of the algorithm’s user for handling the trade-offs for different reward signals while ensuring that the new policy performs at least as well as the baseline policy along each individual objective. We build on traditional SPI algorithms and propose a novel method based on Safe Policy Iteration with Baseline Bootstrapping (SPIBB, Laroche et al., 2019) that provides high probability guarantees on the performance of the agent in the true environment. We show the effectiveness of our method on a synthetic grid-world safety task as well as in a real-world critical care context to learn a policy for the administration of IV fluids and vasopressors to treat sepsis.

Multi-Objective SPIBB: Seldonian Offline Policy Improvement with Safety Constraints in Finite MDPs
Harsh Satija
Philip S. Thomas University of Massachusetts
McGill University, Mila
harsh. satija@maill www.ial.mmmml....
pthomas@cs umass edu
Joelle Pineau
Romain Laroche
McGill University, Mila, Facebook Al Research jpineauecs cetpuuuessssmmiil... mogill.ca
Microsoft Research
romain www.oceeemmcooooff.t.ooooo
Abstract
We study the problem of Safe Policy Improvement (SPI) under constraints in the offline Reinforcement Learning (RL) setting. We consider the scenario where: (i) we have dataset collected under a known baseline policy, (ii) multiple reward signals received from the environment inducing as many objectives optimize. We present an SPI formulation for this RL setting that takes into account the prefer- ences of the algorithm's 's user for handling the trade-offs for different reward signals while ensuring that the new policy performs at least as well as the baseline policy along each individual objective. We build traditional SPI algorithms and propose a novel method based on Safe Policy Iteration with Baseline Bootstrapping (SPIBB, Laroche al., 2019) that provides high probability guarantees the performance of the agent in the true environment. We show the effectiveness of our method on a synthetic grid-world safety task well in a weal-world critical care context learn a policy for the administration of IV fluids and vasopressors to treat sepsis.
Introduction
Reinforcement Learning (RL) as paradigm for sequential decision-making (Sutton, 1988) has shown tremendous success a variety simulated domains (Mnih et al., 2015; Silver et al., 2017: OpenAL, 2018). However, there are still quite few challenges between the traditional RL research and real-world tasks. Most of these challenges stem from assumptions that are rarely satisfied in practice (Dulac-Arold et al., 2019), or the inability of the algorithm's 's user to specify the desired behavior of the agent without being a domain expert (Thomas et al., 2019). We focus on the real-world application point of view and posit the following requirements:
Multiple reward functions: Traditional RL methods assume single scalar reward is present in the environment. However, most real- world tasks, have multiple (possibly conflicting) objectives constraints that need to taken into consideration together, such as the signals related to the safety (physical well- well-being of the agent or the environment), budget utilization (energy maintenance costs), etc.
Stakeholder control of the trade-off: The MI. practitioners should have the ability to control the different trade-offs the agent is making and choose the one they consider best for the task at hand.
Offline setting: In many real-world domains (e.g., healthcare, finance or autonomous vehicles), there an abundance of data, collected under sub-optimal policy, but training the agent directly via interactions with the environment is expensive and risky. We assume that we only have access dataset of past trajectories that can be used for training (Lange et al., 2012).
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Preventing unintended behavior: We want the agent to be robust to both extrapolation errors from offline RL and misaligned objectives that are poor proxy of the user's intentions and algorithm's actual performance (Ng et al., 1999; Amodei et al., 2016). We consider the case where the user can specify undesirable behavior in the context of the performance observed in the batch.
Practical guarantees: We want guarantees about the undesirable behavior that might be caused by the agent in the real-world. We care about the results that can be obtained using the finite amount of samples we have in the batch, and aim to provide some measure of confidence in deploying the agents in the environment.
To achieve this set of properties, we adopt the Seldonian framework (Thomas et al., 2019), which is general algorithm design framework that allows high-confidence guarantees for constraint satisfaction in a multi-objective setting. Based on the above specifications, we seek to answer the question: if we are given a batch of data collected under some (suboptimal) behavioral policy and some user preference, can we build policy improvement algorithm that returns policy with practical nigh-confidence guarantees on the performance of the policy w.r.t. the behavioral policy?
We acknowledge that there are other important challenges in RI., such as partial observability, safe exploration, -tationony environments and function approximation in high- -igh-dimensional spaces, that also stand in the way of making RL a more applicable paradigm. These challenges are beyond the scope of this work, which should rather be thought of as taking a step towards this broader goal.
In Section 2, we present our contribution positioned with respect to other related work. In Section 3, we formalize the setting and then extend traditional SPI algorithms to this setting. We then show it is possible to extend the previous work on Safe Policy Iteration (SPI). particularly Safe Policy Iteration with Baseline Bootstrapping (SPIBB. Laroche al., 2019). for the design of agents that satisfy the above requirements. We show that the resulting algorithm is pooorettcallyggruuddee provides practical high-probability guarantees. We extensively test our approach on a synthetic safety- gridworld task in Section 4 and show that the proposed algorithm achieves better data efficiency than the existing approaches. Finally, we show its benefits on a critical-care task in Section 5. The accompanying codebase is available at https: / ////gitaa gighub.......
and
2 Related work
Multi-Objective RL (MORL): Traditional multi-objective approaches (Mannor and Shimkin, 2004; Roijers et al., 2013; Liu et al., 2014) focus on finding the Pareto-frontic of optimal reward functions that gives all the possible trade-offs between different objectives. The user can then select a policy from the solution set according to their arbitrary preferences. In practice, an alternate trial and error based approach of scalarization is used to transform the multiple reward functions into a scalar reward based preferences across objectives (usually, by taking a linear combination). Most traditional MORL. approaches have focused on the online, interactive settings where the agent has access to the environment. While some recent approaches are based on off-policy learning methods (Lizotte et al., 2012: Van Moffaert and Nowé, 2014; Yang et al., 2019; Abdolmalcki et al., 2020), they lack guarantees. In contrast, our work focuses exclusively on learning in the offline setting and gives high-probability guarantees on the performance in the environment.
Constrained-RL. RL under constraints frameworks, such as Constrained MDPs (CMDPs, Altman, 1999), present an alternative way to define preferences in the form of constraints over policy's 's returns. Here, the user assigns single reward function to be the primary objective (to maximize) and hard constraints are specified for the others. The major limitation of this setting is that it assumes the thresholds for the constraints are known priori. Le et al. (2019) study offline policy learning under constraints and provide performance guarantees w.r.t. the optimal policy, but their work relies on the concentrability assumption (Munos, 2003).
Concentrability is strong assumption that upper bounds the ratio between the future state-action distributions of any non-stationary policy and the baseline policy under which the dataset was generated by some constant. From practical perspective, is unclear how to get a tractable estimate of this constant, as the space of future state-action distributions of -tatioati policies is vast. Thus, this constant can be arbitrarily huge, potentially even infinite when the baseline policy fails to cover the support of the space of all non-stationary policies (such as in the low-data regime), leading to the performance bounds given by these methods to blow up (and even be unbounded). Additionally, the guarantees in Le et al. (2019) are only valid with respect the performance of the optimal policy.

---

In this work, we instead focus on the performance guarantees based on returns observed in the dataset, as it does not require making any of the above assumptions.
Reward design: Reward-design (Sorg et al., 2010) and sward-modelling approaches (Christiano al., 2017; Littman et al., 2017; Leike et al., 2018) focus designing suitable reward functions that are consistent with the user's intentions. These approaches rely heavily on the human or simulator feedback, and thus do not carry over easily to the offline setting.
Seldonian-RLe (and Safe Policy Improvement): The Seldonian framework (Thomas et al., 2019) is general algorithm design framework that allows the user to design ML algorithms that can avoid undesirable behavior with high probabbiiyy guarantees. In the context of RL, the Seldonian framework allows to design policy optimization problems with multiple constraints, where the solution policies satisfy the constraints with high-probability. In the off setting, SPI refers to the objective of guaranteeing a performance improvement over the baseline with high-probability guarantees (Thomas et al., 201 5a; Petrik et al., 2016; Laroche et al., 2019). Therefore, SPI algorithms are a specific setting that falls in the general category of Seldonian-B algorithms.
We focus on two categories of SPI algorithms that provide practical error bounds on safety: SPIBB (Laroche et al., 2019) that provides Bayesian bounds and HCPI (Thomas et al., 20152,0 that provides frequentist bounds. SPIBB methods constrain the change in the policy according to the local model uncertainty. SPIBB has been formulated in the context of single reward function, and as such does not handle multiple rewards and by extension also lacks the ability for the user to specify preferences. Our primary focus is to provide construction for extending the SPIBB methodology to the multi-objective setting that handles user preferences and provides high-probability guarantees.
Instead of relying model uncertainty, HCPI methods utilize the high-confidence lower bounds on the Importance Sampling (IS) estimates of a target policy's 's performance to ensure safety guarantees. HCPI has been applied to solve Seldonian optimization problems for constrained-RL setting using an enumerable policy class. Thomas et al. (2019) suggested using HCPI for the MORL setting, and we build that idea. Particularly, we show how HCPI can be implemented with stochastic policies in the context of our setting with user preferences and baseline constraints.
3 Methodology
3.1 Setting
We consider the setting where the agent's interactions with the environment can be modelled as Markov Decision Process (MDP, Bellman, 1957). Let X and A respectively the (finite) state and action spaces. Let p² X x A + (X) denote the true (unknown) transition probability function, where (X) denotes the set of probability distributions on X. Without loss of generality, we assume that the process deterministically begins in the state zo- We define [N] to be the set (0, 1, 1,.... N ---1 1} for any positive integer N. Let there be different reward signals and r* = (rahxe(d) : x x A
[-rr,r, r=]" be the true (unknown) stochastic multi-reward signal. Finally, 7 {nw]eetd E 10, 1)dd is the multi-discount-facee
The MDP, m², can now be defined with the tuple (X, A. p², r" 2o). A policy x X' - (AA) maps state to distribution over actions. We denote by II the set of stochastic policies. We consider the infinite horizon discounted return setting. For any E [d], the 1thh reward value function " X + ER denotes the expected discounted sum of rewards when when following policy π in an MDP m starting from state x. Analogously, we define the state-action value functions for performing action in state in MDP m under for rewards as IT (a, a). Let Adv m. (I, = Im, a) un (#) denote the corresponding advantage function. The expected return of policy π w.r.t. the th reward in the true MDP m² is denoted by JT k u x (2o) (20)
Ex,m E RR, Xo rol. where action A, ~ .(- X.), immediate reward Rx, ~ r+(- X,, At), and state Xe+1 ~ p"
At).
We consider the offline setting, where instead of having access to the environment we have a pre- collected dataset of trajectories denoted by D = (Te (TT))) where (D] denotes the number of trajectories in the dataset. A trajectory T of length T is an ordered set of transition tuples of the form 7 {ar, a4 d/e ry Je(T), where ni denotes the state at the next time-step. We denote the Maximum
'Costs, which are meant be minimized, can be expressed as negative rewards.
3

---

Likelihood Estimation (MLE) of the MDP with m = 7,, To.). where p and f denote the transition and reward models estimated from the dataset's statistics.
Assumption 3.1 (Baseline policy). We assume that we have access to the policy that generated the dataset. We call such policy the baseline policy and denote it by T..
3.2 Problem formulation
We consider safe policy improvement with respect to the baseline according to the dimensions of the multi-objective setting. Therefore, under Bayesian approach, we search for target policies such that they perform better (up to a precision error () than the baseline along every objective function with high probability 1 6, where ( and ō hyper-parameters controlled by the user, denoting the risk that the practitioner is willing to take. We denote by II, the set of admissible policies that satisfy:
IP Vk E [d], Jm. Jm -
<(D > 1 5.
(1)
In the multi-objective case, there does not exist single optimal value, but Pareto frontier of optimal values. One way to evaluate the MORL problems via the multiple-policy approaches (Vamplew et al., 2011; Roijers et al., 2013) 3) that compute the policies that approximate the true optimal Pareto-fronticn However, note that optimality and safety are contradicting objectives. is not clear how (and if) one can make claims about optimality in the offline setting without bringing in additional unrealistic assumptions (Section 2, MORL). Instead, we take an alternate approach inspired by another category of MORL methods called single-policy (Roijers al., 2013; Van Moffaert and Nowé, 2014) 4) where the trade-offs between different objectives are explicitly controlled by the user via providing a scalarization or preferences over objectives. We assume the user preference as an input to our algorithms, and used for scalarization of the objectives, where A E R+ Our objective therefore becomes
{Ak { ke(d]] given
arg max ETT..
Σ kE(d]
(2)
The above formulation gives freedom to the user in terms of what particular quantity they want to optimize via X, but still ensures that the solution policy performs as well as the baseline policy across all d objectives. Note that our explicit goal is to maximize the objective specified by the user. However, the user might make mistakes in specifying this objective (Section 2, Reward design), and the above formulation offers guarantees that prevent deteriorating the performance of the policy across any of the objectives. This allows the user to to experiment with different reward design strategies in safety-critiaal settings without worrying about the risks of ill-defined scalarizations. naive approach would be applying the user scalarization to also define the safety constraints. However, this construction fails to prevent undesirable behavior for the individual objectives (shown in Appendix A).
3.3 Multi-Objective SPIBB (MO-SPIBB)
Robust MDPs (lyengar, 2005; Nilim and El Ghaoui, 2005) can be regarded as an approximation of the Bayesian formulation by partitioning the MDP space M into two subsets: the subset of plausible MDPs E and the subset of implausible MDPs. The plausible set is classically constructed from concentration bounds over the reward and transition function:
E E= E m, s.t. Vx, a,
|p((|a,, a) p(1|1, a) S e(x, a), ) ||[[ a) f(x, a)l S e(x, a)r- }
where is upper bound on the state-action error function of the model that are classically obtained with concentration bounds, such that the true environment vn* E with high probability - 6. In the single objective framework, Laroche al. (2019) empirically show that optimising the worst-case performance policy in E\t provides policies that are too conservative. Petrik et al. (2016) prove that it NP-hard find the policy æ that maximises the worst-case policy improvement over
Instead, the SPIBB methodology (Laroche et al., 2019) consists in searching for a policy that maximizes the safe policy improvement in the MLE MDP, under some policy constraints: SPIBB
Simão et al. (2020) proved that SPIBB/Soft-SPIBB bounds may be obtained with an estimate of

---

and Soft-SPIBB (Nadjahi et al., 2019) policy search constraints both revolve around the idea that we must only consider policies for which the policy improvement may be accurately estimated. Using %b as reference, SPIBB allows policy changes only in state-action pairs for which more than samples have been collected. Soft-SPIBB extends this by applying soft constraints that allow slight changes in the policy for the uncertain state-action pairs, which are controlled by an error bound related to model uncertainty, As such, on low-confidence transitions, this class of methods provides a mechanism that prevents the agent from deviating too much from rTs In this work, we build on Soft-SPIBB because has yielded better empirical results. Formally, its constraint on the policy class is defined by:
1 nr s.t. Vx, Ee(xs [<aa))) - =o(a|>)] S
where is a hyper-parameter that controls the deviation from the baseline policy.
We define x (x, a) = ExE(q A (2t a) to be the state-action value function associated with the linearized À parameters. The same notation extension is used for u ( and Jm.x. The application of Soft-SPIBB to multi-objective safe policy improvement is therefore direct:
arg max Jm wIlI,nII,
(3)
which is always realizable since Th E II, llls.
We show that the construction of the plausible set required for the application of SPIBB is technically sound by deriving the concentration bounds for the multi-objective case. In Appendix B.I, we show with Hoeffding's inequality that grows the square root of the logarithm of d (the number of reward functions), iee. almost imperceptibly. From there, all the SPIBB theoretical results from Laroche et al. (2019); Nadjahi et al. (2019); Simão al. (2020) may be generalized at negligible SPI guarantee cost to the multi-objective setting, by applying their theorems separately to every objective function.
Now, the problem Equation (3) can be transformed into a policy improvement procedure that solves for every state E X the following optimization problem':
Ts-OPT
arg max(((((11) (-)x)) [z),, x (2. .))
(S-OPT)
Tell
s.t.
Σ e( (x(aa)) - no (a|x))) Se,
(T m (T,)
acd
Vk E [d]. Σ (alx)) Adv (z, a) 2 0. m..k
(π E II,)
REAA
The above procedure requires us to make additional algorithmic modifications that are not present in the original SPIBB algorithms. In particular, we need to explicitly incorporate advantage constraints for safety-guarantes for the individual objectives (proof given Appendix B.2). The classic single- objective SPIBB algorithms do not need to check the advantage conditions because it is automatically guaranteed by the arg max and the fact that Tr E IIş.
Using the construction above, we directly get the following result on the performance guarantees for each objective function that satisfies the desired property Equation (1):
Proposition 3.1.
The policy returned from solving the S-OPT satisfies the following property in
every state I E X' with probability at least (1 - 8):
EUmax
Vk E [d]. v. ae (x) - em & (x) >
(4)
1
where Umax I Fr is the maximum of the value function.
1-
The proof is presented in Appendix B.3. The solution of S-OPT is computed by solving the Linear Program using standard solvers, such evxpy (Diamond and Boyd, 2016). There is an increase in the computational cost proportional to the number of reward functions. Compared to Soft-SPIBB,
In practice, also need check Vo that .[( (r) valid probability distribution: positive and sums
5

---

the value and advantage functions estimation cost increases by factor of d: respectively ( and [|] There is ((D)) cost for estimating the error bounds, and we also require solving a Linear Program for each state that approximately amounts to additional ((( d))
steps to the total computational cost (Boyd al., 2004).
Remark (Extension to Constrained -LL. The above methodology also be extended to the Constrained- RL setting for offline policy improvement in general CMDPs. Recall that SPIBB algorithms offer guarantees in the form of: 1/2 tb 2 02 Ob {. where i2 and U/ are respectively the true values of the target and baseline policies, 6, and ig are their estimates in the MLE MDP, and & is an error term due parametric uncertainty. As consequence, any constraint such that c I 1% + 6 0g - { may be guaranteed (vp- - ig may easily be bounded with Hoeffding s inequality), and when e is larger, we can return no solution found as with other Seldonian algorithms.
3.4 Multi-Objective HCPI (MO-HCPI)
We briefly recall how the HCPI methodology (Thomas et al., 2015a,b) can be applied directly for solving the objective in Equation (2). For target policy, Te, we use IS, Te, mT) to denote the estimated returns for the kth reward component (rk) using any IS based off-policy estimator (Precup, 2000). A high-confidence lower bound on Jm can be defined as:
Pr PT(TTA 2 (D, Te, mb) (D, 2 21-8/d, 8/d.
(5)
where CI2 > 0 denotes the terms associated with the choice of concentration inequality employed (and typically lim CD, = 0).
The dataset D is first split into train (Dtr) and test (D.) by the user. Let IS^ denote the IS estimator associated with the user-specified reward scalarization A. Given the user specified parameters: X, 6, CI, IS, Dtr. D, and To. the policy improvement problem in Equation (2) is transformed to the following optimization problem:
TH-OPT = arg max (D TT, Tt))
(H-OPT)
=E
S.t.
Vk E [d], (D, TT xb)
Cl (D, 2
where fa denote the empirical returns for under mho The policy returned by H-OPT will only violate the safety guarantees with probability at most 8. Proof of this claim and additional details are provided in Appendix C.
Although we only focus on finite MDPs in this work, the HCPI based approach relies on IS estimates and therefore it can also be used for infinite MDPs or POMDPs. Unfortunately, the IS estimates are typically known to suffer from high variance (Guo et al., 2017). Furthermore, the optimization problem in H-OPT is more challenging, and we need to resort to regularization based heuristics.
4 Synthetic Experiments
The main benefits of working in a synthetic domain are: (i) we can evaluate the performance on the true MDP instead of relying on off- aff-policy evaluation (OPE) methods, (ii) we have control over the quality of the dataset. We test both MO-SPIBB (S-OPT) and MO-HCPI (H-OPT) on variety of parameters: the amount of data, quality of baseline and different user reward scalarizations.
Env details: We take standard CMDP benchmark (Leike et al., 2017; Chow et al., 2018) which consists of a 10 x 10 grid. From any state, the agent can move to the adjoining cells in the 4 directions using the 4 actions. The transitions are stochastic, with some probability (generated randomly for each state-action for every environment instance) the agent is successfully able to reach the next state, and with (1 a) the agent stays in the current state. The agent starts the bottom-right corner, and the goal is to reach the opposite corner (top-left). The pits are spawned randomly with some uniform probability (Πрa 0.3) for each cell. The reward vector consists of two rewards signals. A primary reward ro that is related the goal and is + +1000.0 on reaching the goal and -1.0 at every other time-step. The secondary reward ri is related to pits, for which the agent gets -1.0 for any action taken in the pit. The constraint threshold for this CMDP is 220 and 7 = 0.99. Maximum length of an episode is 200 steps. Therefore, the task objective is to reach the goal in the least number of steps, such that the agent does not spend more than 2 steps in the pit cells.

---

Dataset collection procedure: For every random CMDP generated, we first find the optimal policy I by using the procedure described in Appendix D.I. The baseline policy is generated using a convex combination of the optimal policy and uniform random policy (Trand), i.e., Th = px + [[--)]rand where p controls how close TT 's performance to π². Different datasets with varying sizes and p are then collected under Ty and given as input to the methods.
Baselines: We compare against the following baselines:
Linearized: This baseline transforms the rewards into a single scalar using X and then applies the traditional policy improvement methods on the linearized objective, i.,, arg max Tm.x
Adv-Linearized: This method has the same objective as the Linearized baseline, with the additional constraints based on advantage estimators built from m, i.c. Vr E X:
arg max(T (-/r)(((((),)))))<<)) (x)) I,, (2,-)) =I1
(6)
s.t. vk E [d], Σ (a|p) Ady a) 2 O.
ma,k
aEA
Evaluation:
Using m", we can directly calculate the returns for any solution policy. Only tracking the scalarized objective can be misleading, so we track the following metrics:
Improvement over Th: This denotes the difference between the scalarized return of the solution policy and the baseline policy, i.c., Jm A - Tm A- Mean improvement over mh captures on average improvement over Ty in terms of the scalarized objective.
Failure-rate: The failure rate over n runs captures the number of times, on average, the solution policy ends up violating the safety constraints in Equation (1), and thus performs worse than the baseline. In the context of this task, safety constraints are violated if either the agent takes longer reach the goal, steps into more number of pits compared to TT.
We test on different combinations of user preference (A) and baseline's quality (p) on 100 randomly generated CMDPs, where X, E {0, } / E {0.1, 0.4, 0.7, 0.9} and [D] E (10, 50, 500, 2000). We evaluate under two settings: (i) we use a fixed set of parameters across different (A, p) combinations, where we run S-OPT with {0.01, 0.1, 1.0} and H-OPT with Doubly Robust IS estimator (Jiang and Li, 2015) and Student's t-test concentration inequality; (ii) we treat them as preer-paraeeters that can be optimized for partieular (A, p) combination. The best syper-parameters are tuned in single environment instance and then they are used to benchmark the results on 100 random CMDPs.
The mean results with fixed parameters and 5 = 0.1 can be found in Figure la. The high failure rate of Linearized baseline, regardless of the size of the dataset, is expected as it optimizes the scalarized reward directly and is agnostic of the individual rewards. Adv-Linearized performs better, but in the low daga-regime, we see a high failure rate that eventually decreases as the size of dataset increases. This is expected because with more data, more reliable advantage functions estimates are calculated that are representative of the underlying CMDP. Compared to the basclines, both S-OPT and H-OPT maintain failure rate below the required confidence parameter 8, regardless of the amount of data. Also, as the size of dataset increases, we see an increase in improvement over Tb, that makes sense the methods only deviate from baseline when they are sure of the performance guarantees. We expect S-OPT to violate the constraints with increasing value of c, as it relaxes the looser guarantee on performance. This
Results:
constraint on the policy-class (Section 3.3) and leads again is reflected in our experiments where S-OPT with 1.0 has a higher failure-rate than e = 0.1. We observed similar trends for different 8 values. A more detailed plot corresponding different À and p combinations as well as results for a riskier value of 5 = 0.9 are given in Appendix D.2.
The results with optimized hyper- parameerrsss can be found in Figure 1b. We notice that when the parameter is tuned properly, S-OPT has better performance in terms of improvement over nú for the same amount of samples when compared to H H-OPT, while still ensuring the failure rate is less than 8. These observations are consistent with the results in the single-objective setting in the original SPIBB works (Laroche et al., 2019; Nadjahi et al., 2019). The general trends and observations from the eaeed-arrmmtter case are also valid here. Additional details, including results for X, p combinations, hyper-parameters considered and qualitative analysis can be found in Appendix D.3.
We also compare our methods against Le et al. (2019) in Appendix D.4. We show the advantage of our approach over Le et al. (2019), particularly in the low-data regime, where our methods can
7

---

6-0.1
5 001
175
1.2
5-0PT--
Improvemers over 60
153
H OPT
Lisarizzd
S.0PT
0.5
Fallure
0.4
50
500
[D] numn of trajectories
[d] - num of trajectories
(a) Fixed hyper-parameters
6-0,1
S-OPT IOPP
375
Unearized
Adv-
Improvement over
0.5
125
0.6
100
0.4
9,0
ttoject
|D]
trajectories
(b) Optimized hyper- syper-parameters.
Figure I: Results on 100 random CMDPs for different À and p combinations with 0.1. The different agents are represented by different markers and colored lines. Each point on the plot denotes the mean (with standard bars) for 12 different À, P combinations for the 100 randomly generated CMDPs (1200 datapoints). The x-axis denotes the amount of data the agents were trained on. The y-axis for left subplot in each sub-figure represents the improvement over baseline and the right subplot denotes the failure rate. The dotted black line in the right subplots represents the high-confidence parameter = 0.1. Figure Ia denotes when the hyper-paranen paremeearmeterss are fixed = (0.01, 0.1, 1.0) and IS Doubly Robust (DR) estimator with student's 's 1-test concentration inequality. Figure lb is the version with tuned hyper- prper-parameters for each combination.
improve over the baseline policy while ensuring low failure rate. This makes sense the method in Le et al. (2019) relies on the concentrability coefficient which can be arbitrarily high in the low data setting, and therefore their performance guarantees do not hold anymore. We also provide experiments on the scalability of methods with the number of objectives d in Appendix D.5.
5 Real-world experiment
In order to validate the applicability of our methods on a real-world task, we consider recent works on sepsis management via RL, where we only have access to pre-collected patient dataset and goal to recommend treatment strategies for patients with sepsis in the ICU (Komorowski al., 2018: Tang et al., 2020). Sepsis is defined as life-threatening organ dysfunction caused by dysregulated host response to an infection (Singer et al., 2016). The main treatment method sepsis involves the repeated administration of intravenous (IV) fluids and vasopressors, but how to manage their appropriate doses at the patient level is still key clinical challenge (Rhodes et al., 2017).
The problem is anfety-critical as our methods need to be cautious about using the data that was possibly collected under unobservable confounders and that can lead to biased model estimates. For instance, a study by Ji et al. (2020) of the model used in Komorowski et al. (2018) found that learned model suggests clinically implausible behavior in the form unexpectedly aggressive treatments. We show that our methodology can be applied here to prevent such behavior that results from small sample sizes. We propose to do so by incorporating safety constraints to prevent recommending the treatment decisions that were never or rarely performed in the datasct.

---

Data and MDP Construction: We use the publicly available ICU dataset MIMIC-III (Johnson et al.. 2016), with the setup described by Komorowski et al. (2018); Tang et al. (2020) and build on top of their data prepproeessing and MDP construction methodology." This leaves us with a cohort of 20,954 unique patients. The state-space consisting of clinical variables summarizing features like demographics, physiological condition, laboratory values, etc., is discretized using a k-means based clustering algorithm to map the states to 750 clusters. The actions include administration of IV fluids and vasopressors, which are categorized into 5 dosage bins each, leading y is set to 0.99. The reward is based on patient mortality. The agent gets a reward, ro, of 100 at the end of the episode based the survival of the patient. More details can be found in Appendix E.
total of |A| 25. The
In the original work, the rare state-actions taken by the clinicians (state-action pairs observed infrequently in the training set) are removed from the dataset. Instead of removing them, we define an additional reward, r1, based on the rarity the state-action pair. We define rare state-action pairs to be those that are taken less than 10 times throughout training dataset, and the agent gets a reward of 10 for every such rare state-action taken, i.e., ri (I., a) = 10.0 if Count(x;, a) < 10. The final task objective then becomes to suggest treatments that handles the trade-off between prioritizing improving the survival vs prioritizing commonly used treatment decisions.
Evaluation: We compare our approach with the same baselines from Section on different À combinations. We run our methods for 10 runs with different random seeds, where for each run the cohort dataset was split into train/valid/test sets in the ratios of 17/00.100.2. We evaluate the performance of the solution policies returned by different methods on the test sets using two different OPE methods, Doubly Robust (DR) (Jiang et al., 2015) and Weighted Doubly Robust (WDR) (Thomas and Brunskill, 2016). We acknowledge that these methods are proxy of the actual performance of the deployed policies. Hence, these results should not be misinterpreted as us claiming that the policies returned by our methods are now ready to be used in the ICU,
Table 1: Performance of various methods using DR and WDR estimators with mean and standard deviation on 10 random splits of the cohort dataset. The red cells denote the corresponding safety constraint violation, i.e, either Th" < To or J > -TT.
User preference (A)
Policy
Survival return (Jn)
hare-treatment return (-J))
DR
WDR
DR
WDR
Clinician's (m))
64.78 = 0.90
64.78 + 0.90
13.58 + 0.19
13.58 + 0.19
Linearized
97.68 + 0.22
97.58 + 0.20
27.64 1.11
27.84 + 1.09
Adv-Linearized
91.62 + 0,46
92.68 + 0.23
15.18 = 0.59
13.56 + 0.42
[Ao 1, 0]
S-OPT
66.11 + 0.87
66.05 + 0.86
13.42 + 0.20
13.46 + 0.20
H-OPT
65.95 + 0.00
65.95 + 0.00
13.37 + 0.00
13.37 + 0.00
Linearized
87.17 + 0.48
89.11 1 0.37
2.41 1 0.47
1.52 0.41
Adv-Lincarized
86.77 = 0.49
88.58 + 0.25
2.53 - 0.50
1.57 + 0.43
Do0 1011->1 1]
S-OPT
86.77 + 0.49
88.58 + 0.25
2.53 +.50
1.57 + 0.43
H-OPT
86.37 = 0.00
88.03 + 0.00
2.58 + 0.00
1.43 0.00
Linearized
-89.39 + 0.43
-9.900 + 0.29
22.99 - 0.40
22.81 + 0.30
Adv-I Adv-Linearized
60.27 + 0.49
61.44 + 0.85
18.40 1.40-02. 0027
15.36 + 0.58
= [ =
S-OPT
67.73 + 0.82
67.22 1 0.88
13.24
13.55 1: 0.33
H-OPT
65.95 = 0.00
65.95 + 0.00
13.37 0.00
13.37 + 0.00
Linearized
58.27 + 2.18
60.52 = 2.07
0.04 0.03
0.02 0.01
Adv-Lincarized
76.05 = 0.65
76.85 + 0.72
0.07 0.05
0.04 0.03
(Ao = 00==1
S-OPT
76.07 = 0.65
76.87 + 0.73
0.07 0.05
0.04 1 0.03
H-OPT
76.54 + 0.00
77.55 + 0.00
0.09 + 0.00
0.05 0.00
We refer to the return associated with the mortality reward (ra) as survival return (To). and the negative return associated with rare state-action reward (r) ) as rare-treatment return (TTi. Higher survival return implies more successful discharges, and lower rare-treatment return implies more adherence to common practice treatment decisions. We present the results on survival and rare-treatment returns in Table 1. As expected, we observe both the Linearized and Ady-Linearize basclines violates constraints across different X, whereas S-OPT and H-OPT are able to respect the
Results:
AA caveat here is regarding the underlying assumption that the MDP construction methodology by Ko- morowski et al. (2018); Tang et al. (2020) maintains the Markovian property in the discretized state-space.

---

safety constraints irrespective of the X. 5 The validation set was used to tune the apper-paramtters and we report how the performance varies with different hyper-parameters in Appendix E.2.
Qualitative Analysis: We conclude with a qualitative analysis of the policies returned from our setting and the traditional RL approach of maximizing just the survival return. Ji et al. (2020) found that the RL-policies for sepsis-managemi treatments, particularly high vasopressor doses for states where the common practice (according to most frequent action chosen by the clinician for that state) is to give no vasopressors at all. The common practice involves giving zero vasopressors for 722 of the 750 states. However, the policy RL baseline recommends vasopressors in 562 (77. 84%)
task usually end up recommending aggressive
returned by the traditional of those 722 states, with 295 of those recommendations being large doses (upper 50th percentile of nonzero amounts or > 0.2 pg/kg/min). We compare these statistics for two of the policies returned by MO-SPIBB that deviate the most from Th. The policy returned by S-OPT (A (1, 1)) recommends vasopressors in only 93 of those states (12.88 %)., with 47 of those recommendations belonging to high dosages. The other policy, S-OPT (A = [0, 1). recommends vasopressors in 1 134 (18.56 %) of those states and 70 of those recommendations fall in large dosages. Therefore, the policies returned by our approach, even when they deviate from the baseline, are less aggressive in recommending rare treatments. In Appendix E.3, we present an additional qualitative analysis that demonstrates our methods recommend lesser rare-action treatments than the traditional oingle-objective RL approach.
An argument can be made against the case when all rare state-action pairs are removed from the training data itself. This will ensure that any learned policy will have near 0 rare-tree treetreetment return. However, it is not always clear how to define the cut-off criteria for rare-actions, and it might be possible that some of these rare state-action pairs are actually crucial for finding a better policy, For instance, we did an experiment where we assigned state-actions pairs with frequency < 100 to be rare state-action pairs and filtered those from the training set. The clinician's 's performance on the test set using a DR estimator for survival return is 65.95. In this case, the traditional single-objective RL baseline gives the survival return of 11 11226, which shows that removing such transitions from the dataset actually hampers the solution quality. Our approach of assigning a separate reward for rare state-action pairs is able to find a solution with a survival return of 86.75 even in this scenario.
6 Conclusion
We present new Seldonian RL algorithm that takes the user preference based scalarization into account while ensuring the solution policy performs reliably in context to the baseline policy across all objectives. On both synthetic and real-world tasks, we show that the proposed approach can improve the policy while ensuring the safety constraints are respected.
Our setting can accommodate any general form of scalarizations (e.g. non-linear convex) as well as objectives (such as fairness), making it applicable to a wide variety of real-world tasks. The only assumption we made is regarding the dataset being collected under single known baseline policy. An exciting line of future work can be to relax this assumption and consider the scenario where the dataset comes from variety of unknown policies with different qualities. We did not make any claims about the optimality of the solutions as often optimality and safety are contradicting objectives. It is not clear how (and if) one can make claims about optimality in the offline setting without bringing in additional unrealistic assumptions (Section 2). The extension to infinite MDPs and the
setting is also left for future work. It is important to note that when it comes to practical application, it is not unusual for continuous domains to be discretized to enable better interpretability, especially when interactions with humans are necessary. If the Markovian property is valid in the discretized space, SPIBB-based guarantees will also hold true.
7 Acknowledgements
The authors would like to thank NSERC (Natural Sciences and Engineering Research Council), IVADO (Institut de valorisation des données) and CIFAR (Canadian Institute for Advanced Research) for funding to McGill in support of this research. Philip S. Thomas was funded in part by NSF award
'In Table 1, À = [1, 1] represents a rare case of reward scalarization that allows all the methods to find a good solution policy that satisfies the constraints. In general, it is difficult to find such scalarization parameters seen in synthetic experiments (Appendix D.2).
10

---

#2018372. The computational component of this research was enabled in part by support provided by Calcul Québec (www.cam cal IDT team.
ca), Compute Canada (www. computecanada ca) and Mila's
We would also like to thank Emmanuel Bengio and Koustuv Sinha for many helpful discussions about the work, and the anonymous reviewers for providing constructive feedback.
References
Abdolmaleki, A., Huang. S., Hasenclever, L.., Neunert, M., Song, F., Zambelli, M., Martins, M., Heess, N., Hadsell, R., and Riedmiller, M. (2020). A distributional view multi-objective policy optimization. In International Conference on Machine Learning, pages 11-22. PMLR.
Altman, E. (1999). Constrained Markov decision processes, volume 7. CRC Press.
Amodei, D., Olah, C., Steinhardt. J., Christiano, P., Schulman, J., and Mané, D. (2016). Concrete problems in Al safety. arXiv preprint arXiv:1606.06565
Bellman, R. (1957). A Markovian decision process. Journal of mathematics and mechanics, pages 679-684.
Boyd, S., Boyd, S. P., and Vandenberghe, (2004). Convex optimization. Cambridge university press.
Chow, Y., Nachum, O., Duenez-Guzmanl E., and Ghavamzadeh, M. (2018). A Lyapunov-based approach to safe reinforcement learning. arXiv preprint a 1805.07708.
Christiano. P., Leike, J.. Brown, T. B., Martic, M., Legg, S., and Amodei, D. (2017). Deep reinforce- ment learning from human preferences, arXiv preprint arXiv:1706.0377 03741.
Diamond, and Boyd, S. (2016). CVXPY: Python-embeddee modeling language for convex optimization. Journal of Machine Learning Research, 17(83):1-5.
Dulac-Arnold, G., Mankowitz, D., and Hester, T. (2019). Challenges of real-world reinforcement learning. arXiv preprint arXiv:1904 12901.
Guo, Z. D., Thomas, P. S., and Brunskill, E. (2017). Using options and covariance testing for long horizon off-policy policy evaluation. arXiv preprint arXiv:1703.0345.
Hansen, N. (2006). The ema evolution strategy: comparing review, In Towards a New Evolutionary Computation, pages 75-102. Springer.
Iyengar, G. N. (2005). Robust dynamic programming. Mathematics of Operations Research, 30(2):257-280.
Ji, C. X., Oberst, M., Kanjilal, S., and Sontag, D. (2020). Trajectory inspection: A method for iterative clinician-driven design of reinforcement learning studies. arXiv preprint arXiv:2010.04273
Jiang, N., Kulesza, A., Singh, S., and Lewis, R. (2015). The dependence effective planning horizon on model accuracy. In Proceedings of the 2015 International Conference on Autonomous Agents and Multiagent Systems, pages 1181-1189.
Jiang, N. and Li, L. (2015). Doubly robust off-policy value evaluation for reinforcement learning. arXiv preprint arXiv:1511.0322
Johnson, A. E., Pollard, J., Shen, L., Li-Wei. H. L., Feng. M., Ghassemi, M., Moody, B., Szolovits, P., Celi, L. A., and Mark, R. G. (2016). MIMIC-III, freely accessible critical care database. Scientific Data, 3(1)1199.
Kakade, S. and Langford, J. (2002). Approximately optimal approximate reinforcement learning, In In Proc. 19th International Conference on Machine Learning. Citeseer.
Kivinen, and Warmuth, K. (1997). Exponentiated gradient versus gradient descent for linear predictors. information and computation, 132(1):1-63.

---

Komorowski, M., Celi, L. A., Badawi, O., Gordon, A. C., and Faisal, A. A. (2018). The artificial intelligence clinician learns optimal treatment strategies for sepsis in intensive care. Nature Medicine, 24(1 17:1116-7720.
Lange, S., Gabel, T., and Riedmiller, M. (2012). Batch reinforcement learning. In Reinforcement Learning, pages 45-73. Springer.
Laroche, R., Trichelair, P., and Tachet des Combes, R. (2019). Safe policy improvement with baseline bootstrapping. In Proceedings of the 36th International Conference on Machine Learning (ICML).
Le, H., Voloshin, C., and Yue, Y. (2019). Batch policy learning under constraints. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 3703-3712, Long Beach, California, USA. PMLR.
Leike, J., Krueger, D., Everitt, T., Martic, M., Maini, V., and Legg, S. (2018). Scalable agent alignment via reward modeling: A research direction. arXiv preprint arXiv:1811.0787
Leike, J., Martic, M., Krakovna, V., Ortega, A., Everitt, T., Lefrancq, A., Orseau, L., and Legg, (2017). AI safety gridworlds. arXiv preprint arXiv:1711.09883
Littman, M. L., Topeu, U., Fu, J., Isbell, C., Wen, M., and MacGlashan, J. (2017). Environment- independent task specifications via gltl. arXiv preprint arXiv: arXiv:1704.0453 1704.04341.
Liu, C., Xu, X., and Hu, D. (2014). Multiobjective reinforcement learning: comprehensive overview. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 45(3):385-39)
Lizotte, J. J., Bowling, M., and Murphy, A. (2012). Linear fitted-q iteration with multiple reward functions. The Journal of Machine Learning Research, (3(1):3253-329)
Mannor, S. and Shimkin, N. (2004). geometric approach multi-criterion reinforcement learning. The Journal of Machine Learming Research, 5:325-360.
Maurer, A. and Pontil, M. (2009). Empirical Bernstein bounds and sample variance penalization. arXiv preprint arXiv:0907.3740..
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015), Human-level control through deep reinforcement learning. Nature, 518(7540):529-533
Munos, R. (2003). Error bounds for approximate policy iteration. In ICML, volume 3, pages 560-567.
Nadjahi, K., Laroche, R., and Combes, R. T. d. (2019). Safe policy improvement with soft baseline bootstrapping. arXiv preprint veXiv:/907.055
Ng, A. Y., Harada, D., and Russell, S. (1999). Policy invariance under reward transformations: Theory and application to reward shaping. In Proceedings of the Sixteenth International Conference on Machine Learning, volume 99, pages 278-287.
Nilim, A. and El Ghaoui, L. (2005). Robust control of Markov decision processes with uncertain transition matrices. Operations Research, 53(5):780-798
OpenAl (2018). Openai five. htttp::/blog..peett
com/openai-fivel
Petrik, M., Chow, Y., and Ghavamzadeh, M. (2016). Safe policy improvement by minimizing robust baseline regret. arXiv preprint arXin:1607.03842.
Precup, (2000). Eligibility traces for off-policy policy evaluation. Computer Science Department Faculty Publication Series, page 80.
Rhodes, A., Evans, L. E., Alhazzani, W., Levy, M. M., Antonelli, M., Ferrer, R., Kumar, A., Sevransky, J. E., Sprung, C. L.,, Nunnally, M. E., et al. (2017). Surviving sepsis campaign: International guidelines for management of sepsis and septic shock: 2016. Intensive Care Medicine, 43(3):304- 377.
12

---

Roijers, D. M., Vamplew, P., Whiteson, S., and Dazeley, R. (2013). A survey of multi- sequential decision makking. Journal of Artificial Intelligence Research, 48:67-113.
Silver, D., Schrittwieser, Simonyan, K., Antonoglou. I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., et al. (2017). Mastering the game of go without human knowledge. Nature, 550(7676):354-33)
Simão, T. D., Laroche, R., and Tachet des Combes, R. (2020). Safe policy improvement with an estimated baseline policy. In Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems, pages 166911277.
Singer, M., Deutschman, S., Seymour. C. W., Shankar-Han Harii M., Annane, D., Bauer, M., Bellomo, R., Bernard, G. R., Chiche, J.-D., Coopersmith, C. M., et al. (2016). The third international consensus definitions for sepsis and septic shock (Sepsis-3). Jama, 315(8):001-80.
Sorg, J., Singh. S. P., and Lewis, L. (2010). Internal rewards mitigate agent boundedness. In Proceedings of the 27th International Conference on Machine Learning (ICML-10), pages 1007- 1014.
Sutton, R. S. (1988). Learning predict by the methods of temporal differences. Machine learning, 3(1):9444.
Tang, S., Modi, A., Sjoding, M., and Wiens, J. (2020). Clinician-in-the-loog decision making: Reinforcement learning with near-optimal set-valued policies. In International Conference on Machine Learning, pages 9387-9396. PMLR.
Thomas, P. and Brunskill, E. (2016). Data-efficient off-policy policy evaluation for reinforcement learning. In International Conference Machine Learning, pages 2139-2148.
Thomas, P., Theocharous, G., and Ghavamzadeh, M. (2015a). High confidence policy improvement. In International Conference on Machine Learning, pages 2380-2383
Thomas, P. S., Castro da Silva, B., Barto, A. G., Giguere, S., Brun, Y., and Brunskill, E. (2019).
Preventing undesirable behavior of intelligent machines. Science, 366(6466))999910042
Thomas, P. S., Theocharous, G., and Ghavamzadeh. M. (2015b). High- off- off-policy evalua- tion. In Twenty-Ninth AAAI Conference on Artificial Intelligence.
Vamplew, P., Dazeley, R., Berry, A., Issabekov, R., and Dekker, E. (2011). Empirical evaluation methods for multiobjective reinforcement learning algorithms. Machine learning, 84(1):51-80
Van Moffaert, K. and Nowé, A. (2014). Multi-objective reinforcement learning using sets of pareto dominating policies. The Journal of Machine Learning Research, 15(1)3383-3512.
Walpole, R. E., Myers, R. H., Myers, S. L., and Ye, K. (1993). Probability and statistics for engineers and scientists, volume 5. Macmillan New York.
Yang, R., Sun, X., and Narasimhan, K. (2019). A generalized algorithm for multi-objective reinforce- ment learning and policy adaptation. arXiv preprint arXiv: 1a0..00088..55
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] See Sections 3.3 and 3.4 for the methodology and theoretical claims, and Sections 4 and 5 for the empirical results.
(b) Did you describe the limitations of your work? (Yes) See Section for the limitations and scope of this work.
13

---

(c) Did you discuss any potential negative societal impacts of your work? [Yes] As we mentioned Sections and 3.2, our goal to maximize the objective specified by the user while ensuring that the solution policy avoids causing harmful effects after deployment the true environment in comparison to the existing baseline policy. We aim to bridge the gap between traditional RL methods and high-stake real-world applications, but, as with any general technology, we acknowledge that some RL applications can have the potential of misuse, and our methods do not prevent that.
(d) Have you read the ethies review guidelines and ensured that your paper conforms them? [Yes] We discuss the societal impacts in the point above. We use pre-existing publicly available data and libraries and give more details about them in the Point below.
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Section 3.1 for the assumption regarding access to baseline policy.
(b) Did you include complete proofs of all theoretical results? [Yes] The complete proofs are provided in the Appendix B and Appendix C.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed reproduce the main ex- perimental results (either in the supplemental material or as a URL)? [Yes] The code required produce the results provided in the supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] The training details are mentioned in Sections 4 and 5 in the main text, and Appendices and E.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] The details about the error bars are provided in Sections 4 and 5.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Details about the compute and resources can be found in appendices D and E.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See Sections and for the appropriate references.
(b) Did you mention the license of the assets? [Yes] We the (MIMICIIII, Johnson et al., 2016) dataset and provide the appropriate reference. The explicit link to the license is here: https: / http//hpysiieneee org/ wwtpeeettiiiiiiiiiieeiieieeitttt htt//vieewlcceeseeee www.enecceeeeeeeeeeeeee .4/.
(c) Did you include any new assets either the supplemental material or as URL? [Yes] We provide the accompanying code for running the experiments in the supplemental material.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A] We are using publicly available libraries and dataset.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A] We are using the publicly available dataset (MIMIC-III, Johnson et al., 2016) that already deidentifies the data in accordance with Health Insurance Portability and Accountability Act (HIPAA) standards using structured data cleansing and date shifting. We refer to the original text for more details on the deidentification process.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "wdXOonZO5TD", "reviewer": "Reviewer_RGWk", "summary": "This paper presents the Safe Policy Iteration with Baseline Bootstrapping (SPIBB) in the offline RL setting with multiple objectives, i.e. multiple rewards. The authors apply the SPIBB set-up to handle the trade-offs for different rewards standing for different user preferences using the multi-objective framework while ensuring that the new policy has the performance guarantee that is at least as well as the behavior policy for each reward function. The authors provide theoretical guarantees of the approach that shows that safe policy improvement is guaranteed for each objective. Finally, the method is evaluated on a grid-world safety task as well as a real-world health care experiment, where the proposed method outperforms previous approaches.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The method is theoretically grounded. The paper is also well written and easy to understand. The experiment results in finite MDPs suggest that the method is both effective in synthetic domains as well as real-world clinical care settings. I think the approach is of great significance to the field of offline RL.\n\nHowever, I do have a few concerns, which I will discuss as follows.\n\nFirst, I think the novelty of the paper is somewhat limited. The method seems to be a direct application of SPIBB and HCPI to the multi-objective setting without much modification. The theoretical results seem also to be straightforward extensions to results shown in SPIBB and HCPI, which makes the contribution rather incremental.\n\nMoreover, regarding the experiments, while the authors admit that the results are focused on finite MDPs, I think it is reasonable to apply it to infinite MDPs in empirical evaluations. I would be curious to see how the proposed method can perform in standard offline RL benchmarks such as D4RL where a few domains exhibit the multi-objective structure e.g. antmaze and kitchen. Moreover, I think it would be good to compare the method to previous offline RL methods [1,2,3,4,5,6] combined with vanilla multi-task RL such as shared networks or multi-headed networks. I think vanilla offline MTRL should in principle tackle the proposed problem setting and including such a comparison would be important.\n\nBased on the above comments, I would currently vote for a weak reject.\n\n=================\nPost-rebuttal update: After reading the response, I think my concerns are clarified and hence increase my score to a 6.\n\n[1] Kumar,  A.,  Zhou,  A.,  Tucker,  G.,  and Levine,  S.   Conservative q-learning for offline reinforcement learning.Conference on Neural Information Processing Systems,2020.\n[2] Ofir Nachum, Bo Dai, Ilya Kostrikov, Yinlam Chow, Lihong Li, and Dale Schuurmans.  Al-gaedice: Policy gradient from arbitrary experience.arXiv preprint arXiv:1912.02074, 2019.\n[3] Advantage-weighted regression:Simple and scalable off-policy reinforcement learning.arXiv preprint arXiv:1910.00177, 2019.\n[4] Scott Fujimoto, David Meger, and Doina Precup.   Off-policy deep reinforcement learningwithout exploration.arXiv preprint arXiv:1812.02900, 2018.\n[5] Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policyq-learning via bootstrapping error reduction.  InAdvances in Neural Information ProcessingSystems, pages 11761–11771, 2019.\n[6] Yao Liu, Adith Swaminathan, Alekh Agarwal, and Emma Brunskill. Off-policy policy gradientwith state distribution correction.CoRR, abs/1904.08473, 2019.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours"}, "extra_text": {}}, {"review_id": "tq1kUgHHaQs", "reviewer": "Reviewer_q8Um", "summary": "The paper describes the multi-objective extension of the SPIBB (Laroche et al, 2019) and Soft-SPIBB (Nadjahi et al., 2019) methods for policy improvement with respect to a number of constraints. It is performed by incorporative multi-objective state-action value functions into the original Soft-SPIBB method.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "*Pros: *\n\nInteresting and important problem to address (see below)\n\n*Cons:*\n\n Insufficient experimental analysis, novelty is not clear as the narrative describes SPIBB extension for multi objective training by incorporating linear combination of state-action value functions; the clarity and reproducibility could be improved .\n\nWhile the reviewer thinks that multi-objective reinforcement learning with safety constraints is a very important problem, the reviewer also wonders whether evaluating improvements of the policy over one user-chosen preference (as opposed to building a Pareto front) is sufficient and is enough for solving a multiobjective problem. If we only find some of the solutions for some pre-defined constraint, it may not explain the trade-offs between multiple objectives, which, as the reviewer understands, is a motivation behind multi-objective optimisation. An introduction in a popular book chapter in multiobjective optimisation [1] explains this motivation as follows:\n\n*'Many real-world search and optimization problems are naturally posed as non-linear programming problems having multiple objectives. Due to the lack of suitable solution techniques, such problems were artificially converted into a single-objective problem and solved. The difficulty arose because such problems give rise to a set of trade-off optimal solutions (known as Pareto-optimal solutions), instead of a single optimum solution. It then becomes important to find not just one Pareto-optimal solution, but as many of them as possible. This is because any two such solutions constitutes a trade-off among the objectives and users would be in a better position to make a choice when many such trade-off solutions are unveiled.'*\n\n*Reproducibility*\n\nIt is not clear how exactly the baseline policy was generated: what were the hyperparameters of the method described in D1, what optimisation method was used and how was the policy function defined (tabular?)? Line 198 in the description of the proposed method states: \"The solution of S-OPT is computed by solving the Linear Program using standard solvers, such as cvxpy (Diamond and Boyd, 2016).\" Was the same method used for generating the policy, what exact solvers  have been used, and what were the hyperparameters of the solvers? It would be good to state it in the appendix and reference it from the experimental section. \n\n\n*Other comments:*\n\n- The reviewer thinks that it would be good to structure the paper in a way that outlines novel contribution to answer the following: apart from changing the optimisation to multiobjective, what would differentiate this work from Soft-SPIBB? The reviewer finds it hard to figure it out from 3.3 and 3.4.  One approach could be to define it as an algorithm and outline the new parts of it. \n\n- Line 216: “The dataset D is first split into train (Dtr ) and test (Ds) sets by the user.” How exactly does it happen and why is it not possible to use randomisation instead?\n\n- In Eq. 1 we have the constraint expressed via parameters $\\delta$ and $\\zeta$; from line 172 onwards, we see a different set of constraints without parameter $\\zeta$. Could the authors explain the transition between eq. 1 and the problem (S-OPT)?\n\n==\nThe score updated to 6 taking into account the rebuttal (see follow up discussion). l the confidence decreased from 4 to 3 due to large number of changes discussed during the rebuttal.\n\n[1] Deb K. (2005) Multi-Objective Optimization. In: Burke E.K., Kendall G. (eds) Search Methodologies. Springer, Boston, MA. https://doi.org/10.1007/0-387-28356-0_10", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors correctly addressed limitations such as only considering finite Markov Decision Processes (MDPs); the authors addressed in their comment potential negative impact of their work such as misuse of reinforcement learning. ", "review_text": "", "extra_scores": {"ethical_concerns": "All concerns acknowledged and addressed by the authors", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "rzjUVvZ_9CM", "reviewer": "Reviewer_VHxZ", "summary": "The paper proposes a new Safe Policy Improvement problem formulation to deal with multi-objective offline settings. \n\nAuthors consider a problem where a batch of data, collected by following a known baseline policy, is given and the goal is to return a new policy that, achieves (with high confidence) better results than the baseline, considering also user’s preferences. \n\nTaking inspiration from the Seldonian approach [Thomas et al., 2019] and offline SPI algorithms (SPIBB [Laroche et al., 2019], Soft-SPIBB [Nadjahi et al., 2019], HCPI [Thomas et al., 2015a,b]), the authors then show that this can be formulated as particular optimization problems which can be solved, by using common techniques with a computational penalty that scales with the number of objectives, to obtain a policy with the required safety guarantees. \n\nFinally, they compare their methods (MO-SPIBB and MO-HCPI) against two baselines on two experimental settings, a synthetic problem and real-world scenario, showing that their approaches are, when optimized, very competitive but also safe, obtaining results that are consistently better than the baseline up to the accuracy desired, especially in a small sample regime.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n\nThe paper is well written: it is clear what the authors want to achieve and the context of the work. \n\nThe paper addresses a relevant problem, safe algorithms in RL, in a setting which has not been extensively considered before (multi-objective).\n\nThe authors also present a well-rounded experimental section: a synthetic environment and an experiment on real-world data. The latter is especially interesting because it gives an idea on how the methods can be actually be applied to real-life problems.\n\nWeaknesses\n\nThe work is extensively based on existing approaches and its contribution (in term of novelty ) is not major: the proposed methods amount to solve a new optimization problem and lead to solutions that can only be applied to fully-observable, discrete multi-objective environments, as noted by the authors; considering the paper’s focus on real-world applications, this is a limiting factor.\n\nAlso, the experimental settings are on the small side and thus do not give a clear idea on how the methods would behave on more realistic models, especially in terms of their performance.\n\nClarity\nThe paper is well written. In particular, it is clear what problem the authors are addressing, the properties that they want their method to have, what they want to achieve and the paper’s contributions.\n\nThe experimental methodology and results are also well explained.\n\nHowever, there are some small notation issues or confusing details that I think should be considered by the authors: \n\n1) Line 115: the multi-reward signal is defined to be stochastic but is given as a vector-valued function from state-action pairs to a d-cube, i.e. mapping pairs to d scalars. Shouldn’t be to a distribution over scalars?\n\n2) Same line: $r_{\\top}$ is not defined, although not hard to infer\n\n3) Equation (S-OPT), after line 186: as in 2), I would specify that the brackets $\\langle \\rangle$ are to be read as an inner product. Moreover, it would be better to separate the two operands by a comma or a vertical line, or use the dot notation instead\n\n4) Same equation: it is not immediately clear from the main paper why the third line of this equation is something necessary to guarantee the condition of belonging to $\\Pi_A$. A brief explanation would significantly help here.\n\n5) Section 4, Dataset collection procedure: do you collect only one dataset for dataset size for each one of the 100 random CMDPs and twelve $\\rho$?\n\n6) Why do some lambdas parameters sum to 2 or 0 in Section 5, Table 1? In line 141 lambda is defined over the d-simplex. Also, for the case $\\lambda_0 = \\lambda_1 = 0$, does this mean that the [action-]value function is zero for every state[-action pair] (line 174)?\n\nOriginality\n\nThe methods extend known SPI algorithms to deal with a new problem and, although not completely original, they differ from existing techniques in the sense that they amount to solve new optimization problems, with an added constraint; also, it is clear from the paper how these new methods differ from previous work, and how it is situated with respect to related contributions.\n\nSignificance\n\nThis paper presents an extension of common SPI approaches that can handle multi-objective environments, so its contribution could be definitely very useful to researchers, perhaps as a stepping stone for other methods with slightly different assumptions or approximation techniques.\nHowever, the fact that this paper doesn’t consider relevant real-life modeling problems, such as partial observability or function approximation, somewhat limits its significance, especially for practitioners.\n\nQuality\n\nThe claims are supported, even though the mathematical proofs are completely relegated to the supplementary material; the experimental methodology is also technically sound, and extensively explained. The authors are also clear about the limitations of their contribution, its strengths and weaknesses, although I would have liked some comments about the computational limits of these approaches, and how they would fare in very large experiments.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations and societal impacts are clearly stated by the authors, in the main paper and in the checklist.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10"}, "extra_text": {}}, {"review_id": "cSGHd2uR9RS", "reviewer": "Reviewer_VZ8L", "summary": "The authors study the safe policy improvement problem in the offline RL settings where the known baseline policy is available. In this problem settings, the dataset has been already collected by the baseline policy and contains multiple reward signals from the environment. Under this problem setting, the authors aim to achieve a policy that balances the different rewards while guaranteeing that the new policy is better than the baseline policy. The authors proposed a new algorithm based on SPIBB and show its effectiveness in a synthetic environment and the one with sepsis management.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think this paper is well-written. The problem considered in this paper is reasonable and important for the research community. Though the assumption on the baseline policy (i.e., Assumption 3.1) is rather strong, there are indeed scenarios where we can assume the availability of the known baseline policy. \n\nPros\n- Well-written texts. Though I think there is space for improvement as noted in Minor Comments, this paper is mostly easy to understand. Motivations are clear and algorithms are well-presented.\n- The numerical experiments are well-designed and the results are promising (though I have several criticisms as noted in Cons).\n- Reproducibility. I read through the source code (did not run, though), and I found it easy to read. This source would be helpful for future research.\n\nCons\n- Incremental theoretical analysis. Theoretical results are simple extensions of the previous work. I don't think there are strong contributions in the theory parts.\n- I do feel that the authors should have (also) tested their algorithm in the standard benchmark. (e.g., MuJoCo tasks) as in Urpi et al. (2020).\n    - Urpí, Núria Armengol, Sebastian Curi, and Andreas Krause. \"Risk-Averse Offline Reinforcement Learning.\" ICLR. 2020.\n- Lack of comparison with previous work. The authors compared their algorithms with very simple baselines (i.e., minor modification with theirs). For example, the authors should have compared their method with such methods as in Le et al. (2019).\n    - Le, H., Voloshin, C., and Yue, Y. (2019). Batch policy learning under constraints. In ICML.\n\n### Minor Comments:\n- I think this paper would be easier to read if the authors use the same characters as the one typically used in the RL paper. The original SPIBB paper (https://arxiv.org/pdf/1712.06924.pdf) is much easier to read, at least for me. For example, I would like the authors to change as follows:\n    - $q \\rightarrow Q$\n    - $v \\rightarrow V$\n    - $r_{\\top} \\rightarrow R_{\\max}$\n\n- The authors may consider including Reward-free RL papers in the related work. As a notable one, \n    - Jin, Chi, et al. \"Reward-free exploration for reinforcement learning.\" In ICML. 2020.\n\n### Questions\n- The descriptions in lines 74 - 81. Could you elaborate more? Why can you say that the assumptions in this paper are milder than those in the previous work? And, could you provide the reason why the authors did not have to make the concentrability assumption in this paper?\n\n\n---\nScore is updated according to the new experimental result.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "limitations and societal impact have been fully addressed. ", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposed a SPIBB algorithm for offline constraint RL. This is an interesting problem to address. Most of the reviewers think the theoretical analysis is rather weak in the paper, but the experiments give good complementation to the theory. Hence the overall recommendation is a weak acceptance. 
Decision: Accept (Poster)