An Examination of Preference-based Reinforcement Learning for Treatment Recommendation
Authors: Nan Xu
Nitin Kamra
Yan Liu
Venue:  (2021)
Keywords: Preference-based Reinforcement Learning
Treatment Recommendation
healthcare

Abstract: Treatment recommendation is a complex multi-faceted problem with many conflicting objectives, e.g., optimizing the survival rate (or expected lifetime), mitigating negative impacts, reducing financial expenses and time costs, avoiding over-treatment, etc. While this complicates the hand-engineering of a reward function for learning treatment policies, fortunately, qualitative feedback from human experts is readily available and can be easily exploited.  Since direct estimation of rewards via inverse reinforcement learning is a challenging task and requires the existence of an optimal human policy, the field of treatment recommendation has recently witnessed the development of the preference-based ReinforcementLearning (PRL) framework, which infers a reward function from only qualitative and imperfect human feedback to ensure that a human expert’s preferred policy has a higher expected return over a less preferred policy. In this paper, we first present an open simulation platform to model the progression of two diseases, namely Cancer and Sepsis, and the reactions of the affected individuals to the received treatment. Secondly, we investigate important problems in adopting preference-basedRL approaches for treatment recommendation, such as advantages of learning from preference over hand-engineered reward, addressing incomparable policies, reward interpretability, and agent design via simulated experiments. The designed simulation platform and insights obtained for preference-based RL approaches are beneficial for achieving the right trade-off between various human objectives during treatment recommendation.

Under review as a conference paper at ICLR 2021
AN EXAMINATION OF PREFERENCE-BASED REIN- FORCEMENT LEARNING FOR TREATMENT RECOM- MENDATION
Anonymous authors Paper under double-blind review
ABSTRACT
Treatment recommendation is a complex multi-faceter problem with many con- flicting objectives. c.g., optimizing the survival rate (or expected lifetime). mit- igating negative impacts, reducing financial expenses and time costs, avoiding over-treatment, etc. While this complicates the aand-engineering of reward func- tion for learning treatment policies, fortunately, qualitative feedback from human experts readily available and can be easily exploited. Since direct estimation of rewards via inverse reinforcement learning is challenging task and requires the existence of an optimal human policy, the field of treatment recommendation has recently witnessed the development of the Learning (PRL) framework, which infers a reward function from only qualitative and imperfect human feedback to ensure that human expert's preferred policy has a higher expected return over a less preferred policy. In this paper, we first present an open simulation platform to model the progression of two discases, namely Cancer and Sepsis, and the reactions of the affected individuals to the received treat- ment. Secondly, we investigate important problems adopting preference- based RL approaches for treatment recommendation, such as advantages of learning from preference over hand-engineered reward, addressing incomparable policies, reward interpretability, and agent design via simulated experiments. The designed simulation platform and insights obtained for preference- RL approaches are beneficial for achieving the right trade-off between various human objectives during treatment recommendation.
fference-based Reinforcement
INTRODUCTION
With recent advances in deep learning and open access to large-scale Electronic Health Records (EHRs), Deep Reinforcement Learning (RL) approaches have gained popularity for treatment recom- mendation (Raghu et al. 2017: Martzn et al., 2019)). But the success of RL applications often crucially depends the prior knowledge that goes into the definition of the reward function (Wirth) jet al. 2017). However, treatment recommendation is a multi-faceted problem where the reward function is hard engineer and requires quantifying the trade-off between diverse types of realistic objectives. For instance, clinicians often aim to optimize the survival rate (or expected lifetime) while mitigating negative impacts of the treatment (Raghu et al. 2017 Lopez-Martinez al., 2019 Wang et al. 2018). However, they also keep in mind, the patient's ' considerations of financial expenses and time costs in accepting treatment strategies (Faissol al. 2007 Denton et al. 2009). Moreover, unnecessary or over-treatment needs to be avoided and certain agreements based on the patient's medical insurance plan also need to followed for an affordable treatment (Nemati et al. 2016).
To explicitly reflect human's objectives in the reward function, prior work jointly considers multiple objectives weighted linearly to reduce the problem to a single-objective MDP (Faissol et al. 2007] Denton et al. 2009). However, the linearly weighted reward function induces negative interference between objectives, especially when representations are learned using neural networks and shared among different objectives, which goes against human's 's actual intentions (Pham et al.) 20188 Schaul jet al., 2019). Further, given clinicians' treatment strategies, the intrinsic reward function cannot be inferred accurately with existing inverse reinforcement learning (IRL.) methods (Abbeel & Ng

---

Under review as a conference paper at ICLR 2021
2004; Ho & Ermon 2016), since they require access to samples from optimal policy, which is not guaranteed in reality (Komorowski et al. 2018 Saria 2018).
Fortunatcly, qualitative feedback according to humans' preferences be easily obtained and efficiently leveraged to infer reward functions. In this paper, we investigate preference- based Re- inforcement Learning approaches (Fürnkranz et al. 2012 Cheng et al.) 2011 Akrour al.) 2012 Schäfer & Hüllermeier 2018 Christiano et al.) (017) for treatment recommendation, where the reward is estimated based on preferences over a pair of treatment strategies. Specifically, the reward estimator ensures that in a policy pair, the policy preferred according to a human's 's objectives has higher expected return. However, acceptance of PRL approaches for treatment recommendation requires significant exploration of their practical utility, reliability and interpretability.
Contributions: In this work, we first present an open simulation platform to investigate the preference-basee Reinforcement Learning approaches from the above aspects. The constructed simulator models the dynamic state transitions of different individuals with Cancer or Sepsis and their reactions to the received medication operation treatment, which enables efficient model training and reliable performance evaluation. Next, we conduct comprehensive simulated experiments to address the following questions: 1) Does the preference-based qualitative feedback really benefit the policy learning compared to handerafted rewards and other existing treatment recommendation approaches? 2) How to better optimize human's 's objectives by learning reward representation which can deal with policies which are incomparable? 3) Is the reward function inferred by PRI. interpretable and does it faithfully follow human intentions? and How to design agent types so that resulting policies together with the preference feedback lead to more accurate reward estimation and better treatment outcomes? Our experiments provide useful insights and guidance in developing preference- -assd RL approaches to realize the right trade-off between human objectives during treatment recommendation.
PROBLEM DEFINITION
We cast the treatment policy learning as a Markov Decision Process (MDP). At time-step I, is vector composed of multiple health-related features, a is either a scalar value representing dosage amount a boolean value denoting whether to perform conducted actions, features the state influence each other's progression, which simulated by the state transition probability function Ps* |st, ax). The agent targeted learning the optimal policy x' that maximizes the expected return V-- (8o) = max, E 7're). where E [0. is the discount factor and ri is the estimated reward based on preference feedback. Given two policies Tm and T. starting with the same initial state nm (s:) > (x:))) represents the preference of policy T. n. based on human's 's objectives. Rather than using hand-crafted reward functions of the MDP, we are aimed at finding parameterized reward function Гe,, that approximates the true reward function r underlying human's preference.
operation. Besides effects from the
SIMULATION PLATFORM DESIGN
3.1 GENERAL CANCER AND DRUG TREATMENT SIMULATION
Following prior work FFürkkranz et al. 2012), we use the mathematical model proposed by Zhao et al. (009) to simulate the general cancer evolution and drug treatment effects.
State Transformation: The values of the next tumor size 1/+1 and the toxicity level Fz+1 are determined by the current drug amount d, their current values Vt, and initial values yo, To:
1+++ ReLU y: [a: max(x,, - b1 (de m1)] 1 0))
.t+1 ReLU (z; a2 max(y;, yo) + b2 (de m2)),
where / is the indicator function which outputs 1 if the current tumor size yı > 0 and 0 otherwise.
3.2 SEPSIS INFECTION AND BLOOD PURIFICATION SIMULATION
We employ the mathematical model derived by Song et al. (0012) to simulate the acute inflammation process in response an infection. There 19 physiological features that govern sepsis dynamics,
2

---

Under review as a conference paper at ICLR 2021
8 of which observable while the remaining 11 are unmeasurable conceptual variables. Whenever blood purification operation is made, three components in the circulation are eliminated, i... activated neutrophils N, and the pro- and anti- mnliminflammatrry mediators PI and AI. Besides effects from the blood purification operation, the variables influence each others' progression through Ordinary differential equations (ODEs).
State Transformation:
There are 18 ODEs to describe feature interactions and 3 ODEs for hypo- thetic mechanism of blood purification. For simple demonstration, we only list the equations of activated neutrophils (N,) with(out) blood purification operation here.
dNa di
N₂P 1
N1
1
No
No operation:
+
NN, + PI" TN.-N.
1 NN,-N. + P/ TN,-Na
TN.
NaPI"
NN.-N.
PI" TN.-N. Na/Noo
dNaHA
dNa
With operation:
HAIBA (Na/Na0)
where N,, Np, Na resting, primed and activated blood neutrophils respectively, P/ is the systemic pro-inftammatory response, TN,-Na, NN,-Na: TN.-N, are constant parameters and - N., NN,-N. -N,, hASHA are hill equations.
METHOD
4.1 EXISTING TREATMENT RECOMMENDATION APPROACHES
When the optimization objective is to maximize the clinical efficacy alone, the reward function for intermediate timesteps is either 0 or hand-crafted based on indicators of patient health, while the rewards for positive and negative outcomes terminal timesteps are normally the same scale, but of opposite directions (Raghu et al., 2017 Wang et al.) 2018 Nemati et al. 2016). Besides obtaining the optimal clinical efficacy, another line of work has also included auxiliary objectives like mitigating negative impacts or improving health conditions. Most of existing work specified linear scalarization functions based domain knowledge project the multi-objective MDP to a single-objective MDP (Denton et al., 2009: Lopez-Martinez et al. 2019 Zhao et al. 2009p. Due to the sensitivity of the learned policy and resulted performance relative values of the manually specified rewards, the employed reward functions in these approaches are difficult be quantified by experts achieved distinct goals in treatment recommendation.
Learning from Hand-crafted Reward:
Learning from Human Feedback: Given demonstrations from domain experts, inverse Reinforce- ment Learning (IRL) methods have been proposed to seek the reward function that models the intention of the demonstrator first and then train RL agents to match the demonstrations (Abbeel &
Ng 2004 Ho & Ermon 2016). Though explicit quantitative reward signals are no longer needed in IRL settings, learning treatment policies from clinicians' demonstrations is challenging, since optimal demonstrations are difficult to provide by clinicians while the general treatment regimens in demonstrations hardly reflect the actual intentions (Gao et al., 2018; Brown et al. 2019).
Fortumately, even non non-experts can provide feedback in the form of preference, which has been utilized replace conventional numerical reward signals with relative utility values (Fürnkranz al. 2012
Cheng et al. 2011 Akrour et al. 2012 Schäfer & Hüllermeier 2018 Christiano et al. 2017
4.2
PREFENCE-BASED REINFORCEMENT LEARNING FRAMEWORK
We display the framework in Algorithm T to show the reward and policy learning procedure given preference feedback. Firstly, we are aimed learning reward function, based which the preference between two policies could be approximated.
Learning from Qualitative Feedbacks: We denote by (m:)) > (s:) the case that given 8;, Tm is preferred to nT. We here treat the qualitative feedback learning problem as a classic binary classification task, where two policies are given and model learns approximate the preference between the two. We assume the probability that one policy is preferred to the other is a function of their received reward estimations (explained later in Preference Probability Representation). then the
3

---

Under review as a conference paper at ICLR 2021
Algorithm 1 PREFERENCE-BASEI RL FOR TREATMENT RECOMMENDATION
Require:
S": initial states sampled subjects
N: number of training iterations
the maximum simulation time to treat each subject
1: Randomly initialize 0p, 0) 84
2: D 0,
0,1 0 Initialize empty lists to store samples for reward and agent learning 3: for 0 to N 1 do
4:
for all E S" do
5:
$ < 8, s 4- 8, - 0² 0 Initialize state vector and trajectory list
6:
for / = to
do
7:
oj - x(s): 6),, s - SIMULATE (s) r
LEWARD(s),
8:
af t xss;: 844 SIMULATE (t. az) r + REWARD(S?, ap;0p)
9:
+ - ((8) at, rbp.)) (1.1)4111 72 < 7 ((87, r2, 8 x1+1)
10:
end for
r p U (T ( pre(T) 72) + --BLLLATERREEREENCCEE ,²2) Preference feedback from humans + ( pppe((T), ,7))
12:
13:
end for
15:
Drawing minibatches r ~ r1, T2 ~ 12, D Update reward network 0p with Da by PREFERENCE-BASEE REWARD LEARNING// See Algorithm@(AApent
17:
Update agent network 02 with I 0% with 1 by PREFERENCE-GUIDED AGENT LEARN- ING// See Algorithm (Appendix))
18:
end for
cross-entropy loss function is:
L(Op) EEss (m Tn(s)) log logp(Tm(si) > (s.):::pp)
+ [TT (s:) (s))))) log p (T (8:) > m. (s:):))p))
where [(. > .) is an indicator function equal to if the first policy is preferred to the second, otherwise.
Preference Probability Representation: Bradley-Terry model (Bradley & Terry 1952) is widely
used probability model predict the preference of a paired comparison: p( j)
pp where ps
a positive real-valued score assigned individual order compute the probability that Tm preferred to n, given state we employ its implementation introduced in (Agresti & Kateri 2011
exp R(mm, 8;; 0p)
(s)) X nn (s:)) exp R(Tm s; 0p) + exp R (T, Sp;0pp
where capital R denotes the expected return of conducting a policy given one specific initial state.
Given the learned reward, parameters of the RL agent are updated with either of the following two methods.
Action-based Reward Modification (AbRM):: Hand-crafted rewards assigned to different outcomes influence the agent performance a lot, even if we keep the ratio but change the magnitude only. Instead of designing the scalar rewards for different outcomes manually, we send the preference-based reward Tep (St, ax) to the agent at each time-step.
State-based Reward Modification (ShRM): We derive a new state value hep from rop to represent how good the state is: hep (st) max, Tép (st, a). We further compute the advantage value of the current state over the previous one, hep (St) hap (st-1), as the instant reward to encourage appropriate behaviors in accordance with preference.
Before applying preference- based RL approaches to treatment recom- to ensure that the weference-baseed reward estimation is consistent with human's objectives and the well-trained agent provides reliable and interpretable policies clinicians:
Problems to Resolved:
mendation, we need to address the following problems
4

---

Under review as a conference paper at ICLR 2021
Table Performance for Cancer medication recommendations. The best result per metric is marked in boldface. We present avg + stdev values for all experiments averaged over 10 independent runs.
Clinical Efficacy
Other Factors
Method Type
Method Name
Survival Rate
Tumor Toxicity
Constant Best (0.4)
19.91 % +..88%
2.22
Constant Worst (0.1)
4.89% F0.68%
3.72
Non-learning
Random
17.81%10.911
2.2310.04
Preference Learning
PBPI
21.79%:t0.66%%
2.21+0007
Single-objective RL
26.96%:+3.02%
1.16+00.48
Reinforcement
RL (Ensemble)
27.38%:t3.32%
1.14+0.49
Learning (handerafted reward)
Existing Multi-objective RL
18.84%:15.77%%
2.266
Grid-search Multi-objective RL
28.98%
0.66:10.45
AbRM (CE)
31.52% 11.38%
0.46+10.06
Reinforcement
AbRM (CE & OF-I)
31.33% 1..8%%
0.39
Learning
SbRM (CE)
30.54% 4.%
0.68
(Prefcrence-basee reward)
SbRM (CE&OF-I)
31.72% 1100%
0.43
Does the prefreenc-baasee qualitative feedback really benefit the policy learning compared to handcrafted rewards and other existing treatment recommendation approaches?
How to better optimize human's objectives by learning reward representation which can deal with policies which are incomparable?
Is the reward function inferred by PRL interpretable and does it faithfully follow human intentions?
How to design agent types so that resulting policies together with the preference feedback lead to more accurate reward estimation and better treatment outcomes?
EXPERIMENTS
5.1 SETTINGS
Medication Recommendation for General Cancer: For 6-month simulation, the agent makes dosage amount decisions in each month. 10, 000 subjects are randomly sampled for training. 2, 000 for validation and 2, 000 for testing. We are aimed at learning optimal policies with three kinds of intentions: 1) maximizing survival rate to obtain optimal clinical efficacy (CE); 2) and mitigating negative effects represented by the sum of the tumor size and the toxicity level after treatment (CE&OF-I); 3) and mitigating negative effects represented by two separate health signs, the highest toxicity level during the treatment and the final tumor size (CE&OF-II).
During the 100-hour simulation, the agent is asked whether to perform a 2-hour operation in every hours. We randomly sample 000 subjects for training, 1, 0000 for validation and testing. This is partially observable MDP and LSTMs are utilized for agent modeling. The agent learns policies fulfill two intentions: maximizing survival rate to obtain optimal clinical efficacy (CE); 2) and avoiding too frequent operations (CE&OF).
Blood Purification Recommendation for Sepsis:
5.2 COMPARED APPROACHES
We benchmark results from the following existing approaches from treatment recommendation literature:
Non-learning (Zhao et al., 2009 Fürnkranz et al. 2012): 1) Constant: A static dosage amount is given all the subjects throughout the six months; 2) Random: One of the four dosage options is randomly selected at each time-step: 3) Upper Bound: The subjects with Sepsis receive operations all the time throughout the simulation period.
Preference Learning (Fürnkranz et al.] (012): in Preference-Basee Policy Iteration (PBPI), one action is preferred the other based on their outcomes after certain times of simulations. Every time the dosage with the highest preference is selected.
Reinforcement Learning with handcrafted Reward: 1) Single-objective RL (Schulman et al. 2013): the conventional policy gradient approach; it receives +1 for survival outcome, -1 for death, and 0 for intermediate steps. 2) Single-objective RL (Ensemble): among two agents, the one with better performance on the validation set evaluated on the testing set. It is developed for fair comparison
5

---

Under review as a conference paper at ICLR 2021
Figure 1: Performance for Sepsis blood purification recommendation.
(a) Optimize clinical efficacy only
(b) Optimize clinical efficacy & mitigate negative impacts
SBAN (CTACPF)
SBRM [CE&OF] SHII
SUMM
$1.09
6.69
AbRN (F)&
01.9
ABRM (CESOF)
6.17
01.8
6.99
Mult--lnjective RL
80.0
R.
4.37
81.09
Upper
Upper
7.0
80.0
80.5
81.0
61.5
82.0
50.5
65.0
83.5
$4.0
4.5
5.0
5.5
6.0
6.5
7.0
Sunvoal
Number Operations
as two agents with different parameter initializations are used in the PRL framework. 3) Existing Multt-objective RI. (Zhao et al.) 2009): manually defined rewards based on key factors are assigned to each time-step. 4) Grid-search Multi-ohjective RI. (Kim & De Weck (006): both the clinical efficacy and the negative impacts from other factors are treated as objectives and the linear scalarization with the best weight retrieved from grid search is employed.
Reinforcement Learning with Preference-basee Reward: to guide the RL agent learning, both AbRM and SBRM are trained based preference determined by human's intentions. Since the preference- based reward is a non-stationary value approximated by a neural network, we implement agents with the policy gradient, which is robust changes in the reward function (Ho & Ermon 2016
Christiano et al. 2017).
5.3 BENCHMARK RESULTS
Medication Recommendation for General Cancer: We evaluate different approaches to pursue treatment goals in terms of maximizing survival rate CE, mitigating negative impact CE& OF-I (sum of tumor size and toxicity level) in Table ! and CE&OF-Il (highest toxicity level and final tumor size) in Table ( (Appendix). Considering the Survival Rate the only metric to derive preference on two policies, agents learning from either action-based (31.52%) or state-based (30.54%) preference reward have much better performance in saving lives than Single-objective RL (26.96%), where the handcrafted reward is used to penalize policies with death outcomes. When negative impacts expected be mitigated besides saving lives, agents receiving rewards from preference are capable to maintain the performance on the clinical efficacy with much fewer negative impacts at the same time. When human's 's preference is defined as CE&OF-II, the highest toxicity level during the treatment and the final tumor size are two contradictory objectives to minimize. In Table 2 we observe that preference-bane based reward guides the agent to policies with the highest survival rates, while one negative impact gets reduced but the other increases compared with other approaches.
Blood Purification Recommendation for Sepsis: Performance bar charts of different approaches evaluated by Survival Rate and Number of Operations are illustrated in Fig. [ When guided by preference-based reward rather than manually crafted reward, a slightly higher Survival Rate is achieved by both AbRM and SHRM, while the average number of operations has fallen considerably. by 6.79% with AbRM and 14.50% with SbRM. Note that although the approach Multi-objective RL leads to the fewest number of operations, the performance in Survival Rate has dropped to make undesired trade-offs between clinical efficacy optimization and negative impacts mitigation.
5.4 EXAMINATION OF PREFERENCE-RASEE RI. APPROACHES
Advantages of Preference-based Reward over handcrafted Reward: In Fig. (Appendix), we first show the sensitivity of policy behaviors to small changes in handcrafted rewards leads to unstable clinical efficacy even when the relative importance of obtaining positive outcomes against negative ones keeps unchanged. The three heatmaps of the agent's 's performance in response to different reward scalars reflect the difficulty in specifying an appropriate reward function to enable policy learning with the optimal clinical efficacy during treatment recommendation. In Fig. 6 (Appendix), we further show the difficulty of selecting reward scalars for three factors survival rate, last tumor size and maximum toxicity level in the grir-search Multi-objective RL approach to appropriately prioritize the clinical efficacy over negative impacts. To study whether the preference- based reward estimation can fully capture the human's intentions, we provide the RI. agents with linear combinations of

---

Under review as a conference paper at ICLR 2021
Figure 2: Sepsis treatment strategies recommended by the SbRM agent: (a) clinical efficacy and expected return during training, (b) reward transferability among different configurations.
(a) ShRM Training
(b) SbRM Transferability
0.5
80
80
Expected Return
0.4
70
Survival Rate
Survival Rate
60
0.2
50
Training Expected
0.1
Validation Expected
40-
Training Survival
40
Transfer
Validation Survival
30
With Transfer
10
Epochs
Epoch
preference-based reward and handcrafted reward. Results in Table (Appendix) show that learning from preference-basee reward alone is adequate to achieve high clinical efficacy, while its combination with handcrafted rewards distracts the RL agent from optimizing the human's actual intentions and finally leads to inferior performance. To study the generalizability of the reward estimator, we extract the well-trained reward model in 2-hour operation configuration for Sepsis treatment and use the pre-trained model for 4-hour operation experiments. In Fig. 2b and Fig. (Appendixx). the reward estimator with knowledge transfer helps the agent speed up learning: compared with learning from scratch, the reward estimator with good initialization from a different configuration can provide better guidance to the agent.
Figure 3: Cancer treatment recommendation for 10,000 training subjects with the PRL framework. Curves in green describe scenarios when incomparable policies are discarded while curves blue show cases when comparable policies are preserved and efficiently utilized.
Prodicted brefmene
10:0
6.6
4000
6.5
200
395
390
35
200
150
203
100
353
are 200
250
Epoch
Epoch
Epoch
Addressing Incomparable policies: Given two policies for one sampled subject, if they have identical performance according to human's objectives, then the two policies are deemed to be incomparable. Since no clear preference conclusion can be drawn between the two incomparable policies, the majority of existing work in preference learning disregarded them directly (Fürnkranz
et al. 2012; Cheng et al.] 2011; Akrour et al.) 2012 Schäferr & Hüllermeter] 2018 Christiano
jet al. 2017). Only comparable pairs, either nm preferred to म. (M(T (8:) > T. (8x)) 1) or m.
preferred to Tm (s.) > (s:)) = 1), are included in the training set to optimize preference approximation. However, preference learning based comparable policies alone achieves quite unsatisfactory clinical efficacy in our treatment recommendation tasks. As shown in Fig. 3C for Cancer treatment recommendation, the survival rate (green curve) progresses with little improvement but great fluctuation during 400 epochs of training. Two reasons are likely to contribute to the failure: 1) polarized preference (one preferred with probability 0.85, and the other 0.15 in Fig. 33) inferred between two incomparable policies although the preference label is never provided in the training set; 2) only around one-fifth of the policy pairs (2,000 comparable from 10,000 sampled subjects) are leveraged in each epoch for preference model update (green line in Fig. (p)) After the above performance analysis, we find that excluding incomparable pairs from the training set leaves the parameterized model exploring the preference space arbitrarily and inferring random preference over two policies although they are incomparable. To avoid arbitrary exploration in the preference space, handle the incomparable pairs with simple approach: treating both policies from the incomparable pair equally, i..., I (s;) > n. (s;)) = (T, > Tm (*:)) = 0.5. With the small but important augmentation the preference indicator function, incomparable policies are efficiently
7

---

Under review as a conference paper at ICLR 2021
Figure 4: Cancer treatment recommendation: (a) clinical efficacy and expected return during training, and (b, c) expected return of policies ending with different negative impacts during testing.
Survve
Survive
-2
Dic
6.-- 64- Espected
Rate
Expected Resuum
-2
65-
3.03
6.-
62-
Traniig Axpecad
-359
200
2X Epocts
250
350
Negotive impacts (Last Tumor Toxidit)
Negathic Impocts (aat Tumor Toxldhy!
(a) SbRM Training
(b) PG on Negative Impacts
(c) SbRM on Negative Impacts
utilized for better preference space exploration (preference approaching 0.5 expected in Fig. B)), more samples for preference model update (all the 10,000 samples from the training set participate in the loss function minimization in Fig. (p), and much higher clinical efficacy (more than 30% survival rate achieved after the model converges in Fig. Bc).
To demonstrate whether the preference-based reward match human's 's actual intentions, we visualize the SbRM and ABRM agent's 's expected return for Cancer treatment during training and its relationship with the resulted negative impacts during testing (OF-I) in Fig. 4 and Fig. 7 (Appendix). respectively. From Fig. 4a, we can observe that the rising trend of the expected return matches the improving Survival Rate quite well, although the parameters of the reward estimator are updated at the same time. The estimated reward offers reasonable explanations for the policy performance: the higher the expected return, the better the policy. After the model converges, we analyze the distribution of expected returns for policies with different negative impacts. Since penalties or rewards are assigned to policies based on their outcomes only, the conventional Policy Gradient approach treats policies ending with survivals but different negative impacts equally (horizontal blue dots in Fig. 4p) After adopting preference-based reward, policies resulting in survival outcomes can distinguish from each other: policies with smaller negative impacts have much higher expected return. In Fig. 4c. policies leading to deaths have extremely low expected return (approaching zero), while the expected return for policies with survival outcomes is negatively proportional to the amount of negative impacts. As shown in Fig. 2a and Fig. Da (Appendix), the expected return received by the agent for Sepsis treatment also shares the common trend with the Survival Rate: if more lives have been saved by the agent, then higher expected return is achieved.
Interpretability in Inferred Rewards:
Influence of Agent Types in Treatment Outcomes: As depicted in Algorithm | the studied preference-bact based RL. framework adopts two RL agents controlled by different parameters to infer the reward and learn the policy that optimizes human's 's intentions. We here study the influence of different agent designs on reward approximation and resulted performance. Specifically, the reward function is estimated to approximate the preference over policies, among which the first policy is performed by one RL agent while the second policy can be executed by different agent types. The clinical efficacy curves shown in Fig. g (Appendix) empirically prove the effectiveness of the current design of two different preference-bace based RL agents.
CONCLUSIONS AND FUTURE DIRECTIONS
To obtain optimal treatment policies based on human's 's diverse objectives, we investigate performance of the preference- based Reinforcement Learning approaches, where higher rewards are automati- cally estimated and assigned to actions following human's actual intentions underlying the provided preference feedback. During interacting with the developed simulation platform, we resolve criti- cal implementation problems and gain a deeper understanding in designing preference-based RL approaches, in order to better aid clinicians in treatment decision making. In future work, we will consider tackling some more practical aspects about human's preferences in adopting treatment strategies: 1) how to efficiently leverage preference in reward learning if human's 's feedback is limited, 2) how fully reflect human's actual intentions in reward learning if both preference feedback and clinicians' demonstrations are provided.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In
Proceedings of the twenty-first international conference on Machine learning, pp. 2004.
Alan Agresti and Maria Kateri. Categorical data analysis. Springer, 2011.
Riad Akrour, Mare Schoenauer, and Michèle Sebag. April: Active preference learning-based reinforcement learning. In Joint European Conference Machine Learning and Knowledge Discovery in Databases, pp. 116-131. Springer, 2012.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-355 1952.
Daniel S Brown, Wonjoon Goo, Prabhat Nagarajan, and Scott Niekum. Extrapolating beyond
suboptimal demonstrations via inverse reinforcement learning from observations. arXiv preprint (eXiv:/904.0650 2019.
Weiwei Cheng, Johannes Fürnkranz, Eyke Hillermeier, and Sang-Hyeun Park. Preference- based policy iteration: Leveraging preference learning for reinforcement learning. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 312-327. Springer, 2011.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems, pp. 4299 4307, 2017.
Brian Denton, Murat Kurt, Nilay D Shah, Sandra C Bryant, and Steven A Smith. Optimizing the start time of statin therapy for patients with diabetes. Medical Decision Making, 29(3):351-367, 2009.
Daniel M Faissol, Paul M Griffin, and Julie I. Swann. Timing of testing and treatment of hepatitis c and other diseases. In Proceedings, pp. 11, 2007.
Johannes Fürnrranz, Eyke Hüllermeier, Weiwei Cheng, and Sang-Hyeun Park. Preference-basee reinforcement learning: a formal framework and a policy iteration algorithm. Machine learning. 89(1-2):123 156. 2012.
Yang Gao, Huazhe Xu, Lin, Fisher Yu, Sergey Levine, and Trevor Darrell, Reinforcement learning from imperfect demonstrations. arXiv preprint arXiv:1802.05315 2018
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in neural information processing systems, pp. 4573, 2016.
I1 Yong Kim and OL De Weck. Adaptive weighted sum method for multiobjective optimization: new method for pareto front generation. Structural and multidisciplinary optimization, 31(2): 105-116, 2006.
Matthieu Komorowski, Leo A Celi, Omar Badawi, Anthony C Gordon, and A Aldo Faisal. The artificial intelligence clinician learns optimal treatment strategies for sepsis in intensive care. Nature Medicine, 24(11):1716, 2018.
Daniel Lopez-Martinez, Patrick Eschenfeldt, Sassan Ostvar, Myles Ingram, Chin Hur, and Rosalind Picard. Deep reinforcement learning for optimal critical care pain management with morphine using dueling double-deep q networks. arXiv preprint v:Xiv:1904.1110 2019.
Shamim Nemati, Mohammad M Ghassemi, and Gari Clifford. Optimal medication dosing from suboptimal clinical examples: A deep reinforcement learning approach. In 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp- 2978-2981. IEEF, 2016.
Tu- HooH Pham, Giovanni De Magistris, and Ryuki Tachibana. Optlayer- constrained opti- mization for deep reinforcement learning in the real world. In 2018 IEEE International Conference Robotics and Automation (ICRA), pp. 6236-6243. IEEF, 2018.

---

Under review as a conference paper at ICLR 2021
Aniruddh Raghu, Matthieu Komorowski, Imran Ahmed, Leo Celi, Peter Szolovits, and Marzyeh Ghassemi. Deep reinforcement learning for sepsis treatment. arXiv preprint Sedi::7111.066000 2017.
Thomas Rimmelé and John A Kellum. Clinical review: blood purification for sepsis. Critical Care, 15(1):205, 2011.
Claudio Ronco, Rinaldo Bellomo, Peter Homel, Alessandra Brendolan, Maurizio Dan, Pasquale Piccinni, and Gluseppe La Greea. Effects of different doses continuous veno-venous haemofil- tration outcomes of acute renal failure: a prospective randomised trial. The Lancet, 356(9223); 26-30, 2000.
Suchi Saria. Individualized sepsis treatment using reinforcement learning. Nature medicine, 24(11): 1641-1642, 2018.
Dirk Schäfer and Eyke Hüllermeier. Preference-based reinforcement learning using dyad ranking. In International Conference on Discovery Science, pp. 161-175. Springer, 2018.
Tom Schaul, Diana Borsa, Joseph Modayil, and Razvan Pascanu. Ray interference: source of plateaus in deep reinforcement learning. arXiv preprint urXiv:/9044 1455, 2019.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02433. 2015.
Sang OK Song, Justin Hogg, Zhi-Yong Peng, Robert Parker, John A Kellum, and Gilles Clermont, Ensemble models of neutrophil trafficking severe sepsis. PLoS computational biology, 8(3), 2012.
Ivan Stojkovic, Mohamed Xi Hang Cao, and Zoran Obradovic. Effectiveness of multiple blood- cleansing interventions sepsis, characterized in rats. Scientific reports, 6:24719, 2016.
Lu Wang, Wei Zhang, Xiaofeng He, and Hongyuan Zha. Supervised reinforcement learning with recurrent neural network for dynamic treatment recommendation. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 2447-2456. ACM, 2018.
Christian Wirth, Riad Akrour, Gerhard Neumann, and Johannes Fürnkranz. survey of preference- based reinforcement learning methods. The Journal of Machine Learning Research, 18(1):4945- 4990, 2017.
Yufan Zhao, Michael R Kosorok, and Donglin Zeng. Reinforcement learning design for cancer clinical trials. Statistics in medicine, 28(26):3294-35 3315, 2009.
10

---

Under review as a conference paper at ICLR 2021
A APPENDIX
A.1 FIGURES AND TABLES
Figure 5: Performance of RL in different hand-crafted reward designs optimize clinical efficacy.
(a) Random Seed 2001
(b) Random Sced 2002
(c) Random Seed 2003
Death Reward
Death Reward
39 30 -4 31
11
304
26
25
24
Survival Reward
Survival Reward
Survival Reward
Figure 6: Performance of Multi-objective RL with linear scalarization three reward factors: survival rate, last tumor size, and maximum toxicity levels. The linear weight assigned to each factor is one of four values: {1, 4, 8).
Gridsearch NOMOP
Preference: ABRM (CE&OF-II, SERM (CESOr-1)
30
1.5
Surviva Rate
Toccoy
20
1.0-
Last
0.5
0.0
10
20
0
1.0
20
30
0
1.0
20
30
Different ralios ameng three reward factors
Different ratios among thrice reward factors
Different ratios amonn three rewrard factors
Figure 7: Cancer treatment strategies recommended by agent AbRM: (a) clinical efficacy and expected return during training, and (b) expected return of policies ending with different negative impacts during testing.
(a) AbRM Training
(b) AbRM on Negative Impacts
165
Survive
-30
Expected Return
Expected Retum
26
1.0
W
Survins
22
0.8
0.6
Training Survival
18
Validation Survival
50
100
150
200
250
300
350
400
Epochs
Negative Impacts (Last Tumor Toxicity)
11

---

Under review as a conference paper at ICLR 2021
Figure 8: Effects of different agent designs performance.
(a) AbRM
(a)
(b) CE&OF-]
Agent
Agent
(b) CE&OF-I1
Survical Rate
30
Survial Rate. Rate
30
Survival Rate
30
20
20
20
Pref
Pref
Pref
Pref
Pref Const
Pref Pref (Twin) (Single)
Pref & Pref
Pref & Env
Pref
Pref Const
Pref Pref (Twin) (Single)
Pref & Pref
Pref & Env
Pref Rand
Pref
Pref
Fref
Rand
Rand
Const
(Twin) (Single)
Different combinations
Different combinations
Different combinations
(b) SbRM
Agent
Agent
(a) CE
(b) CE&OF-I
b) CE&OF-I1
Survical Rate
30
Surviall Rate
30
Survival Rate
30
20
20
Pref
Pref
Pref Rand
Pref
Prif
Priff
Pref
Priff
Prief Const
(Twin) (Single)
(Twin) (Single)
(Twin) (Singla)
Different agent combinations
Different agent combinations
Different agent combirations
Figure 9: Sepsis treatment strategies recommended by the AbRM agent: (a) clinical efficacy and
expected return during training, (b) reward transfcrability among different configurations.
(a) AbRM Training
(b) AbRM Transferability
80
Expected Return
70
70
60
60
Survive
Surviva
Training Expected
Validation Expected
50
40
Training Survival
No Transfer
Validation Survival
30
With Transfer
10
20
30
40
50
2
4
10
12
Epochs
Epoch
12

---

Under review as a conference paper at ICLR 2021
Figure 10: For Cancer experiments, expected return of DQN learning from behavioral policies of Policy Gradient and its estimations from different off-policy evaluation methods.
1.5
1.0
Expected Return
0.0
-0.
-1.0
-1.5
Ground-truth (-0.77)
PDIS (-0.00)
WPDIS (-0.11) Estimators
WDR (0.81)
(0.81)
Table 2: Performance for Cancer medication recommendation considering negative impacts from two factors: the tumor size in the end and the ever experienced maximum toxicity.
Clinical Efficacy
Other Factors
Method Type
Method Name
Survival Rate
Last Tumor
Max Toxicity
Constant Best (0.4)
19.91%
10.11%00.589 0.58%
1.76:10.02
0.5710.01
Constant Worst (0.1)
4.89% +.68%
3.72+60.03
0.48
Non-learning
Random
17.81 10.91%
0.82+00.02
1.64:0.004
Preference Learning
PBPI
20.80%:10.5699
1.38++0.1
1.01+0.12
Single-objective RL
26.96%
0.48
1.37-0022
Reinforcement
Single-objectivg RL (Ensemble)
27.38%
0.4740.29
1.38:0.29
Learning
Existing Multi-objective RL
18.84%15..7%%
0.25:00.10
2.1640..0
(Hand-crafted reward)
Gri-searchh Multi-objective RL
25.10%:1144%
0.6310020
1.2140.16
AbRM (CE)
31.52%+1.3889
0.1140.10
1,74 0.06
Reinforcement
AbRM (CE&OF-II)
30.34%
0.2310022
1.6310019
Learning
SbRM(CE)
30.54%.1..44 3.46%
0.18
1.67:0.22
(Preference-bassee based reward)
S6RM(CE&OF.I)
30.76%+42.6494
0.08:10.09
1.75:10.09
13

---

Under review as a conference paper at ICLR 2021
Table 3: Evaluating the clinical efficacy (survival rate) achieved by the proposed preference- based RL framework when hand-crafted and preference-based rewards are linear combined with different ratios.
AbRM
ABRM
AbRM
SbRM
SbRM
SbRM
Ratios
(CE)
(CE&OF-I)
(CE&OF-II)
(CE)
(CE&OF-I)
(CE&OF-II)
8:1
28.06%
28.07%+3..500 5305%%
28.23%1 3.82%
29.10%
26.51%-1244%
28.94%13.119
4:1
28.92%:12.82%
27.41%133.666
27.9%313.61%%
21.77%=3.16%
28.45%13.18%
27.54%:13.099%
2:1
26.90%=3.19%%
27.50%+t..64%
27.71%::3.47%
27.55%=3.40%
27.81%:3..15%
28.43%++.112%
1:1
27.42%+1.22%%
27.70%
27.62%
27.99%+2..9%
28.49%
28.71%+3.500%
1:2
26.96%
2.95%
27.55% 2..90%
28.66%
1.3. 199
28.59%
2.94%
28.79%
13.21%
28.30% 13.29%
1:4
27.46%+1..5
26.59%-1..400
28.25%
4 +.44%%
29.91%
1.93%
28.75%
1.006%
23.08%++...
+3.64%
1:8
23.05%
29.21%
27.67%
13.38%
29.18%
2.45%
28.55%
22.85%
18.48%13.228
0:1
31.52%
1.38% %
31.33% + 1.18%
30.34% % + 2.71 %
30.54%
3.46 %
31.72% 1.08% %
30.76 % +..64%
(Ours)
14

---

Under review as a conference paper at ICLR 2021
A.2 PREFERENCE-BASE BBSSED REINFORCHMENT LEARNING ALGORIHTM
The preference-based Reinforcement Learning framework is composed of two main modules, Treference-based Reward Learning and Preference-guice Agent Learning. In Preference-bane based Reward Learning, the reward estimator parameterized by 8 delivers step-wise rewards to the two agents parameterized by 0 and 62 based on their policy preference. Learning, the agents update their parameters so as to optimize the clinicians' objectives. The pair of policies performed by the two agents on the sampled subject is stored in the policy pool and leveraged for parameter update in reward estimator, with the aim to ensure higher expected return for the preferred policy. We list the pseudo codes for collaborative learning in Algorithm|] [I] Preference-based Reward Learning in Algorithm 2 and Preference-guice Agent Learning in Algorithm 3 respectively.
Preference-guide
Agent
Collaborative Learning Algorithm I Iilluttraes the collaborative learning process between the two modules in order to estimate reward and learn policies in personalized treatment recommendation. In the beginning. the model parameters are randomly initialized (line T. and the policy pools for the reward estimator and the two agents are created as empty sets (line 2 In each iteration, one subject is sampled from the training set for agent learning (line 3 to 5p. At each simulation step, the two agents are asked to make decisions based on the current state and the reward estimator generates correshonding step-wise reward for each of them (line to TOp. The subject's internal state keeps on updating until the simulation time has reached or the subject dies intermediately according to the underlying mathematical modeling. The policy pools of the two agents are augmented with the trajectories on the newest sampled subject (line[9and [1]. while the policy pool for the reward estimator is also updated (line [3) after computing the ground-truth preference label (2). After all the samples have been utilized for policy generation, the reward estimator minimizes the classification loss during policy preference inference with Algorithm 2 (line 16), while the RL agents optimize the expected return with Al Algorithm 3 (line 17).
Algorithm 2 PREFERENCE-BANE BBASED REWARD LEARNING
Require:
D,, sampled policy pairs in n-th iteration
0p: parameters to update in reward function
Tp: discounted factor on reward
B: step size for parameter update
1: L +00
2: for all (7), 72, pre((r) 7²)) E D, do
3:
R(T': 0p) t 0, R(r2;;pp)) f
4:
for all (5,, (0. Er do
5:
R());0p) t- R(7);0p) + t
6: end for
7: for all (ss, at, r 6p. )4 E 2 do 8: Op) R(72; 2(72;0r) + +
9: end for
10: Compute p(7 > x2)
11:
if > then
12:
L+ L log p(T) > 7
13:
else if 22 > - then
14:
else r I log (1 p(T > +22)
15:
72 then
16:
L L 0.5 log -22 + 0.5 (1 -1-ple p(T) 72))
17:
end if
18:
end for
19:
Update Op c 0p B Ib
20:
return 0p
Preference-based Reward Learning Given pairs of policies with corresponding preferences,
the
reward estimator updates its parameters maximize the probability that the preferred policy achieves
15

---

Under review as a conference paper at ICLR 2021
higher expected return than the other. As shown in Algorithm 2] the discounted expected returns achieved by each agent are firstly calculated respectively for each Then the probability that policy 7 is preferred to r is positively correlated to the expected return of 7-, and is computed as (Agresti & Kateri 2011 introduced (line 10). Hence p(T² > -²) is equal p(r > 7 Then the loss value computed considering different kinds of preference relationships between the two policies (line to
each sampled policy pair (line 3 to (
to [7). Incomparable policy pairs are also leveraged
in reward learning for better preference space exploration (line 13
Algorithm 3 PREFERENCE GUUIDED AGENT LEARNING
Require:
n. sampled policies from one agent in n-th iteration
a: step size for parameter update
.M: one of the two reward assignment methods
Lox: loss function in any deep RL approach parameterized by agent parameters 0₂
1:
2:
for all (st; ae, TB S141) n, do
if MM is Action-based Reward Modification then 4: Tt t- (8p(8 at)
3:
else if is State-based Reward Modification then 6: Tt. + hap hap (11-1)
5:
7: end if
8: <- E E {((s, (1,
9:
end for
10: Update 0^ + O^ aBon Σ Log (1) FEe
11: return 0
Preference-guided Agent Learning Each agent updates their parameters individually as Algo-
rithm 3 depicts. The agent receives rewards computed by either Action-based Reward Modification (line 3 to or State-based Reward Modification (line 5 5 6). Then we leverage (st, 0e, Tt, 814 to update the agent model implemented by any deep Reinforcement Learning approach.
A.3 SIMULATION PLATFORM DESIGN
A.3.1 MEDICATION RECOMMENDATION FOR GENERAL CANCER
Survival Analysis Within time interval (t - 1, t). where (1 < t S 6). the survival status is assumed
to depend on both the current tumor size y/: and the toxicity level xy. The probability of patient's death is modeled as follows:
Hazard function: X(1) = exp( -4 + + xz), Cumulative hazard function: AA (1) = [" M(s)d(s),
--11
Survival function: A F exp( (A(()))
Death probability: Pdenth = A F()).
Implementation Details The action space is discrete and the dosage amount decisions selected among 4 options: 0.1, 0.4, 0.7, 1.0 (Fürnkranz et al.) (012).. For state initialization, the tumor size and the toxicity level in the 0th month are generated independently from the uniform distribution 2000 2). The simulation terminates after = 6th month or if the patient dies intermediately.
Model Implementation and Training For 6-month simulation, we randomly sample 10. 000 subjects for training, 2, 000 for validation, and 2, 000 for testing. The neural networks for all deep learning approaches including preference learning and reinforcement learning share the similar network structure and activation and the second followed by different activation functions for different approaches. In one epoch, the agent gets updated after seeing all the training samples. The learning rate is set to 0.01 and all the networks converge after 400 epochs. For deep RI methods, set the discount factor y 1.
parameters: fully-connected layers, the first followed by ReLU
16

---

Under review as a conference paper at ICLR 2021
A.3.2 BLOOD PURIFICATION RECOMMENDATION FOR SEPSIS
Mathematical Modeling in Simulation Sepsis is initiated by spillover of pathogens into blood, where the pathogen is allowed to spread throughout the organism in which systemic inflammation takes place (Stojkovic et al. 2016). Motivated by the promising results of blood purification in other critical illness conditions like acute kidney failure (Ronco al.] 22000, blood purification has gained attention as a potentially effective solution for septic subjects (Rimmelé & Kellum 2011). In blood purification treatment, the patient is connected to an extracorporeal hemoadsorption device that removes harmful particles from the blood and leads the patient towards healthy state.
We employ the mathematical model derived by Song et al. to simulate the acute inflammation process in response to an infection (Song et al. 2012). Both heuristic knowledge about the mechanism underlying infection and real measurements from experiments on CLP-induced septic rats were leveraged for the model design. The distribution of initial physiological features and their interactions are derived from domain knowledge. The initial physiological features that characterize subject accords with the probability distributions based on real experimental measurements for septic rats. The parameters in transition functions are calibrated so that the generated trajectories closely follow experimentally observed temporal patterns in septic rats.
Figure 1 demonstrates the feature interaction network. There are 19 physiological features that govern sepsis dynamics, 8 of which are observable (features above the horizontal dashed line) while the remaining 11 are conceptual variables (features below the horizontal dashed line). When a blood purification operation is made, three components in the circulation are eliminated (features marked by red dashed ring), i.e., activated neutrophils N, and the pro- and anti-inflammatory mediators P/ and AI. Besides effects from the blood purification operation, the variables influence each others' progression through Ordinary differential equations (ODEs).
State Transition There are 18 ODEs to describe feature interactions and 3 ODEs for the hypothetic mechanism of blood purification. The hypothetic mechanisms of action of the blood purification are implemented by assuming the hemoadsorption device eliminates only three components in the
circulation: activated neutrophils (Na), pro-inflammatory mediators (PI), and mediators (Af) during the treatment period. We here only show the transition equation of these three key features with and without operation, ODEs concerning other features be found in (Song et al. 2012).
The variable PI stands for the extent of the systemic inflammation and progresses follows:
dPT dt
B/Bx
AT 1011-P1 -PT)
hPI + B/BBx hPLD + +DDD D1
(1
hPIAA + AAm
(3)
B/Boo
D
A1 (I)
+ +(1 brL_ B/
Bo
hy
KPL D + Du
h4 + AT
B/Bx
D
Al PI)
1
h + AT -PI)
hpI_8 + B/Bxo h D Du
TPI
PI(()+ + dt
(P)
no operation is performed
PI( + +1)
(4)
PT(L) + dPI (1')
PT hРıнA+РI
Otherwise
where B is the population of bacteria in the peritoneum, D is coarse-graine representation of integrated tissue damage, variables hpr_n. her hpr_AAs Tpy are subject-specific parameters, Bxc predefined upper bound of B, hPIHA 0.3 and n 3.
17

---

Under review as a conference paper at ICLR 2021
Table 4: Configurations for Sepsis treatment simulation. à represents hour in the simulation platform
Operation Time Interval L 5th to 18
Decision-making Frequency f
Duration per Operation /
Step Size
Horizon Length T
0.1
T = 100 h
= 2 h or 4h
1= 2 h or 4h
The variable Al describes the level of the anti-inflammation corresponding to systemically acting anti-inflammatory mediators and gets updated as follows:
dAl dt
PI
NooNoo
PI
Na/N
T1 AILPI PI + PI (1
)+ + (1- h PI + PI bAIN, + Na/No
hALN, + Na/Noo
(5)
P1
Na/No0
AT) 1
W + PT HALN, No
TAS
AI((') + 4AI (t')
no operation is performed
AI(t' + +)
di
(6)
AT((')+ + dAl
di (t')
AT hAIRA+AI
Otherwise
where variables AAN_PI,hAI_NN TAI TAI subject-specific parameters, Noc is predefined upper bound of neutrophils, hASHA 0.3, ny = 1 and n2 3.
The variable No represents the activated blood neutrophils and transits in each simulation step as follows:
dNa
P1
1
1 M MN + PTa No
N1P
Na TN,
NaP 1
di
N. PI TN.N.
W PTn TN....
transmission Nr, to No
transmission from Np N..
transmission N., N,
(a)+) + dNa (t)
If no operation is performed
(t + +1) (a)+)+ + dt (t)
dt
HN.HA+N, & (t') Otherwise
No/Nx
where N, is resting blood neutrophils, No is blood neutrophils, N, is neutrophils sequestered in the lung capillaries, variables h N. IN,,N. IN_NN, No, Nu, TNa_N, are subject-specific parameters, hN.HA = 0.3, and n = 3.
Survival Analysis The survival status of the subject only depends on the value of the systemic pro-inflammatory response PT at the end of the simulation. When the PT value at the last time-step smaller than the pre-defined threshold 0.5, then the subject is assumed to be alive, otherwise dead. Note that after the blood purification process, the PI value reduces as time passes, hence one cannot conclude whether the subject alive in the intermediate time-steps. After the pre-defined simulation horizon is reached, mathematical model is quite different from the general Cancer Treatment model where subjects have probability to die intermediately.
can confirm which subjects survive with the help of treatment. The
Implementation Details Due to phenotype differences, some subjects survive without any blood purification operation while some die. This is consistent with laboratory experiments where 30% of rats survived till seven days while the remaining died between two to five days after CLP (Zhao) 2009). We call the survivor group Survival Population and the non-survivor group Death Population. The survival status of the Survival Population gets no influence from blood purification operations. Subjects from Death Population have the potentials to survive if proper treatment policies are delivered. Since we primarily concerned about the outcomes on subjects from Death Population, we only sample subjects from the Death Population in this paper to train and evaluate treatment policies.
There are few time, the simulator updates the internal status of subjects by computing the ODEs with feature values from the last simulation step and the current action. 2) Simulation horizon length T: we can evaluate the performance of policy by checking outcomes of subjects after time T. 3) Valid time
that should be set in advance: 1) Simulation step size T every
18

---

Under review as a conference paper at ICLR 2021
range for patients to receive treatment: operations take place any time-step (L |0, - 1]) or constrained to predefined time intervals (L _ [0, T 1l). 4) Frequency of decision decision-making f subjects receive operations at each simulation step or less frequently. 5) Duration of each blood purification operation I: it takes some costs to turn on/off the purification device and is also unrealistic attach and detach the device from the subject frequently. Therefore, there should he a pre-defined value for the purification duration to rule out the possibility of too frequent actions.
To generate testable hypotheses that guide future laboratory experiments (Song et al. 2012 Stojkovic et al. 2016), the simulation of sepsis evolution should be configured to make the generated trajectory closely follow experimentally observed temporal patterns (Song al.) 2012). Further, several constraints can be imposed on the simulation in accordance with previous blood purification studies (Song et al. 2012 Stojkovic al. 2016). Therefore we use the configuration listed in Table. 4 for experiments.
Model Implementation and Training We randomly sample 3,000 subjects for training, 1,000 for validation, and 1,000 for testing. Implementation details of the deep RL approaches are similar to those mentioned in the Cancer task, except that the backend network is L.STM-based since this is POMDP.
Learning efficient treatment policies for Septic subjects is more difficult for Cancer due to the larger state space and the partially observable environment. Therefore, adopt the following methods to ensure robust learning: 1) Mini-batch gradient descent with batch size 10,000 is adopted to update parameters in reward estimator and RL agents. 2) The learning rate for RL agents is 0.01 while 0.001 for the reward estimator. 3) As discussed in Experiment Section, experience replay makes the estimated reward positively proportional the Survival Rate. We randomly extract policy pairs from the latest 30,000 samples for model updates.
19

---

Under review as a conference paper at ICLR 2021
Figure 11: Distribution of state features (tumor size and toxicity level) from Cancer subjects without tre
No Treatmant
Treatment
Constom: Best
Constart Best (0.4)
Tumer Sbe
0.0
Texdcity Lave Surweal Kate
Tumer Sbe
04
Teatett Level Surv Rale
0.7
0.7
Sunvial hate
0.7
07
-0.5
1:2
Toxtsty
02
0.5
-0.2
02
0.1
0.0
0.0
03
0.0
0.
Narth
March
Month
Morth
Consiant Worst 10.1]
Constant Worst (6. 3)
Random
Randem
2.5
Sanstal
6.7
Survial Aate
Sundun Rate
Surviaal Rate
0.6
6.6
06
us
Suntill
Toxckyy
0.1
0.0
0.1
0.0
0.0
0.2
0.0
PSP1
FBP1
Single objective RL
Single objective RL
Tumer Sim Sundeal
0.0
Tadetty Lave
Tumer
Teatsty Lovel
Survical
Kate
03
Sanbo
0.
04
1
6.2
Marth
Mench
Morth
Menth
Single-otjective RL (Ersumble)
Singkocbiaccive (Ensemale)
Esisting Mult-otjjctive RL
Existing Mult-ckiiccvee
Tumn Siee
6.0
Tediity Lase
Tumer She
6.3
Tealiity Leww
0.
Sundral hats
6.7
Survaal fula
0.
Surshra tale
Surviall hala
Girvin Site
0.2
9.2
1.2
92
0
6.1
0.5
6.1
0.0
North
Vent
North
Menth
Grid-search Mulli-dbjection
Crid-snarch M dilteeejectiie RI.
Abil (CF)
A3RM (CF)
Level
Suntral Rata
0.7
Survaal Rata
0.7
Sandual Rate
6.7
Survial Rate
0.a
0.5
1:
Toakdyy
02
0.1
2.2
0.2
6.1
0.0
01
0.
Morth
North
ABRN (CE S CF-11
ABRM ICE CFFI)
SbAM (CE)
SaRM (CE)
Turne 2
Telttyy hale
Rrovutt Botee
Lavel
05
1:2
6.1
0.0
01
0.a
6.3
Marth
Mhrchh
ShRM 1CE5.0F-0
SbAM (CESOF-))
Tumor siee
0.8
to Leve
Suntral Rata
Survial Rata
2.0
1.3
9.0
0.5
0.4
2.5
Marth
Monhh
20

---

Under review as a conference paper at ICLR 2021
Interaction network of inflammatory responses and hypothetic hemoadsorption mecha- nisms of action in CLP-induced sepsis. Nodes in green represent components in peritoneum, nodes in orange stand for blood components, and nodes in purple stand for lung components. Edges represent network interactions under blood purification treatment compiled from literature. When blood purification is performed, only features PI, .AI and Na influenced (marked by red dashed rings).
Figure 12:
Lsel
HMGB1
CRT
ALT
TNFa
IL1b
IL6
IL10
Observable features
Unobservable Teatures
Nt
B
Np
Nr
AI
Na
activation
inhibition
mass flow
operation influence
Figure 13: Distribution of 19 state features from Septic subjects without treatment or with treatment from SbRM.
CLP
P1
AJ
0.4
0.75
0.4
0.2
0:5
0.2
0.3
5.0
1000
500
1000
500
1000
1000
1000
Na
Ns
15
Asb
N:
2.0
0.4
0.4
1.5
0.2
0.5
01
0.0
1.8
0.75
0.75
0.50
0.50
0.25
0.25
0.00
0.00
0.00
500
1000
500
1000
1000
1020
HMGB1
0.8
0.5
0.8
3.4
Treatment
0.2
0.2
0.2
2-hour Treatment from SoRM
0.5
0.0
530
1000
500
1000
500
1000
500
1000
21


Reviews:
[{"review_id": "oT0-1khP2xL", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThis is a relatively comprehensive evaluation. The authors know the related literature, and a number of experiments are presented to show different aspects of the methods. The work is promising. Writing is great, and the authors can write beautiful sentences, but the overall structure has some flaws.\n\nI am not happy with the way the authors split their manuscript into the main paper and the appendix. The current version of the main paper is not self-contained, and it does not provide minimum explanation of the methods. I was very perplexed after reading the main part before I read the appendix because the core algorithm is not explained in the main part of the paper. It was hard to follow the experiments in sec. 5, when one is unsure how the methods work. The experiments section itself makes frequent jumps to the appendix. This content would be much more appropriate for a journal paper, in which most of the appendix would be nicely integrated with the main body.\n\nIt is not entirely clear what is the new contribution in this paper. When algorithm 1 is introduced in section 4.2, the authors do not say which components are new or how they extend the existing literature.\n\nThe EVALUATEPREFERENCE procedure in algorithm 1 is not defined, yet it is a very important component. The pseudocode of algorithm 1 indicates that entire trajectories are compared to elicit preferences, but this may lead to suboptimal behaviour. Note that if two policies are suboptimal, they may make suboptimal decisions in different states, so preferences at the level of individual states would be more appropriate. This important component of the algorithm is unclear to me in the paper. \n\nFollowing on the previous question, I should ask what the human expert would need to do if this algorithm was applied in practice. Specifically, what would be displayed to the human expert, and what the human expert would need to do or what they would need to compare and judge? The simulator (which would be the real environment) and the algorithms are tightly entangled in the appendix, and for this reason it is hard to imagine how this solution would work with real data and real human input.\n\nSince the authors' goal is to show that their method provides better rewards than the handcrafted rewards known in the existing literature, perhaps some discussion of those rewards could be added. How were those rewards derived? Did the human designers have the same objectives in mind? Could one derive a better handcrafted reward for these experiments? Sufficient evidence should be provided that the handcrafted rewards used in comparisons are not trivial. I am saying this because the preference-based reward is derived from preferences that match the quantities that constitute the objective of learning, so it is not that surprising that preference-based reward leads to better performance according to the corresponding objectives.\n\nSmall issues:\n\nIn sec. 1, the authors mention open access large-scale Electronic Health Records. Examples of such publicly available records should be provided.\n\nThe authors should clarify what they mean in this sentence \"the linearly weighted reward function induces negative interface between objectives\".\n\nSeveral symbols are not defined in section 2.1, e.g., a1, a2, m1, m2.\n\nFormatting on p. 3 is incorrect.\n\nWhat are \"hill\" equations on p. 3?\n\nIt would be good if the authors could explain equations that are in lines 5 and 8 in Algorithm 2. What would be an intuitive explanation of L in this algorithm? \n\n\n\nThis paper addresses a very important problem, and the results could be significant. I think that the future readers would be confused if this paper was accepted in this form. It would be sensible to turn it into a longer journal paper and clarify the key components.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Xqw-jbJl9cX", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper describes a preference based reinforcement learning framework for treatment recommendation tasks. The authors use a model-based environment to simulate the progression of disease and propose a preference based approach. They claim they can use this framework with any off-the-shelf RL algorithm to generate better survival results. The experiments are carried out with policy gradient. The authors also compared several different ways to design reward functions, and propose to use incomparable samples with a 0.5/0.5 distribution in the Bradley-Terry model, instead of discarding such samples, to improve the performance. The paper is generally well-written and easy-to-follow. Despite the technical contributions listed above, I would suggest rejection due to the following concerns.\n\n1. Innovation\nThe paper lacks citations and comparisons with several recent works in this field, such as [r1, r2]. The approach proposed is similar and the innovation seems marginal. I suggest the authors include a comparison of methodology and, if possible, experiments comparing their approach with state of the art preference based RL algorithms in the revision.\n\n2. Lack of implementation details\nThere are a few details I didn't find reading the manuscript, for example, how to query human responses on line 12 of Algorithm 1, what is the guideline for human experts to rate different treatment plans, implementation details and hyper-parameters used in the agent learning algorithm. The lack of such details makes it difficult for other researchers to reproduce the results shown in the paper.\n\n\n[r1] Christiano, Paul F., et al. \"Deep reinforcement learning from human preferences.\" Advances in Neural Information Processing Systems. 2017.\n[r2] Ibarz, Borja, et al. \"Reward learning from human preferences and demonstrations in Atari.\" Advances in Neural Information Processing Systems. 2018.", "extra_scores": {}, "extra_text": {}}, {"review_id": "RWI9jII764", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#### **Summary**\nThis paper develops a preference-based framework for Reinforcement Learning in Healthcare, focusing on treatment recommendation. The paper highlights several factors where policies trained with preferences may be better suited for use in healthcare applications and extensively evaluates these factors using two simulated care scenarios. \n\n\n#### **Assessment**\nThe foundational concepts underlying this paper are strong and the authors make good points framing the potential advantage of using preference-based rewards over what is termed a “handcrafted reward”. Reward design is an open challenge in Reinforcement Learning, particularly in Healthcare settings, and this paper proposes an intriguing way to perhaps avoid explicitly designing a reward while still providing improved performance. However there are significant gaps in the technical development of the paper that greatly reduce the clarity and perceived significance of the presented work (several examples highlighted in the “Weaknesses” section below). Several claims are made in the introduction and set-up of the experiments that are not fully supported, discussed or justified. Finally, there are some major concerns about the suitability of the simulated environments to adequately evaluate the contributions of the proposed methods (more in the “Weaknesses” section below) in light of the applicability to real world dynamics.\n\n\n#### **Strengths**\n- The paper presents an appealing rationale for the use of preference-based RL in complex sequential decision making problems where reward design is a significant challenge. The paper is firmly grounded in relevant literature. There are several papers that I expected to be included (see Relevant Literature in the “General Comments” section) but their omission is not a critical error.\n- The experimental investigation is ambitious and covers most, if not all, relevant questions regarding the utility of preference-based rewards in treatment recommendation. Several relevant baselines are used to compare and contrast the two proposed preference based RL approaches, highlighting the apparent strengths of the proposed method.\n- Specifically, I found the analysis and experiments about transferability and how the proposed approach handles “incomparable policies” to be really interesting. Only transferring the parameters of the estimated reward functions instead of the policies themselves is a really interesting idea and could stand a far more detailed and extensive study in its own right.\n- The introduction of additional simulated treatment domains will be a nice contribution to the community as there are only a small handful of stable simulators available within the RL for healthcare domain.\n\n\n#### **Weaknesses**\n\nI found several weaknesses in the presentation and development of the proposed approach. Throughout the paper concepts are not fully detailed and are only vaguely referred to, seemingly in expectation that the reader implicitly understands or knows what is meant. The major offense in this direction is that the “human” preference-based rewards are never explicitly defined. It’s not clear from the formulation how the returns or observations from the environments are used to compute the rewards under this framing. I read the paper and appendices several times to look for a formalization of these reward definitions and was disappointed to not find them. The closest I could come was in the clinical efficacy and “other factors” metrics. But I didn’t feel that these could be the defined preferences because the authors continually draw a distinction between hand designed rewards and preference-based rewards determined by human intentions. \n\nAlong this point, there are continual references to human preferences or human feedback guiding reward design. However, isn’t this just a human-in-the-loop version of handcrafting a reward? Also, it’s unclear that a clinician will be able to interpret the neural network policies and learned reward representations to provide adequate guidance and preference determination. In the introduction the following statement is made: “Fortunately, qualitative feedback according to human’s preferences can be easily obtained and efficiently leveraged…” This is unfortunately not true in regards to clinical decision making without resulting in overly biased evaluations. Individual clinicians differ in their interpretation of patient conditions as well as the appropriate route of treatment. Without belaboring the point, it’s not clear how establishing the preferences between policies is not just a different form of handcrafting a reward function. Missing from the paper is a discussion about how these preferences would be obtained and integrated into the development of learning a policy. Based on the conclusion it appears that this wasn’t actually incorporated in this paper leaving major questions about the proposed direction and claims made in the paper about handcrafted vs. preference based rewards. Pessimistically, the observed gains with AbRM and SbRM could be attributed to ensembling (not altogether novel given recent advances in offline RL utilizing multiple agents to deal with overestimation within the learned value functions) or more informative reward design. \n\nOn this point regarding overestimation. There are major concerns that the preference based reward and how the individual reward functions (and agents) are trained will largely suffer from overconfidence, a standard problem in RL. Simulators largely cover over this limitation because unrealistic, non-physical behaviors or actions are still supported. In practice, RL in healthcare will be used in offline, off-policy settings where overconfidence and extrapolation to actions not seen in the training set will lead to ineffective and, in the worst case, fatal treatment policies (see Gottesman, et al [2019]). Simulators can only get us so far without specific guarantees about how realistic and representative they are of clinical and physiological reality. These limitations are not discussed or acknowledged in the paper in any way.\n\n*Other points of weakness in the paper:*\n\n- There are extensive experiments performed to demonstrate the advantages of the proposed learning approach yet the discussion and presentation of the results is unfocused and difficult to follow in places. There are four experimental questions raised in the introduction and at the end of Section 4. The presented results do not address or answer these questions directly. A good place to do this would have been in a discussion section at the end of Section 5. This being said, as written, the fourth question is not really a question at all. It rather stands as a statement of what should be an objective of the paper, demonstrating how to build preference-based agents.\n- Section 3 mentions the development of a platform. This however was never discussed or introduced. Typically the concept of a simulation platform denotes a standard tool or API such as OpenAI Gym or Mujoco. This was not formalized and provided a sense that the authors were trying to make unsupported claims about the paper’s significance.\n- How are the environments interacted with? There are no formalized definitions of what the state or action spaces are. It is never explained how long trajectories are (ie. how many treatment decisions are possibly made for an individual patient). What are the relevant parameters of variation within the simulated dynamics? Equations for the treatment dynamics of the cancer and sepsis simulators are presented without any description of what the variables represent. How are patient physiologies varied? (patient types are mentioned in the paper) How different are the physiological responses? Do they admit different policies (aside from the trivial behavior observed in the sepsis simulator)? Provided the lack of variation in the reported results over the different data sets (namely in Figures 2, 3, 4, etc), it appears that there’s no variation nor differences between the training and validation sets. (Why wasn’t the test set evaluations presented even though it’s mentioned in the baselines?)\n- The parameterizations of the reward estimators $R$ are never described. How does performance change based on the “accuracy” of these estimators?\n- What does it mean to sample a patient/subject? Why is the number of patients fixed for each subset of the data? Are these numbers proxies for the number of training episodes used to learn and evaluate policies with? What does it mean to validate an RL policy when it appears policies are optimized from scratch in Figures 2, 4 and etc.?\n- What is the “upper bound” presented in Figure 1?\n- Why are there different time scales between Figure 2 (a) and (b)? Why isn’t expected return plotted in Figure 2(b)?\n- There isn’t a clear separation between the evaluated metrics and the rewards/expected returns. Of course if one increases, the other should as well, right? \n\n\n#### **General Comments**\nOverall, I found this paper to be incomplete despite the extensive experiments. There are some great ideas at the root of the proposed approach to learning policies for treatment recommendation but there was far too little technical development for me to be confident about the stated contributions. There were not enough clear explanations or definitions of important and focal components of the proposed approach. Also, it’s not clear that the simulators are best suited for comparing the proposed approach with the baselines, the fact that the validation and training curves match exactly raise significant doubts that there is any variation between settings within the simulators. This being said, there are also minor concerns about the use of policy gradient approaches within a healthcare setting. Rollouts using inaccurate policies and approximate dynamics models will not be representative of real-world physiologies. Policies developed from non-physical behavior cannot be deployed in practice and are not at all reliable. While PG approaches are admissible in simulators, their suitability is extremely limited in real settings.\n\nUltimately, I feel that my opinion of the paper could be improved if sufficiently clear descriptions of the following concepts were provided:\n- What explicit preferences were used for extracting the rewards for training AbRM and SbRM.\n- Related, what are the target rewards for the learned reward functions $R$?\n- What is the loss function for the agent? (Alg 3., Line 10)\n- How are patient physiologies varied in the simulators? Are the differences meaningful (ie. do they provide unique policies)?\n\n\n\n**Relevant Literature**\nThere were a few pieces of prior work that I expected to be included in this paper given its focus on reward design and the definition of objectives. Within the healthcare space there has been some notable work done to address reward design for RL approaches. Recently, Prasad, et al [ACM CHIL; 2020] looked into admissible reward functions for acute care.\n\nIn the IRL space, reward design has been a large area of research. Among several good papers I wanted to highlight Hadfield-Menell, et al [NeurIPS; 2017] and Shah, et al [ICML; 2019].\n\nFor Multi-objective Markov Decision Processes, I would refer the authors to Lizotte and Laber [JMLR; 2016].\n\nFinally, Yu, et al [arxiv; 2019] put together a decent survey of RL in Healthcare that might help round out the setting of the proposed preference-based approach.\n\n\nPrasad, Niranjani, Barbara Engelhardt, and Finale Doshi-Velez. \"Defining admissible rewards for high-confidence policy evaluation in batch reinforcement learning.\" Proceedings of the ACM Conference on Health, Inference, and Learning. 2020.\n\nHadfield-Menell, Dylan, et al. \"Inverse reward design.\" Advances in neural information processing systems. 2017.\n\nShah, Rohin, et al. \"On the Feasibility of Learning, Rather than Assuming, Human Biases for Reward Inference.\" International Conference on Machine Learning. 2019.\n\nLizotte, Daniel J., and Eric B. Laber. \"Multi-objective Markov decision processes for data-driven decision support.\" The Journal of Machine Learning Research 17.1 (2016): 7378-7405.\n\nYu, Chao, Jiming Liu, and Shamim Nemati. \"Reinforcement learning in healthcare: A survey.\" arXiv preprint arXiv:1908.08796 (2019).", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers agree that this work is promising. The paper is well-grounded in the literature and different aspects of the considered methods are investigated through a variety of experiments. Unfortunately, this paper does not provide sufficient details to allow the reader to understand what has been done nor how to adequately build from it. For example, details in the Appendix lack sufficient formalization of the equations or concepts used to train the preference-based agents. The paper would benefit from clarifications of the method, procedures, and equations used. Beyond that, a major concern lied within the evaluation of the simulated patients across different initializations. Provided that one of the proposed contributions of this paper is a robust simulation platform for RL research within healthcare, it would be important to report convincing results on the patient physiologies admitted by the simulator and characterizing the behaviors of policies learned using this simulator. Finally, issues regarding the structure of the paper, including the split between the main paper and the Appendix, should be resolved before this paper can be published. Notably, the authors should consider elevating important material from the Appendix into the main paper.
Decision: Reject