Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: #### **Summary**
This paper develops a preference-based framework for Reinforcement Learning in Healthcare, focusing on treatment recommendation. The paper highlights several factors where policies trained with preferences may be better suited for use in healthcare applications and extensively evaluates these factors using two simulated care scenarios. 


#### **Assessment**
The foundational concepts underlying this paper are strong and the authors make good points framing the potential advantage of using preference-based rewards over what is termed a “handcrafted reward”. Reward design is an open challenge in Reinforcement Learning, particularly in Healthcare settings, and this paper proposes an intriguing way to perhaps avoid explicitly designing a reward while still providing improved performance. However there are significant gaps in the technical development of the paper that greatly reduce the clarity and perceived significance of the presented work (several examples highlighted in the “Weaknesses” section below). Several claims are made in the introduction and set-up of the experiments that are not fully supported, discussed or justified. Finally, there are some major concerns about the suitability of the simulated environments to adequately evaluate the contributions of the proposed methods (more in the “Weaknesses” section below) in light of the applicability to real world dynamics.


#### **Strengths**
- The paper presents an appealing rationale for the use of preference-based RL in complex sequential decision making problems where reward design is a significant challenge. The paper is firmly grounded in relevant literature. There are several papers that I expected to be included (see Relevant Literature in the “General Comments” section) but their omission is not a critical error.
- The experimental investigation is ambitious and covers most, if not all, relevant questions regarding the utility of preference-based rewards in treatment recommendation. Several relevant baselines are used to compare and contrast the two proposed preference based RL approaches, highlighting the apparent strengths of the proposed method.
- Specifically, I found the analysis and experiments about transferability and how the proposed approach handles “incomparable policies” to be really interesting. Only transferring the parameters of the estimated reward functions instead of the policies themselves is a really interesting idea and could stand a far more detailed and extensive study in its own right.
- The introduction of additional simulated treatment domains will be a nice contribution to the community as there are only a small handful of stable simulators available within the RL for healthcare domain.


#### **Weaknesses**

I found several weaknesses in the presentation and development of the proposed approach. Throughout the paper concepts are not fully detailed and are only vaguely referred to, seemingly in expectation that the reader implicitly understands or knows what is meant. The major offense in this direction is that the “human” preference-based rewards are never explicitly defined. It’s not clear from the formulation how the returns or observations from the environments are used to compute the rewards under this framing. I read the paper and appendices several times to look for a formalization of these reward definitions and was disappointed to not find them. The closest I could come was in the clinical efficacy and “other factors” metrics. But I didn’t feel that these could be the defined preferences because the authors continually draw a distinction between hand designed rewards and preference-based rewards determined by human intentions. 

Along this point, there are continual references to human preferences or human feedback guiding reward design. However, isn’t this just a human-in-the-loop version of handcrafting a reward? Also, it’s unclear that a clinician will be able to interpret the neural network policies and learned reward representations to provide adequate guidance and preference determination. In the introduction the following statement is made: “Fortunately, qualitative feedback according to human’s preferences can be easily obtained and efficiently leveraged…” This is unfortunately not true in regards to clinical decision making without resulting in overly biased evaluations. Individual clinicians differ in their interpretation of patient conditions as well as the appropriate route of treatment. Without belaboring the point, it’s not clear how establishing the preferences between policies is not just a different form of handcrafting a reward function. Missing from the paper is a discussion about how these preferences would be obtained and integrated into the development of learning a policy. Based on the conclusion it appears that this wasn’t actually incorporated in this paper leaving major questions about the proposed direction and claims made in the paper about handcrafted vs. preference based rewards. Pessimistically, the observed gains with AbRM and SbRM could be attributed to ensembling (not altogether novel given recent advances in offline RL utilizing multiple agents to deal with overestimation within the learned value functions) or more informative reward design. 

On this point regarding overestimation. There are major concerns that the preference based reward and how the individual reward functions (and agents) are trained will largely suffer from overconfidence, a standard problem in RL. Simulators largely cover over this limitation because unrealistic, non-physical behaviors or actions are still supported. In practice, RL in healthcare will be used in offline, off-policy settings where overconfidence and extrapolation to actions not seen in the training set will lead to ineffective and, in the worst case, fatal treatment policies (see Gottesman, et al [2019]). Simulators can only get us so far without specific guarantees about how realistic and representative they are of clinical and physiological reality. These limitations are not discussed or acknowledged in the paper in any way.

*Other points of weakness in the paper:*

- There are extensive experiments performed to demonstrate the advantages of the proposed learning approach yet the discussion and presentation of the results is unfocused and difficult to follow in places. There are four experimental questions raised in the introduction and at the end of Section 4. The presented results do not address or answer these questions directly. A good place to do this would have been in a discussion section at the end of Section 5. This being said, as written, the fourth question is not really a question at all. It rather stands as a statement of what should be an objective of the paper, demonstrating how to build preference-based agents.
- Section 3 mentions the development of a platform. This however was never discussed or introduced. Typically the concept of a simulation platform denotes a standard tool or API such as OpenAI Gym or Mujoco. This was not formalized and provided a sense that the authors were trying to make unsupported claims about the paper’s significance.
- How are the environments interacted with? There are no formalized definitions of what the state or action spaces are. It is never explained how long trajectories are (ie. how many treatment decisions are possibly made for an individual patient). What are the relevant parameters of variation within the simulated dynamics? Equations for the treatment dynamics of the cancer and sepsis simulators are presented without any description of what the variables represent. How are patient physiologies varied? (patient types are mentioned in the paper) How different are the physiological responses? Do they admit different policies (aside from the trivial behavior observed in the sepsis simulator)? Provided the lack of variation in the reported results over the different data sets (namely in Figures 2, 3, 4, etc), it appears that there’s no variation nor differences between the training and validation sets. (Why wasn’t the test set evaluations presented even though it’s mentioned in the baselines?)
- The parameterizations of the reward estimators $R$ are never described. How does performance change based on the “accuracy” of these estimators?
- What does it mean to sample a patient/subject? Why is the number of patients fixed for each subset of the data? Are these numbers proxies for the number of training episodes used to learn and evaluate policies with? What does it mean to validate an RL policy when it appears policies are optimized from scratch in Figures 2, 4 and etc.?
- What is the “upper bound” presented in Figure 1?
- Why are there different time scales between Figure 2 (a) and (b)? Why isn’t expected return plotted in Figure 2(b)?
- There isn’t a clear separation between the evaluated metrics and the rewards/expected returns. Of course if one increases, the other should as well, right? 


#### **General Comments**
Overall, I found this paper to be incomplete despite the extensive experiments. There are some great ideas at the root of the proposed approach to learning policies for treatment recommendation but there was far too little technical development for me to be confident about the stated contributions. There were not enough clear explanations or definitions of important and focal components of the proposed approach. Also, it’s not clear that the simulators are best suited for comparing the proposed approach with the baselines, the fact that the validation and training curves match exactly raise significant doubts that there is any variation between settings within the simulators. This being said, there are also minor concerns about the use of policy gradient approaches within a healthcare setting. Rollouts using inaccurate policies and approximate dynamics models will not be representative of real-world physiologies. Policies developed from non-physical behavior cannot be deployed in practice and are not at all reliable. While PG approaches are admissible in simulators, their suitability is extremely limited in real settings.

Ultimately, I feel that my opinion of the paper could be improved if sufficiently clear descriptions of the following concepts were provided:
- What explicit preferences were used for extracting the rewards for training AbRM and SbRM.
- Related, what are the target rewards for the learned reward functions $R$?
- What is the loss function for the agent? (Alg 3., Line 10)
- How are patient physiologies varied in the simulators? Are the differences meaningful (ie. do they provide unique policies)?



**Relevant Literature**
There were a few pieces of prior work that I expected to be included in this paper given its focus on reward design and the definition of objectives. Within the healthcare space there has been some notable work done to address reward design for RL approaches. Recently, Prasad, et al [ACM CHIL; 2020] looked into admissible reward functions for acute care.

In the IRL space, reward design has been a large area of research. Among several good papers I wanted to highlight Hadfield-Menell, et al [NeurIPS; 2017] and Shah, et al [ICML; 2019].

For Multi-objective Markov Decision Processes, I would refer the authors to Lizotte and Laber [JMLR; 2016].

Finally, Yu, et al [arxiv; 2019] put together a decent survey of RL in Healthcare that might help round out the setting of the proposed preference-based approach.


Prasad, Niranjani, Barbara Engelhardt, and Finale Doshi-Velez. "Defining admissible rewards for high-confidence policy evaluation in batch reinforcement learning." Proceedings of the ACM Conference on Health, Inference, and Learning. 2020.

Hadfield-Menell, Dylan, et al. "Inverse reward design." Advances in neural information processing systems. 2017.

Shah, Rohin, et al. "On the Feasibility of Learning, Rather than Assuming, Human Biases for Reward Inference." International Conference on Machine Learning. 2019.

Lizotte, Daniel J., and Eric B. Laber. "Multi-objective Markov decision processes for data-driven decision support." The Journal of Machine Learning Research 17.1 (2016): 7378-7405.

Yu, Chao, Jiming Liu, and Shamim Nemati. "Reinforcement learning in healthcare: A survey." arXiv preprint arXiv:1908.08796 (2019).
review_title: Interesting concepts and extensive experimental justification; Unfortunately incomplete and unclear technical development

final_score: rating: 4: Ok but not good enough - rejection
review_content: 
This is a relatively comprehensive evaluation. The authors know the related literature, and a number of experiments are presented to show different aspects of the methods. The work is promising. Writing is great, and the authors can write beautiful sentences, but the overall structure has some flaws.

I am not happy with the way the authors split their manuscript into the main paper and the appendix. The current version of the main paper is not self-contained, and it does not provide minimum explanation of the methods. I was very perplexed after reading the main part before I read the appendix because the core algorithm is not explained in the main part of the paper. It was hard to follow the experiments in sec. 5, when one is unsure how the methods work. The experiments section itself makes frequent jumps to the appendix. This content would be much more appropriate for a journal paper, in which most of the appendix would be nicely integrated with the main body.

It is not entirely clear what is the new contribution in this paper. When algorithm 1 is introduced in section 4.2, the authors do not say which components are new or how they extend the existing literature.

The EVALUATEPREFERENCE procedure in algorithm 1 is not defined, yet it is a very important component. The pseudocode of algorithm 1 indicates that entire trajectories are compared to elicit preferences, but this may lead to suboptimal behaviour. Note that if two policies are suboptimal, they may make suboptimal decisions in different states, so preferences at the level of individual states would be more appropriate. This important component of the algorithm is unclear to me in the paper. 

Following on the previous question, I should ask what the human expert would need to do if this algorithm was applied in practice. Specifically, what would be displayed to the human expert, and what the human expert would need to do or what they would need to compare and judge? The simulator (which would be the real environment) and the algorithms are tightly entangled in the appendix, and for this reason it is hard to imagine how this solution would work with real data and real human input.

Since the authors' goal is to show that their method provides better rewards than the handcrafted rewards known in the existing literature, perhaps some discussion of those rewards could be added. How were those rewards derived? Did the human designers have the same objectives in mind? Could one derive a better handcrafted reward for these experiments? Sufficient evidence should be provided that the handcrafted rewards used in comparisons are not trivial. I am saying this because the preference-based reward is derived from preferences that match the quantities that constitute the objective of learning, so it is not that surprising that preference-based reward leads to better performance according to the corresponding objectives.

Small issues:

In sec. 1, the authors mention open access large-scale Electronic Health Records. Examples of such publicly available records should be provided.

The authors should clarify what they mean in this sentence "the linearly weighted reward function induces negative interface between objectives".

Several symbols are not defined in section 2.1, e.g., a1, a2, m1, m2.

Formatting on p. 3 is incorrect.

What are "hill" equations on p. 3?

It would be good if the authors could explain equations that are in lines 5 and 8 in Algorithm 2. What would be an intuitive explanation of L in this algorithm? 



This paper addresses a very important problem, and the results could be significant. I think that the future readers would be confused if this paper was accepted in this form. It would be sensible to turn it into a longer journal paper and clarify the key components.


review_title: When reinforcement learning is considered for treatment recommendation, there is a trade-off between survival rate and other factors and side effects. In this paper, the authors use preferences to learn the reward that will guide the RL agents towards behaviour that matches the objectives that are used to assess agents' performance.

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper describes a preference based reinforcement learning framework for treatment recommendation tasks. The authors use a model-based environment to simulate the progression of disease and propose a preference based approach. They claim they can use this framework with any off-the-shelf RL algorithm to generate better survival results. The experiments are carried out with policy gradient. The authors also compared several different ways to design reward functions, and propose to use incomparable samples with a 0.5/0.5 distribution in the Bradley-Terry model, instead of discarding such samples, to improve the performance. The paper is generally well-written and easy-to-follow. Despite the technical contributions listed above, I would suggest rejection due to the following concerns.

1. Innovation
The paper lacks citations and comparisons with several recent works in this field, such as [r1, r2]. The approach proposed is similar and the innovation seems marginal. I suggest the authors include a comparison of methodology and, if possible, experiments comparing their approach with state of the art preference based RL algorithms in the revision.

2. Lack of implementation details
There are a few details I didn't find reading the manuscript, for example, how to query human responses on line 12 of Algorithm 1, what is the guideline for human experts to rate different treatment plans, implementation details and hyper-parameters used in the agent learning algorithm. The lack of such details makes it difficult for other researchers to reproduce the results shown in the paper.


[r1] Christiano, Paul F., et al. "Deep reinforcement learning from human preferences." Advances in Neural Information Processing Systems. 2017.
[r2] Ibarz, Borja, et al. "Reward learning from human preferences and demonstrations in Atari." Advances in Neural Information Processing Systems. 2018.
review_title: Lacks comparison with sota

Meta-review: Reviewers agree that this work is promising. The paper is well-grounded in the literature and different aspects of the considered methods are investigated through a variety of experiments. Unfortunately, this paper does not provide sufficient details to allow the reader to understand what has been done nor how to adequately build from it. For example, details in the Appendix lack sufficient formalization of the equations or concepts used to train the preference-based agents. The paper would benefit from clarifications of the method, procedures, and equations used. Beyond that, a major concern lied within the evaluation of the simulated patients across different initializations. Provided that one of the proposed contributions of this paper is a robust simulation platform for RL research within healthcare, it would be important to report convincing results on the patient physiologies admitted by the simulator and characterizing the behaviors of policies learned using this simulator. Finally, issues regarding the structure of the paper, including the split between the main paper and the Appendix, should be resolved before this paper can be published. Notably, the authors should consider elevating important material from the Appendix into the main paper.
Decision: Reject