Nested Policy Reinforcement Learning for Clinical Decision Support
Authors: Aishwarya Mandyam
Andrew Jones
Krzysztof Laudanski
Barbara Engelhardt
Venue: ICLR 2022 Submitted (2022)
Keywords: machine learning for healthcare
reinforcement learning

Abstract: Off-policy reinforcement learning (RL) has proven to be a powerful framework for guiding agents' actions in environments with stochastic rewards and unknown or noisy state dynamics. In many real-world settings, these agents must operate in multiple environments, each with slightly different dynamics. For example, we may be interested in developing policies to guide medical treatment for patients with and without a given disease, or policies to navigate curriculum design for students with and without a learning disability. Here, we introduce nested policy fitted Q-iteration (NFQI), an RL framework that finds optimal policies in environments that exhibit such a structure. Our approach develops a nested $Q$-value function that takes advantage of the shared structure between two groups of observations from two separate environments while allowing their policies to be distinct from one another. We find that NFQI yields policies that rely on relevant features and perform at least as well as a policy that does not consider group structure. We demonstrate NFQI's performance using an OpenAI Gym environment and a clinical decision making RL task. Our results suggest that NFQI can develop policies that are better suited to many real-world clinical environments. 

Under review as a conference paper at ICLR 2022
NESTED POLICY REINFORCEMENT LEARNING
Anonymous authors
Paper under double-blind review
ABSTRACT
Off-policyy reinforcement learning (RL) has proven to be powerful framework for guiding agents' actions in environments with stochastic rewards and unknown or noisy state dynamics. In many weal-world settings, these agents must operate in multiple environments, each with slightly different dynamics. For example, we may be interested in developing policies to guide medical treatment for patients with and without given discase, or policies to navigate curriculum design for students with and without a learning disability. Here, we introduce nested policy fitted Q-iteration (NFQD, ronments that exhibit such a structure. Our approach develops a nested Q-value function that takes advantage of the shared structure between two groups of obser- vations from two separate environments while allowing their policies to be distinct from one another. We find that NFQI yields policies that rely on relevant features and perform at least as well as policy that does not consider group structure. We demonstrate NFQI's performance using an OpenAl Gym environment and clin- ical decision making RI task. Our results suggest that NFQI can develop policies that are better suited to many real-world clinical environments.
RL framework that finds optimal policies in envi-
INTRODUCTION
Off-policy reinforcement learning (RL) has proven to be a powerful framework for guiding agents' actions in environments with stochastic rewards and unknown or noisy state dynamics [3] 2). Pre- vious off-policy RL algorithms (9) learn single optimal policy designed to maximize agents' long- term expected rewards. However, such a learning scheme is not always ideal. many real-world settings, agents' environments can be divided into multiple groups for the same task. These groups may share rewards but have diffcrent state dynamics, which often entails distinct optimal policies.
For example, say that are designing clinical decision support algorithm to treat ischemic stroke in two groups of patients: one group has a large number of patients with healthy kidneys and the other has a smaller number of patients with renal (kidney) disease. Note that there are similarities in the treatment plans for these two groups; in particular, both sets of patients exhibit the effects of disturbed blood flow the brain due to stroke, and may require similar procedures. However, patients with renal disease must follow a treatment plan that considers the inability of their kidneys to properly filter out toxins. Traditional off-policy RL algorithms would learn a single policy that does not consider the underlying presence of renal disease. This policy is likely to overfit to the patients with functioning kidneys and overlook any known differences in optimal treatment plans. In such scenarios, it would be beneficial to learn distinct policies for pre-defined groups of patients, while leveraging any shared environmental structure and reward functions between the groups to ensure large and effective sample size.
To address this problem, we introduce nested policy fitted Q-iteration (NFQD, an off-policy RI. al- gorithm that extends fitted Q-iteration (FQI) @ to account for nested environments. Here, nested refers to an environment structured with two predefined groups that share some characteristics; we will refer to the two groups as background and foreground. The background group often contains far more samples. In the clinical example above, the background group includes patients with healthy kidneys and the foreground group includes renal disease patients. Both groups share reward func- tion, in this case successfully treating a stroke, but have different state dynamics.
The contributions of this paper off-policy RL in the presence of nested environments are threefold:

---

Under review as a conference paper at ICLR 2022
1. We introduce NFQI, supervised learning method that uses nested Q-value function to learn policies that are suited to nested environments. NFQI concurrently models both the background and foreground datasets and is agnostic to modeling choices.
2. We develop training procedure inspired by transfer learning (31) to fit this nested Q-value function and allow for group imbalance in the data.
3. We show that NFQI generates policies that outperform baseline methods set of quan- titative and qualitative metrics and rely on relevant state features to do so.
This paper proceeds follows. Section 2 discusses related work. In Section 3 we present frame- work for NFQI. In Section [ we show results for NFQI in simulations and hospital data. Section 5 concludes and discusses potential future work.
RELATED WORK
Multi-task RL. The goal of multi-task RL is to train agent to perform optimally at multi- ple tasks (8) Recent work has developed a method to efficiently learn a task-conditioned policy (als,,) 2), where a is action, 8 is a state, and is task embedding (30). This approach has been observed to perform better than single-task learning because uses the relationships between tasks to improve performance for individual tasks (27). However, the goals of multi-task RL differ from those of NFQI: Multi-task RL optimizes performance for multiple tasks, each with a different Markov decision process (MDP), whereas NFQI most appropriate to compare two environments with identical tasks and reward functions but different transition dynamics. Additionally, the neural networks typically used for multi-task RI. 30 require large training sets that are not always avail- able in the medical and learning applications NFQI was built to address. As such, these networks are likely to suffer from overparameterization in these applications.
Meta RL. Meta RL algorithms train agents to solve new tasks efficiently based experience with other similar tasks. These approaches infer the task from a distribution p(T) using a small set of observations, rather than through explicit task embedding. Much of the effort in meta RL is focused on task design as opposed to model tuning, to make this inference process easier (1)). Other work has shown that meta RL can be appropriate for offline RL (1.. Like multi-task RL, meta RL is different from NFQI because it trains agents to perform multiple tasks with different state and action spaces. Unlike meta RL and multi-task RI,, NFQI expects all samples to arise from MDPs that have the same state and action spaces and follow the same reward function. Since the dynamics of the two environments vary, the MDPs will have different state transition functions, entailing different optimal policies for the groups
Transfer learning. Transfer learning methods fine-tune pre- traitrained model for target dataset. Often, the target dataset is relatively small: transfer learning using a larger and similar dataset may result in better performance than training model on just the target dataset (6). common way to use these methods is to first train neural network using the large dataset; then, some of the network layers are frozen, and the rest of the network is fine-tuned by training on the target dataset. way, the network is adapted to the target problem by learning a nonlinear map from the solution estimated for the larger dataset to the outcome for the target dataset. This strategy can be applied to reinforcement learning (31) and may improve resource usage when the target dataset is small (4). The two datasets that NFQI uses are likely to have substantial sample size imbalance; for this reason, train the parameters for NFQI using procedure inspired by transfer learning.
this
3 METHODS
In this section we present nested policy fitted Q-iteration (NFQI), off-policy RL algorithm for environments that have a nested group structure. We first review Q-learning and fitted Q-iteration (FQI) and then describe how to extend them to NFQI.
3.1 NOTATION AND PRELIMINARIES
A Markov decision process (MDP) is a discrete- time stochastic control process defined by a 4- tuple (S,A, P, Rt,, where S is the state space; A is the set of actions; P(s (st, at) is a function
2

---

Under review as a conference paper at ICLR 2022
governing state transition probabilities from time t to t + 1; and R S R is reward function.
We assume a finite number of actions. A policy н : S - A is a mapping from states to actions that describes an agent's strategy within MDP. In general,
assume S, E R' and a, E R7.
3.2 Q-LEARNING AND FITTED Q-ITERATION (FQI)
Fitted Q-iteration (FQI) is an off-policy RL algorithm based on Q-learning (9,29 that is designed to find an optimal policy for given MDP. Q-learning approaches define function Q SS x A + R that maps state-action pairs to an estimate of the cumulative expected reward obtained by tak- ing particular action in particular state. For given policy л time step I, Q is defined as (= (x) r + (V*T()) where the expectation is taken over the possible next states given the transition probabilities P. V is the value function corresponding to π, and Y [0, 1] is a dis- count factor. The principle of Bellman optimality (5) states that policy x' is optimal for given MDP if and only if, for every state Broadly, Q Q-learning algorithms attempt to approximate the Q function using agents' observed inter- actions with an environment, as the state transition function is typically not directly observable. This approximation is used to recommend optimal actions for given state. The data used to estimate the Q function typically consist of samples representing steps taken within an MDP. In particular, a sample time is 4-tuple (st, a, 7t) Te) representing the agent's state, chosen action, observed next state, and observed reward.
S, holds that V**(s) = max, R(s, a) + y "V= (s').
FQI approximates Q using offline batch optimization approach (9) Specifically, it first posits family of functions fo, a), parameterized by 8, that are meant to approximate the expected future reward after taking action from state s. Common choices for f are regression trees (9) and neural networks (25). FQI then employs an iterative algorithm that alternates between updating the current Q-values and fitting a regression problem that optimizes the parameters 0. Suppose we have set of T samples from an MDP, represented as a set of one-step transition tuples {Se, a,, St+1, (1) 1?) On step of the algorithm, FQI first updates its estimate of the Q-function according to the Bell- man equation: (S,, ax) = r++1 + y max Sêx_1 (Se+ a') for = I...., ,T.
The second step minimizes loss measuring the discrepancy between the current Q- 2-function estimate and the ap- proximating function f. The optimization problem is mine, Em L (Qu(s) a²"),
) Soc (S7 a()) where C(y, 7 is the loss function comparing the "true" value to predicted value 7. After K iterations, FQI's approximation to the optimal policy is given by x (s)) = arg max fox (s, a').
3.3 NESTED POLICY FQI
We now introduce NFQI, an extension of FQI that estimates group-specific policies and accounts for nested group structure in the data. NFQI is framework for off-policy reinforcement learning that can use an arbitrary function class to approximate the Q function. Below, we describe the general NFQI framework with minimal assumptions about its parameterization. Additional experiments show that neural networks are well suited to parameterize Q (see Appendix 6.2 and Appendix 6.5).
Recall that nested datasets are structured with two predefined groups the background, and fore- ground. To account for nested datasets, NFQI imposes structure on the family of functions used to approximate the Q function. We define the approximating function f as
(1) where Is is function modeling the shared Q-value structure between the background and fore- ground samples, 9, models foreground-speee
f(s, a, 2) = (s. a) + (1 (==1))97($ a),
structures, and 1 (2-1) is an indicator function that returns when z 1 (foreground) and 0 otherwise (background). Note that, despite the uncon- ventional structure of the approximating function, it is still guaranteed to converge to the Bellman optimal Q-value function; see Appendix 6.T for further details.
NFQI uses two-stage training procedure inspired by transfer learning (Appendix Algorithm T In the first stage, we train ga using all of the foreground and background training samples, while ignoring the foreground-speeffic model gf. Note that this stage is equivalent to setting = = 0 for all samples (including foreground samples) temporarily. In the second stage, we train the foreground- specific function 91 using only foreground training samples. Under this approach, we first estimate the shared structure the network using both groups of samples in our training set, and we fine-tune the network's foreground predictions in ff using just the foreground training samples.
3

---

Under review as a conference paper at ICLR 2022
4 EXPERIMENTS
We demonstrate the performance of NFQI through the following experiments. Our experiments seek to answer the following questions:
1. How well does a joint policy learned using NFQI perform in comparison to two policies learned independently (one for each dataset) using FQI?
2. How does joint policy learned using NFQI perform compared a joint policy learned by applying FQI or transfer learning to the union of background and foreground datasets?
3. Do the policies learned by NFQI rely on group-specifie features to make predictions?
4. Are policies learned using NFQI able to achieve good performance when the sample size of the background dataset is much larger than that of the foreground dataset?
5. When there is no group structure in the dataset and the application of NFQI is technically inappropriate, does NFQI perform as well as standard FQI?
We present results from NFQI applied to a nested Cartpole environment 7 and to a clinical decision support task using MIMIC-IV, a benchmark electronic health record (EHR) dataset (44
4.1 NESTED CARTPOLE
We first demonstrate the performance of NFQI using the Cartpole environment, as implemented in the OpenAl gym (7). The Cartpole environment consists of an unstable pole attached to the top of a cart. An agent's 's objective is to keep the pole balanced upright (in this case for 1000 steps); this can be done by strategically applying a force on the cart the left right at each time step (see Appendix 6.44for the full MDPs and Appenxix (6.6) for training details).
In this study, we adapt the Cartpole environment to accommodate nested structure. We call our setup the "nested Cartpole" environment. The background environment is the original Cartpolc setup, while the foreground environment includes a constant force of e Newtons that pushes the cart to the left. Unless otherwise specified, = 5. Although the objectives for the two simulators is identical, an agent will require a different policy to succeed in the background and foreground envi- ronments because the dynamics of each system are different. Intuitively, an agent in the foreground environment will need to develop a policy that counteracts the consistent leftward force.
To construct datasets we sample Cartpole episodes from the foreground and background environ- ments. Each trajectory starts in a random state and chooses actions uniformly at random until the pole falls over. We use these trajectories to train NFQI and other off-policy algorithms. We evalu- ated the performance of each policy by applying it in the nested Cartpole simulator and computing the number of steps the cart maintains the pole in the goal state (with maximum of 1000 steps).
4.1.1 NFQI IMPROVES PERFORMANCE WHEN THE FOREGROUND SAMPLE SIZE IS SMALL
We first sought to benchmark NFQI's performance against FQI. To do so, we generated a training set of trajectories that reflect the group imbalance that we expect see in the nested setting. In par- ticular, this dataset contains 200 background samples and 50 foreground samples. The foreground environment has force of c Newtons pushing the cart the left. We trained FQI separately on the background and foreground dataset; this yielded two independent policies. We fit NFQI with both datasets using the procedure in Appendix Algorithm I We tailored the neural networks for each algorithm so that they both use the same number of parameters (see Appendix (.6), thus pre- cluding either algorithm from having a more computationally expressive family of functions. We evaluated each policy 50 times in each of the foreground and background environments, computing the mean and confidence interval of these repetitions.
We find that the policy learned by NFQI performs substantially better than FQI in the foreground environment and slightly better than FQI in the background environment (Fig. I. FQI's poor performance in the foreground environment can likely be explained by the small training set size. However, NFQI is able to leverage information from both foreground and background trajectories, which allows it to overcome this limited sample size. This result suggests that NFQI is viable
4

---

Under review as a conference paper at ICLR 2022
algorithm for learning optimal policies for two groups, especially when the training set sizes unbalanced between the groups.
1000
800
8000
Steps Survived)
600
600
400
Steot
400
200
200
FgI ned on background
NFQ trained
-QI trained on foreground
NFOI traincd background and foreground
background and foreground
(a) Background environment performance
(b) Foreground environment performance
Figure Performance of NFQI and FQI in the background (Panel and foreground (Panel b) environments. The bar height represents the mean number of steps in the goal state (with a maximum of 1000 steps); ticks represent 95% % confidence intervals over repetitions. NFQI trained with both datasets performs better on test data from each of the environments, and substantially better for the foreground test data.
4.1.2 NFQI OUTPERFORMS FQI AND TRANSFER LEARNING
Given that we want to train policy using both datasets the same time, we next compared NFQI in the nested Cartpole environment to two other bespoke approaches: FQI trained jointly with tra- jectories generated from both the foreground and background environments, and transfer learning- The FQI and transfer learning algorithms find single policy that does not consider a group label at training or inference time (further details in Appendix (6.6).
Cartpole performance, background
Cartpole performance, foreground
1000
Steps Surviee
600
600
400
400
200
200
(Foorce Left)
[Forrc
(a) NFQI performance, background dataset
(b) NFQI performance, foreground dataset
Figure 2: NFQI outperforms related algorithms in nested Cartpole environment. Increasing the force on the cart (x-axis) increases the difference in the foreground and background environments. As the environments become distinct, NFQI enables agents survive longer (y-axis) than FQI and transfer learning for both the background (Panel a) and foreground (Panel b) data.
To evaluate the algorithms, we created eleven variations of the nested Cartpole environment, each corresponding to different value for the constant force applied in the foreground, 10.1....,110 10). We evaluate each algorithm times for each environment.
We found that NFQI outperforms FQI and transfer learning in this setting, especially as the fore- ground and background environments become more different from one another (Fig. 2). When the foreground and background environments were the same (i.e., 0), NFQI and FQI performed similarly, while transfer learning performed slightly worse. As the background and foreground dynamics became more distinct (i.e., increased), NFQI outperforms FQL. In particular, NFQI maintained near wear-perfect performance in the background, and the performance degraded more slowly in the foreground environment compared to competing approaches. These results suggest that NFQI
5

---

Under review as a conference paper at ICLR 2022
NFQI background SHAP values
NFQI foreground SHAP values
righ
Cart position
Cart velocity
Cart velocity
angie
Pole angle
Feature
Polo velocity
Pale valocity
Action
Action
0.03
0.00
0.05
0.00
0.15
9.25
0.25
0.50
Low
0.0
SHAP value (impact on model outputt
SHAP value (impact on model outputt)
Figure 3: SHAP plots for background (Panel a) and foreground (Panel b) samples from the Cartpole environment. Each point represents the SHAP value for single sample and single feature. The magnitude of SHAP values is similar in both contexts, but foreground SHAP values shifted right, suggesting that the foreground policy is compensating for the foreground-specific dynamics.
is able to model the differences between the background and foreground while exploiting shared structure to improve its performance in both environments.
4.1.3 NFQI YIELDS POLICIES THAT RELY ON RELEVANT FEATURES
In addition to obtaining high-performancd policies, it is also desirable to have these policies rely on relevant features. To investigate this, we computed estimates of feature importance for both the shared and sengrooudd-peciffe model components using SHapley Additive exPlanations (SHAP), a game theoretic approach to quantify the extent to which the features of each sample impact the model's predictions (16). Features with larger SHAP absolute values indicate a greater predictive power. In the nested Cartpole environment, we compute SHAP values for each of the five features used estimate Q-values: cart position, cart velocity, pole angle, pole velocity, and action.
We observed several patterns in the SHAP values for NFQI that agree with prior intuition. First, we found that across all samples in both datasets, the features rank in the same order of importance. This ordering corresponds to the mean SHAP value across samples. We see that cart position shows the highest SHAP values, meaning it the most predictive feature, while action is the least predictive (Fig. 3. The model's emphasis on carl position is likely due to the constant force being applied in the foreground Cartpole environment; NFQI's policy adapts to the relative importance of this feature for both environments. We see that the foreground cart position SHAP magnitude is much larger relative to that in the background, indicating that the foreground policy is compensating for the boregroun--ppecffe dynamies, which include constant force pushing the cart towards the left. These results suggest that the policies found by NFQI rely on relevant features, as demonstrated in part by the SHAP values.
4.1.4 NFQI IS ROBUST TO GROUP SIZE IMBALANCE
In many real-world settings, we have access far less data in the foreground condition than in the background. For example, it is much easier to collect medical records from healthy (background) patients than from patients with specific, chronic disease (Foreground). Thus, it is important that nested RL methods be resilient to group imbalance and able to leverage the statistical strength of the background data to estimate foreground-specific structure even when foreground data are rare. In order to test NFQI's performance with imbalanced data, we conducted an experiment with im- balanced group sizes in the nested Cartpole environment. Similar previous experiments, we gave the foreground different dynamics than the background by adding a constant leftward force of c = 5 Newtons. Here, we fix the total number of training trajectories across both conditions to be 400, and we set the fraction of samples that come from the foreground data to 10%, 20%, 30%, 40%, and 50% (where 50% corresponds to balanced sizes). We choose these fractions because they reflect percentages of imbalance that we in medical data. We fit NFQI before, using neural network as our Q-value approximation function. For comparison, we also train FQI on the combined dataset of background and foreground trajectories. We repeat this experiment 50 times.
NFQI outperforms FQI in this Cartpole setting with imbalanced group sizes. Specifically, across the range of data imbalance, the policies found by NFQI perform better than the FQI policies in both

---

Under review as a conference paper at ICLR 2022
Performance with imbalanced datasets
800
FQI (joint)
NFQI
Steps survived
Dataset
Background
Foreground
400
200
0.1
0.2
0.4
0.5
Fraction foreground samples
Figure 4: NFQI is robust to imbalance in foreground and background sample sizes. We fix the total number of training trajectories across both conditions to be 400, and we set the fraction of samples that come from the foreground data to 10%, 20%, 30%, 40%, and 50% (where 50% corresponds to balanced sizes). Error bars correspond to 95% confidence intervals. NFQI maintains superior performance to FQI despite increasing levels of class imbalance. The foreground dataset is harder to model than the background, but a nested policy better reflects the environments.
the background and foreground (Fig. 4.. This is likely due to the fact that NFQI accounts for the group structure in the dataset, while also exploiting the shared structure between the background and foreground to increase the effective sample size. On the other hand, FQI must model the groups with a single joint model, resulting in poor performance for both datasets. This result implies that NFQI is successful in common setting in which we have access to small proportion of foreground samples relative to the background samples.
4.1.5 NFQI ONLY LEARNS DIFFERENT POLICIES IF THE ENVIRONMENTS ARE DIFFERENT
When there is no meaningful group structure in the data, we want NFQI to find the same optimal policies for two (identical) groups We demonstrate that NFQI behaves in this way by testing its performance when there is nested structure. We call this experiment the "structureless" test. We first generate a dataset from single simulator and give it false group structure by randomly assigning group labels to samples. Then, we fit NFQI to this dataset and analyzed the policies for each group, again using SHAP plots.
Structureless test
SHAP values for structureless test
Foreground
Background
8000
velocity
Count
6000
angle
4000
Pole valocity
2000
Action
Receground
Foreground (150)
0.19
Stops Survived
mean[[[HAP valuel)
(a) Structureless test for the Cartpole environment.
(b) SHAP plot for the structureless test.
Figure 5: NFQI does not estimate practically different policies for groups when there is no group structure. Here, we find that NFQI finds policies that yield similar performance for both the foreground and background group (Panel a) and that NFQI uses sample features similarly across groups to predict rewards (Panel b).
We find that NFQI finds two policies that show similar behavior between the artificial background and foreground datasets (Fig. Sab. The policies use the sample features similarly to make predictions, with the same rankings of mean absolute SHAP values across artificial groups (Fig. [bb. These results suggest that NFQI learns indistinguishable policies when the samples originate from the same simulator with identical dynamics.
7

---

Under review as a conference paper at ICLR 2022
4.2 CLINICAL DECISION MAKING TASK USING ELECTRONIC HEALTH RECORDS (EHR)
After validating NFQI on the Caripole environments, next set out to test NFQI's ability identify
more complex policies for patient treatment in a hospital setting. To do so, we leveraged electronic health record (EHR) data from the Medical Information Mart for Intensive Care (MIMIC) series (22
26 15 14),
which contains de-identified records from patient hospital admissions.
2
1.5
Creatinine
Action
Creatinine
Action
1.0
1.0
6
8
0.5
0.5
10
7.0
3.0
3.5
70
0.0
Fotassium Level Blood (mmol/L)
Potassium Level Blood (mmoliL)
(a) FQI policy non-renal patient.
(b) FQI policy for renal patient.
2
1.5
Creatiinee
Creatinine
Action
1.0
8
3.0
3.5
0.0
1.0
0.0
Potasslum Level In Blood (mmolt.)
Potassium Level Blood (mmoli))
(c) NFQI policy for non-renal patient
(d) NFQI policy for renal patient
Figure 6: Visualizing FQI and NFQI policies for non-renal and renal patients. Heatmaps indicate the policy's repletion recommendation for patients with given potassium and creatinine level.
In this experiment, we aim to identify policy for maintaining healthy levels of electrolytes in hospital patients. Determining when to prescribe or "replete" electrolytes is a difficult task; the overuse of medication intended to balance electrolyte levels in patients has been linked to patient mortality and complications post-surgery (28). Here, we use NFQI to identify electrolyte repletion policies for two groups of intensive care unit (ICU) patients that require different repletion strategies: a foreground group of 123 patients who have kidney disease (renal patients) and background group of 400 patients with healthy kidney function (non-renal patients). Our goal is determine the dosage of potassium to administer to a patient across the duration their hospital stay. We discretize the action space into three levels of repletion: repletion, low repletion, and high repletion.
Table I: Evaluating RL algorithms using action matching. We evaluate FQI and NFQI on their ability to predict the clinician's 's actions on test samples from the foreground (FG) and background (BG) datasets. We report accuracy (%) and F1 score across the three classes of repletion, both with 95 % confidence intervals.
Algorithm
FG Accuracy (%)
FG Score
BG Accuracy (%)
BG Fl Score
FQI
36.09 +: 8.5
0.26 + 0.05
27.7 + 6.6
0.20 + 0.03
NFQI
45.5 12.2
0.25 + 0.04
35.7 + 6.3
0.26 + 0.03
Building off previous RL approaches for this task 244) we discretize patient trajectories into 6- hour intervals. Each sample contains information about the patient's vitals, labs, and prescribed medicines during that interval, well as comorbidities and metadata, such age. The reward function for this dataset is a manually parameterized function of the patient state and repletion action that produces higher values when patient's measured potassium level is within a healthy range (3.5 and 4.5 mmol/L), and lower reward values otherwise (24). This reward function must be constructed manually because there is notion of reward in EHR records, and cannot derive an empirical one because it is not safe or ethical to evaluate a policy on patients (see Appendix 6.7 for further details).
8

---

Under review as a conference paper at ICLR 2022
We fit NFQI and FQI to this dataset and examine the resulting policies, performing both quantitative and qualitative analyses on our policies. We quantitatively evaluate a policy's 's performance based on its level of agreement with doctor's 's actions in test data (Tab. (Tab.[].. We find that the policies estimated using NFQI better reflect doctor's actions than those estimated with FQI.
To give better insight into why NFQI quantitatively outperforms FQI, visualize the policies qualitatively through heatmaps. We create synthetic patient states by densely sampling a grid with two features, potassium and creatinine, while holding all other state features fixed. Creatinine levels capture the kidney's ability to filter out toxins (18); higher levels of creatinine indicate worse kidney function. Given patient's state features, we then use the trained Q-value function select the action that generates the highest expected reward. NFQI recommends higher levels of repletion for renal patients versus non-renal patients, especially when the patient's potassium is below the healthy range (19) (Fig. (6). Moreover, NFQI's 's policy recommends more repletion for renal patients with higher creatinine level, indicating that electrolyte repletion should increase as kidney function worsens. Meanwhile, FQI's policy shows little difference between these two patient groups, and recommends lower levels of repletion across the grid. These results suggest that NFQI is able to discriminate between the patient groups and act according to prior intuition.
is also imperative that NFQI policies are in- terpretable since the actions recommended can affect other aspects of a patient's physiology. Thus, we studied which features NFQI found most predictive in the foreground and back- ground policies. To do so, we computed SHAP values from each group (Fig. 7 We find that NFQI identifies clinically relevant characteris- ties to inform both foreground and background policies. Patients with renal disease tend to have higher creatinine values and fluctuating electrolyte levels; the foreground policy for re- nal patients relies on potassium and creatinine measurements more than the background pol- icy. This suggests that the NFQI policies reflect the group-specifis dynamics, while also sharing strength across the foreground and background samples.
NFQI SHAP values
Magresium (non-M)
Potassium
Patient weight
Hours Magnesium (IV)
10:05
Magnesium (MI
Non-renal
Hours Phosphate (IV)
1001
Reral
Hemoglobin
+1.03
Creatinine phosphokinese
+6.02
Creatinine
+3.00
Sum of 56 other features
1031
MSaA/|SHAP valuel)
Figure 7: NFQI mean SHAP values for renal (blue) and non-renal (red) patients. Y-axis shows clinically relevant covariates.
DISCUSSION AND FUTURE WORK
In this work, we present nested policy fitted Q-iteration (NFQI), an off-policy RL framework that estimates optimal policies for nested environments. We demonstrate that NFQI policies outperform those estimated from FQI and related approaches. When applied to simulated and medical data, these policies are performant, robust to sample size imbalance. learn justifiable differences in the datasets, and gracefully revert to FQI policies when there is no relevant group structure.
Several future directions remain. From a methods perspective, other approaches for the nested Q- value function could be considered, such as Gaussian processes, random forests, and other neural network architectures. This may improve the fit to new, larger datasets by increasing the expres- siveness of the Q-value approximation function. Additionally, extending the nested model to allow for multi-grou, structure may be useful when there are more than two groups in the data. This may involve changing the structure of the approximating function f to incorporate other compositions of g. and gy. In medical settings we anticipate future work in enabling NFQI as a clinical decision- support tool (i.c., to recommend treatments with qualitative justification) to be used in conjunction with a physician's 's expertise.
REFERENCES
[1] On-line learning in neural networks. Publications of the Newton Institute. Cambridge Univer- sity Press, 1998.

---

Under review as a conference paper at ICLR 2022
[2] Pieter Abbeel and Andrew Y. Ng. Apprenticeship learning via inverse reinforcement learning. In Twenty-first international conference on Machine learning ICML '04, page 1. ACM Press, 2004.
[3] Greg M. Allenby, Peter Rossi, and Robert E. McCulloch. Hierarchical Bayes Models: Pructitioners Guide. Jan 2005.
[4] Jordan T Ash and Ryan P Adams. On warm-starting neural network training. arXiv preprint 0rXiv:1910.08477 2019.
[5] R. Bellman. On the theory of dynamic programming. Proceedings of the National Academy of Sciences, 38())s7168)))77771 Aug 1952.
[6] Stevo Bozinovski and A Fulgosi. The influence of pattern similarity and transfer of learning upon training base perceptron b2. 1976. (original Croatian: Utjecaj slicnosti likova i transfera ucenja obucavanje baznog perceptrona B2), Proc. Symp. Informatica 3-121-5, Bled.
[7] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv: 1606. 540, 2016.
[8] Rich Caruana. Multitask learning. Machine Learning, 28, 07 1997.
[9] Damien Ernst, Pierre Geurts, and Louis Wehenkel. Tree-based batch mode reinforcement learning. J. Mach. Learn. Res., 6:503-556, 12 2005.
[10] Damien Ernst Pierre Geurts, and Louis Wehenkel. Tree-based batch mode reinforcement learning. Journal of Machine Learning Research, 6:503-556, 2005.
[11] Ary Goldberger, Luis AN Amaral, Leon Glass, Jeffrey M Hausdorff, Plamen Ch Ivanov, Roger G Mark, Joseph E Mietus, George Moody, Chung-Kang Peng, and H Eugene Stanley. Physiobank, physiotoolkit, and physionet: components of new research resource for complex physiologic signals. Circulation, 101(23)e215-e2200 2000.
[12] Abhishek Gupta, Benjamin Eysenbach, Chelsea Finn, and Sergey Levine. Unsupervised meta- learning for reinforcement learning, 2020.
[13] Junyan Hu, Hanlin Niu, Joaquin Carrasco, Barry Lennox, and Farshad Arvin. Voronoi-based multi-robot autonomous exploration in unknown environments via deep reinforcement learn- ing. IEEE Transactions on Vehicular Technology, (9(12):14413-144) 2020.
[14] Alistair Johnson, Lucas Bulgarelli, Tom Pollard, Steven Horng, Leo Anthony Celi, and Roger Mark. Mimic-iv (version 0.4), 2020.
[15] Alistair E.W. Johnson, Tom J. Pollard, Lu Shen, Li-wei H. Lehman, Mengling Feng, Moham- mad Ghassemi, Benjamin Moody, Peter Szolovits. Leo Anthony Celi, and Roger G. Mark. MIMIC-III, a freely accessible critical care database. Scientific Data, 3(1):160035, 2016.
[16] Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In Guyon, U. V. Luxburg, S. Bengio, Wallach, R. Fergus, Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 4765-4774, Curran Associates, Inc., 2017.
[17] Aishwarya Mandyam, Elizabeth C. Yoo, Jeff Soules, Krzysztof Laudanski, and Barbara E. Engelhardt. Cop-e-cat: Cleaning and organization pipeline for ehr computational and analytic tasks. In Proceedings of the 12th ACM Conference Bioinformatics, Computational Biology, and Health Informatics, BCB '21, New York, NY, USA, 2021. Association for Computing Machinery.
[18] MayoClinic. Creatinine tests. Accessed: 2021-05-27.
[19] MayoClinic. Low potassium (hypokalemia): Symptom - overview covers what can cause this blood test result. Accessed: 2021-05-27.
10

---

Under review as a conference paper at ICLR 2022
[20] Robert A. McLean, William L. Sanders, and Walter W. Stroup. A unified approach to mixed linear models. The American Statistician, 45(1):54, Feb 1991.
[21] Eric Mitchell, Rafael Rafailov, Xue Bin Peng. Sergey Levine, and Chelsea Finn. Offline meta- reinforcement learning with advantage weighting. In Marina Mcila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Pro- ceedings of Machine Learning Research, pages 7780-7791. PMLR, 18 24 Jul 2021.
[22] G.B. Moody and R.G. Mark. A database support development and evaluation of intelligent intensive care monitoring. Computers in Cardiology 1996, pages 657-660, 1996.
[23] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. arXiv preprint arXiv: 2019.
[24] Niranjani Prasad. Methods for reinforcement learning in clinical decision support. 2020.
[25] Martin Riedmiller. Neural fitted q iteration-first experiences with a data efficient neural rein- forcement learning method. In European Conference Machine Learning, pages 317-328. Springer, 2005.
[26] M Saeed, C Lieu, G Raber, RG Mark, and Mohammed Saeed. MIMIC II: A Massive Temporal ICU Patient Database Support Research in Intelligent Patient Monitoring. Computers in Cardiology, pages 641-644, 2002.
[27] Shagun Sodhani. Amy Zhang, and Joelle Pineau. Multi-task reinforcement learning with context-based representations, 2021.
[28] Anders Winther Voldby and Birgitte Brandstrup. Fluid therapy in the perioperative setting clinical review. Journal of Intensive Care, 4(1):27, 04 2016.
[29] Marco Wiering and Martijn Van Otterlo. Reinforcement learning. Adaptation, learning, and optimization, 12(3), 2012.
[30] Ruihan Yang, Huazhe Xu, Yi Wu, and Xiaolong Wang. Multi-task reinforcement learning with soft modularization, 2020.
[31] Zhuangdi Zhu, Kaixiang Lin, and Jiayu Zhou. Transfer learning in deep reinforcement learn- ing: A survey, 2021.
APPENDIX
6.1 BELLMAN OPTIMALITY OF NEQI'S Q- FUNCTION
In its original formulation, FQI was shown to converge to a Bellman-optimal Q-value function using concepts from dynamic programming theory (IO). Here, show that NFQI reduces to particular type of FQI, and thus remains within this Bellman-optimal family.
Recall that FQI approximates the Q-value function with function fro a) taking state-action pair as input, while NFQI uses an approximating function fs a, 2) that accepts an additional argument representing the sample's 's group identity. Because we can think it as a dimension of the state space that takes two possible values, and constant over given trajectory. Thus, trivially we can augment the state as (s .22 and describe NFQI as particular case of FQI where the approximating function is fF a). We can write the additive form of NFQI Eqquation
an attribute of the environment,
as
(5, = (s, + (s
where is p-vector, the (p + 1)th element of 5p+1 is

---

Under review as a conference paper at ICLR 2022
6.2 NEURAL NETWORK Q- Q-VALUE FUNCTIONS
Here, we present one instance of NFQI using a neural network as a nonlinear Q-value function approximator.
For simplicity, we describe an implementation using a multilayer perceptron (MIP) with one hidden layer: the nested network can be generalized to arbitrarily large architectures. Let be an element- wise nonlinear activation function, and suppose that each state-action pair is p-dimensional, which we write as single vector X, [s, a '1 R". Then, the output of the network for back-
ground sample is given by
S(s ar, 0) = oxx W Wls)
where W1 E go*k and W2 1* are linear network weights separated by the nonlinear function Note that, because this is background sample, only apply 9s to the sample. To model foreground samples, we also include a foreground additive component in the network. This component, 9f, only operates foreground samples. The output of the network for foreground sample is given by: [(r a,, 1) = (x, W1s) W2e + ox Wir)War,
(8,a)
By (8,,a))
where We and Wes are the shared and foreground-speciin weights for layer E {1,2), respec- tively.
In our experiments below, we use variant of this type of neural network (6.6). We optimize the network parameters = {Wee, Werlez using a squared error loss function,
min Σ (i(s)) ar) -s,( Bc,
6.3 CODE
Before NFQI can be deployed in partieular application area, its hyperparameters, and possibly its model structure, should be tuned. We release code with recommendations for tuning these settings. All of our experiments were run on an internally-hosted cluster using a 320 NVIDIA P100 GPU whose processor core has 16 GB of memory hosted. Our experiments used a total of approximately 100 hours of compute time. The code for our experiments is based on prior implementation of FQJl We implement our nested neural network models in Py PyTorch (23) and use an SGD-based optimizer (.) For all experiments, we use 80% of our data train and 20% of our data test. We use default learning rate of 10-3, We use the same hyperparameters for nested- and group-label agnostic methods.
6.4 CARTPOLE ENVIRONMENT
The Cartpole environment contains cart with an inverted pendulum pole on top, and an agent's goal in this environment is to move the cart left and right such that the pole remains upright and the cart itself stays within the bounds of the window box. More formally, we can describe the Cartpole environment as Markov decision process (MDP) represented by a tuple (S,A, P. R). Here, S = [x, é 0, 0 is a four-dimensional state space, where and á are the cart's position and velocity, and 0 and á are the pole's angle and angular velocity; AA {a,,a,} a, } is the action space, containing actions for applying a force of 10 Newtons to the left or right; P a deterministic function determining state dynamics (based on simple Newtonian mechanies in this case); and R is a reward function, which returns -1 when the pole falls or the cart exits the frame (at which point the episode is terminated). and returns 0 otherwise. In the nested Caripole environment, we adjust the dynamics described by P to simulate different dynamics for the foreground and background environments. Specifically, in the foreground environment, we include a constant leftward force of Newtons. In this case, choosing the action "push left" results in a leftward force of 10 +
12

---

Under review as a conference paper at ICLR 2022
Newtons, while the action "push right" results in rightward force of 10 Newtons. This asymmetry is the driving difference between the background and foreground environments. As such, each environment has the same state space, action space, and reward function, but different state transition function.
6.5 LINEAR MODEL Q-VALUE FUNCTION
We tested the effect of the choice of the approximating function ʃ on the performance of the NFQI framework. To do so, we fit NFQI and FQI using a linear model and neural network the Cartpole environment.
For the linear model, we consider Ia and g to be linear models in Equation equation | each sample by a vector containing state and action, [s] a where S, is the state vector and a, is the action vector. We assume a finite action space throughout this study. Then, we have the following model:
We represent
9s (s,, ax)
St
B, + 1 Bos +
9f
(s ar)
St
G +
St ag
G, +1 Bos +
a.
where 1 represents a column vector of ones, and is zero-mean random noise. Here, the coefficients 8, and By capture the background and foreground's (linear) relationship between state and action pairs, and expected reward. The terms Sus and Buy capture context-specifit intercepts. Note that this approach similar hierarchical model approach (0 33
As expected, we find that the linear model is unable to successfully find policy that solves the Cartpole environment (Figure 8). In contrast, neural network is able to find good policy most of the time. We anticipate this because a neural network can approximate functions that are nonlinear and that the Cartpole environment requires nonlinear function approximate reward. We also expect the MIMIC-IV environment to require a nonlinear function. For this reason, we consider neural networks throughout this study.
Approximation functions
1000
Successful steps
600
400
200
Figure 8: Performance of FQI in the Cartpole environment using two different approximation func- tions, Performance of each model is measured as the number of steps before the pole fell or the cart exited the frame. Performance shown here for a linear model and neural network.
13

---

Under review as a conference paper at ICLR 2022
Cartpole performance, background
Cartpole performance, foreground
1000
1000
Steps Survived
800
Scepp Survived
800
600
600
400
400
200
200
10
2
|Forcel Left)
(Force Left)
Figure 9: A neural network-based version of NFQI outperforms related algorithms and a linear model-based NFQI in nested Cartpole environment. Linear NFQI likely underparameterized for the Cartpole problem in both the background (Panel a) and foreground (Panel b) environments.
14

---

Under review as a conference paper at ICLR 2022
Algorithm 1: NFQI training procedure
Randomly initialize shared and foregroundspecifie model parameters 0, f: while background loss La not converged do
Compute Ls:
Update 0, using SGD;
end
while foreground loss Ly not converged do
Compute Lj;
Update , using SGD;
end
(s) < arg max, (s, a') // background policy
(s) + arg max, (8) // foreground policy
return Tb; f
6.6 ALGORITHMM
We use three algorithms our experiments: NFQI, FQI, and transfer learning. We use the same network with the same number of parameters for all three algorithms. The difference between the algorithms can be explained using the training procedures as described below. The network structure follows:
input S, a,
X [s, a] Linear(10) - ReLU - Linear(5) - ReLU
x_shared = Linear(1)
x fg==xxx =x - Linear(10) - ReLU - Linear(S) - ReLU - Linear(1)
output x shared
With the exception the imbalanced dataset experiment (Section 4.1.4, all algorithms use the same number of training and test samples; the network is trained using only training samples, and tested using only test samples.
To train NFQI, follow Algorithm I group label to the network. Depending on the group label, the network uses either the shared layers or both the shared and foreground layers estimate Q-value. To train FQI, do not consider the group label of the sample. The network uses all of the layers to estimate Q-value for each training sample. This is as if all of the group labels were
To perform inference, input state-action pair and the
We also do not consider the group label when training a transfer learning algorithm. We first use the background training samples to train the shared layers in our network. Then, we freeze the shared layers and use the foreground training samples to train the foreground specific layers. When performing inference, this network does not consider group label; such, the network uses every layer to estimate Q-value for sample.
All of our algorithms use mini-batch SGD optimizer and an MSE loss. random seed used to ensure reproducibility. We evaluate the network on one test sample for every train sample in the batch; the previous three evaluations were successful (i.e., the pole in the Cartpole environment stayed up for 1000 steps), we deem the layers converged.
6.7 MIMIC-IV ENVIRONMENT
The MIMIC-IV EHR dataset was collected between 2008 and 2019 from the Beth Israel Deaconess Medical Center in Boston, Massachusetts, The latest version of this dataset, MIMIC-IV, contains de-identified data from over 524,000 distinct hospital admissions and over 257,000 patients (14).
Our preprocessing steps follow the framework suggested by prior work 24 17).
Our background dataset consists of 400 patients which healthy kidneys and our foreground dataset contains 123 patients with renal disease. Intuitively, electrolyte repletion policies must account for kidney function. For example, patients who admitted for renal condition have baseline
15

---

Under review as a conference paper at ICLR 2022
imbalance of electrolytes and receive electrolyte repletion much differently than patients who have functioning kidneys.
Below is list of all patient characteristics used in our experiments.
Table 2: Features used predict electrolyte repletion.
Feature
Description
Age
Age of patient admission
Gender
Gender discretized to male/female
Expired
Binary flag indicating whether patient is dead/alive
Patient Weight
Patient Weight (kg)
Length of Stay
Length patient's stay (days)
Heart Rate
Heart Rate (bpm)
Respiratory Rate
Rate of breathing (breaths per minute)
Oxygen Saturation
Oxygen saturation in blood (%)
Temperature
Body temperature F)
Systolic BP
Systolic Blood Pressure (mmHg)
Diastolic BP
Diastolic Blood Pressure (mmHg)
Potassium (IV)
Potassium administered through IV (mL)
# Hours Potassium (IV)
# hours potassium is administered through IV Potassium administered orally (mg)
Potassium (nonIV)
# Hours Potassium (nonlV)
# hours potassium is administered orally Potassium measured blood Calcium administered through IV (mL)
Potassium
Calcium (IV)
Hours Calcium (IV)
# hours calcium administered through IV Calcium administered orally (mg)
Calcium (non-IV)
Hours Calcium (nonlV)
# hours calcium administered orally
Calcium
Calcium measured in blood
Phosphate (IV)
Phosphate administered through IV (mL.)
# Hours Phosphate (IV)
# hours phosphate administered through IV Phosphate administered orally (mg)
Phosphate (non-IV)
Hours Phosphate (nonIV)
# hours phosphate administered orally
Phosphate
Phosphate measured in blood
Magnesium (IV)
Magnesium administered through IV (mL)
# Hours Magnesium (IV)
# hours magnesium is administered through IV Magnesium administered orally (mg)
Magnesium (non-IV)
# Hours Magnesium (nonIV)
# hours magnesium is administered orally
Magnesium Sodium
Magnesium measured blood Sodium measured in blood
Vasopressors
Administered to constrict blood vessels Administered reduce blood pressure Administered to reduce blood pressure Administered treat hypertension, edema Administered to promote absorption of glucose from blood Administered increase blood sugar Orally administered nutrition supplements
Beta Blockers
Calcium Blockers
Loop Diuretics
Insulin
Dextrose
Oral Nutrition
Parenteral Nutrition
Non-orally administered nutrition supplements
Total Parenteral Nutrition
IV-based fluids used for nutrition
Dialysis
Binary indicator of dialysis procedure
Blood Transfusion
Binary indication of red blood cell transfusion
Coronary Artery Disease
Binary indication of disease
Atrial Fibrillation
Binary indication of disease
Congestive Heart Failure
Binary indication of disease Binary indication of disease
Chronic Kidney Disease
Renal Disease
Binary indication of non-chronic kidney disease
Paralysis
Binary indication of loss of ability move
Parathyroid
Binary indication of disease
Rhabdomyolysis
Binary indication of disease
16

---

Under review as a conference paper at ICLR 2022
Table 2: Features used predict electrolyte repletion.
Feature
Description
Sarcoidosis
Binary indication of disease
Alanine Aminotransferase
Measure in blood
Anion Gap
Measure in blood
Blood Urea Nitrogen
Measured in blood
Creatine Phosphokinase
Measured in blood
Hemoglobin
Measured in blood
Glucose
Measured in blood
Creatinine
Measured in blood
Lactic Acid Dehydrogenase
Measured in blood
White Blood Cell
Count measured in blood
The Markov decision process (MDP) for this MIMIC-IV environment is tuple (S, P. R). Here,
S is 61-dimensional vector containing information corresponding to each feature in Tab. [ A = {0, 0j, [0, 10), }] is the action space, each action corresponding to no repletion, low
repletion and high repletion respectively; P contains the (unknown) state transition probabilities guiding a patient's state dynamics during their hospital visit; and R is a reward function, which returns a value based on whether the patient's observed potassium level is outside the normal potas- sium range and the cost of repletion. R is the sum of a four dimensional vector; each element in this vector corresponds to the cost of oral repletion, the cost of intravenous repletion, penalty for the patient's potassium level being too high, and a penalty for the patient's potassium level being too low, respectively. The reward function can be written as r; = w di (St, des $141) where & is four-dimensional vector function such that:
1 Lai rouce rouce(
0 0.
ag [intravenoue]
1
vc(+)=
(1 (KK Kmax -1)]
[K|>
10 0) 10, 0)
sl [K] (i>KKnn Kmax 10 (1
Kmar -17
Here, K is the known measurement of potassium, and Kmaz and Kmin define the upper and lower bounds of the target potassium range, respectively. The first two elements of the reward vector correspond to whether the repletion administered orally through intravenous line. For
experiments, we use = 1,1
17


Reviews:
[{"review_id": "u_Pe0LRu7XI", "reviewer": "Reviewer_gMUj", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The challenge of being able to identify shared problem structure between subgroups and leverage that information to improve sample efficiency in reinforcement learning is an important and difficult task. This work provides an interesting perspective and intuitive approach for solving it.\n- I thought the questions and experiments laid out in Section 4 (top of page 4) were interesting and well-defined. Overall, their experimental design(s) in the CartPole setting were compelling.\n\nWeaknesses:\n- My main critique of the paper is that it takes an unnecessarily complicated approach to an otherwise easy problem and does not compare performance of the proposed algorithm to one of the more obvious/intuitive benchmark approaches. First, I'll address the \"unnecessarily complicated approach\" comment. The proposed approach is to essentially create two separate sets of weights for modeling a group-specific $Q$-value estimator: one shared set of weights for all groups, and one set of weights unique to the \"foreground\" group. But the authors themselves state that their approach is equivalent to just augmenting the state space with an indicator of one-hot encoded group assignment. In this light, an alternative summary of the paper's contributions could be written as, \"We find that FQI performs better when it has access to all of the variables that actually specify the MDP\". Fundamentally, when group assignment modifies the MDP but is not included in the state space, the resulting stochastic decision process is no longer Markovian. I suppose you could think of the resulting problem setting as a POMDP. The resulting problem is interesting when group assignment is not observable (see e.g., Killian and Daulton [1]), but trivial when group assignment is available: just expand the state space to include the state space and the problem is solved. In my mind, then, to the extent that NFQI is not always just a simple state space augmentation (see bullet below), the best comparison here would just be FQI with an augmented state space. I would bet, though, that any differences in performance between NFQI and state-augmented FQI (when there is actually a difference between them) are negligible at best. The other baseline to compare against in class-imbalanced settings would be to use FQI + augmented state space + sample weighting to address the class imbalance (so upweighting the loss of the smaller class by ~1/P(a random sample is in the smaller class)). \n- Related to the above, I'll note that the authors' comment in Appendix 6.1 about equivalence of their proposed $f_{\\text{NFQI}}$ with state-augmented FQI is _not_ true for the neural network approach that they describe in Appendix 6.2. This discrepancy is not mentioned, but should be. More importantly, however, the authors gloss over the fact that convergence of FQI (see Section 3.5 of [2]) is heavily dependent on the use of an induced kernel that does not depend on the output values of the training sample. This is certainly not the case when using neural networks as function approximators, as the authors have proposed. Indeed, _none_ of the instantiations of NFQI proposed by the authors carry such a convergence guarantee.\n- In the discussion of the related work, the authors claim that their approach differs from Meta-RL and Multi-task RL by arguing that these approaches deal with different tasks _in addition to_ different MDPs, but the NFQI approach only handles modeling the _same_ task with different MDPs. I would argue that Multi-task RL and Meta RL are thus more general than the proposed NFQI approach and in fact subsume it. To that extent, one relevant comparison here would be to compare Multi-task RL and Meta-RL approaches to the proposed NFQI approach. But honestly, this is probably overkill because with known group identity you can likely get most of the potential gains just by augmenting the state space and running FQI.\n- I have significant concerns about the quantitative evaluation approach for the MIMIC dataset/task. Using alignment between clinician behavior and the learned RL policy as a proxy for policy quality is fraught in many ways (What if clinicians behave suboptimally? What if the differences are just due to chance?). See Gottesman et al [3] for some pointers to best practices in off-policy policy evaluation. \n- Two central premises of the proposed approach is that there are many important real-world problem settings in which (assuming some shared RL task/reward function), (1) there are precisely two known/observable conditions for which the underlying transition dynamics of the associated MDP are similar but slightly different; and (2) there are significantly more data available for one condition relative to the other. \n- While premise (1) is plausible, it is poorly (if at all) justified. The much more likely scenario in my mind is one in which there are _many_ conditions/groups, all of which are _unobservable_, each of which is associated with a slightly altered MDP. Consider the experiment the authors ran on MIMIC, where they split the cohort into individuals with kidney disease and individuals without kidney disease. Who's to say that this is the right way to split that cohort into two groups? Would it be more appropriate to split patients on age? Diabetes status? Prior history of an Acute Kidney Injury? Also, does the specific kind of kidney disease matter? What about the severity? Is there some grouping that is unobserved/latent but perhaps correlated with age, diabetes status, and kidney disease that would be a more accurate description of how patient-specific MDPs vary? The limitation of the presently proposed framework to exactly two observed groups is quite constraining and limits the significance of the work. That being said, in the spirit of \"All models are wrong, but some are useful\", I'll put that aside for now and focus on the merits of the approach for improving performance in the proposed problem setting. \n- Premise (2) is also plausible, but also poorly justified. The justification for this claim is essentially an intuitive argument in the second sentence of Section 4.1.4 \"NFQI is robust to group size imbalance\": \"For example, it is much easier to collect medical records from healthy (background) patients than from patients with a specific, chronic disease\". I would actually argue that the opposite is true in the medical setting. Medical record data for healthy individuals are relatively sparse because healthy patients don't go to the hospital. On the other hand, individuals with chronic conditions will tend to visit their primary care provider (PCP) or specialist with much more regularity. I'm not arguing with the premise itself, but rather saying it needs more justification and the one given is inadequate (and, in my view, just wrong).\n\nI'm not going to get into really minor points here, because I think the paper will require a major overhaul before it can be accepted at a conference like ICLR. That being said, I do think there are some interesting directions here for future work, specifically in settings where there is more than one group and group identity is unknown. There's already a fair bit of existing work on Meta RL and Multi-task RL that is quite relevant. Another interesting direction on the theory side would be to better understand how \"close\" MDPs from two or more groups have to be in order for weight-sharing approaches like the one proposed provide sample efficiency improvements.\n\n[1] Killian, Taylor, et al. \"Robust and efficient transfer learning with hidden parameter Markov decision processes.\" Proceedings of the 31st International Conference on Neural Information Processing Systems. 2017.\n\n[2] Ernst, Damien, Pierre Geurts, and Louis Wehenkel. \"Tree-based batch mode reinforcement learning.\" Journal of Machine Learning Research 6 (2005): 503-556.\n\n[3] Gottesman, Omer, et al. \"Evaluating reinforcement learning algorithms in observational health settings.\" arXiv preprint arXiv:1805.12298 (2018).", "summary_of_the_paper": "This paper proposes nested policy fitted Q-iteration (NFQI), an adaptation of the well-established Fitted Q-Iteration (FQI) algorithm originally proposed by Ernst, Geurts, and Wehenkel in 2005 [1] to the setting where the agent is required to learn policies for two different MDPs while sharing information between them. At a high level, the authors' proposed modification involves learning a function $f$ which approximates the Q-function, $Q^{\\pi}(s_t, a_t) = r_t + \\gamma \\mathbb{E}_{s'}[V^{\\pi}(s')]$ by decomposing the $Q$-estimate into a shared component and a subgroup-specific component. \n\nConcretely, they represent $f$ as $f(\\mathbf{s}, \\mathbf{a}, z) = g_s(\\mathbf{s}, \\mathbf{a}) + \\mathbb{1}[z=1]g_f(\\mathbf{s}, \\mathbf{a})$ where $g_s$ models the $Q$-value using parameters shared across groups and $g_f$ models a group-specific modification to the estimated $Q$-value.\n\nThrough both simulated experiments (CartPole) and an observational dataset (electrolyte replenishment optimization on MIMIC) the authors demonstrate that their NFQI approach performs better than do alternative approaches which do not take known subject-specific group information into account.\n\n[1] Ernst, Damien, Pierre Geurts, and Louis Wehenkel. \"Tree-based batch mode reinforcement learning.\" Journal of Machine Learning Research 6 (2005): 503-556.", "summary_of_the_review": "This paper proposes an approach for intelligent weight sharing in offline RL when the dataset available consists of two groups with slightly different MDPs, and where group identity is known/observable. This problem can be trivially solved by simply including group identity in the state representation. The experimental design is good and the research questions are interesting. However, the problem is simple enough and the solution trivial enough that there's not a whole lot of novelty or significance in this paper. I recommend rejection.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "uXXdFgHHLA9", "reviewer": "Reviewer_rPvd", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### **Strengths**\n- Exposition is clear and easy to follow. \n- Simulated experiments on CarPole are extensive and seek to answer various interesting research questions that justify the usefulness of the proposed approach. \n- Good use of SHAP for understanding which features were used by the policy’s recommendation and showing the difference of foreground/background. \n\n### **Weaknesses**\n- This paper only considers two groups (background / foreground) that share state dynamics to some extent. Extensions to more than two groups could be more generally useful, but are only mentioned in future work and not discussed concretely. \n- There doesn’t seem to be anything special about the proposed approach that’s specific to the offline setting. One could parameterize the Q-function as $f(s,a) = g_s(s,a) + 1_{z} g_f(s,a)$ in an online setting as well. Could you elaborate whether this approach will also provide benefit in the online setting, and if not, what makes it special for offline RL? \n- The baselines were not clearly described in the main text: what is the “transfer learning” baseline (first appeared on page 4)? \n- Re Real-world EHR data experiments: I don’t believe “agreement with doctor’s actions” is the right quantitative metric. Consider using an OPE (offline policy evaluation) method [1][2]. \n- It wasn’t clear to me whether there was a train/val/test split because Sec 4.2 on page 8 describes the total number of patients yet Table 1 says “predict … on test samples”\n\n### **Detailed Comments**\n- Re a possibly inaccurate claim on page 11 Appx 6.1: “In its original formulation, FQI was shown to converge to a Bellman-optimal Q-value function using concepts from dynamic programming theory.” The classical results only guarantees that iterative application of Bellman optimality operator (without function approximation, or equivalently, with perfect function approximation) leads to Q*. But in practice the convergence of FQI [3] depends heavily on (i) the completeness/realizability of the function class, and (ii) distribution shift in limited data. I also hope to see some theoretical support for why $f_{NFQI}(\\\\tilde s, a)$ is a “good” choice of function class. \n- The proposed setting seems to be a variation/special case of factored MDP (with factored state spaces). I would recommend reading and citing the following references [4][5] (plus possibly their followup works) and comment on how the proposed approach is related and where it differs. \n- Presentation of Fig 6: Can you please clarify what are the values in the heatmap? Since there are only 3 actions (no, low, high), I was expecting only three colors in the heatmap. I would also recommend using a discretized color bar with only 3 colors. Furthermore, the y-axis label for creatinine should be flipped so that lower values are at the bottom which is what a typical Cartesian axis looks like. \n- Naming: it’s true that the presented setting has “nested” dynamics, but I’m not sure if “nested” is the best way to describe the value function and policy. \n\n### **References**\n1. Taylor W. Killian, Haoran Zhang, Jayakumar Subramanian, Mehdi Fatemi, Marzyeh Ghassemi. An Empirical Study of Representation Learning for Reinforcement Learning in Healthcare. ML4H @ NeurIPS 2020. https://arxiv.org/abs/2011.11235\n2. Shengpu Tang, Jenna Wiens. Model Selection for Offline Reinforcement Learning: Practical Considerations for Healthcare Settings. MLHC 2021. https://arxiv.org/abs/2107.11003\n3. Jinglin Chen, Nan Jiang. Information-Theoretic Considerations in Batch Reinforcement Learning. ICML 2019. https://arxiv.org/abs/1905.00360\n4. Daphne Koller, Ronald Parr. Computing factored value functions for policies in structured MDPs. IJCAI-99. https://dl.acm.org/doi/abs/10.5555/1624312.1624408\n5. Carlos Guestrin, Daphne Koller, Ronald Parr, Shobha Venkataraman. Efficient solution algorithms for factored MDPs. JAIR 2003. https://doi.org/10.1613/jair.1000\n", "summary_of_the_paper": "This paper considers decision making in nested MDPs where there are two distinct groups having partially shared (but unknown) state dynamics. A new algorithm called NFQI (nested fitted-Q iteration) is presented, and it’s based on a simple modification to FQI, a commonly used offline RL algorithm. Baseline comparisons and sensitivity analyses were done in a set of simulated experiments and using OpenAI gym CartPole. The proposed approach is then applied to a real-world RL task formulated from MIMIC-IV EHR data. ", "summary_of_the_review": "The presented setting is interesting to look at and potentially useful in clinical decision making, and the simulation results justified the benefit of the proposed approach to a limited extent. However, experiments on real EHR data need to be improved, and there seems to be a lack of theoretical understanding for when and why this approach works well; this will also help generalize the results to more than two groups. There needs to be better framing regarding whether the conclusions are limited to offline settings. Lastly, it's important to include a discussion that relates to past work on factored state spaces and factored value functions. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "aG8WWlSY5bv", "reviewer": "Reviewer_EMFG", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n- A simple approach to account for a structured and imbalanced data when learning off-policy RL\n- NFQI works well in the case of extreme imbalance between the foreground and background environments\n- The interpretability experiments using SHAP values validate that the NFQI finds policies that identify the foreground environment as an important component\n- The model is applied to real-world clinical data (MIMIC-IV) \n\n\nCons:\n- The novelty of NFQI is somewhat limited. The proposed training approach is very similar to transfer learning, where  shared and specific components are trained.\n- Limited baselines in the experimental analysis\n\n", "summary_of_the_paper": "The paper proposes an off-policy RL called NFQI (Nested Fitted Q-Iteration) as an extension of Fitted Q-iteration to estimate group-specific policies and account for group structure in the data having two pre-defined groups of observations (background and foreground). NFQI imposes a structure on the family of function to approximate the Q function with guarantees to converge to Bellman optimal Q-value function. NFQI is trained in two stages: The first stage the shared component is trained with all samples. In the second stage, the foreground component is trained using only the foreground samples. The NFQI method is validated on a nested cart-pole environment where the background is the original environment and foreground includes a constant force that pushes the cart to the left. NFQI is also validated using a real-world clinical data from MIMIC dataset. ", "summary_of_the_review": "NFQI has many real-world applications, including the use in healthcare. However, the novelty of the method is rather limited. Also, comparisons with additional baselines would be required to further validate the approachg.\n\nMinor comments:\n- background loss L_b is not defined in Algorithm 1\n- Table 2 is duplicated, so which list of features has been used in the experiments shown in the paper ?\n- In figure 5-a superimposed bars are not clear as mixed color is showing up. Separate bars would be better for clarity.\n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "CUbJDC3RLIp", "reviewer": "Reviewer_KSWV", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "*Strengths*\n---\n\nThe paper nicely outlines the setting for NFQI. It's clear to understand how the Nested MDPs are modeled and where they may be found in the real world. Concerns about novelty and the necessity of this formulation are listed below among the weaknesses I found in the paper.\n\nThe paper is well structured and well written for the most part. At times I felt that the discussion was a little too high level, specifically in Sections 2 and 3 (Related Work and Methods; more in the 'Weaknesses' section below) but for the most part necessary explanations were sufficiently through and clearly explained. A particular strength in the writing was evident in how the core insights or experimental objectives were clearly outlined at the beginning of Section 4 (Experiments). \n\nI was also impressed at the extent by which the learned policies were evaluated and compared to standard FQI. The interpretation of how different features contribute to policy decisions using SHAP values was nice to see (some concerns about clarity are discussed below). \n\n*Weaknesses*\n---\n\nI have several concerns about the necessity of this approach as proposed by the authors and presented in this paper, as submitted.\n\nFirst, by focusing only on two subpopulations of these Nested MDPs the authors are investigating the simplest base case of what could be termed multi-task or meta learning. There is very little discussion to justify why taking this stripped down approach is necessary. Beyond the tendency of the multi-task or meta-learning literature to focus on large overparametrized networks, there's no reason to expect why the methods and algorithms presented in these papers wouldn't be applicable and perform as well as NFQI.\n\nSecond, there is a large body of work on transfer within RL that has been overlooked and should be cited appropriately. The Nested MDP setting seems to be a special case of a Hidden-parameter MDP (HiP-MDP) or Block MDP (references below). Each of these branches of literature cover the exact setting where families of MDPs, differentiated only by changes in their dynamics, are treated with joint policies or otherwise personalized to individual settings through a latent or contextual parameter. A discussion differentiating this paper from this body of literature is warranted.\n\nThird, it's not surprising that NFQI outperforms FQI given the extra contextual information the algorithm is provided through the contextual group variable $z$. A more fair comparison would be providing this extra feature to FQI and evaluating the policy performance in that setting. Additionally, it's severely unclear what is actually done in the Transfer Learning baseline. Are policies learned on the background dataset and they applied directly to the foreground dataset? Is there any finetuning on the foreground at all? \n\nBeyond these points I was disappointed with how high-level and devoid of detail the discussion in Section 3 (Methods) was. There is very little concrete development of NFQI where I wasn't entirely sure what was being presented in the experimental results and how much significance to ascribe to the performance gains or differences between FQI and NFQI. Clearly, something exciting is happening by leveraging the contextual information of the nested MDP in Equation 1 within the function approximator $f$. But without a clear link or more formal presentation of where Eqt. 1 fits into FQI to produce NFQI its not easy to immediately follow the results and insights developed in Section 4 (Experiments). It's also unclear, after reading the Appendix, what the separate Losses represent. Based on Eqt. (1) and Algorithm (1) in the appendix, it seems that $g_b$ is doubly accounted for. So is $g_b \\rightarrow \\pi_b$ having double the gradients applied to it? If so, it's unsurprising that it's performance is so high relative to FQI on the background dataset on the cartpole task.\n\n**Questions from Cartpole experiments**\n\nIt's not apparent why different dataset sizes were used between the various experiments for Cartpole (also, in Section 4.1.1 does \"samples\" mean \"episodes\"?). The lack of consistency and the policies achieving different levels of performance between experiments is unnecessary and only provides more opportunity for confusion. It would possibly be better if the experiment in Section 4.1.1 had the same total number of episodes as the other experiments in Section 4.1 but still could be used to demonstrate the effects of a small portion of those episodes belonging to the foreground dataset with a more thorough exploration of this effect as is done in Section 4.1.4. What sparked this question was the diminished performance in Figure 4 in comparison to Figure 1.\n\nThe use of SHAP values to differentiate policies is a really neat idea. I think that there are further insights behind Figure 3 that probably deserve to be explored. As discussed, there are qualitative differences in the importance of features between the background and foreground policies. What is shown in addition to this is that the feature values that a deemed important are inverted between the datasets. High feature values seem to be more important in the foreground dataset. This is interesting given the definition of the foreground MDP. Does this relation shift as you reduce the magnitude of the force applied in the foreground dataset?\n\nIt's not described what's varied between runs of the experiments, are these random seeds? Sampling strategies? I.e. what is driving the variance of the performance? Is it the method itself?\n\nIn Section 4.1.4 it is mentioned that the proportion of data that makes up the foreground dataset is varied to mirror proportions seen in medical datasets. This is unecessarily vague. What types of datasets? What makes up these proportions in the medical datasets? What conditions? \n\n**Questions from the experiments using the MIMIC renal cohort**\n\nThe cohort definition was unclear beyond the number of patients. The number of features and how they're derived is relevant information that should be included in the main body of the paper. The specific action definition is also not very clearly outlined. What goes into repleting electrolytes? Is there more than one medication for this beyond prescribing potassium? After looking through Table 2 in the appendix, it appears that several of the features are associated with treatment decisions or medications administered to the patients. There is likely some overlap with comorbidities and the presence of these medications. How was this accounted for? Were treatment/action decisions incorporated into the patient's state representation?\n\nThe potassium treatment decisions are rightfully binned into a categorical vector (none, low, high) which corresponds to three discrete actions, as I understood the definition. However in Figure 6 there seems to be a continuum of treatments that are administered. Which was it? On this point, it's not clear what the numerical values in Figure 6 correspond to in the categorical sense. This was confusing because, as stated, the healthy range for potassium is defined as 3.5-4.5 mmol/L but in Figure 6d, the NFQI policy appears to recommend 'High' replenishment treatments within that range (assuming a value of 2 = 'High'). That seems counterintuitive.\n\nNow, for Table 1. If there are only three discrete actions, then the results of predicting the clinician's actions aren't performing much better than random. It's also curious that the action prediction performance for the background dataset is so much lower than the foreground dataset. Perhaps this is due to greater heterogeneity in the background data? I would expect the performance to be better on the background dataset due to there being more data. \n\nIn Figure 7, it's interesting to see how the foreground and background policies differ from each other. I'm curious if there's some causal leakage informing these results however. For example, one of the top rated features is Potassium, which is directly correlated to the treatment decisions, right? Is this filtering through as a top rated feature in the foreground dataset because there is a higher frequency of these labs/tests being ordered? Is this the same for HGB or the amount of phosphate being administered via IV? What is the significance of adding the other features together. This imbalances the presentation of Figure 7 and my immediate impression was that the other 56 features are more important since their combined SHAP values are significantly higher than the others presented. I would perhaps recommend removing that last row and making a comment in the caption (or main body) that the top 5-10 features from NFQI are provided...\n\n\n----\n`References`\n\n**HiP-MDPs**\n\nDoshi-Velez, F., & Konidaris, G. (2016, July). Hidden parameter markov decision processes: A semiparametric regression approach for discovering latent task parametrizations. In IJCAI: proceedings of the conference (Vol. 2016, p. 1432). NIH Public Access.\n\nKillian, T., Daulton, S., Konidaris, G., & Doshi-Velez, F. (2017, December). Robust and efficient transfer learning with hidden parameter Markov decision processes. In Proceedings of the 31st International Conference on Neural Information Processing Systems (pp. 6251-6262).\n\nYao, J., Killian, T., Konidaris, G., & Doshi-Velez, F. (2018). Direct policy transfer via hidden parameter markov decision processes. In LLARLA Workshop, FAIM (Vol. 2018).\n\nPerez, C., Such, F. P., & Karaletsos, T. (2020, April). Generalized hidden parameter mdps: Transferable model-based rl in a handful of trials. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 34, No. 04, pp. 5403-5411).\n\n**Block MDPs**\n\nMisra, D., Henaff, M., Krishnamurthy, A., & Langford, J. (2020, November). Kinematic state abstraction and provably efficient rich-observation reinforcement learning. In International conference on machine learning (pp. 6961-6971). PMLR.\n\nZhang, A., Lyle, C., Sodhani, S., Filos, A., Kwiatkowska, M., Pineau, J., ... & Precup, D. (2020, November). Invariant causal prediction for block mdps. In International Conference on Machine Learning (pp. 11214-11224). PMLR.\n\nZhang, A., Sodhani, S., Khetarpal, K., & Pineau, J. (2020). Learning Robust State Abstractions for Hidden-Parameter Block MDPs. arXiv preprint arXiv:2007.07206.\n\nZhang, A., Sodhani, S., Khetarpal, K., & Pineau, J. (2020). Multi-Task Reinforcement Learning as a Hidden-Parameter Block MDP. arXiv e-prints, arXiv-2007.", "summary_of_the_paper": "This paper introduces a learning paradigm to handle related MDPs (called Nested MDPs) that share the same structure and definition, varying only in their dynamics. Based on Fitted Q-Iteration (FQI), the authors propose an algorithm known as Nested FQI (NFQI) to learn from the shared structure of the Nested MDPs while also being able to adapt to the specific dynamics of the separate MDPs. NFQI is developed and analyzed empirically in the simplest setting where there are only two variants of dynamics are present, where there is a nominal imbalance between expected and out of distribution observations. NFQI is compared to standard FQI and standard transfer learning on an augmented Cartpole task as well as medical treatment task derived from retrospective EHR data. Extensive analysis on the learned policies is performed to establish the anticipated benefits of using NFQI in these settings.", "summary_of_the_review": "There are several concerns about significance of the problem setting and proposed algorithmic approach. Additionally, there are severe gaps in clear exposition outlining how the proposed algorithm is set-up and run. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper provides a method for offline RL in settings where the environment may exhibit significant similar structure, such as one part having nearly the same dynamics as other parts. The work is motivated in part by healthcare settings. The reviewers appreciated the potential applications to areas like healthcare but also thought there is a strong body of related work (e.g. transfer learning, meta-RL and other related papers) and it was unclear how novel the approach was within that related work, or how it would compare. The authors did not respond to the reviewers’ reviews. We hope their input is useful to the authors’ in revising their work for the future.
Decision: Reject