Off-Policy Selection for Initiating Human-Centric Experimental Design
Authors: Ge Gao
Xi Yang
Qitong Gao
Song Ju
Miroslav Pajic
Min Chi
Venue: NeurIPS 2024 poster (2024)
Keywords: Off-policy selection (OPS)
Offline reinforcement learning and OPS for human-centric experimental design
intelligent tutoring
sepsis treatments
Primary area: machine_learning_for_other_sciences_and_fields

Abstract: In human-centric applications like healthcare and education, the \textit{heterogeneity} among patients and students necessitates personalized treatments and instructional interventions. While reinforcement learning (RL) has been utilized in those tasks, off-policy selection (OPS) is pivotal to close the loop by offline evaluating and selecting policies without online interactions, yet current OPS methods often overlook the heterogeneity among participants. Our work is centered on resolving a \textit{pivotal challenge} in human-centric systems (HCSs): \textbf{\textit{how to select a policy to deploy when a new participant joining the cohort, without having access to any prior offline data collected over the participant?}} We introduce First-Glance Off-Policy Selection (FPS), a novel approach that systematically addresses participant heterogeneity through sub-group segmentation and tailored OPS criteria to each sub-group. By grouping individuals with similar traits, FPS facilitates personalized policy selection aligned with unique characteristics of each participant or group of participants. FPS is evaluated via two important but challenging applications, intelligent tutoring systems and a healthcare application for sepsis treatment and intervention. FPS presents significant advancement in enhancing learning outcomes of students and in-hospital care outcomes.

Off-Policy Selection for Initiating Human-Centric Experimental Design
Ge Gao #1
Xi Yang?
Qitong Gao
Song Ju
Miroslav Pajic
Min Chil
Abstract
In human-centric tasks such as healthcare and education, the helerogeneity among patients and students necessitates personalized treatments and instructional terventions. While reinforcement learning (RL) has been utilized in those tasks, off-po selection (OPS) is pivotal to close the loop by offline evaluating and selecting policies without online interactions, yet current OPS methods often over- look the heterogeneity among participants. Our work is centered on resolving pivotal challenge in human-centric systems (HCSs): how to select policy deploy when a new participant joining the cohort, without having access to any prior offline data collected over the participant? We introduce First-Glance Off- Policy Selection (FPS), novel approach that systematically addresses participant heterogeneity through sub-group segmentation and tailored OPS criteria each sub-group. By grouping individuals with similar traits, FPS facilitates personalized policy selection aligned with unique characteristics of each participant or group of participants. FPS is evaluated via two important but challenging applications, intelligent tutoring systems and healthcare application for sepsis treatment and intervention. FPS presents significant advancement in enhancing learning outcomes of students and in-hospital care outcomes.
Introduction
Human-centric systems (HCSs), e.g., used in healthcare facilities [53 16 and intelligent education (IE) [5, 26 68). have widely employed reinforcement learning (RL) enhance user experience by improving outcomes of disease treatment, knowledge gaining, etc. Specifically, RL has been used in healthcare automate treatment procedures [53), in IE that induce policies automatically adapting difficulties course materials and helping students to setup and refine study plans to improve learning outcomes 34 84). Though various existing offline RL methods can be adopted [9] 29]
for policy optimization, validation of policies' performance is often conducted by online testing [61 74 69 10] Given the long testing horizon (e.g., several years, semesters, in healthcare, and IE, respectively) and the high cost of recruiting participants, online testing is considered exceedingly time- and desource-consuming and sometimes could even be hindered by protocols overseeing human involved experiments, e.g., performance and safety justifications need to be provided before new medical device controllers be tested on patients (51].
Recently, off-policy evaluation (OPE) methods have been proposed to tackle such challenges by estimating the performance of target (evaluation) RL policies with offline data, which only requires the trajectories collected over behavioral polices given priori; similarly, off off-policy selection (OPS) targets determine the most promising policies, of the ones trained with different algorithms
or hyper-parameter sets, that can be used for online deployment 3 g 46 76, 81]. However, most
existing OPS and OPE methods designed in the context of homogenic agents, such in robotics
"Stanford University. The work was done at North Carolina State University.
Contact:
segaolstanford.edd mchi@ncau.edu.
IBM Research
Duke University
8 North Carolina State University
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

or games, where characteristics of the agents can be captured by their specifications. which are in general assumed fully known (e.g., degree of freedom, angular constraint of each joint).
The pivotal challenge in OPE/OPS for HCSs. In contrast, in HCSs, the participants can have highly diverse backgrounds, where each person may be associated with unique underlying characteristics that are not straightforward to be captured individually; due to the partial observability of participants' mind states and the limited size of the cohort that can be recruited for experiments with HCSs. For example, patients participated in healthcare research studies could have different health/disease records, while the students using an intelligent tutoring system in IE may have different mindsets toward studying the course. As result, the optimal criteria for selecting the policy to be deployed to each participant can vary, and, more importantly, it would be intractable for existing OPS/OPE frameworks to determine what the policy selection criteria would be for a new participant who just joined the cohort. Consequently, there lacks a framework that can resolve the pivotal challenge in facilitating real-world HCSs - how to select policy to deploy when a new participant joining the cohort, without having access to any prior offline data collected over the participant?
In this work, we introduce First-glance off- Selection (FPS), to address the problem of determining the OPS criteria needed for each new participant joining the cohort (i.e., at a 0 only, or without using information obtained from >= onwards), assuming that we have access to offline trajectories for small batch of participants a priori, i.e., the offline data. Specificallyy, it first partitions the participants from the offline dataset into sub-groups, clustering together the ones pertaining to similar behaviors. Then, an unbiased value function estimator, with bounded variance, developed to determine the policy selection criteria for each sub-group. At last, when new participants join, they will be recommended with policies selected according to the sub-groups they fall within. Note that FPS is distinguished from typical off-policy selection (OPS) setup in the sense that, the major goal of prior OPS approaches is to select the best policy over the entire population, while FPS aims to decide the best policy for each student who arrives to the HCS on-the-fly, leveraging the information observed at the initial step (t = 0) only.
The key contributions of this work are summarized as follows: (i) We introduce the FPS framework which is critical for closing the gap between OPS and applications pertaining HCSs, i.e. selecting the policy that would maximize the gain of the new participants at the point of joining cohort. To the best of our knowledge, this is the first framework that considers the new participant arrival's problem in the context of OPS in HCSs. (ii) We conduct extensive experiments to evaluate FPS in real- real-world IE system, with 1,288 students participating over years. Results have shown that, with the help of FPS, it improved the learning outcomes by 208% compared to policy selection criteria hand-crafted by instructors. Moreover, it leads to 136% increased outcome compared to policies selected by existing OPS methods. (iii) FPS is also evaluated against an important healthcare application, i.e. septic shock treatment
[45 48 53 where it can accurately identifying the best treatment policies to be deployed to incoming patients, and outperforms existing OPS methods.
2 First-Glance Off-Policy Selection (FPS)
In this section, we introduce the FPS method, which determines the policy to be deployed to new participants that join an existing cohort, conditioned only their initial states. Specifically, the participants pertaining to the offline dataset are partitioned into sub-groups according to their past behavior. Then, variational auto-encoding (VAE) model is used to generate synthetic trajectories for each sub-group, augmenting the dataset and improving the state-action coverage. Moreover, an unbiased value function estimator, with bounded variance, is developed to determine the policy selection criteria for each sub-group. At last, when new participants join, they will be recommended with the policies conditioned on the sub-groups they fall within respectively. We start with sub- section that introduces the problem formulation formally.
2.1 Problem Formulation
The HCS environment is formulated as a human-centric Markov decision process (HC-MDP), which is a 7-tuple ASP. R, 7). Specifically, S is the state space, A is the action space, P x A - S defines transition dynamics from the current state and action to the next state, So defines the initial state distribution, R : S x A + R is the reward function, I is the set of participants involved in the HCS, 7 E (0, 1] is discount factor. Episodes are of finite horizon T. At each time- step L. in online policy deployment, the agent observes the state 8, E S of the environment, then chooses an action a; E A following the target (evaluation) policy n. The environment accordingly provides reward rt R(st, at), and the agent observes the next state $1+1 determined by P. A
2

---

Problem 2.3. The goal of FPS to select the best policy aT from a set of pre-trained (candidate) policies II. E II. for each of the new participants i' E {N + 1, N + +2.... joining (i.e., arriving at) the HCS with an observable initial state 50 ~ -So(i') (but the rest of the trajectory remain (0) (0)
trajectory is denoted as T [..., (0) -ij , -6 (4.1).... 1E=1. Moreover, we consider having access to a historical trajectory set (i.e., offline dataset) collected under a behavioral policy 3 o x, : == which consist of N trajectories. We first make two assumptions in regards to the correspondence between trajectories and participants, and the initial state distribution for each participant, respectively.
(2)
D3 = ( (6) IN
Assumption 2.1 Sinjectoor-Paatiiippee Correspondence). As participant in human-centric experi- ments is in general unlikely to undergo exactly the same procedure more than once under the topic being studied, we assume that there exist a unique correspondence between each trajectory 7() and the participant (i E I) from which the trajectory is logged.
We henceforth can use i to refer to index either a trajectory from the offline dataset, or the correspond- ing participant, depending on the context.
Assumption 2.2 (Independent Initial State Distributions). The initial state of each trajectory so 7(), corresponding to unique (the i-th) participant following from the assumption above, is sampled from an initial state distribution (i) conditioned on i-th participant's 's characteristics and past records (i.e, specific to the i-th trajectory), and is independent from all other (o)'s 's where j E [1, N >
The assumptions above reflect the scenarios that are specific HCS for example, patient is unlikely to be prescribed the same surgery twice. Even if the patient has to undergo a follow-up surgery that is of the similar type (e.g., mostly seen in trauma or orthopedics departments), the second time when the patient comes in he/she will start with a rather different initial state, since the pathology may have already been intervened as result of the last visit. Consequently, one can treat such a visit as new (synthetic) participant who just join and has the health record same as the one updated after the last visit. other words, a participant being considered in this paper can be generalized, e.g., to hospital visit, or a student participating in specific course supported by intelligent education (IE) systems, depending on the context. Moreover, assumption 2.2 directly follows from the philosophy illustrated in assumption 2.1 - the initial state of each trajectory depend on the corresponding participant's s unique characteristics and historical records before joining the experiment/cohort, and can be considered mutually independent across all participants. Now define the goal for FPS.
unobservable), that maximizes the expected accumulated return V max V over the full horizon T; here v (e,)) (r "e>a)>="" R~ Ei=1 ((--reel)) and p" is the state-action visitation distribution under π from step t = onwards.
Note that the problem formulation here is different than the typical OPS/OPE setup used in existing works (23 66 7 77 79 [4 30) as only the initial state Su is available for policy selection. Such
a formulation is aligned with use cases under HCSs, e.g., treatment plan needs to be laid out soon after a new patient is admitted to the intensive care unit (ICU) in medical centers. However, most indirect OPS methods such as importance sampling (IS) 152 7 and doubly robust (DR) (23 66)
require the entire trajectory to be observed, in order to estimate V", Though direct methods like fitted-Q evaluation (FQE) (30 could be used as workaround, they do not take into account the unique characteristies for each participant that plays crucial role in HCS applications; results in Section show that they in general underperform in the real-world IE experiment. To address both challenges, we introduce the FPS approach, starting with the sub-group partitioning step introduced below.
2.2 Sub-Group Partitioning
In this sub-section, we introduce the sub-group partitioning step that partition the participants in the offline dataset into sub-groun sub-groups. Furthermore, value functions over all candidate policies x E II are learned respectively for each sub-group, to be leveraged as the OPS criteria for each sub-group.
The partitioning is performed over the initial state of each trajectory in the offline dataset, т3 E DD Given assumptions 2.1 and 2.2 and the fact that So('''s in general only share limited support across
'Without loss of generality, in the rest of the paper, we use So to represent the marginal distribution of the initial states over all participants, while Si( represents the distribution specific to the i-th participant.
3

---

participants (i.e., every human has unique characteristics and past experience), such partitioning is essentially performed at per--eeriice per-participant level. Specifically, we consider partitioning the participants into M sub-groups. Then for all sub-groupe Km 's in the set of sub-groups, K. = (K....., KM] J. we have U Kn, Sn and Km n K 0, Vm of n. The total number of groups M needed can be determined using silhouette scores [21]. Denote the partition function k(-)= So + K. We then define the value function specific to each sub-group.
Definition 2.4 (Value Function per Sub- group) The value function over policy x, VR.. specific to the sub-group Km, is the expected accumulative return over the initial states that correspond to the set of participants Imm = {ilk(s))) Km; E I} residing in the same sub-group. Vž =o)=p"" r~ >> r [T], with so ~ Unio((So E Im.))
(S) representing that Sy is sampled from a uniformly weighted mixture of distributions over (So(t)li J. pertaining to sub-group Km-
The goal of sub-group partitioning is to learn the partition function k(-), such that the difference between the value of the best policy candidate, en VR,, and the value of the behavioral policy, Vi is maximized for all participants E I and bub-ggoups Km. E IC, i.e.
Σ IEI
( max V TELL
]
V Km Km==(<5) (..9))
max
K.. <m.=((())))
The objective ([) is designed in the sense that participants may benefit more from the type of policies that fit better for their individual characteristics. For example, in IE, different candidate lecturing policies may be used toward prospective high- and low-performers respectively. as justified by the findings from provided by different policies for a specific type of learners (i.e., sub-group) could be different, measured by VÆ V for all π E II; here, V captures the expected return from a instructor- designed, one-size-fit-al baseline (i.e., behavioral) policy that is used to collect offline data [38 68 84 I.. Then, it would be crucial to identify to which group each student belongs, as it can maximize the returns collected by each student throughout the horizon.
real-world IE experiment (centered around Figure 2 in Section 3.2). The value
Practical off-policy deployment to initiate human-centric experiments over the sub-group objective (T). Our focus is to select the policy that can possibly work the best for each incoming individual from a set of policy candidates that are pre-given, which is critical for RL policy deployment in real-world HCSs, considering online policy optimization can be high-stake. The overall practical off-policy deployment can be achieved using a two-step approach, i.e., (1) pre-partitioning with offline dataset, followed by (ii) deployment upon observation of the initial states of arriving participants. Due to space limitation, the specific steps can be found in Appendix D.I
Proposition 2.5. Define the estimator D as, i.e.,
T
T
D
= Im
W. Σ
>
Σ 2
(6) ri
(2)
TE....
here, Im follows the definition above, which is the set of participants grouped in Kmi Wi II TEIT(ai) (9) st
(0) )/(( (s()) is the IS weight for the i-th trajectory in the offline dataset; st at ,r, are the states, actions, rewards logged in the offline trajectory, respectively. Then, D unbiased, with its variance bounded by, i.e.
(1) (i) (1)
1 N
(Da. <
(3)
ESS
with ESS being the effective sample size (27
The proof of proposition 2.5 is derived from 25 and provided in Appendix D.2
2.3 Trajectories Augmentation within Each Sub-Group
In HCSs, each sub-group may only contain a limited number of participants, due to the high cost of recruiting participants as well as time constraints in real-world experiments. For example, in

---

one sub- group only contains 45 students as result from sub- group partitioning. Consequently, the overall offline trajectories within each group may cover limited visitations of the state and action spaces, and make the downstream policy selection task challenging [46]. Latent-model-basee data augmentation has been commonly employed previous [4], to resolve similar issues. For this sub-section, we specifically consider the variational auto-encoder (VAE) architecture introduced [14,, as it originally designed for offline setup well. Now we briefly introduce the VAE setup, which can capture the underlying dynamics and generate synthetic offline trajectories to improve the state-action visitation coverage within each subgroup. Specifically, given the offline trajectories Tm specific to the subgroup Km, the VAE consists three major components, i.e., (i) the latent prior p(20) that represents the distribution the initial latent states over Tm (łi) the encoder 9, (2 a1-1, that encodes the MDP transitions into the latent space; (iái) the decoders Pe (=< Qt-1). ( 2x), Pe(rt-11 1 22,) that reconstructs new samples. The training objective is formulated derived for the architecture above. More details can be found in Appendix [D3] Consequently, for
the IE experiment in Section
offline RI. [20 31 58
an evidence lower bound (ELBO) specifically
the trajectories in each subgroup. Tm. the VAE can be trained generate set of synthetic samples, denoted as Tm- In the Section 3.2] we further discuss and justify the need of trajectory augmentation through an real-world intelligent education (IE) experiment.
2.4 The FPS Algorithm
Algorithm FPS.
Require: set of target policies II, offline dataset D.
Ensure:
// Training Phase.
1: Calculate the number of subgroups M needed for D. using silhouette scores (21).
2: Obtain the sub-group partitioning K {K1....,KKa KM] following Section 2.2
3: for each sub-group Km do
4: Augment sub-group samples Tm with Tm-
5: Use the estimator in Proposition 2.5 to obtain 8 for all candidate target policies π II, over
Tm U Tm.
6: Select the best candidate target policy ni that maximizes D as one to deployed over Km.
// Deployment Phase.
7: while the HCS receives the initial state from participant do
8: Determine the sub-group Km for the new participant.
9:
Deploy to the participant the best candidate policy n. specific to sub-group Km.
The overall flow of the FPS framework is described in Algorithm I The training phase directly follow from the sub-sections above. Upon deployment, FPS can help HCSs monitor each arriving participant, determine the sub-group the participant falls within, and select the policy to be deployed according to the initial state. Such real-time adaptability is important for HCSs in practice, and is different from existing OPS works which in general assume either the full trajectories or population characteristies are known 25 76 82). For example, in practical IE, students may start learning irregularly according to their own schedules, hence can create discrepancies their start times. Such methods fall short in cases when selecting policies based on population or sub-group information in the upcoming semester - they requires the data from all arriving students are collected upfront, which would be unrealistic. Note that, to the best of our knowledge, we are the first work that formally consider the problem of sub-typing arriving participants, and FPS is the first approach that solves this practical problem by introducing framework that can work with HCSs in the real-world.
Experiments
FPS is tested over two types of HCSs, i.e., intelligent education (IE) and healthcare. Specifically, the real-world IE experiment involves 1,288 student participating college entry-level probability course across 6 academic semesters. The goal is to use the data collected from the students of the first semesters, to assign pre-trained RL lecturing policies to every student enrolled in the 6-th semester, in order to maximize their learning outcomes. The healthcare experiment targets for selecting pre-configured policies that can best treat patients with sepsis, over a simulated environment
widcly adopted in existing works [45 46 65 36 12]
5

---

30
Return of the selec policy 10
True reward
Benchmark policy's s true reward
Estimated reward
Policy reward
20
15
wis, PDIS, FGE (wl)
wrs frew. +VRRS).
WDR ARS
FPS-no TA
FPS-P
FPS
OPS
OPS RRS
OPS VRRS
FPS-noTA
FPS-P
FP5
MAGIC (al)
Figure 1: Analysis of main results from the real-world IE experiment. (a) Overall performance of the 6-th semester's student cohort. Methods that selected the same policy are merged in one bin, i.., all refers to all three variations (raw, +RRS, +VRRS) of the existing OPS baselines. (b) Estimated and true policy performance using each method. For OPE, OPE+RRS, OPE+ VRRS, results with the least gap between estimated and true rewards among OPE methods (i.e., WIS, FQE+RRS, and FQE+ BRE+S respectively) are shown in the figure. True reward refers the returns averaged over the cohort of the 6-th semester, obtained by deploying the policy selected for each student correspondingly.
3.1 Baselines
Existing OPS/OPE. The most straightforward approach to facilitate OPS HCSs is to select policies via existing OPS/OPE methods, by choosing the candidate target policy II that achieves the maximum estimated return over the entire offline datasel, i.e., indiscriminately across all potential sub aub-groups. Specifically, 6 commonly used OPE methods are considered. ie.. Weighted IS (WIS) [52, Per- Decision IS (PD1S) [52], Fitted-Q Evaluation (FQE) [30], Weighted DR (WDR) (66], MAGIC 666 and Dual stationary DIstribution Correction Estimation (DualDICE) 44).
Existing OPS/OPE with vanilla repeated random sampling (OPS+RRS). We also compare FPS against classic data augmentation method in order to evaluate the necessity of the VAE-based method introduced in Section - repeated random sampling (RRS) with replacement of the historical data to perform OPE. RSS has shown superior performance in some human-related tasks, such as disease treatment [66. Specifically, all OPS/OPE methods considered above are applied to the RRS-augmented offline dataset, where the value of each candidate target policy is obtained by averaging over 20 sampling repetitions. However, note that RRS does not intrinsically consider the temporal relations among state-action transitions captured by MDP.
Existing OPS/OPE with VAE-based RRS (OPS+VRRS). This bascline perform OPS with RRS on augmented samples resulted from the VAE introduced in Section 2.3 in order to allow RRS to consider MDP-typed transitions, hence improve state-action visitation coverage of the augmented dataset. This method can, to some extent, be interpreted as an ablation baseline of FPS, by removing the sub-group partitioning step (Section 2.2 and slightly tweaking the VAE-based offline dataset augmentation step (Section 2.3) such that it does not need any sub-group information. Specifically, we set the amount of augmented data identical the amount of original historical data, i.e., IT| |T| N, and RRS N samples from both set 7 to perform OPE. Final estimates are averaged results from 20 repeated sampling processes.
FPS without trajectory augmentation (FPS-noTA). This is the ablation baseline that completely removes from FPS the augmentation technique introduced Section 2.3
FPS for the population (FPS-P). We consider additional ablation baseline that follows the same training steps as FPS (i.e., steps 1-7 of Alg. [], but rather select a single policy that is identified (by FPS) as the best for majority of the sub-gn uub-groups, to be deployed to all participants. In other words, after training, FPS produces the mapping h IC + II, while FPS-P P will always deploy to every arriving participant the policy that appears most frequently in the set (h(Km) ) h K)..

---

3.2 The Real-World IE Experiment
The IE system has been integrated into a undergraduate-level introduction to probability and statistics course over 6 semesters, including a total of 1.288 student participants. This study has received approval from the Institutional Review Board (IRB) at the institution to ensure ethical compliance. Additionally, oversight provided by a departmental committee, which is responsible for safeguard- ing the academic performance and privacy of the participants. In this educational context, each learning session revolves around a student's engagement with a of 12 problems, with this period referred to as an "episode" (horizon 7 = 12). During each step, the IE system offers students three actions: independent work, utilizing hints, or directly receiving the complete solution (primarily for study purposes). The states space is constituted by by 140 features that have been meticulously extracted from the interaction logs by domain experts, which encompass various aspects of the students' activities, such as the time spent each problem and the accuracy of their solutions. The learning outcome is issued as the environmental reward at the end of each episode (0 reward for all other steps), measured by the normalized learning gain (NI.) quantified using the scores received from two exams, i.e., one taken before the student start using the system, and another after. Data collected from the first 5 semesters (over lecturer-designed behavioral policy) are used to train FPS for selecting from a set of candidate policies to be deployed to each student in the cohort of the 6-th semester, including 3 pre-trained RL policies and benchmark policy (whose performance benchmark the lower-bound boond of what could be tested with student participants). See Appendix for the definition of NLG, details pre- pre-trained RL policies, and more.
Main results. Figure presents students' perfor- mance under policies selected by different methods.
40
Overall, FPS was the most effective policy selec- tion leading to the greatest average student perfor-
mance. The return difference between FPS and the two ablation, TP and FPS-P, illustrate the importance of augmenting offline trajectories (as introduced in Section 2.3) and assign to arriv- ing students policies that better fit the character-
Students performance
20
istics shared within their sub-groups, respectively. Moreover, most existing OPS/OPE methods tend to select sub-optimal policies that resulted in bet- ter learning gain than the benchmark policy. Note
-2
that we also observed that DuaIDICE could not dis- tinguish the returns over all target policies; thus,
is unable be used for policy selection in this experiment and we omit its results. It is also impor- tant evaluate how accurate the value estimation V= would be for the best candidate policy selected across all methods, over the arriving student cohort at the 6-th semester, as illustrated in Figure [(b) FPS provided more accurate policy estimation by achieving the smallest error between true and es-
OPS (raw, +RRS, -VRRS)
40
FPS
K₁
K₂
K3
K4
Subgroup
Figure 2: Performance of students (mean over all four sub-groups under selected policies in the 6-th semester.
timated policy rewards. With VRRS, most OPS methods improved their policy estimation performance, which was benefited from the richer state- action visitation coverage provided by the synthetic samples generated by VRRS. However, even with such augmentations, existing OPS methods still chose sub-optimal policies, which justified the importance of considering participant-sper partitioning in FPS (Section 2.2
characteristics in HCSs, which is tackled by sub- soougroup
More discussions. For more comprehensive understanding of student behaviors affected by the policy being deployed in IF, we further investigate how the sub-groups are partitioned and how the policies being assigned to each sub-group perform. Specifically, FPS identified four subgroups (i.e., KI, K2, Ka, K4) result of Section 2.2 Under the behavioral policy, the average NLG across all students is 0.9 with slight improvement after tutoring. Specifically, K (NNtrvnn 345, N Nest = 30) and K2 [95% CI. 0.6, 0.8] under the behavioral policy, respectively. In the testing (6-th) semester, FPS
678, Ntest = 92) achieved average NLG of 1.9 /95% CI. 1.7, 2. and 0.7
*CI stands for confidence interval.
7

---

Table 1: The absolute errors (AEs) and returns resulted from deploying to each patient the corre- sponding candidate policy selected by FPS against baselines, as well as the top- regret (regret I) of the selected policy, averaged over 10 different simulation runs. Standard errors are rounded.
HPS
WIS
PIDIS
FQR
WDR
MAGIC:
N=2,500
AE
0.026:0.00
0.054+0.00
0.109:0.00
0.07010.01
8.281+0.00
5.68110.00
Return
0.132+0.02
0.121#0.01
0.121#0.01
0.129+0.00
0.121#0.01
0.12940.00
Regret
0.042+0.01
0.066#0.00
0.066#0.00
0.106+0.09
0.066+0.00
0.106=0.09
N=5,000
AE
0.00
0.046+0.00
0.082+0.00
0.073+0.01
3.443+0.01
3.238+0.01
Return
0. 0.14
0.123:0.01
0.123=0.01
0.121
0.123+0.01
0.12310.01
@
0.020=0.00
0.050=0.00
0.050=0.00
0,208=0.13
0.050=0.00
0.050=0.00
N=10,000
AE
0.022+0.00
0.022=0.00
0.097
0,105#0.00
0.995#0.01
1.210=0.01
Return
0.130::.000
(..1290..00
0.121#0.00
0.121+0.00
0.121+0.00
0.121#0.00
Regret
0.01620.00
0.019+0.00
0.029+0.01
0.029+0.01
0.029+0.01
0.029+0.01
constantly selected the best performing policy for students identified sub-groups K₁ and K2, with learning outcomes improvement quantified as 17.7 [95% CI, -1.7, 37./] and 13.6 (95% CI, -5.2, 32.4] in terms of NLGs, respectively, as shown in Figure 2 This is on the same level achieved by the best possible baseline combinations worked for each student, regardless of the base OPS algorithm used, i.e., the union of the best performance reported from the 18 baselines involving existing OPS methods (introduced in the first 3 paragraphs of Section 3.1 over each student. However, note that in reality one does not have access to such an oracle in terms of which out of the 18 baseline methods would work well for each arriving student upfront (i.e., at the beginning of the semester). In contrast, FPS achieved the same level of performance without the need for an oracle. Note that sub-groups K₁ and 22, were less sensitive to target policies and achieved positive NL.G in both training and testing semester. On the other hand, offline data over the behavioral policy showed that Ka(Nerai
101, Ntest = 12) and K2 (Nerain = 24, Ntest = -6) are associated with negative average NLGs of -0.5 (95% CI, 1.7, 2.1] and -1. 195% CI. -0.3, 1.2). respectively, which can be considered low performers. It is observed that students in sub-group K3 performed kept moving rapidly among questions while working on the IE system, indicating that they were not seriously tackling any one of the questions; while participants in K₁ abused hints, but still made much more mistakes in the meantime. Figure 2 also presents the NLG of students from the low- pewforrooning subgroups K3 and K₃ under policies selected by the best existing- OPS baselines (following the oracle as above) and FPS. Under FPS, both subgroups achieved significant improvement (average NLGs 24.0 /95% CI, 10.2, 37.7] and 31.7 (95% CI, 10. 1, 53.3), respectively) compared to students in historical semesters. However, the sub-optimal policy chosen by baselines had a negative effect on both sub-groups (average NLGs -1.7 (95% CI, -18.1 -5.3] and 1999 (95% CI, 172.1, 32.4], respectively); see Figure [(a) Such an observation particularly justifies the need for personalizing the policies deployed different type of participants (i.e., students), especially for the sub-groups (i.e., (ow-performers), since they can be more sensitive to policy selection. Based on the statistics reported above, FPS improved the NLG of students by 208% over the lecturer- designd behavioral policy, and by 136% over the union of the best performance achieved across existing- OPS-based baselines. Though the difference may not be large for potential gigh-perfrrmin sub- sub-groups (e.g.. K1 & K2), we observed that the baselines can even have a negative effect on some sub-groups (e.g..K3 & K4), which is undesired in human-centric experiments. In empirical human-centric scenarios, such as education, the behavior policy is generally highly regularized by department committees strictly following guidance from human-centric experiments, such that the target policy would not be dramatically opposed to the behavior policy- - suggesting the underlying assumption that the divergence between behavior and target policies could be intrinsically bounded. To this end, the FPS framework has the potential to facilitate fairness in RL-empowered HCSs in general we have discussed this in details in Appendix A.5
3.3 The Healthcare Experiment
In this experiment, we consider selecting the policy that can best treat sepsis for each patient in the ICU, leveraging the simulated environment introduced by 48], which has been widely adopted in Specifically, the state space is constituted by a binary indicator for diabetes, and four vital signs (heart rate, blood pressure, oxygen concentration, glucose level) that take values in a subset of (very high, high, normal, low, very low); size of the state space is [S] = 1440. Actions are captured by combinations of three binary treatment options, (antibiotics, vasopressors, mechanical ventilation). which lead to |A| = 23. Three candidate target policies are considered and provided by [45). i.c., (i) without antibiotics (WOA) which does not administer antibiotics right
existing works [2 46 65
36 45)
8

---

after the patient is admitted, (ii) with antibiotics (WA) that always administer antibiotics once the patient is admitted, (tit) an RL policy trained following policy iteration (PI). Note that as pointed by [45 the true returns of WA and PI are usually close, since antibiotics are in general helpful for treating sepsis, which is also observed in our experiment; see [ Moreover, a simulated unrecorded comorbidities is applied to the cohort, capturing the uncertainties caused by patient's s underlying diseases (or other characteristics), which could reduce the effects of the antibiotics being B for more details in regards to the environmental setup.
administered. See Appendix
Given the simulated environment, we mainly consider using this experiment to evaluate the source of improvement brought in by the sub-group partitioning step (Section 2.2) in FPS. Specifically, multiple scaled offline datasets are generated, representing different degrees of the state-action visitation coverage we vary the total number of trajectory N= 5,000, 10,000), in lieu of performing trajectory augmentations for both FPS and existing OPS baselines. In other words, this experiment, we consider the FPS without the VAE augmentation step introduced in Section 2.3 as well as the 6 original OPS baselines (without any RRS/VRRS) introduced in Section [31] We believe this setup would help isolate the source of improvements brought in by sub-group partitioning. The average absolute errors (AEs), in terms of OPE, and returns, in terms of OPS, resulted from deploying to each patient the corresponding candidate policy selected by FPS against baselines, are reported in Table It can be observed that FPS achieved the lowest AE and highest return regardless of the size of the offline dataset. We additionally evaluate the top-1 regret (i.e., regret 1) of the selected policy It can be observed that FPS achieved
following FPS and baselines, which are also reported in Table !| exceedingly low regrets compared to basclines. Both observations emphasize the effectivenesss of the sub-group partitioning technique leveraged by FPS, as the environment does capture comorbidities as part of the participant characteristics. Moreover, the AEs and regrets of most methods decrease when the size of offline dataset increase, justifying that improved state-action visitation coverage provided by the offline trajectories crucial for reducing estimation errors and improving policy selection outcomes (i.e., the motivation of trajectory augmentation introduced in Section 2.3
4 Related Works
Off-policy selection (OPS). OPS are typically approached via OPE in existing works, by estimating
the expected return of target policies using historical data collected under a behavior policy. A
variety of contemporary OPE methods has been proposed, which can be mainly divided into three
categories [72: (i) direct methods that directly estimate the value functions of the evaluation
policy [44 67 79 76), including but not limited to model-based estimators (MB) [17 14, 15 79].
value-based estimators (30 such as Fitted Q Evaluation (FQE), and minimax estimators
35
80 70
such as DuaIDICE (77): (if) inverse propensity scoring, or indirect methods [52]. such as Importance
Sampling (IS) 77: (ii)) hybrid methods combine aspects of both inverse propensity scoring and direct
methods (66). such as DR (23). In practice, due to expensive online evaluations, researchers generally selected the policy with the highest estimated rewards via OPE. For example, Mandel et al. selected the policy with the maximum IS score to be deployed to an educational game (38] Recently, some works focused on estimator selection or hyperparameter tuning in off- off-policy selection [46] 75 63 43
28 32 65 50). However, retraining policies may not be feasible in HCSs as online data collection
time- and resource-consuning More importantly, prior work generally selected policies without considering the characteristics of participants, while personalized policy is flavored towards the needs specific to HCSs.
RL-empowered automation in HCSs. In modern HCSs, RL has raised significant attention toward enhancing the experience of human participants. Previous studies have demonstrated that RL can induce IE policies M 38 60, 73). For example, Zhou al. [84] applied hierarchical reinforcement learning (HRL) to improve students' normalized learning gain in a Discrete Mathematics course, and the HRL-induced policy was more effective than the Deep Q-Network induced policy. Similarly, in healthcare, RL has been used to synthesize policies that can adapt high-level treatment plans [33] 45
(36) or to control medical devices and surgical robotics from a more granular level [66] 55). Since
online evaluation/testing is high-stake in practical HCSs, effective OPS methods are important in closing the loop, by significantly reducing the resources needed for online testing/deployment and preemptively justifying safety of the policies subject to be deployed.
9

---

Conclusion and Limitation
In this work, we introduced the FPS framework that facilitated policy selection in real-world HCSs; tackled the off-pofft off-policy deployment with new arrivals problem that is pivotal for RL policy deployment in HCSs. Unlike existing OPS methods, FPS customized the policy selection criteria for each sub- group respectively. FPS was tested in real-world IE experiment and simulated sepsis treatment environment, which significantly outperformed baselines. Though the future it would be possible to extend FPS to offline RL policy optimization framework, however, in this work we specifically focus on the OPS task in order to isolate the source improvements brought by sub-group partitioning and trajectory augmentation. Future avenues along the line of FPS also include deriving estimators (for Proposition 2.5) that allow bias- varianar trade off, e.g., by integrating WDR or MAGIC (to substitute the IS weights). Societal and broader impacts are discussed in Appendix C
Compared IE systems, HCSs in healtheare would be considered even more high-stakes, thus may further limit the options (i.e., policies) that are available to facilitate sub-grouping experiments, due to stricter clinical experimental guidelines. However, FPS has demonstrated its extraordinary capabilities over a real-world experiment that involved >1,200 participants with years of follow-ups, which showed its efficacy and scalability toward working with more challenging systems and larger cohorts in healthcare, as the assumptions needed by FPS across these two systems would not change fundamentally. Moreover, potential underlying confounding may exist across the patient's s initial states in healthcare, and it is also important to consider inputs from healthcare professionals during sub-grouping. As a result, one may further extend our framework toward such a direction, allowing it to function better in the healthcare domain.
Acknowledgments
This research was supported by the NSF Grants: Integrated Data-driven Technologies for Individ- ualized Instruction in STEM Learning Environments (1726550), CAREER: Improving Adaptive Decision Making in Interactive Learning Environments (1651909). Generalizing Data-Driven Tech- nologies to Improve Individualized STEM Instruction by Intelligent' Tutors (2013502), NATAD Award (2332744), and National AI Institute for Edge Computing Leveraging Next Generation Wireless Networks, Grant CN5-2112562. This research was also sponsored in part by the AFOSR under award number FA9550-19- 19-1-0169. We would also like to thank the anonymous reviewers for insightful comments that lead improved paper presentations.
References
[1] Mark Abdelshiheed. John Wesley Hostetter, Tiffany Barnes, and Min Chi. Leveraging deep reinforcement learning for metacognitive interventions across intelligent tutoring systems, In International Conference on Artificial Intelligence in Education, pages 291-303. Springer, 2023.
[2] Karo Wuntch, Alireza Ahadi, and Andrew Petersen. Evaluating neural networks method for identifying students in need of assistance. In Proceedings of the 2017 ACM SIGCSE technical symposium computer science education, pages 11 111-116, 2017.
[3] Yash Chandak, Shiv Shankar, Nathaniel Bastian, Bruno da Silva, Emma Brunskill, and Philip Thomas. Pof-py evaluation for eetion-dependent non-stationary environments. Advances in Neural Information Processing Systems, 35:9217-9232, 2022.
[4] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 14:45084-5 2021.
[5] Min Chi, Kurt VanLehn, Diane Litman, and Pamela Jordan. Empirically evaluating the applica- tion of reinforcement learning to the induction of effective and adaptive pedagogical strategies. User Modeling and Adaptdd Interaction, 21(():137-180, 2011.
[6] Corina Cortes, Yishay Mansour, and Mehryar Mohri. Learning bounds for importance weighting, Advances in neural information processing systems, 23, 2010.
10

---

[7] Shayan Doroudi, Philip S Thomas, and Emma Brunskill. Importance sampling for fair policy selection. Grantee Submission, 2017.
[8] Salma Elmalaki. Fair-iot: Pairness-aware human- sin-thh-loop reinforcement learning for har- nessing human variability in personalized iot. In Proceedings of the International Conference on Internet-of-Thing Design and Implementation, pages 119-132, 2021.
[9] Justin Fu, Mohammad Norouzi, Ofr Nachum, George Tucker, Alexander Novikov, Mengjiao Yang, Michael R Zhang, Yutian Chen, Aviral Kumar, Cosmin Paduraru, et al. Benchmarks for deep off-policy evaluation. In International Conference on Learning Representations, 2021.
[10] Justin Fu, Mohammad Norouzi, Ofir Nachum, George Tucker, Ziyu Wang, Alexander Novikov, Mengjiao Yang. Michael R Zhang, Yutian Chen, Aviral Kumar, et al. Benchmarks for deep off-policy evaluation. arXiv preprint arXiv:2103. 2021.
[11] Ge Gao, Qitong Gao, Xi Yang, Song Ju, Miroslav Pajic, and Min Chi. On trajectory aug- mentations for off- offlpo evaluation. In The Twelfth International Conference on Learning Representations, 2024.
[12] Ge Gao, Song Ju, Markel Sanz Ausin, and Min Chi. Hope: Human-centric off-policy evalu- ation for e-learning and healthcare. In Proceedings of the 2023 International Conference Autonomous Agents and Multiagent Systems, pages 1504-1513, 2023.
[13] Ge Gao, Samiha Marwan, and Thomas W Price. Early performance prediction using inter- pretable patterns in programming process data. In Proceedings of the 52nd ACM technical symposium on computer science education, pages 342-348, 2021.
[14) Qitong Gao, Ge Gao, Min Chi, and Miroslav Pajic. Variational latent branching model for off-policy evaluation. In The Eleventh International Conference Learning Representations, 2022.
[15] Qitong Gao, Ge Gao, Juncheng Dong, Vahid Tarokh, Min Chi, and Miroslav Pajic. Off-policy evaluation for human feedback. In Proceedings of the 37th International Conference on Neural Information Processing Systems, pages 9065-9091 2023.
[16] Qitong Gao, Michael Naumann, Ilija Jovanov, Vuk Lesi, Karthik Kamaravelu, Warren M Grill, and Miroslav Pajic. Model-based design of closed loop deep brain stimulation controller using reinforcement learning. In 2020 ACM/IEEE / Ith International Conference on Systems (ICCPS), pages 108-118. IEEE, 2020.
[17] Qitong Gao, Stephen L Schmidt, Karthik Kamaravelu, Dennis A Turner, Warren M Grill, and Miroslav Pajic. Offline policy evaluation for learning-based deep brain stimulation controllers. In 2022 ACM/IEEE 13th International Conference on Cyber-Physical Systems (ICCPS), pages 80-91. IEEE, 2022.
[18] Karan Goel, Albert Gu, Yixuan Li, and Christopher Re. Model patching: group performance gap with data augmentation. In International Conference on Learning Representations, 2021.
Closing the sub-
[19] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pages 1861-1870. PMLR, 2018
[20] Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In International Conference on Learning Representa- tions, 2020.
[21] David Hallac, Sagar Vare, Stephen Boyd, and Jure Leskovec. Toeplitz inverse covariance clustering of multivariate time series data. ACM SIGKDD Int. Conf. on Knowledge Discovery and Data Mining, pages 215-223, 2017.
[22] Botao Hao, Xiang Ji, Yaqi Duan, Hao Lu, Csaba Szepesvari, and Mengdi Wang. Bootstrapping fitted q-evaluation for off-policy inference. In International Conference on Machine Learning, pages 4074-4084. PMLR, 2021.

---

[23] Nan Jiang and Lihong Li. Doubly robust off off-policy value evaluation for reinforcement learning. In International Conference on Machine Learning, pages 652-661. PMLR, 2016.
[24] Song Ju. Identify critical pedagogical decisions through adversarial deep reinforcement learning. In In: Proceedings of the 12th International Conference on Educational Data Mining (EDM 2019), 2019.
[25] Ramtin Keramati, Omer Gottesman, Leo Anthony Celi, Finale Doshi- and Emma Brun- skill. Identification of subgroups with similar benefits in off-policy policy evaluation. In Conference Health, Inference. and Learning, pages 397-410. PMLR, 2022.
[26] Kenneth Koedinger, John R. Anderson, William H. Hadley, and Mary A. Mark. Intelligent tutoring goes to school in the big city. International Journal of Artificial Intelligence in Education, 8(1):30-43, 1997.
[27] Augustine Kong. A note on importance sampling using standardized weights. University of Chicago, Dept. of Statisties, Tech. Rep, 348, 1992.
[28] Aviral Kumar, Anikait Singh, Stephen Tian, Chelsea Finn, and Sergey Levine. A workflow for offline model-free robotic reinforcement learning, In Conference on Robot Learning, pages 417-428. PMLR, 2022.
[29] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1 1 1191, 2020.
[30] Hoang Le, Cameron Voloshin, and Yisong Yue. Batch policy learning under constraints. In International Conference on Machine Learning, pages 3703-3712. PMLR, 2019.
[31] Alex X Lee, Anusha Nagabandi, Pieter Abbeel, and Sergey Levine. Stochastic latent actor-critic: Deep reinforcement learning with a latent variable model. Advances Neural Information Processing Systems, 33:741-752, 2020.
[32] Jonathan Lee, George Tucker, Ofir Nachum, and Bo Dai. Model selection in batch policy optimization. In International Conference on Machine Learning, pages 12542-12569. PMLR, 2022.
[33] Bruno Lepri, Nuria Oliver, and Alex Pentland. Ethical machines: The human-centric use of artificial intelligence. IScience, 24(3), 2021.
[34] Evan Liu, Moritz Stephan, Allen Nie, Chris Picch, Emma Brunskill, and Chelsea Finn. Giv- ing feedback on interactive student programs with meta-exploration. Advances in Neural Information Processing Systems, 35:36282-36294, 2022.
[35] Qiang Liu, Lihong Li, Ziyang Tang, and Dengyong Zhou. Breaking the curse horizon: Infinite-horizon off-policy estimation. Advances in Neural Information Processing Systems, 31, 2018.
[36] Guy Lorberbom, Daniel D Johnson, Chris J Maddison, Daniel Tarlow, and Tamir Hazan. Learn- ing generalized gumbel-max causal mechanisms. Advances in Neural Information Processing Systems, 24:26792-26805 2021.
[37] Meili Lu, Xile Wei. Yanqiu Che, Jiang Wang. and Kenneth A Loparo. Application of reinforee- ment learning to deep brain stimulation in a computational model of parkinson's 's disease. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2831)339,339, 2019.
[38] Travis Mandel, Yun-Eu En Liu, Sergey Levine, Emma Brunskill, and Zoran Popovic. Offline policy evaluation across representations with applications educational games. In AAMAS, volume 1077, 2014.
[39] Ye Mao. One minute is enough: Early prediction of student success and event-level difficulty during novice programming tasks. In In: Proceedings of the 12th International Conference on Educational Data Mining (EDM 2019), 2019.
12

---

[40] Samiha Marwan, Anay Dombe, and Thomas W Price. Unproductive help-seeking in pro- gramming: What it and how address In Proceedings of the 2020 ACM conference on innovation and technology in computer science education, pages 54-60, 2020.
[41] Alberto Maria Metelli, Matteo Papini, Francesco Faccio, and Marcello Restelli. Policy opti- mization via importance sampling. Advances in Neural Information Processing Systems, 31, 2018.
[42] Blossom Metevier, Stephen Giguere, Sarah Brockman, Ari Kobren, Yuriy Brun, Emma Brun- skill, and Philip S Thomas. Offline contextual bandits with high probability faimess guarantees. Advances in neural information processing systems, 32, 2019.
[43] Kohei Miyaguchi. A theoretical framework of almost hyperparameter--fie free hyperparameter selection methods for offline policy evaluation. arXiv e-prints, pages arXiv-2201, 2022.
[44] Ofir Nachum, Yinlam Chow, Bo Dai, and Lihong Li. Dualdice: Behavior- estimation of discounted stationary distribution corrections. Advances in Neural Information Processing Systems, 32, 2019.
[45] Hongseok Namkoong, Ramtin Keramati, Steve Yadlowsky, and Emma Brunskill. Off-policy policy evaluation for sequential decisions under unobserved confounding. Advances Neural Information Processing Systems, 33: 33:18818 19-1188,3, 2020.
[46] Allen Nie. Yannis Flet-Berliac, Deon Jordan, William Steenbergen, and Emma Brunskill. Data- efficient pipeline for offline reinforcement learning with limited data. Advances in Neural Information Processing Systems, 35: 15:10411-448,,,, 2022.
[47] Allen Nie, Ann-Katrin Reuel, and Emma Brunskill. Understanding the impact of reinforcement learning personalization subgroups of students in math tutoring. In International Conference on Artificial Intelligence in Education, pages 688-694, Springer, 2023.
[48] Michael Oberst and David Sontag. Counterfactual off- off-policy evaluation with gumbel-max structural causal models. International Conference on Machine Learning, pages 4881-4899 PMLR, 2019.
[49] Art Owen. Monte carlo theory, methods and examples. 2013.
[50] Tom Le Paine, Cosmin Paduraru, Andrea Michi, Caglar Gulcehre, Konrad Zolna, Alexander Novikov, Ziyu Wang, and Nando de Freitas. Hyperparameter selection for offline reinforcement learning. arXiv preprint arXiv:2007.088 000005000090..... 2020.
[51] Bahram Parvinian, Christopher Scully, Hanniebey Wiyor, Allison Kumar, and Sandy Weininger. Regulatory considerations for physiological closed-loop controlled medical devices used for automated critical care: food and drug administration workshop discussion topics. Anesthesia and analgesia, 126(6):1916, 2018
[52] Doina Precup. Eligibility traces for off- off-policy policy evaluation. Computer Science Department Faculty Publication Series, page 80, 2000.
[53] Aniruddh Raghu, Matthicu Komorowski, Imran Ahmed, Leo Celi, Peter Szolovits, and Marzych Ghassemi. Deep reinforcement learning for sepsis treatment. arXiv preprint arXiv:1711.09603 2017.
[54] Alfréd Rényi. On measures of entropy and information. In Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability, Volume 1: Contributions the Theory of Statistics, volume 4, pages 547-562. University of California Press, 1961.
[55] Florian Richter, Ryan K Orosco, and Michael C Yip. Open-sourced reinforcement learning environments for surgical robotics. arXiv preprint arXiv: 1903900009990 2019.
[56] Sherry Ruan, Allen Nie, William Steenbergen, Jiayu He, JQ Zhang, Meng Guo, Yao Liu, Kyle Dang Nguyen, Catherine Wang, Rui Ying, al. Reinforcement learning tutor better supported lower performers in a math task. arXiv preprint 2304::23000099333 2023.
13

---

[57] Havard Rue and Leonhard Held. Gaussian Markov random fields: theory and applications. CRC press, 2005.
[58] Oleh Rybkin, Chuning Zhu, Anusha Nagabandi, Kostas Danilidis, Igor Mordatch, and Sergey Levine. Model-based reinforcement learning via latent-space collocation. In International Conference Machine Learning, pages 9190-9201. PMLR, 2021.
[59] Rolf Schwonke, Alexander Renkl, Carmen Krieg, Jörg Wittwer, Vincent Aleven, and Ron Salden. The worked-example effect: Not an artefact of lousy control conditions. Computers in human behavior, 25(2):258- 266, 2009.
[60] Shitian Shen and Min Chi. Reinforcement learning: the sooner the better, or the later the better? In Proceedings of the 2016 conference on user modeling adaptation and personalization, pages 37-44, 2016.
[61] David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362(6419):1140 1400) 1144, 2018.
[62] Robert R Sinclair, James E Martin, and Robert P Michel. Full-time and part-time subgroup differences in job attitudes and demographic characteristics. Journal of Vocational Behavior, 55(3):337-357, 1999.
[63] Yi Su, Pavithra Srinath, and Akshay Krishnamurthy. Adaptive estimator selection for off-policy evaluation. In International Conference on Machine Learning, pages
PMLR, 2020.
[64] John Sweller and Graham A Cooper. The use of worked examples substitute for problem solving in learning algebra. Cognition and instruction, 2(1):59 89, 1985.
[65] Shengpu Tang and Jenna Wiens. Model selection for offline reinforcement learning: Practical considerations for healthcare settings. In Machine Learning for Healthcare Conference, pages 2-35. PMLR, 2021.
[66] Philip Thomas and Emma Brunskill. Data-efficient off-policy policy evaluation for reinforce- ment learning. In International Conference on Machine Learning, pages 2139-2148. PMLR, 2016.
[67] Masatoshi Uehara, Jiawei Huang, and Nan Jiang. Minimax weight and q-function learning for off-policy evaluation. In International Conference on Machine Learning, pages 9659-9668. PMLR, 2020.
[68] Kurt VanLehn. The behavior of tutoring systems. International Journal Artificial Intelligence in Education. 16(3):227-265. 2006.
[69] Oriol Vinyals, Igor Babuschkin. Wojeiech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Jun- young Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(77822:300-354 2019.
[70] Cameron Voloshin. Nan Jiang, and Yisong Yue. Minimax model learning. In International Conference on Artificial Intelligence and Statistics, pages 1612-1620. PMLR, 2021.
[71] Cameron Voloshin, Hoang M Le. Nan Jiang, and Yisong Yue. Empirical study of off off-policy policy evaluation for reinforcement learning. arXiv preprint 19/1v:11/110006 2019.
[72] Cameron Voloshin, Hoang Minh Le, Nan Jiang, and Yisong Yue. Empirical study of off- off-policy
policy evaluation for reinforcement learning. In Thirty-fftth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021.
[73] Pengcheng Wang, Jonathan Rowe, Wookhee Min, Bradford Mott, and James I Lester. Interactive narrative personalization with deep reinforcement learning. Proceedings of the Twenty-Sixth International Joint Conference Artificial Intelligence, 2017.
14

---

[74] Peter R Wurman, Samuel Barrett, Kenta Kawamoto. James MacGlashan, Kaushik Subrama- nian, Thomas J Walsh, Roberto Capobianco, Alisa Devlic, Franziska Eckert, Florian Fuchs. et al. Outracing champion gran turismo drivers with deep reinforcement learning. Nature, 202(7896):223-228, 2022.
[75] Tengyang Xie and Nan Jiang. Batch value-function approximation with only realizability. In International Conference Machine Learning, pages 1 11404-1110 11413. PMLR, 2021.
[76] Mengjiao Yang, Bo Dai, Ofir Nachum, George Tucker, and Dale Schuurmans. Offline policy selection under uncertainty. In International Conference on Artificial Intelligence and Statistics, pages 4376-4396. PMLR, 2022.
[77] Mengjiao Yang, Ofir Nachum, Bo Dai, Lihong Li, and Dale Schuurmans. Off-policy evaluation via the regularized lagrangian. Advances in Neural Information Processing Systems, 33:6551- 6561, 2020.
[78] Xi Yang, Yuan Zhang, and Min Chi. Multi-series time-aware sequence partitioning for disease progression modeling. In IJCAI, 2021.
[79] Michael R Zhang, Tom Le Paine, Ofir Nachum, Cosmin Paduraru, George Tucker, Ziyu Wang, and Mohammad Norouzi. Autoregressive dynamics models for offline policy evaluation and optimization. arXiv preprint 1rX.v:2104.3385 2021.
[80] Shangtong Zhang, Bo Liu, and Shimon Whiteson. Gradientdice: Rethinking generalized offline estimation of stationary values. In International Conference on Machine Learning, pages 11194-11203. PMLR, 2020.
[81] Siyuan Zhang and Nan Jiang. Towards hyperparameter free policy selection for offline rein- forcement learning. Advances in Neural Information Processing Systems, 34:12864-12875, 2021.
[82] Rujie Zhong, Duohan Zhang, Lukas Schäfer, Stefano Albrecht, and Josiah Hanna. Robust on-policy sampling for data-efficient policy evaluation in reinforcement learning. Advances in Neural Information Processing Systems, 35:37376-37388, 2022.
[83] Guojing Zhou, Hamoon Azizsoltani, Markel Sanz Ausin, Tiffany Barnes, and Min Chi. Hierar- chical reinforcement learning for pedagogical policy induction. In International conference on artificial intelligence in education, pages 544-556. Springer, 2019.
[84] Guojing Zhou, Hamoon Azizsoltani, Markel Sanz Ausin, Tiffany Barnes, and Min Chi. Lever- aging granularity: Hierarchical reinforcement learning for pedagogical policy induction. Inter- national journal of artificial intelligence in education, 32(2):454-505 2022.
15

---

List of Appendices
Appendix 17
Detailed Setup of the IE Experiment and Additional Discussions
17
A.1
The IE System for the College Entry- Level Course.
17
A.2
Classroom Setup)
17
A.3
Environmental Setup of the System
18
A.3.1
State Features.
18
A.3.2
Actions & rewards.
18
A.3.3
Behavior policy.
18
A.3.4
Target (evaluation) policies.
18
A.3.5
Sub-group identification,
19
A.4
Data Pre-Processing for Sub-Group P Partitioning with the IE Experiment
19
A.5
More Discussions over the Results from Section 3.2
20
B
Detailed Setup of the Healthcare Experiment
23
States & actions.
23
B.2
Rewards.
23
B.3
Optimal policy.
23
B.4
Behaivor policy.
23
B.5
Target policies.
23
Societal and Broader Impacts
23
C.1
Societal Impacts
23
Broader Impact on Facilitating Fairness Rl.-Eepowereed HCSS
23
D More the Methodology
24
D.I Practical Off-Policy Deployment in HCSs over the Sub-Group Objective (1).
24
D.2 Proof of B
24
D.3 Detailed Formulation of VAE in MDP
25
D.4 Proof of Equation
25
More Experimental Setup
27
Training Resources
27
E.2
Implementations and parrmparrmeters
27
OPE Standard Evaluation Metrics
27
16

---

A Detailed Setup of the IE Experiment and Additional Discussions
A.1 The IE System for the College Entry-Level Course.
Though the problem setting and our method are general and can be applied to other interactive IE systems, we primarily focus on the system specifically used in an undergraduate probability course at a university, which has been extensively used by over 1, 288 students with ~800k recorded interaction logs through 6 academic years. The IE system is designed to teach entry-level undergraduate students with ten major probability principles, including complement theorem, mutually exclusive theorem, independent events, De Morgan's theorem, addition theorem for two events, addition theorem for three events, conditional independent events, conditional probability, total probability theorem, and Bayes' rule.
Each students went through four phases, including (i) reading the textbook; (ii) pre-
Problom
Given event AentAAee kadd B with p/Ap-0., 516-0055 and p--A--66-0.2
2. Determine P/A (B)
exam; (ii) studying on the IE system; and
(iv) post-exam. During the reading text-
Hialory
(IA)
book phase, students read a general de-
PiB) 0.5
RE-AAT-B)
Passs quidos Magir Aulif Juan lutto
scription of each principle, review exam-
p(AuB)) Nome MATAAGEET VARAABLE E
ples, and solve some training problems to
Ri-(-An-B10 None
get familiar with the IE system. Subse-
Response
quently, they take a pre-exam comprising total of 14 single- and multiple-prine multiple-principle problems. During the pre-exam, students not provided with feedback on their answers, nor are they allowed to go back to earlier questions (so the post-exam). Then, students proceed to work on the IE system, where they receive the same 12 problems in predetermined order. Af- ter that, students take the 20- 20-problem post- exam, where 14 of the problems are iso- morphic to the pre-exam and the remain- ders are non-isomorphic multiple-principle problems. Exams are auto-graded following the same grading criteria set by course instructors.
"JAMIEEU vdi piCi for pot Actioon Thaser
Figure 3: Graphical user interface (GUI) of the IE sys- tem. The problem statement window (top) presents the statement of the problem. The dialog window (middle right) shows the message the tutor provides to the stu- dents. Responses, e.g., writing an equation, are entered in the response window (bottom right). Any variables and equations generated through this process are shown on the variable window (middle left) and equation win- dow (bottom left).
Since students' underlying characteristics and mind states are inherently unobservable [38], the IE system defined its state space with 142 features that could possibly capture students' learning status based on their interaction logs, as suggested by domain experts. While tutoring, the agent makes decisions on two levels of granularity: problem-level first and then step-level. For problem-level, it first decides whether the next problem should be a worked example (WE) [64], problem-solving (PS), or a collaborative problem-sol solving worked example (CPS) (59 In WEs, students, observe how the tutor solves a problem; in PSs, students solve the problem themselves; in CPSs, the students and the tutor co-construct the solution. If CPS selected, the tutor will then make step-level decisions on whether to elicit the next step from the student or to tell the solution step to the student directly. Besides post-exam score, another important measure of student learning outcomes is their normalized learning gain (NLG), which is calculated by their pre- and post-exam scores LL score scoreposteram scorepresnam
The NLG defined in [5) represents the extent to which students have benefited from the IE system in terms of improving their learning outcomes.
VI www.e
A.2 Classroom Setup
Participants recruitment. All participants were entry-level undergraduates majoring in STEM and enrolled in the Probability course in college. They were recruited via invitation emails and told the procedure of the study and their data were used for research purpose only, and the study was an opt-in without influence on their course grades. Participants can also opt-in not recording their logs and quit the study any time. No demographics data or course grades were collected. All participants had acknowledged the study procedure and future research conducted using their logs.
17

---

Principles taught by the IE system. Table 3 shows all ten principles for the IE system to teach designed for the undergraduate entry-level students with STEM majors.
Pre- and post-exams. As introduced in Section 2 to which students have benefited from the IE system for improved learning outcomes. Tables 4 contain all problems in pre- and post-exams during our experiment with the IE system.
we use pre- and post-exams to measure the extent
The set of candidate target policies under consideration. For safety concerns, only three RL- induced target policies that passed expert sanity checks can be deployed, while the expert policy still remained in the semester as the control group. For faimess concerns, the IE system randomly assigned a policy to each student, while we tracked the policies selected by FPS on each subgroup to evaluate its effectiivnesss The chi-squared test was employed to check the relationship between policy assignment and subgroups, and it showed that the policy assignment cross subgroups were balanced with significant relationship (p-value=0.479). In the testing semester, 140 students accomplished all problems and exams.
We provide the design of each problem regarding principles coverage for readers' interests. Detailed problem descriptions are omitted for identity and anonymity, which are only accessible within the research groups under IRB. An example problem description is shown in Figure 3
A.3
Environmental Setup of the IE System
A.3.1 State Features.
The state features were defined by domain experts that could possible capture students' learning status based their interaction logs. In sum, 142 features with both discrete and continuous values extracted, we provide summary descriptions of the features characterized by their systematic functions: (i) Autonomy (10 features): the amount of work done by the student, such as the number of times the student restarted a problem; (ii) Temporal Situation (29 features): the time-related information about the work process, such as average time per step; (iti) -roviig (35 features): information about the current problem-solving context, such as problem difficulty; (iv) Performance (57 features): information about the student's performance during problem-solve solvins such as percentage of correct entries; (v) Hints (11 features): information about the student's 's hint usage, such as the total number of hints requested.
A.3.2 Actions & rewards.
See A. above.
A.3.3 Behavior policy.
The behavior policy follows an expert policy commonly used in e-learning the next problem as a worked example (WE), problem-solving by students (PS), or collaborative problem-solving working examples (CPS). Note that the three decision choices are designed by domain experts that are found can support students' learning in prior works 55 thus the expert policy is considered as effective.
S3
randomly taking
A.3.4 Target (evaluation) policies.
In total, four target policies, including three RL-induced and the expert policy, were examined in testing semester. The three RL-induced policies were trained using off- off-policy DQN-based algorithm, and passed expert sanity check. In this study, expert sanity check were conducted by departments and independent instructors for pre-examination of the target policies.
Specifically, we employed the DQN-based algorithm designed by domain researchers 2 called Critical-RL, that have achieved empirical significance in woal-world classrooms, and passed expert sanity check by our institutions. First, for each problem, a pair of adversarial policies using vanilla DQN algorithm were induced, including an original policy induced using the original rewards and an inversed policy induced using the inversed rewards (i.e., the negative value of the original rewards). Then, critical decisions are identified following two rules: (1) Given the state, the two policies make opposite decisions; and (2) the decision is important (critical) for both policies. For a given state, rule (1) is tested first. If the ad versaril policies make the same decisions, it is not critical. Otherwise.
18

---

rule (2) is tested. In order to measure the importance of the decision for each policy, we calculate the absolute Q-value difference between the two alternative actions. If this difference is greater than a threshold, the decision is considered important (critical). A critical-RI. policy carries out the original policy when a decision is recognized as critical, and expert policy for the rest. Overall, in this study, we examined one critical-RI. policy and two variations of it, i.e., policy carrying out original policy when a decision is not critical, and a policy carrying out original policy over all decisions. We set the threshold to be the median Q-value difference for all decisions in our training data set following the settings of the original Critical-RL work [24]. Each pair of of adversarial policies considered all parts of the training data were identical, such as state representation and transition samples, except the rewards. We use the learning rate Ir = 1e- - 3 for inducing DQN policies.
A.3.5 Sub-group identification.
Specifically, learn the subgroups in the IE system, we leverage off-the-shelf algorithm called Toeplitz inverse covariance- based clustering (TICC) [21] to map initial logs Sy into M clusters based on the values of student-sensittve features (as defined in Appendix [A.4), where each so So is associated with a cluster from the set K = [K,,.., KM}, over the offline dataset, from which the number of individuals within each cluster is intrinsically determined. Specifically, TICC initially clusters all states from the offline dataset, where the states that are mapped to the same cluster can be considered share the graphical connectivity structure of cross-fcatures and temporal information captured by TICC. Then the clusters of initial states can be determined accordingly from the clustering outcomes. We consider using TICC because of its superior performance in clustering compared to traditional distance- based methods such as K- K-means, especially with human behavior- related tasks (8), such that the clusters of initial logs could be more scalable and capable of the evolving individuals and their behaviors in real- world HCSs. For a new participant arriving in the testing period, the cluster where the participant may belong to is the cluster exhibiting the least averaging distance between the initial states of the participant and samples within the cluster captured from the offline dataset.
The size of clusters is determined by data-driven procedure following the original TICC work (i.e., it
determined with the highest silhouette score in clustering historical trajectories) (21). Note that we exhibit TICC as an example in our proposed pipeline, while it can be replaced by other partitioning approaches if needed. Then, we assume subgroup partitioning consistent with cluster assignments associated with initial logs, i.e., students whose initial logs are associated with the same cluster index are considered from the same subgroup.
TICC Problem.
Each cluster m E [1, M] is defined as a Markov random field [57] or corre- lation network, captured by its Gaussian inverse covariance matrix E-1 E Rexx where is the dimension of state space. We also define the set of clusters K = {KI,..., KM} C R as well as the set of inverse covariance matrices Σ = " ...... Enr). Then the objective is set to
be: max ..K
m=
M
(0) Emm) - c1(ss $ Km])] where the first term defines the EnSeE (C(s))
log- ligllikel of s/ coming from Kn, as C(s (1) Emm =
pm Im. (s Mmm) (2(( -
log det 1m - | log(2n) with pmm being the empirical mean of cluster Km, the second term (2) & 1
Km,} penalizes the adjacent events that are not assigned to the same cluster and e is a constant balancing off the scale of the two terms. This optimization problem can be solved using the popectatio-mmxxiiiaaln family of algorithms by updating Σ -- and K alternatively (1].
A.4 Data Pre- Pre-Processing for Sub-Group Partitioning with the IE Experiment
The initial logs (serving as the initial states in the MDP) of students are used for sub-group partitioning. which is now only benefited from the FPS framework design but over two educational perspectives. First, initial logs may reflect not only the background knowledge of students but their interaction habits [3. without specific information related to behavior policies that may be distracting for sub-group partitioning. Though some existing works utilize demographics or grades of students from their prior taken courses to identify student subgroups 2 62 it may not be feasible in practice due to the protection of student information by institutions. Second, prior works have found that initial logs can be informative to indicate learning outcomes of students [39], which makes possible for the IE system to customize the policies with the goal of improving learning outcomes for each subgroup.
19

---

However, there is challenge with sub-group partitioning over the initial logs of students. The state space of student logs in the IE system usually nigh-dimensional, due the detailed capture of each step taken during interaction and associated timing information [
38 For example, in this study, 142 features have been recorded. While some features might irrelevant for downstream data mining tasks, it is challenging to determine their relevance priori (38]. To solve this, we used data-driven feature taxonomy over the state features of students, then perform subgroup partitioning with distilled features based on the feature taxonomy.
The data-driven feature taxonomy over state features of students. Educational researchers have used feature taxonomy in qualitative ways to support instructors subgroup students and understand behaviors of students [40]. Unlike prior approaches that are expensive requiring much effort from human experts, we used data-driven feature taxonomy for straightforward student subgroup partitioning that may reflect the knowledge background and dynamic learning progress students. Specifically, we define three major categories of features according to their temporal and cross- student characteristics: (i) gestem-assigned the features, which are static across students on the same problem, are assumed to be assigned by the system; (ii) Student-centric: the features, which differ across students from the initial logs and may change over time, is assumed to be students-centric and reflect both students' initial knowledge background and the changes of individual underlying mindset during learning; (iii) Interaction drrive:: the features, which contain characteristics from both satem-assigned and student-centre types, are assumed to be mixed-style features that affected by both system and individuals. For example, the number of tells since elicit is set with a default value by the system while changing over time depending students' progress.
Sub-group partitioning with dis- tilled features via feature taxonomy. Since system-assigned features are mainly dominated by system design and remain static across students on each problem, for the purpose of sub- group partitioning, we focus on the two types of features, student-centric and interaction-drive and behaviors, for which we call ctudent-sensinee features. In this work, identified 82%(117) from overall 142 features as student-sensitive features and used them for subgroup partitioning.
Table 2: Feature taxonomy with examples and percentage in the the IE system.
Taxonomy
Examples
Perc.
gystem-assignee Student-centric Interaction-driven
Problem difficulty
18%
Number of hints requested
48%
Number of tells since elicit
34%
since both could be highly associated with students' underlying mental status
A.5 More Discussions over the Results from Section 3.2
100
WIS
PDIS
MAE (four target policies)
140-
WDR
MAGIC
120
80
40-
Number of problem
10
Figure 4: Mean absolute error (MAE) of OPE AugRRS with subgroup partitioning over problems in historical data.
Would sub-group partitioning over longer trajectory improve the performance of the OPS+VRRS baselines? Recall that OPS+VRRS deployed the sub-optimal to most students, while
20

---

their estimation accuracy (i.e. absolute enror) was improved compared to purely OPS and OPS+RRS (Figure [[b]], which outperformed by FPS over a slight margin. We further investigate the effects of subg-roup partitioning with longer trajectory information on OPS+VRRS performance. We conduct sub-group partitioning over the length of trajectories, i.e., perform sub-group partitioning on the averaged states' features associated with the first A problems across historical trajectories, where E [1, 11] excluding the final problem. Then we augment the same amount of samples for each subgroup K, i.e., |Tx = |TK = | and perform OPS+RRS. We observe that in all 55 conditions 11. PDIS+VRRS Á and FQE+VRRS A 7, 8). all OPS+ still select the sub-optimal policy. Figure presents the mean absolute error (MAE) of the OPS+VRRS methods over the four target policies. shows the trend of improved MAE over the number of problems for most methods. Those indicate that more information over a longer trajectory does have some positive effects on OPS+VRRS, but their policy selection hard to be improved and stabilized. More ceudent and robust OPS methods are needed for IE policy selection.
except the five (i.e.. WIS+VRRS
21

---

Table 3: Principles taught by the IE system for undergraduate entry-level students.
Name of principle
Expression
Complement
P(A) P(-4)
MET
Mutually Exclusive Theorem
P(A = 0iff and are mutually sxclusive events
IE
Independent Events
P(4BB) (()))(B) if and are independent
Morgan's Theorem
PI-AU -B. P(-(An B)) P(-AU-II)
Addition Theorem for two events
P(A P(A) P(D) P(An
A3
Addition Theorem for three events
B - P(A) P(R) P(C) P(An P(AnC) ((BN)) P(An
CIE
Conditional Independent
PC)]BCC P(AIC)
independent events given
Conditional Probability
P(A" P(2) P(B ()PAA
TPT
Total Prohahility Thearem
P(A) P(A)) P( 1... P(A
Ba....
exclusive
U.
Bayes Rulo
P(B,14) P,A ((P)) PAA)) (BB) P(A +... P(P.).))(E.)
if D, Bz.. .D are munully exclusive events and Dj U Bg U.. Bz
Table 4: Pre-exam problems in the IE system.
Problem
CT
MET
IE
DMT
A2
A3
CIE
CP
TPT
BR
2
4
6
8
V
V
10
12
13
14
Table Post-cxam problems in the system.
Problem
CT
MET
IE DMT
A3
CIE
CP
TPT
BR
1so-Test-Problem
11
2
7
3
4
5
3
6
10
7
8
13
9
N/A
10
11
12
12
13
14
N/A
15
N/A
16
N/A
17
18
8
19
N/A
20
N/A
22

---

B Detailed Setup of the Healthcare Experiment
We use the sepsis simulator developed by 48 and benchmark settings of
45
B.1 States & actions.
The definition of states and actions are introduced Section 3.3
B.2 Rewards.
We also follow the benchmark [45] in terms of configuring the reward function and behavioral policy, Specifically, a reward of is received at the end of horizon (T'= = 5) if the patient is deceased (i.e., at least three vitals are out of the normal range), or +1 discharged (when all vital signs are in the normal range without treatment).
B.3 Optimal policy.
To learn the optimal policy, 45 used policy iteration to learn the optimal policy, and created a near- optimal (soft optimal) policy by having the policy take a random action with probability 0.05, and the optimal action with probability 0.95. The value function (for the optimal policy) was computed using value iteration. The discount factor 0.99.
B.4 Behaivor policy.
The behaviour policy is a mixture of two policies: 85% the soft optimal policy and 15% of a sub-optimal policy that is similar to the soft optimal but the vasopressors action is flipped.
B.5
Target policies.
See Section
3
C Societal and Broader Impacts
C.1 Societal Impacts
All real-world data employed in this paper were obtained anonymously through exempt IRB-approved protocols and were scored using established rubrics. No demographic data or class grades were collected. All data were shared within the research group under IRB, and were de-identified and automatically processed for labeling. This research seeks to remove societal harms that come from lower engagement and retention of students who need more personalized interventions and developing more robust medical interventions for patients.
C.2 Broader Impact on Facilitating Fairness in RL-Empowered HCSs
47 56 8. The FPS frame- work can be potentially extended promote fairesss certain extent, by helping minority/under- represented groups to boost their utility gain through deployment of customized policy specific to the group. Specifically, following FPS, the sub- sub-group partitioning step can identify the small-scaled yet important groups, then the policy that is most beneficial for the group can be deployed to maximize the gain. As illustrated by Figure 2] FPS effectively identifies the low-performing students (group K4), which only constitute < 5% of the overall population at the 6-th (testing) semester, without leveraging any subjective information a priori (i.e., sub-group -roug partitioning uses exactly the same features across all students). FPS then significantly boosts their performance by deploying the policy most suitable for the group. Similarly, one can easily extend the FPS framework to intelligent HCSs oriented toward other applications, in order to identify the groups that potentially need more attention, and help all participants achieve similar gain indiscriminately by deploying the right policy each participant.
Fairness in Al-empowered HCSs has been a long-standing concern 33 42
23

---

D More on the Methodology
D.1 Practical Off-Policy Deployment in HCSs over the Sub-Group Objective T
The overall off policy deployment can be achieved using a two-step approach, f.e., (1) pre-partitioning with offline dataset, followed by (ii) deployment upon observation of the initial states of arriving participants.
To facilitate (i), clustering methods, such TICC 21), is used to learn preliminary partitioning
: Su IC Then, the value function estimator D T. <(,o=(+oo) is trained for estimating V =
v =m=o(()) using part of the offline trajectories whose initial state so falls in the corresponding group Km 1(50) following Proposition 2.5 for all Km E < At last, one can learn a mapping : Sp x IT K. - I to reconstruct D estimation using the (8o, Km)) triplets as the inputs (e.g., by minimizing squared error).
In step (ii), plug into d((s)) Km) all policy candidates π E and all possible partitions Km E K. Then, one can determine which policy æ satisfies maX d( (6) ,,, Km). Assuming the pre- partitioning over offline dataset is knowledgeable about initial logs of participants, for each arriving participant i' 2 N with their initial log s determine the cluster they may most likely belong to. For example, the cluster Km. with the least averaging distance between Su (t) and 96 for every 86 E Km- Then, assign to participant i' the corresponding group k(*(()) Km as captured by the mapping function SS0 - K. Deploy the corresponding policy that is estimated achieving best performance for group Km, to i'.
D.2 Proof of Bound 3
Rényi divergence. For e 2 O, the Rényi divergence for two distribution and 3 is defined by [54]
1
(xx)
(T]13)
c log2 Σ ((x)
3(x)
6 denote the exponential in base by (T (x
Proof. D
1 EiEI
wi WT 7 ET 7 with
Im
can be upper bounded by the variance of importance sam- pling weights. Denote Ji ΣT ((() Following (25), since Var(G) s [E(² (following the definition of variance),
(0) |s| )/30 (o( (a
|19 1 N2
Var(Dx)))
E
(w) 1)2]
| Var(w),
with the last equality following the fact E[w] = 1. Moreover, Var(w) = dz(x111)) as pointed out
by (6). following the Rényi divergence [54] Thus, the variance of the estimator Var (1x..) is:
|g (T(
T. R Kim Var
de (T
1g
N
The expression can be related to the effective sample size (ESS) of the original data given the target policy 41). resulting
Var(D km = 11
1
ESS
which completes the proof.
Note that D (e= essentially approximates the of value difference in each Im.

---

Remark. Note that in the special case that behavior policy being the same as target policy, the bound evaluates to zero, Moreover, as noted by [25], denote the right-hand side of inequality 3 by Var,()) it can be used in each sub-group as a proxy of variance of the estimator in the subgroup, i.e.,
1
V (Dr km,
l19m I
())(Km)
K
here, Im refers the total return of the trajectories pertaining the sub-group Km, and E.SS (Km)
can be estimated by ESS using (Km) Ciezo 9:)²
49
Diez of
D.3 Detailed Formulation of VAE in MDP
The latent prior p(20) ~ ~N 1) representing the distribution of the initial latent states (at the beginning each PST in the T where is the identity covariance matrix.
Encoder.
4n (2 Q1 is used to approximate the posterior distribution 1.12-11011122.80
where 2 Rm and is the dimension. 011-18)-1, Juez P35-1
(2(20s))) IT [T (n(=)lee-,,--11 st), both distributions (z)(55)) and 9n (2 (- a2 ax11,8) follow diagonal Gaussian, where mean and diagonal covari- ance determined by multi-layer perceptrons (MIPs) and long short-term memory (LSTM), with neural network weights y Thus, one can infer =3 ~ (2(2000)) z ~ q, (2)))))) with hg = Sn (h² =l- 0,-1, where fn represents LSTM layer and h2 represents LSTM recurrent hidden state.
Given that (n(2o:]/su:1) 00:r-1)
Decoder.
Pe (2 St, Tt- 1 a-- is used sample new trajectories.
Given where oe
Pę(=1.T,80:77 80:7; To:7-1 |2, = Pr PT (2 are determined following the behavioral policy i, distributions (=)))) and pe(rr-1lze) follow diagonal Gaussian with mean and covariance determined by MLPs and (z 0(11) follows diagonal Gaussian with mean and covariance determined by LSTM.
Thus, the generative process can be formulated as, i.e., at initialization, of p(zo), o Pe Pe ag ((oo) (5);; followed by =f pe (hτ), r-_1 Pe (r ( of (s( ar B(ar with h5 = 9: ( = 1 1)] where dE represents an MLP.
Training objective. The training objective for the VAE in MDP is maximize the evidence lower bound (ELBO), which consists the log-likelinhe likelihood of reconstructing the states and rewards, and regularization of the approximated posterior, i.e.,
ELBOO(, = Egg Σ 1 log (ogge()))ze) + Σ" T
1== log (rg
-K[((((xo/b))ll((aa) - Σ (, 0r- (z
The proof of Equation is provided in Appendix D.4
models such as VAE have been commonly used for augmentation in offline RL, while they general rarely come with error bounds provided [20 31 (8). Prior works have also found that applying generative models to data augmentation can learn more robust predictors that are invariant especially with subgroup identity (18). Though generative augmentation models may perfectly model the subgroup distribution and introduce artifacts, as noted by [18), we can directly control the deviations of augmentation from original data with translation or consistency loss as in Equation 4 Our experimental results further show that off-policy selection can benefit more with combination of augmented samples and raw data compared to using raw (original) data only.
More discussions on trajectory augmentations.
D.4 Proof of Equation 4
The derivation of the evidence lower bound (ELBO) for the joint log-likelihood distribution can be found below.
25

---

Proof.
log pa (80:T, ro:r-1)
(5)
log
(3(ST:T 21:T; Norr-1)d2
Pa(sorr, (orr,
log
qo U0:T-1d22
FIrEZ Ya (20er) 80:r: 10:T-1)
Jensen's s inequality
Eg oog p(z)) + log (a 21:7; No:T. (20) log qa (zu:t (orr, co:--1)
(8)
Eg
log (0) logps (20)
Σ t=
log (s 22, T- Te 2t
-log (20 (so)
log qa (2el (11
(9)
Ega
log p(z)(()))))))))))))) log (20l + log (s + Σ 1=1
log baseeblellllloorraaaaaaaaaaaaaal
(10)
-gg
logpo($a1*))
1=1 log logpo(rc-12z)
[o((a(20)s))ll((zo) E._
(11)
26

---

E More Experimental Setup
E.1 Training Resources
All experimental workloads are distributed among 4 Nvidia RTX A5000 24GB, 3 Nvidia Quadro RTX 6000 24GB, and 4 NVIDIA TITAN Xp 12GB graphies cards.
E.2
Implementations and Hyper-parameters
For FQE, as in 30 we train a neural network to estimate the values of the target polices т E II by bootstrapping from the learned Q-function. For DualDICE, use the open-sourced code in its original paper. For MAGIC, we use the implementation [71] For trajectory augmentation, for the components involving LSTMs, which include (2 ac- St) and pa (Zg 2t-1, 0(-1). their architecture include one LSTM layer with 64 nodes, followed by a dense layer with 64 nodes. All other components do have LSTM layers involved, so they are constituted by neural network with 2 dense layers, with 128 and 64 nodes respectively. The output layers that determine the mean and diagonal covariance of diagonal Gaussian distributions use linear and softplus activations, respectively. The ones that determine the mean of Bernoulli distributions (e.g., for capturing early termination of episodes) are configured to use sigmoid activations. For training, in subgroups with sample size greater than 200, the maximum number of iteration is set to 1000 and minibatch size set to 64, and 200 and respectively for subgroups with sample size less than or equal to 200. Adam optimizer is used to perform gradient descent. To determine the learning rate, we perform grid search among {le 4, 3e - 3, 3e 4, 5e - 4, 7e - 4}. Exponential decay is applied to the learning rate, which decays the learning rate by 0.997 every iteration. For OPE, the model-basee methods are evaluated by directly interacting with each target policy for 50 episodes, and the mean of discounted total returns
(7 0.9) over all episodes is used as estimated performance for the policy.
E.3 OPE Standard Evaluation Metrics
Absolute error The absolute error is defined as the difference between the actual value and the estimated value of policy:
AE = V" V"
(12)
where 4 represents the actual value of the policy æ, and V/8 represents the estimated value of ..
Mean absolute error (MAE) The MAE is defined as the average value of absolute error across [II target (evaluation) policies:
MAE
Σ AE(T).
(13)
II
27

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: To the best of our knowledge, we are the first work that targeted and solved practical challenge often encountered in off-policy selection (OPS) when deploying RL policies to new human arrivals in practical human-centric systems (HCSs). We have provided an algorithm and theoretical analysis to address the practical problem interest, with extensive empirical justifications as provided by two human-centric environments across two domains (education and healthcare).
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please see Section 5 Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
28

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: Please Section 2 Appendix D.2
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
Justification: Please Section and Appendices A
B
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
29

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: Please Supplementary Materials. Real-world human data is not publicly released under IRB protocols.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
CodeSuomi ssinnpolicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: Please see Appendices
E
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Please see Figures 2 and Table
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
30

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please see Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't I make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: The authors make sure to preserve anonymity (e.g., there is a special consideration due to laws regulations in their jurisdiction).
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please see Appendix
C
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
31

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper has not been found pose such risks yet.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
The authors respectfully cited the original paper that produced the sepsis
dataset.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
32

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
latasets.papersstt
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The authors provided the details of the code as part of their submissions via structured templates.
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details the dataset/code/mmodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: Please see Appendices
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: Please see Appendix C All real-world data employed in this paper were obtained anonymously through exempt IRB-u IRB-approved protocols and were scored using established rubrics. No demographic data or class grades were collected. All data were shared within the research group under IRB, and were de-identified and automatically processed for labeling. This research seeks to remove societal harms that come from lower engagement and retention of students who need more personalized interventions and developing more robust medical interventions for patients.
33

---

Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
34


Reviews:
[{"review_id": "2dA6b0Juzc", "reviewer": "Reviewer_wmCx", "summary": "This paper introduces First-Glance Off-Policy Selection (FPS), a novel approach to off-policy selection (OPS) in human-centric systems (HCSs) such as healthcare and education, where the heterogeneity among participants requires personalized interventions. \n\nFPS addresses this by segmenting participants into sub-groups based on similar traits and applying tailored OPS criteria to each sub-group. This method is evaluated in two real-world applications: intelligent tutoring systems and sepsis treatment in healthcare. The results demonstrate significant improvements in both learning outcomes and in-hospital care, highlighting FPS's ability to personalize policy selection and enhance system performance.", "questions": "Questions are combined in the weaknesses section. Please see above.", "limitations": "The paper does not thoroughly address FPS's scalability and applicability in large-scale, real-world settings, especially in healthcare. I suggest the authors clarify the scalability and emphasise the importance of clinical guidance. One to two sentences will be enough.", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- **Novel Approach**: The introduction of the First-Glance Off-Policy Selection (FPS) framework is a significant innovation. By systematically addressing participant heterogeneity through sub-group segmentation, FPS offers a fresh perspective on OPS in human-centric systems (HCSs).\n- **New Problem Formulation**: The paper tackles the unique challenge of selecting policies for new participants without prior offline data. This problem formulation is distinct from existing OPS/OPE frameworks, which typically assume homogeneity among agents.\n\n- **Methodological Rigor**: The paper demonstrates a high level of methodological rigor. The use of variational auto-encoding (VAE) for trajectory augmentation and the development of an unbiased value function estimator with bounded variance reflect thorough and robust algorithm design.\n- **Comprehensive Experiments**: The experimental evaluation is extensive, covering both real-world educational systems and healthcare applications. This diversity in testing scenarios strengthens the validity of the results. The paper provides a detailed analysis of the results, including comparisons with various baselines and ablation studies. This thoroughness ensures that the findings are well-supported and credible.\n\nAlso, the paper is well-organized, with clear sections that guide the reader through the problem formulation, methodology, experiments, and results. Each part builds logically on the previous one.", "weaknesses": "The paper is generally well-written. I will combine the weaknesses and questions into one section. \n\n1. Assumption of Independent Initial State Distributions\nThe FPS framework assumes that the initial state distributions for each participant are independent and can be uniformly sampled from the offline dataset. This assumption may not hold true in real-world scenarios where participants’ initial states can be influenced by various contextual factors and past interactions. The independence assumption may oversimplify the complexity of human-centric systems and may lead to suboptimal policy selections. The authors may consider addressing the potential dependencies in initial states.\n\n2. Lack of Consideration for non-stationary state transition\nFPS focuses on the initial state for policy selection without considering longitudinal data that captures the progression of participant states over time. This, in the AI community's language, is a non-stationary state transition issue faced by meta-RL. Is it possible that the state transition for each patient is also independent of each other? In other words, chances are that the state transition for each patient $p_i$ is sampled from a distribution. Would this FPS work in such a case?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and efforts on evaluating our work, and your positive comments that the paper is making an important impact. Please find our point-by-point response below.\n\nQ1. Assumption of Independent Initial State Distributions The FPS framework assumes that the initial state distributions for each participant are independent and can be uniformly sampled from the offline dataset. This assumption may not hold true in real-world scenarios where participants’ initial states can be influenced by various contextual factors and past interactions. The independence assumption may oversimplify the complexity of human-centric systems and may lead to suboptimal policy selections. The authors may consider addressing the potential dependencies in initial states.\n\nR1. Thank you for the insightful comment. That independence assumption we made was inspired by the use of uninformative prior in Bayesian methods where we do not have much prior knowledge in terms of sub-grouping's potential outcome before the experiment starts. We agreed with the reviewer that investigating the potential confoundings in initial states would be important -- and we plan to do so as a separate work in the future, as it could be very challenging to capture those in practical scenarios, e.g., there exist a few literature attempting to soly address that issue [1-3]. \n\nQ2. Lack of Consideration for non-stationary state transition FPS focuses on the initial state for policy selection without considering longitudinal data that captures the progression of participant states over time. This, in the AI community's language, is a non-stationary state transition issue faced by meta-RL. Is it possible that the state transition for each patient is also independent of each other? In other words, chances are that the state transition for each patient 𝑝𝑖 is sampled from a distribution. Would this FPS work in such a case?\n\nR2. Great point. Initially we did not consider the progression/transition of participant states since the overall scenario our work considered was that each individual needs to be assigned a group upon arrival where the initial states would be the only observation available. However, we agreed that our framework can be further extended toward also considering each participant's follow-up visits once they are available. And if the state transitions are also independent from each other, a simple variation of our method may work, as one could re-run the partitioning algorithm at each new visit; this way it would not violate the assumptions needed by our work. Further, in the future we plan to explore the possibilities of temporally dynamic sub-group partitioning, where the agent can consider all the historical states of a participant upon arrival of each visit. It would make the problem setup more challenging as it falls more under a POMDP schema. We greatly appreciate both of these comments as they indeed helped us to shape the future work down the avenue and led us to think of more challenging, interesting and realistic setups. We also hope that our work could possibly inspire more researchers to recognize and emphasize the empirical challenges faced by deploying RL/OPS in real-world systems, if accepted.  \n\nQ3. The paper does not thoroughly address FPS's scalability and applicability in large-scale, real-world settings, especially in healthcare. I suggest the authors clarify the scalability and emphasise the importance of clinical guidance. One to two sentences will be enough.\n\nR3. We sincerely appreciate your suggestion. We will add the following discussion in the camera-ready, if accepted, i.e., *\"Compared to IE systems, HCSs in healthcare would be considered even more high-stakes, thus may further limit the options (i.e., policies) that are available to facilitate sub-grouping experiments, due to stricter clinical experimental guidelines. However, FPS has demonstrated its extraordinary capabilities over a real-world experiment that involved >1,200 participants with years of follow-ups, which showed its efficacy and scalability toward working with more challenging systems and larger cohorts as in healthcare, as the assumptions needed by FPS across these two systems would not change fundamentally. Moreover, potential underlying confoundings may exist across the patient's initial states in healthcare, and it is also important to consider inputs from healthcare professionals during sub-grouping. As a result, one may further extend our framework toward such a direction, allowing it to function better in the healthcare domain.\"* \n\nWe hope these answers provide some explanations to address your concerns and showcase that our work is solving a significant challenge in a satisfying manner. We are happy to answer any followup questions or hear any comments from you.\n\nReferences\n\n[1] Namkoong et al. Off-policy policy evaluation for sequential decisions under unobserved confounding. NeurIPS 2020.\n\n[2] Xu et al. An instrumental variable approach to confounded off-policy evaluation. ICML 2023.\n\n[3] Tennenholtz et al. Off-policy evaluation in partially observable environments. AAAI 2020.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zn1NW5ZkDJ", "reviewer": "Reviewer_2e7j", "summary": "This paper studies off-policy selection in healthcare settings where users are heterogeneous and in situations where new users can appear in the policy deployment phase. To deal with new participants, this paper proposes a two-stage evaluation procedure: (1) learning a partitioning function of users and (2) choosing different policies for each subgroup via OPE. The proposed method (sub-group partitioning) achieves better performance than random user partitioning.", "questions": "- How does the proposed method estimate the value during the sub-group partitioning phase?\n\n- How does the proposed method deal with the variance in the data partitioning?\n\n- If Keramati et al. (2022) is a skyline (with full knowledge about trajectory under the logging policy), how much can the proposed method achieve?\n\n- This may be due to random seeds, but I wondered why FPS-P can be quite pessimistic while FPS and FPS-noTA are rather optimistic.", "limitations": "Ambiguity about the sub-group partitioning phase and light comparison with Keramati et al. (2022) (one of the most related work). See weaknesses for the details.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- **Reasonable approach**: Using similar features and trajectory augmentation sounds like a reasonable approach, given that the dataset is sparse and only initial states are available for some users.\n- **Simple and easy-to-implement algorithm**: The proposed method is not too complicated, and it seems to be easily implementable in real-world applications.\n- **Ablations are informative**: Ablations, especially FPS-noTA and FPS-P, are instructive in telling the benefits of TA and sub-group partitioning, respectively.", "weaknesses": "- **Feasibility of sub-group partitioning**: Section 2.2 (Definition 2.3) states the objective function of sub-group partitioning, and it indicates that the partitioning process requires knowledge about V^{\\pi} for every candidate policy. In my understanding, this procedure itself requires OPE. However, looking at Algorithm 1, it seems the algorithm determines partitioning before applying OPE. How is this sub-group partitioning actually conducted? If using OPE, I believe this partitioning procedure can have a high variance. If simple clustering based on the features of initial states is used, the partitioning may not align with the objective function.\n\n- **Variance in the sub-group partitioning phase**: Related to the above point, how does this algorithm deal with the variance in the sub-group partitioning phase? Also, using silhouette scores is distance-based and does not consider the variance. It would be useful if there is a way to determine the number of clusters (M) in a data-driven way, taking both bias and variance into account.\n\n-  **Improvement over Keramati et al. (2022) seems incremental**: The paper states that the benefit of the proposed method over Keramati et al. (2022) is that the proposed method does not require full-trajectory information, and thus applicable when only initial states are accessible. I understand the practical advantages, however, the technical progress is not convincing, as the way the proposed method overcomes the challenges in sub-group partitioning *without* full trajectory is not well-described. Also, if Keramati et al. (2022) is a skyline, how does the performance of the proposed method differ from Keramati et al. (2022)? A more detailed discussion of comparing the proposed method with existing work would be appreciated.\n\nKeramati et al. (2022): Identification of Subgroups With Similar Benefits in Off-Policy Policy Evaluation. Ramtin Keramati, Omer Gottesman, Leo Anthony Celi, Finale Doshi-Velez, Emma Brunskill. CHIL, 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your time and efforts on evaluating our work. Please find our point-by-point response below.\n\nQ1. How is this sub-group partitioning actually conducted?\n\nR1. We used an off-the-shelf algorithm called Toeplitz inverse covariance-based clustering (TICC) [1] to obtain the initial partitioning as described in Appendix A.3.5, to jumpstart the iterative optimization process toward objective (1) that follows. It could be our writing that causes such a confusion, and we will clarify that in the camera-ready version, if accepted. Please see the response below for the concern over variance.\n\nQ2. Variance in the sub-group partitioning phase: \n\n> Q2.1. How does this algorithm deal with the variance in the sub-group partitioning phase? \n\nR2.1. We thank the reviewer for this thoughtful comment where we carry out additional analyses to address it, which lead to additional findings. Specifically, we alter the total number of subgroups, $M \\in \\{4,5,6\\}$, and re-run the sub-group partition. It is observed that a minimal percentage of students have had their group changed over different $M$'s -- see the table below. This may not be surprising, because in empirical human-centric scenarios, such as education, the policies are generally highly regulated and scrutinized by departmental committees strictly enforcing the guidelines for human-centric experiments, so that both behavioral and target policies deployed at scale would not interfere students' learning greatly. There it implies the underlying assumption that the divergence between behavior and target policies could be intrinsically bounded. Such a finding is important and can be potentially generalized to common human-centric environments, and we plan to further pursue such an avenue in broader contexts both empirically and theoretically in the future.\n\n| Changes of # of Sub-groups          | 3->4         | 4->5         | 5->6         | 3->5         | 3->6         | 4->6         |\n|-----------------|-----------------|-----------------|-----------------|-----------------|-----------------|-----------------|\n| Perc. of students changing sub-groups        | 2%             | 4%             | 3%             | 6%             | 10%             | 9%             |    \n\n> Q.2.2. Using silhouette scores is distance-based and does not consider the variance. It would be useful if there is a way to determine the number of clusters (M) in a data-driven way, taking both bias and variance into account.\n\nR2.2. Given our work is the first targeting and solving a practical challenge often encountered in OPS, we provided the framework in a straight-forward manner, and silhouette score has been broadly employed and exhibits good performance across human-related tasks [1-3]. However, we greatly appreciate this comment and agreed that it would be an interesting topic to investigate in the future toward bias and variance in partitioning.\n\nQ3. Improvement over Keramati et al. seems incremental: \n\n> Q3.1. I understand the practical advantages, however, the technical progress is not convincing, as the way the proposed method overcomes the challenges in sub-group partitioning without full trajectory is not well-described. \n\nR3.1. We respectfully disagree with the opinion that the work is incremental over Keramati et al. Though the theoretical analysis of bound of variance in both work may be grounded on [4-5], please note that the objective of our work is to identify the policy that can possibly work the best for each arriving individual of HCSs from a set of policy candidates, **without access to any prior offline data or entire trajectory collected over the individual**, which targets to address a common bottleneck in empirical human-participated studies. In contrast, Keramati et al. assumed that there existed a population-level (optimal) policy and then identified the sub-group that may benefit most of that policy, which is the opposite side of our problem statement. Moreover, they required the entire trajectory for each individual to be pre-given before sub-grouping, which would not be practical for HCSs in the real-world, as the initial state would be the only observation available at the time an individual arrives at the HCS right after which a group needs to be assigned (e.g., doctors need to sketch out diagnoses/treatment plan for patients soon after they stepped into the triage room). \n\n> Q3.2.  how does the performance of the proposed method differ from Keramati et al.? \n\nR3.2. As we mentioned above, given the complicated real-world human-centric environments and highly limited information of each incoming individual, it’s hard to directly implement Keramati et al. work to our empirical experiments, as the assumption required would not be met, nor the availability of the trajectory beyond the initial state for their method to use for sub-grouping.   \n\nQ4. Why FPS-P can be quite pessimistic while FPS and FPS-noTA are rather optimistic.\n\nR4. There could be some unobserved confounding that exists in human-centric experiments, leading the method that assumed sequential ignorability for the behavior policy to underestimate or overestimate, and similar findings were observed in [6].\n\nWe hope our responses sufficiently addressed your concerns and clarified that our work is solving a significant practical challenge effectively. We are happy to respond to any follow-ups you may have.\n\nReferences\n\n[1] Toeplitz inverse covariance-based clustering of multivariate time series data. KDD 2017.\n\n[2] Domain generalization via model-agnostic learning of semantic features. NeurIPS 2019.\n\n[3] A Hierarchical Clustering algorithm based on Silhouette Index for cancer subtype discovery from genomic data. Neural computing and applications 2020.\n\n[4] Learning bounds for importance weighting. NeurIPS 2010.\n\n[5] Policy optimization via importance sampling. NeurIPS 2018\n\n[6] Off-policy policy evaluation for sequential decisions under unobserved confounding. NeurIPS 2020.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EGspNCHWqP", "reviewer": "Reviewer_aKHU", "summary": "The paper presents the First-Glance Off-Policy Selection (FPS) framework, aimed at improving policy selection for human-centric systems (HCSs) like education and healthcare, by addressing participant heterogeneity. FPS groups participants with similar traits, augments each sub-group with trajectories generated by a variational auto-encoder (VAE), and selects the optimal policy based on an estimator of the policy selection criteria. The proposed method is tested in a real-world intelligent education (IE) system and demonstrates significant improvement in the learning outcomes by personalizing tutoring policies based on initial student states. The framework's effectiveness was also demonstrated in selecting optimal treatment policies for sepsis patients in a simulated healthcare environment.", "questions": "-\tWhat is the error bar in Figures 1 and 2? Is it the standard deviation, standard error, or confidence interval? If it is the standard error, the difference between the baseline and the proposed method is small compared to the variation within each method.\n-\tIn Figure 1(b), most methods overestimate the reward with the proposed estimator. Could the authors explain the reason for this? Is there a systematic bias in the estimator?\n-\tIn Figure 2, FPS and the best possible baseline combinations perform exactly the same in subgroups K1 and K2. Is this because the two methods assign the same policy to every student in these first two subgroups?", "limitations": "The authors have addressed the limitations of their work.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "-\tThe paper introduces a new framework that selects policies for each new participant joining the cohort based solely on the initial state. \n-\tThe proposed method has been tested in a real-world IE system with 1,288 students and in a simulated healthcare environment.\n-\tAlthough each component of the proposed framework has been previously studied, the framework itself demonstrates advantages over baseline methods concerning the addressed problems.", "weaknesses": "-\tThe proposed method is applicable only to problems with a finite number of policies, as policy selection is based on evaluating each candidate target policy separately.\n-\tThe real-world experiment was not a randomized controlled trial that directly compared the proposed method against baseline methods. Instead, in the IE system, a policy was randomly assigned to each student. The authors then tracked the outcomes for students in each subgroup who were assigned the policy recommended by each method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your time and efforts on evaluating our work. Please find our point-by-point response below.\n\nQ1. The proposed method is applicable only to problems with a finite number of policies, as policy selection is based on evaluating each candidate target policy separately.\n\nR1. Good question. We focused on a common scenario when deploying RL policies to real human participants in practical human-centric systems, where all deployed policies have to be highly regulated and scrutinized by departmental committees enforcing the guidelines for human-centric experiments, where the total number of deployed policies would be generally limited [1-4],   \n\nQ2. The real-world experiment was not a randomized controlled trial that directly compared the proposed method against baseline methods. Instead, in the IE system, a policy was randomly assigned to each student. The authors then tracked the outcomes for students in each subgroup who were assigned the policy recommended by each method.\n\nR2. Unfortunately, as we had to follow the pre-defined guidelines agreed with a departmental committee, that each student has to be treated equally during the experiment (i.e., they either opted-in for testing out all the methods or none). As a result, we were not able to run typical randomized control experiments. However, the chi-squared test was employed to check the relationship between policy assignment and sub-groups, and it showed that the policy assignment cross sub-groups were balanced with no significant relationship (p-value=0.479), as described in Appendix A.2.\n\nQ3. What is the error bar in Figures 1 and 2? Is it the standard deviation, standard error, or confidence interval? If it is the standard error, the difference between the baseline and the proposed method is small compared to the variation within each method.\n\nR3. The error bar represents standard error. Though the difference may not be large for potential high-performing sub-groups (e.g., $K_1$ & $K_2$), we observed that the baselines can even have a negative effect on some sub-groups ($K_3$ & $K_4$) as in Figure 2, which is undesired in human-centric experiments. In empirical human-centric scenarios, such as education, the behavior policy is generally highly regularized by department committees strictly following guidance from human-centric experiments, such that the target policy would not be dramatically opposed to the behavior policy – suggesting the underlying assumption that the divergence between behavior and target policies could be intrinsically bounded. We will definitely clarify this in the camera-ready version, if accepted.\n\nQ4. In Figure 1(b), most methods overestimate the reward with the proposed estimator. Could the authors explain the reason for this? Is there a systematic bias in the estimator?\n\nR4. This is probably due to the unobserved confounding existing in human-centric systems, leading the method that assumed sequential ignorability for the behavior policy to overestimate or underestimate, and similar findings were observed in [5-6].\n\nQ5. In Figure 2, FPS and the best possible baseline combinations perform exactly the same in subgroups K1 and K2. Is this because the two methods assign the same policy to every student in these first two subgroups?\n\nR5. Both FPS and baselines selected the same policy for sub-groups $K_1$ and $K_2$.\n\n\nWe hope these answers provide some explanations to address your concerns and showcase that our work is solving a significant challenge in a satisfying manner. We are happy to answer any followup questions or hear any comments from you.\n\nReferences\n\n[1] Mandel et al. Offline policy evaluation across representations with applications to educational games. AAMAS 2014.\n\n[2] Gao et al. Offline policy evaluation for learning-based deep brain stimulation controllers. International Conference on Cyber-Physical Systems 2022.\n\n[3] Zhou et al. Hierarchical reinforcement learning for pedagogical policy induction. International Conference on Artificial Intelligence in Education  2019.\n\n[4] Abdelshiheed et al. Leveraging deep reinforcement learning for metacognitive interventions across intelligent tutoring systems. International Conference on Artificial Intelligence in Education 2023.\n\n[5] Namkoong, et al. Off-policy policy evaluation for sequential decisions under unobserved confounding. NeurIPS 2020.\n\n[6] Fu et al. Benchmarks for Deep Off-Policy Evaluation. ICLR 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper formulates the problem of off-policy selection, relevant to healthcare and social settings where the goal is to select a policy to deploy for a new patient/sample with no prior history. To enable this task, the authors assume independent initial state distributions. A subgroup partitioning using clustering is enabled. The value function corresponding to each subgroup's policy is designed such that the difference between the value corresponding to the policy of a subgroup assignment and the behavior policy is maximized. An estimator is proposed for value function estimation. To enable subgroup partitioning, a variational auto-encoder based data augmentation strategy is used for initial clustering. During deployment, cluster membership is inferred and appropriate policy is deployed.

Reviewers consider the problem formulation and the approach novel and easy to implement, with comprehensive evaluation, including ablations. There are certain clarity issues in the writing (on my reading) which the author response clarified. Reviewer aKHU raised valid points regarding the experiment design in the real world experiment as well as bias in the reward function estimates. Authors have attempted to clarify these concerns. One of the major concerns raised was that of potential distinction over prior work, particularly Keramati et al, which the authors have clarified in their rebuttal. This distinction is satisfactory to me. The other concern raised was regarding analysis of the proposed method re- bias/variance tradeoff of the proposed estimator for value function estimation. There is some analysis in the paper but nowhere nearly as exhaustive. However considering the largely empirical nature of the contribution, I have weighted this criticism appropriately. The empirical evaluation is interesting and demonstrates the utility of the simple approach proposed in the paper.

Considering all of these factors, I recommend and accept.
Decision: Accept (poster)