Off-Policy Evaluation for Human Feedback
Authors: Qitong Gao
Ge Gao
Juncheng Dong
Vahid Tarokh
Min Chi
Miroslav Pajic
Venue: NeurIPS 2023 poster (2023)
Keywords: Off-policy evaluation (OPE)
Variational latent model for trajectory representation learning
Reinforcement learning and OPE for adaptive neurostimulation

Abstract: Off-policy evaluation (OPE) is important for closing the gap between offline training and evaluation of reinforcement learning (RL), by estimating performance and/or rank of target (evaluation) policies using offline trajectories only. It can improve the safety and efficiency of data collection and policy testing procedures in situations where online deployments are expensive, such as healthcare. However, existing OPE methods fall short in estimating human feedback (HF) signals, as HF may be conditioned over multiple underlying factors and are only sparsely available; as opposed to the agent-defined environmental rewards (used in policy optimization), which are usually determined over parametric functions or distributions. Consequently, the nature of HF signals makes extrapolating accurate OPE estimations to be challenging. To resolve this, we introduce an OPE for HF (OPEHF) framework that revives existing OPE methods in order to accurately evaluate the HF signals. Specifically, we develop an immediate human reward (IHR) reconstruction approach, regularized by environmental knowledge distilled in a latent space that captures the underlying dynamics of state transitions as well as issuing HF signals. Our approach has been tested over *two real-world experiments*, adaptive *in-vivo* neurostimulation and intelligent tutoring, and a simulation environment (visual Q&A). Results show that our approach significantly improves the performance toward estimating HF signals accurately, compared to directly applying (variants of) existing OPE methods.

Off-Policy Evaluation for Human Feedback
Qitong Gao"
Ge Gaot
Juncheng Dong'
Vahid Tarokh
Min Chit
Miroslav Pajic
Abstract
Off-policy evaluation (OPE) is important for closing the gap between offline train- ing and evaluation of reinforcement learning (RL), by estimating performance and/or rank of target (evaluation) policies using offline trajectories only. It can improve the safety and efficiency of data collection and policy testing procedures in situations where online deployments ever, existing OPE methods fall short in estimating human feedback (HF) signals, multiple underlying factors and only sparsely
expensive, such healthcare. How-
HF may be conditioned available; as opposed to the agent-defined environmental rewards (used in policy optimization), which are usually determined over parametric functions distri- butions. Consequently, the nature of HF signals makes extrapolating accurate OPE estimations to be challenging. To resolve this, introduce an OPE for HF (OPEHF) framework that revives existing OPE methods in order to accurately evaluate the HF signals. Specifically, we develop an immediate human reward (IHR) reconstruction approach, regularized by environmental knowledge distilled in a latent space that captures the underlying dynamics state transitions as well issuing HF signals. Our approach has been tested over two real-world experiments, adaptive in-vivo neurostimulation and intelligent tutoring, as well as a simulation environment (visual Q& Results show that our approach significantly improves the performance toward estimating HF signals accurately, compared directly applying (variants of) existing OPE methods.
Introduction
Off-policy evaluation (OPE) aims to estimate the performance of reinforcement learning (RL) policies using only fixed set of offline trajectories [6]), i.e., without online deployments. considered to be critical step in closing the gap between offline RL training and evaluation, for environments and systems where online data collection expensive or unsafe. Specifically, OPE facilitates not only offline evaluation of the safety and efficacy of the policies ahead of online deployment, but policy selection as well; this allows one to maximize the efficiency when online data collection possible, by identifying and deploying the policies that are more likely to result in higher returns. OPE has
been used in various application domains including healthcare 68 [3 23 22 robotics 18
intelligent tutoring [64 45 17), recommendation systems 50 43
The majority of existing OPE methods focus on evaluating the policies' performance defined over the environmental reward functions which are mainly designed for use in policy optimization (training). However, as an increasing number of offline RL frameworks are developed for human-involved
48 16]. existing OPE methods lack the ability to estimate how human users would evaluate the policies, e.g., ratings provided by patients (on scale of 1-10) over the procedure facilitated by automated surgical robots; as human feedback (HF) can be noisy and conditioned over various confounders that could be difficult to be captured explicitly 7 44]. For example, patient satisfaction over specific diabetes therapy may vary across the cohort, depending many subjective
systems 64 45
Duke University. Durham, NC, USA. Contact: {qitong saatong.gao, miroslav. miroslaw.pajic) pajicc North Carolina State University. Raleigh, NC, USA.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

factors, such as personal preferences and activity level of the day, while participating in the therapy. in addition to the physiological signals (e.g., blood sugar level, body weight) that are more commonly used as the sources for determining environmental rewards toward policy optimization [70 33 21 19 Moreover, the environmental rewards are sometimes discrete to ensure optimality of the learned policies [67], which further reduces its correlation against HF signals.
In this work, we introduce the OPE for human feedback (OPEHF) framework that revives existing OPE approaches in the context of evaluating HF from offline data. Specifically, we consider the challenging scenario where the HF signal is only provided at the end of each episode - -ie. no per-step HF signals, referred to as immediate human rewards (IHRs) below, are provided - benchmarking the common
situations where the participants are allowed to rate the procedures only at the end of the study. The goal is set to estimate the end-of-episode HF signals, also referred to as human returns, over the target (evaluation) policies, using a fixed set of offline trajectories collected over some behavioral policies. To facilitate OPEHF, we introduce an approach that first maps the human return back to the sequence of IHRs, over the horizon, for each trajectory. Specifically, this follows from optimizing over an objective that consists of necessary condition where the cumulative discounted sum of IHRs should equal the human return, as well as a regularization term that limits the discrepancy of the reconstructed IHRs over state-action pairs that are determined similar over latent representation space into which environmental transitions and rewards are encoded. At last, this allows for the use of any existing OPE methods to process the offline trajectories with reconstructed IHRs and estimate human returns under target policies.
Our main contributions are tri-fold. (i) We introduce novel OPEHF framework that revives existing OPE methods toward accurately estimating highly sparse HF signals (provided only at the end of each episode) from offline trajectories, through IHRs reconstruction. (ii) Our approach does not require the environmental rewards and the HF signals to be strongly correlated, benefiting from the design where both signals are encoded to latent space regularizing the objective for reconstructions of IHRs, which is justified empirically over real-world experiments. (iii) Two real-world experiments, i.e., adaptive in-vivo neurostimulation for the treatment of Parkinson's 's disease and intelligent tutoring for computer science students in colleges, as well as one simulation environment (i.e., visual Q&A), facilitated the thorough evaluation of our approach; various degrees of correlations between the environment rewards and HF signals existed across the environments, as well as the varied coverage of the Mate-action space provided by offline data over sub-optimal behavioral policies, imposing different levels of challenges for OPEHF
2 Off-Policy Evaluation for Human Feedback (OPEHF)
In this section, we introduce an OPEHF framework that allows for the use of existing OPE methods to estimate the human returns that are available only at the end of cach episode, with IHRs remaining unknown. This is in contrast to the goal of classic OPE that only estimates the environmental returns following the user-defined reward function used in the policy optimization phase. A brief overview of existing OPE methods can be found in Appendix C
2.1 Problem Formulation
We first formulate the human-involved MDP (HMDP), which is tuple M= (S, RM .0. s0.7), 7), where S is the set of states, AA the set of actions, P: S x A + S is the transition distribution usually captured by probabilities p(*:| (8 (t-1), R S A A is the environmental reward function, (1**,,,, |s, a) is the human reward distribution from which the IHR r. (= dz) are sampled, 80 is the initial state sampled from the initial state distribution p(50). and 7 E [o, 1) is the discounting factor. Note that we set the IHRs to be determined probabilistically, as opposed to the environmental rewards r2- = R(*,,a, a.) that are deterministic; this is due to the fact that many underlying factors have also observed while performing experiments (see Appendix D). Finally, the agent interacts with the MDP following some policy 'T (alss) that defines the probabilities of taking action at state s.
may affect the feedback provided by humans [53] 7 44].
In this work, we make the following assumption over R and R
Assumption 1 (Unknown IHRs). We assume that the immediate environmental reward function R is known and R(s, a) can be obtained for any state-action pairs in S A. Moreover, the IHR distribution RHt is assumed to be unknown, i.e., 7t R² (-|s, a) are unobservable, for all (s,a) E S x A.
2

---

Instead, the cumulative human return G. defined over RH,, is given at the end of each trajectory, i.e., G = r with T being the horizon and r' R (-1s;, az).
The assumption above follows the fact that human feedback (HF) is not available until the end of each episode, as opposed to immediate rewards that can be defined over the environment and evaluated for any (3t, at) pairs at any time. This is especially true in environments such as healthcare where the clinical treatment outcome is not foresecable until therapeutic cycle is completed, or in intelligent tutoring where the overall gain from students over a semester is mostly reflected by the final grades. Note that although the setup can be generalized to the scenario where HF can be sparsely obtained over the horizon, we believe that issuing the HF only at the end of each trajectory leads to a more challenging setup for OPE. Consequently, the goal of OPEHF can be formulated as follows.
Problem 1 (Objective of OPEHF). Given offline trajectories collected by some be- {-((), (11),
havioral policy B. pa
((N-1) at Hit (
3(ar]s.)} with z()
G being a single trajectory, N the total number of offline trajectories, and rt 's being unknown, the objective is to estimate the expected total human return over the unknown state-action visitation distribution pt of the target
[(s)") ..b". .To .TD
(1) 7C(1)
)...,..... (s...., aT T-1: T-1: -1: T-1: ST
(1)
(evaluation) policy i.e., ((aa)=p' RH
Ei
2.2 Reconstrunction of IHRs for OPEHF
We emphasize that the human returns are only issued at the end of each episode, with IHRs remaining unknown. One can set all IHRs from i 0 to 1 T T-2 2 to be zeros (i.e. r 00 = 0), and rescale the cumulative human return to be the IHR at the last step (i.e.. - = G / 7- to allow the use of existing OPE methods toward OPEHF (Problem [}. However, the sparsity over PH here may impose difficulties for OPE to estimate the human returns accurately over the target policies. For OPEHF, we start by showing that for the per-decision importance sampling (PDIS) method a auriance-reductio variant of the importance sample (IS) family of OPE methods [6]] if IHRs
to be available, they could reduce the variance in the estimation compared to the rescale approach above.
Recall that the PDIS estimator follows GTDDS = + E Lt= 7 (1) H0), where wott 116)
IIk-
are the PDIS weights for offline trajectory (()) Moreover, the estimator of the (1+5))
rescale approach above is G Rescale = N E w G
M()) which is equivalent to the vanilla IS
estimator 661 72). We now show the variance reduction property of ČyDII in the context of OPEHR.
Proposition 1. Assume that (i) E[r 2 O. and (ii) given the horizon T, consider any 1 s 1 <
k T of any offline trajectory T, wo:k and r wark are positively correlated. Then, 4 (GPDIs) S (tscaee)))) with V()) representing the variance.
The proof can be found in Appendix A Assumption (i) can be easily satisfied in the real world, as
HF signals are usually quantified as positive values, e-g., ratings (1-10) provided by participants Assumption (ii) is most likely to be satisfied when the target policies do not visit low-return regions substantially (46), which is a pre-requisite for testing RL policies in human-involved environments as initial screening are usually required to filter the ones that could potentially pose risks to partici- pants (77.
methods re- quire learning value functions. Sparsity of rewards (following the rescale approach above) in the offline dataset may lead to poorly learned value functions [74], considering that the offline data in OPE is usually fixed (i.e., no new samples can be added), and are often generated by behavioral policies that are sub-optimal, which results in limited coverage of the state-action space. Limited availabilities of environment-policy interactions (e.g., clinical trials) further reduce the scale of the exploration and therefore limit the information that can be leveraged toward obtaining accurate value function approximations.
Besides IS, doubly robust (DR)
69 12 and fitted Q-evaluation (FQE) 40
Reconstruction of IHRs. To address this challenge, our approach aims to project the end-of-episode human returns back to each environmental step, i.e., to learn mapping fo( GM (S A)" x
We call it the rescale approach instead of vanilla IS as the idea behind also generalizes non-IS methods.
3

---

Offiine Trajectories
Spacie
VLM+
Train
Ep sode
IHR
Reconstruction
Stop
Latent Representations
Learnt by VLM-H
Trajestory Inference (Enooding) Trajectory Generaton (Decoding)
Offline Trajactories
Reconstructed
Stop
OPE Estimator
Torpet
Figure 1: (Left) Architecture of the variational latent model with human returns (VLM-H), (Mid) Illustration of the clustering behavior in the latent space using I-SNE visualization [3] where the encoded state-action pairs (output by the encoder of VI.M-H) are in general clustered together they generated by policies with similar human returns (shown in the legend at the top left). (Right) Diagram summarizing the pipeline of the OPEHF framework.
R + RT, parameterized by 0, that maximizes the sum of log-likelihood of the estimated IHRs, (o..... ....... ---1" fo(T, G.rr). following maxo N Ei= Ei= log rt 10, (0), G where Gor and ri 26(6) are respectivelly the human return and IHRs (unknown) of the i-th trajectory in the offline dataset p², and N is the total number of trajectories in p². Given that the objective 7((1), s, we introduce surrogate objective above is intractable due to unknown r'.
H(()
((())
1 N-1
Σ 1=0
logp Σ Y
G2LL (0), G C.
100 -
G2(()) ) (I)
maxx N
Here, the first term is necessary condition for fl 's to be valid for estimating r as they should sum G Since many solutions may exist if one only optimizes over the first term, the second term frrggu serves as a regularization that imposes constraints on r s to follow the properties specific to their corresponding state-action pairs; egg., (s, a) pairs that are similar to each other a representation space, defined over the state-action visitation space, tend yield similar immediate rewards (18).
The detailed regularization technique introduced sub-section below. Practically, we choose fe to be bi-directional long-short term memory (LSTM) 32), since the reconstruction of IHRs can leverage information from both previous and subsequent steps as provided in the offline trajectories.
2.3 Reconstruction of [HRs over Latent Representations (RILR) for OPEHF
Now, we introduce the regularization technique for the reconstruction of IHRs, i.e., reconstructing IHRs over latent representations (RILR). Specifically, we leverage the representations captured by variational auto-encoders (VAEs) 35). learned over p² to regularize the reconstructed IHRs, A.f.
VAFs have been adapted toward learning a compact latent space over offline state-action visitations, facilitating both offline policy optimization 442 specifically consider building on the variational latent model (VLM) proposed in [8] since originally proposed to facilitate OPE, as opposed to others that mainly use knowledge captured in the latent space to improve sample efficiency for policy optimization. Moreover, the VLM has shown to be effective for learning an expressive representation space, where the encoded state-action pairs are clustered well in the latent space, as measured by the difference over the returns of the policies from which the aate-action pairs sampled; see Figure I (mid) which uses t-SNE to visualize the encoded acteonction pairs in trajectories collected from a visual Q&A environment (Appendix
27 26 28 and OPE [88] In this work, we
Note that VLM originally does not account for HF signals (neither r nor G 's), we intro- duce the variational latent model with human returns (VLM-H) below, building on the architecture introduced in [18. VLM-H consists of prior p(z) over the latent variables 2 E 2 C RL, with 2 rep- resenting the latent space and L the dimension, along with a variational encoder qoi (2 22-11, 1;-1, 8t). a decoder Ps (=e, St, 7 / 4x-1) .) for generating per-step transitions (over both state-action and latent space), and separate decoder pa ( for the reconstruction of the human returns at the end of each episode. Note that encoders and decoders are parameterized by ta and á respectively. The overall architecture is illustrated in Figure (left).

---

Trajectory inference (encoding). VLM-H's encoder approximates the intractable posterior -1
1( Ja,zz - by avoiding to integrate over the unknown latent space priori. The inference (or encoding) process can be decomposed as, i.e., 9,-1,, st); here, (2(20]50)) encodes initial states so into latent variables 20. and qu (2g] 21-1, at-1, 8t) captures all subsequent environmen- tal transitions the latent space over 2 In general, both zy 's are represented as diagonal Gaussian distribution: with mean and variance determined by neural network y/, in [8 42
qu (20: (errr---
(2)2)|50) IIr= Qu(22(22-1, Qt-1,
27
26
28].
Trajectory generation (decoding). The generative (or decoding) process follows, i.., 20: x)
Po(21.T SO.T, 11.T-1, Gorl
(c lTT) I1.- 1o
Ie (<((s here, p=(=)(4-1,,--1) enforces the transition of latent variables over time, Pa (8 and po (o(rr-1(z)) are used to sample the states and immediate environmental rewards, while Pe (G - (2r) generates the human return issued at the end of each episode. Note that here we still use the VLM-H to capture environmental rewards, allowing the VLM-H to formulate latent space that captures as much information about the dynamics underlying the environment as possible. All Pa ss are represented as diagonal Gaussians with parameters determined by network Φi
To train and v/, one maximize the evidence lower bound (ELBO) of the joint likeliheoood log P T0:T-1, Forlo 1b p²)) i.e.,
Togps(Got 0:T (T) + Σ r=
log logpo(<el2)) + Σ (r
max 6
E 8g
(p(()) - - 1-1 (I(qu(21/2-1. s (2)](())))1,,- (2)
the first three terms are the log- likelihoods of reconstructing the human return, states, and environ- mental rewards, and the two terms that follow are Kullback-Leible (KL) divergence (38 regularizing the inferred posterior 9,: Derivation of the ELBO can be found in Appendix B In practice, if e and s are chosen to be recurrent networks, one can also regularize the hidden states of Φ, 10 by including the additional regularization term introduced in [T8]
Regularizing the reconstruction of IHRs. Existing works have shown that the latent space not only facilitates the generation of synthetic trajectories but demonstrated that the latent encodings of
state-action pairs form clusters, over some measures in the latent space (73], if they are rolled out from policies that lead to similar returns
As a result, we regularize A? following
min Cregu (F |0, wh, 0zt 0.t- (
(i) (i)
Σ
log P(f (1 - G1
o.1 10, v. S Ood 10 G
HG)
(i) (
jEJ
(3)
for each step I; here, (Noxt: 0o:c-1) E (1) p. .jo....,jj-. jK-1) are the indices of offline
trajectories that correspond to the latent encodings
{ (x)) go (-11% (i)) 1 E J, E
(i) [o, - 1}} that are K-neighbours of the latent encoding 2f pertaining to (s a0--1). defined over some similarity/distance function d(-ll-). following, iee.
Σ d(z) (122111200)) Amm
(ha) corresponding (s (l)) E ((+) ~ P
min J.EJ
s.t. 24
(4)
In practice, we choose d(-ll-) to follow stochastic neighbor embedding (SNE) similarities 773 as has been shown effective for capturing Euclidean distances in high-dimensional space 775)
Overall objective of RILR for OPEHE, As result, by following ( and leveraging the Lregu from above, the objective for reconstructing the IHRs is set to be, i.e.,
T-1
max-
log P
G
(,, H(i)
C
(4) ( Erega ( v. Sax 0a
GH(
"This helps facilitate an orthogonal basis of the latent space, which would improve the expressiveness of the model.
If needed, one can project the states over to the orthogonal basis, to ensure that they follow a diagonal co- variance.
5

---

Move from RILR to OPEHE. In what follows, one can leverage any existing OPE methods to take as inputs the offline trajectories, with the immediate environmental rewards r, replaced by the reconstructed IHRs P. ss to achieve the OPEHF's objective (Problem []. Moreover, our method does not require the IHRs to correlated with the environmental rewards, as the VL.M-H learns to reconstruct both by sampling from two independent distributions, Po(rt-1 (22) and p& ( D:r l=r) respectively, following (2): this is also illustrated empirically over the experiments introduced below (Sections where exceedingly low correlations are found in specific scenarios.
The overall pipeline summarizing our method is shown Figure (right).
3 Real- Experiments with Human Participants
In this section, we validate the OPEHF framework introduced above over two real-world experiments, adaptive neurostimulation, and intelligent tutoring. Specifically, we consider four types of OPE methods to be used as the downstream estimator following the RILR step (Section (.3),, including per-
decision importance sampling (IS) with behavioral policy estimation (30] doubly robust (DR) (7]]. distribution correction estimation (DICE) [78] and fitted Q-evaluation (FQE) 40]. A brief overview of
these methods can be found Appendix C and the specific implementations we use are documented
in Appendix D In Appendix E we have also tested our method within a visual Q&A environment [10] 66), which follows similar mechanisms in the two real-world experiments, i.e., two types of return signals are considered though no human participants are involved.
Baselines and Ablations. The base- lines include two variants for each of the OPE methods above, i.e., (i) the rescale approach discussed in Section 2.2 and (ii) another variant that sets all the IHRs to be equal to the environ- mental rewards corresponding steps, r = Ti Vt E T 2). and then let by TT-1 = Pr-1 rr-1+ (G Go:r)/>T-1 E.rr, being the environmental return, which is re- ferred to as fusion below this baseline may perform better when strong correlations existed between environmental and human rewards, as intrinsically decomposes the human returns into IHRs. Conse- quently, each experiment below, we compare the performance of the OPEHF framework extending all four types of OPE methods above, DIS/DD/DDCCEFFFS-PPPRR against the corresponding baselines, S///DR/DICE/FFEEE
Intemal Pubso Gene (PO)
Stimulation
Feedttock jovallable
Feedbbcck (end -f-eo--d
Sensing
Tremor
LFPs
Stimulation
Detection
Wearable
Electroee
Amplitudes
Toblet
Rating
Tremor Severity
Beta PwT. Stim Amps
Bradykinesia Test Results
with Go.T
Historical Trajectories
Amp.
Bradykinesia Ratlng
Epience
Pulse Generator
Eqixoci
[+T
I=T
Figure 2: Setup of the neurostimulation experiments, as
well the formulation offline trajectories. Environmen- tal rewards and human returns are captured in streams and 2-3 respectively.
/eeeiionn/eeee.....ee We also include the VLM-H as an ablation bascline, as if it model-based approach standalone; this is achieved by sampling the estimate returns from the decoder, GH. ~ pppppppaCCCC (
Metrics. Following a recent OPE benchmark [5], three metrics are considered to validate the performance of each method, including mean absolute error (MAE), rank correlation, and regret @ 1. Mathematical definitions can he found in Appendix D Also, following [55,, each method is evaluated over 3 random seeds, and the mean performance (with standard errors) is reported.
3.1 Adaptive Neurostimulation: Deep Brain Stimulation
Adaptive neurostimulation facilitates treatments a variety of neurological disorders A 13 55
Deep brain stimulation (DBS) is a type of neurostimulation used specifically toward Parkinson's 's disease (PD), where an internal pulse generator (IPG), implanted under the collarbone, sends electrical stimulus to the basal ganglia (BG) area of the brain through invasive electrodes; Figure 2 illustrates

---

MAE
Rank
Regret J
3.93
3.93
255
75
0.25-
VUX H Ablition
2.0-
4:45
Baselines
18-
0.10-
05
3:45
OPEN
[Our)
OPENE
oie poro bR OPEN
me Eraao DICE OPEIE
FoeE pooor FOE OPENP
gle Foston OPENN
DEC-- Frsion DiCk OPEIN
Cot- Frston FIL araiie
ts is OPENP
ne bm OPENF
Bert OPEN
OPENP
Figure 3: Results from the adaptive neurostimulation experiment, i.e., deep brain stimulation (DBS). Each method is evaluated over the data collected from each patient, toward corresponding target
policies, respectively. The performance shown above are averaged over all 4 human participants affected by Parkinson's 's disease (PD). Raw performance over each patient can be found in Appendix D
the setup. Adaptive DBS aims to adjust the strength (amplitude) of the stimulus in real-time, to respond to irregular neuronal activities caused by PD, leveraging the local field potentials (LFPs) as the immediate feedback signals, i.e., the environmental rewards. Existing works have leveraged RL for adaptive DBS over computational BG models 225 20 52 59], using rewards defined over a physiological signal - beta-band power spectral density of LFPs (i.e., the beta power) since physiologically PD could lead to increased beta power due to the irregular neuronal activations it causes [39]. However, in climical practice, the correlation between beta power and the level of satisfaction reported by the patients varies depending on the specific characteristics of each person, as PD can cause different types of symptoms over a wide range of severity [56] 37 [5] 76). Such findings further justify the significance of evaluating HF/human returns in the real world using OPEHF
In this experiment, we leverage OPEHF to es- timate the feedback provided by 4 PD patients who participate in monthly clinical testings of RL policies trained adapt amplitudes of the stimulus toward reducing their PD symp- toms, i.e., bradykinesia and tremor. A mixture of behavioral policies is used to collect the of- fline trajectories p². Specifically, the state st is a historical sequence of LFPs cap- turing neuronal activities, and the action a, up- dates the amplitude of the stimulus to be sen Then, an environmental reward = R(st, 0z) gives a penalty if the beta power computed from the latest LFPs is greater than some threshold (to promote treatment efficacy) as well improve battery life of the IPG). At the end of each episode, the human returns G are determined from three sources (weighted by 50%, 25%. 25% respectively), ie.., (i) a satisfaction rating (between 1-10) provided by the patient, (i) hand grasp speed as a result of the bradykinesia test [63], and (iti) level of tremorcalculated over the data from a wearable accelcrometry [60] 61. Each session lasts more than 10 minutes, and each discrete step above corresponds to seconds in the real world; thus, the horizon T Z 300 (more details are provided in Appendix DD. Approval of an Institutional Review Board (IRB) is obtained from Duke University Health System, as well as the exceptional use of the DBS system by the US Food and Drug Administration (FDA).
Table I: Correlations between the environmental and human returns of the 6 target DBS policies associated with each PD patient.
Patient #
2
3
Pearson's Spearman's -0.2
-0.396
-0.477
-0.599
-0.275
every step,
-0.6
0.086
0.086
penalty proportional to the amplitudes of the stimulus being sent (to
For each patient, OPEHF and the baselines are used to estimate the human returns of 6 target policies with varied performance. The ground-truth human return for each target policy is obtained result of extensive clinical testing following the same schema above, over more than 100 minutes. Table shows shows the Pearson's 's and Spearman's correlation coefficients [4. measuring the linear and rank correlations between the environmental returns Ge,T and the human returns G over all the target DBS policies considered for each patient. Pearson's 's coefficients are all negative since the environmental reward function only issues penalties, while human returns are all captured by positive values. It can be observed that only weak-to-moderale degrees of linear correlations exist for all four patients, while ranks between Go 's and G 's are not preserved across patients; thus, it highlights
'RL policies only adapt the stimulation amplitudes within a safe range as determined by neurolo- gists/neurosurgeons, making sure they will not lead to negative effects participants.
7

---

Patient
Patient
Patient
Patient #3
Figure 4: t-SNE visualizing the VLM-H encodings of the state-action pairs rolled out over DBS policies with different human returns (shown in the legend). It can be observed that distances among the encoded pairs associated with the policies that lead to similar returns are in general smaller, justifying the RILR objective (5).
Table Results from the intelligent tutoring experiment, i.e, performance achieved by our OPEHF framework compared to the ablation and baselines over all four types of downstream OPE estimators.
DR
Ablation
OPEHF (our)
OPEHF (our)
Fusion
Rescale
Fusion
Rescale
VLM-H
MAE Rank
0.7++0.14 0.47+0011 0.36+0.16
0.77++0.08
0.57++0.09
1.03=0.07
1.03++0225
0.86++0.04
1.00=0.01
0.4+:0.09
0.8=0.09
0.33=0.05
0.4+=0.0
0.53+0022
0.41
Regret @1
0.36+00.16
0.41
0.41=0.0
0.41+=0.0
0.41+0000
0.28=0.19
DICE
FQE
OPEHF
OPEHF
Fusion
Rescale
Fusion
Rescale
(our)
(our)
MAE
3.19-10.57
2.33:10.59
1.01
0.74
0.98-10.1
0.59.1..1
Rank
0.47-00.2
0.33
0.53-10,22
0.27-0.14
0.4
0.47
Regret@)
0.55=#0.06
0.45=0..18
0.37
0.36=0.16
0.41=00.0
0.41=10.0
the need for leveraging OPEHF to estimate human returns, which is different than the classic OPE that focus on estimating environmental returns.
The overall performance averaged across the 4-patient cohort, is reported in Fig. Raw performance over every single patient can be found Appendix [ significantly improves MAEs and ranks compared to the two baselines, for all 4 types of downstream OPE methods we considered (IS, DR, DICE, and FQE). Moreover, our method also significantly outperforms the ablation VLM-H in terms of these two metrics, the VLM-H's performance is mainly determined by how well it could capture the underlying dynamics and returns. contrast, our OPEHF framework not only leverages the latent representations learnt by the VLM-H (for regularizing RILR), it also inherits the advantages intrinsically associated with the downstream estimators: e.g., low-bias nature of IS, low-variance provided by DR. Moreover, the fusion baseline in general performs worse than the rescale baseline as expected, since no strong correlations between environmental and human returns found, reported in Table
be observed that our OPEHF framework
Note that the majority the methods lead to similar (relatively low) regrets, as there exist a few poli- cies that lead to human returns that are close over some patients (see the raw statistics in Appendix [). The reason is that all the policies to be extensively tested in clinics are subject to initial screening, where clinicians ensure they would not lead to undesired outcomes pose significant risks to the patients; thus, the performance of some target policies tends to be close. Nonetheless, low MAEs and high ranks achieved by method show that it can effectively capture the subtle differences in returns resulting from other HF signals, i.e., levels of bradykineste and tremor. Morcover, Figure 4 visualizes the FLM-H encodings over the trajectories collected from the 6 target DBS policies for each participant and shows that encoded pairs associated with the policies that lead to similar returns in general clustered together, which justifies the importance of leveraging the similarities latent representations to regularize the reconstruction of IHRs as the RILR objective
3.2 Intelligent Tutoring
Intelligent tutoring refers system where students can actively interact with an autonomous tutoring agent that can customize the learning content, tests, etc., to improve engagement and learning

---

OPEHF is important in such a setup for directly estimating the potential outcomes that could be obtained by students, as opposed environmental rewards that are mostly discrete; see detailed setup below. Existing works have explored this topic over classic OPE setting in simulations [49
outcomes 2
64 45
The system is deployed in an undergraduate-leve introduction to probability and statistics course over 5 academic years at North Carolina State University, where the interaction logs obtained from 1,288 students who voluntarily opted-in for this experiment are recorded Specifically, each episode refers to student working on set of 12 problems (i.e. horizon T 12), where the agent suggests the student approach each problem through independent work, working with the hints provided, or directly providing the full solution (for studying purposes) - these options constitute the action space of the agent. The states are characterized by 140 features extracted from the logs, designed by domain experts; they include, for example, the time spent on each problem, and the correctness of the solution provided. In each step, an immediate environmental reward of +1 is issued if the answer submitted by students, for the current problem, is at least 80% correct (auto-graded following pre-defined rubrics). A reward of 0 is issued if the grade is less than 80% or the agent chooses the action that directly displays the full solution. Moreover, students are instructed to complete two exams, one before working on any problems and another after finishing all the problems. The normalized difference between the grades of two exams constitutes the human return for each episode. More details are provided in Appendix D
The intelligent tutoring agent follows different policies across academic years, where the data collected from the first 4 years (1148 students total) constitutes the offline trajectories p (as result of a mixture of behavioral policies). The 4 policies deployed in the 5th year (140 students total) serve as the target policies, whose ground- truth performance is determined by averaging over the human returns of the episodes that are associated with each policy respectively. Table Table3Jdocmments the Pearson's 's and Spearman's 's correlation coefficients between the environmental and human returns from data collected over each academic year, showing weak linear and rank correlations across all 5 years. Such low correlations are due to the fact that the environmental rewards are discrete and do not distinguish among the agent's s choices, i.e., a +1 reward can be obtained either if the student works out solution independently or by following hints, and a 0 reward is issued every time the agent chooses to display the solution even if the student could have solved the problem. As a result, such setup makes OPEHF be more challenging; because human returns are only available at the end of each episode, and the immediate environmental rewards do not carry substantial information toward extrapolating IHRs.
Table 3: Correlations between the environmental and human returns from data collected over each academic year.
Year #
3
4
Pearson's
0.033
0.176
0.089
0.154
0.183
Spearman's 's 0.082
0.156
0.130
0.161
0.103
Table 2 documents the performance of OPEHF and the baselines toward estimating the human returns of the target policies. It can be observed that OPEHF framework achieves state-of-the thet-h-rrt performance, over all types of downstream OPE estimators considered. This result echos the design of the VLM-H where both environmental information (state transitions and rewards) and human returns are encoded into the latent space, which helps formulate a compact and expressive latent space for regularizing the downstream RILR objective (5) Moreover, it is important use the latent information to guide the reconstruction of IHRs (as regularizations in RILR), as opposed to using the VLM-H to predict human returns standalone; since limited convergence guarantees/error bounds can be provided for VAE-based latent models, which is illustrated in both Figure 3 and Table 2 where OPEHF largely outperforms the VLM-H ablation over MAE and rank.
4 Related Works
OPE. Majority of existing model-free OPE methods can be categorized into one of the four types, i.e., IS, DR, DICE, and FQE. Recently, variants of IS and DR methods have been proposed for variance or bias reduction 34
12 69 as well adaptations toward unknown behavioral policies (30).
7 An IRB approval is obtained from North Carolina State University. The use/test of the intelligent tutoring system is overseen by departmental committee, ensuring it does not risk the academic performance and privacy of the participants.

---

DICE methods are intrinsically designed to work with offline trajectories rolled out from a mixture of behavioral policies, and existing works have introduced the DICE variants toward specific envi- ronmental setups
FQE extrapolates policy returns from the approximated There also exist model-based OPE methods 82 [X] that first captures the dynamics underlying the environment, and estimate policy performance by rolling out trajectories under the target policies. A more detailed review of existing OPE methods can be found in Ap- pendix C Note that these OPE methods have been designed for estimating the environmental returns. In contrast, the objective for OPEHF is to estimate the human returns which may not be strongly correlated with the environmental returns, they are usually determined under different schemas.
84
83
78
51 g
Q-values [3
36
VAEs for OPE and offline RL. There exists a long line of research developing latent models to capture the dynamics underlying environments in the context of offline RL as well as OPE. Specifically, PlaNet 27 uses recurrent neural networks to capture the transitions of latent variables over time. Latent representations learned by such VAE architectures have been used to augment the state space in offline policy optimization to improve sample efficiency, e-g. in Dreamers [26] 28) SOLAR [8]] and SLAC [2]. On the other hand, LatCo 665] attempts improve sample efficiency by searching in the latent space which allows by-passing physical constraints. Also, MOPO [80),
COMBO 77., and LOMPO [22] train latent models to quantify the confidence of the environmental transitions as learned from offline data and prevent the policies from following transitions over uncertain regions during policy training. Given that such models are mostly designed for improving sample efficiency in policy optimization/traiiing we choose to leverage the architecture from [18] for RILR it is the first work that adapts latent models to the OPE setup.
Reinforcement learning from human feedback (RLHF). Recently, the concept of RLHF has been widely used in guiding RL policy optimization with the HF signals deemed more informative than the environmental rewards [8 85 47). Specifically, they leverage the ranked preference provided by labelers to train reward model, captured by feed-forward neural networks, that is fused with the environmental rewards to guide policy optimization. However, in this work, we focus on estimating the HF signals that serve as direct evaluation of the RL policies used in human-involved experiments, such as the level of satisfaction (e.g., on a scale 1-10) and the treatment outcome. The reason is that in many scenarios the participants cannot revisit the same procedure multiple times, e.g., patients may not undergo the same surgeries several times and rank the experiences. More importantly, OPEHF's setup is critical when online testing of RL policies may be even prohibited, without sufficient justifications over safety and efficacy upfront, as illustrated by the experiments above.
pursue similar ideas of decomposing the delayed and/or sparse rewards (e.g.. the human return) into immediate rewards, they fundamentally rely transforming the MDP to such that the value functions can be smoothly captured and high-return state-action pairs can be quickly identified and frequently re-visited. For example, RUDDER [3] leverages the transformed MDP that has expected future rewards equal to zero. Though the optimization objective is consistent between pre- and post-transfone transformed MDPs, this approach likely would not converge to optimal policy in practice. On the other hand, the performance (i.e., returns) of sub-optimal policies is not preserved across the two MDPs. This significantly limits its use cases toward OPE which requires the returns resulted by sub-optimal policies to be estimated accurately. As a result, such methods are not directly applicable to the OPEHF problem we consider.
Reward shaping. Although reward shaping methods [3 [58 29
5 Conclusion and Future Works
Existing OPE methods fall short in estimating HF signals, as HF can be dependent upon various confounders. Thus, in this work, we introduced the OPEHF framework that revived existing OPE methods for estimating human returns, through RILR. The framework was validated over two real-world experiments and one simulation environment, outperforming the baselines in all setups. Although in the future it could be possible to extend OPEHF to facilitate estimating the HF signals needed for updating the policies similar to RL.H., we focused on policy evaluation which helped to isolate the source of improvements; as policy optimization's 's performance may depend on multiple factors, such as the exploration techniques used as well as the objective/optimizz chosen for updating the policy. Moreover, this work mainly focuses on the scenarios where the human returns are directly provided by the participants. So under the condition where the HF signals are provided by 3-rd parties (e.g, clinicians), non-trivial adaptations over this work may be needed to consider special cases such as conflicting HF signals provided by different sources.
10

---

Acknowledgements
This work is sponsored in part by the AFOSR under award number NIH UH3 NS103468 award, and by the NSF CNS-1652544 DUE-1726550, IIS-1651909 and DUE-2013502 awards, as well as the National AI Institute for Edge Computing Leveraging Next Generation Wireless Networks, Grant CNS-21 12562.. Investigational Summit RC+S systems and technical support provided by Medtronic PLC. Apple Watches were provided by Rune Labs. We thank Stephen I.. Schmidt and Jennifer J. Peters from Duke University Department of Biomedical Engineering, as well as Katherine Genty from Duke University Department of Neurosurgery, for the efforts overseeing DBS experiments in the clinic.
by the
References
[1] Saminda Wishwajith Abeyruwan, Laura Graesser, David B D' A'mbrosio, Avi Singh, Anish Shankar, Alex Bewley, Deepali Jain, Krzysztof Marcin Choromanski, and Pannag R Sanketi. i-sim2real: Reinforcement learning of robotic policies in tight human-robot interaction loops. In Conference on Robot Learning, pages 212-224, PMLR, 2023.
[2] John R Anderson, C Franklin Boyle, and Brian J Reiser. Intelligent tutoring systems. Science, 22844698) 1985.
[3] Jose A Arjona- Medina, Michael Gillhofer, Michael Widrich, Thomas Unterthiner, Johannes Brandstetter, and Sepp Hochreiter. Rudder: Return decomposition for delayed rewards. Ad- vances in Neural Information Processing Systems, 32, 2019.
[4] Alim Louis Benabid. Deep brain stimulation for parkinson's disease. Current opinion in neurobiology, 13(6):696-706, 2003.
[5] Peter Brown, Antonio Oliviero, Paolo Mazzone, Angelo Insola, Pietro Tonali, and Vincenzo Di Lazzaro. Dopamine dependency of oscillations between subthalamic nucleus and pallidum in parkinson's 's disease. Journal of Neuroscience, 21(3):1033-1038, 2001.
[6] Witney Chen, Lowry Kirkby, Miro Kotzev, Patrick Song, Ro'ee Gilron, and Brian Pepin. The role of large-scale data infrastructure in developing next-generation deep brain stimulation therapies. Frontiers in Human Neuroscience, 15:11700, 2021.
[7] Nicholas C Chesnaye, Vianda S Stel, Giovanni Tripepi, Friedo W Dekker, Edouard L Fu, Carmine Zoccali, and Kitty J Jager. An introduction to inverse probability of treatment weighting in observational research. Clinical Kidney Journal, 15(1):14-20, 2022.
[8] Paul Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
[9] Bo Dai, Ofir Nachum, Yinlam Chow, Lihong Li, Csaba Szepesvári, and Dale Schuurmans. Coindice: Off-policy confidence interval estimation. Advances in Neural Information Processing Systems, 33:9398 00189991111111 2020.
[10] Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, José M.F. Moura, Devi Parikh, and Dhruv Batra. Visual Dialog. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
[11] Günther Deuschl, Carmen Schade- Brittingr,,, Paul Krack, Jens Volkmann, Helmut Schäfer, Kai Bötzel, Christine Daniels, Angela Deutschländer, Ulrich Dillmann, Wilhelm Eisner, al. A randomized trial of deep-brain stimulation for parkinson's 's disease. New England Journal of Medicine, 355(9):896-908 2006.
[12] Mehrdad Farajtabar, Yinlam Chow, and Mohammad Ghavamzadeh. More robust doubly robust off-policy evaluation. International Conference on Machine Learning, pages 1447-1456. PMLR, 2018
11

---

[13] Kenneth A Follett, Frances M Weaver. Matthew Stem, Kwan Hur, Crystal L Harris, Ping Luo, William J Marks Jr, Johannes Rothlind, Oren Sagher, Claudia Moy. et al. Pallidal versus subthalamic deep-brain stimulation for parkinson's 's disease. New England Journal of Medicine, 362(22):077-2291 2010.
[14] David Freedman, Robert Pisani, and Roger Purves. Statistics (international student edition). Pisani, R. Purves, 4th edn. WW Norton & Company, New York, 2007.
[15] Justin Fu, Mohammad Norouzi, Ofir Nachum, George Tucker, Alexander Novikov, Mengjiao Yang, Michael R Zhang. Yutian Chen, Aviral Kumar, Cosmin Paduraru, et al. Benchmarks for deep off- off-policy evaluation. In International Conference on Learning Representations, 2020.
[16] Ge Gao, Qitong Gao, Xi Yang, Miroslav Pajic, and Min Chi. A reinforcement learninginfoormed pattern mining framework for multivariate time series classification. In International Joint Conference on Artificial Intelligence (IJCAI), 2022.
[17] Ge Gao, Song Ju, Markel Sanz Ausin, and Min Chi. Hope: Human-centric off-policy evalu- ation for e-learning and healthcare. In International Conference on Autonomous Agents and Multiagent Systems (AAMAS), 2023.
[18) Qitong Gao, Ge Gao, Min Chi, and Miroslav Pajic. Variational latent branching model for off-policy evaluation. In The Eleventh International Conference on Learning Representations (ICLR), 2023.
[19] Qitong Gao, Davood Hajinezhad, Yan Zhang, Yiannis Kantaros, and Michael M Zavlanos. Re- duced variance deep reinforcement learning with temporal logic specifications. In Proceedings of the 10th ACM/IEEE International Conference on Cyber-Physical Systems, pages 237-248, 2019.
[20] Qitong Gao, Michael Naumann, Ilija Jovanov, Vuk Lesi, Karthik Kamaravelu, Warren M Grill, and Miroslav Pajic. Model-based design of closed loop deep brain stimulation controller using reinforcement learning. In 2020 ACM/IEEE / Ith International Conference on Cyber-Physiee Systems (ICCPS), pages 108-118. IEBE, 2020.
[21] Qitong Gao, Miroslav Pajic, and Michael M Zavlanos. Deep imitative reinforcement learning for temporal logie robot motion planning with noisy semantic observations. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 8490-8496. IEEE, 2020.
[22] Qitong Gao, Stephen L Schmidt, Afsana Chowdhury, Guangyu Feng, Jennifer Peters, Kather- ine Genty, Warren M Grill, Dennis Turner, and Miroslav Pajic. Offline learning of closed-loop deep brain stimulation controllers for parkinson disease treatment. In Proceedings of the ACM/EEEE 14th International Conference on Syber-Physicad Systems (with CPS-loT Week 2023), pages 44 55, 2023.
[23] Qitong Gao, Stephen L Schmidt, Karthik Kamaravelu, Dennis A Turner, Warren M Grill, and Miroslav Pajic. Offline policy evaluation for learning- based deep brain stimulation controllers. In 2022 ACM/IEEE 13th International Conference Cyber-Physical Systems (ICCPS), pages 80-91. IEEE, 2022.
[24] Qitong Gao, Dong Wang, Joshua D Amason, Siyang Yuan, Chenyang Tao, Ricardo Henao, Majda Hadziahmetovic, Lawrence Carin, and Miroslav Pajic. Gradient importance learning for incomplete observations. International Conference on Learning Representations, 2022.
[25] A. Guez, R. D. Vincent, M. Avoli, and J. Pincau. Adaptive treatment of epilepsy via batch-mode reinforcement learning. In AAAI, pages 1671-1678, 2008
[26] Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In International Conference on Learning Representa- tions, 2020.
[27] Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International conference on machine learning, pages 2555-2565, PMLR, 2019.
12

---

[28] Danijar Hafner, Timothy P Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. In International Conference on Learning Representations, 2020.
[29] Beining Han, Zhizhou Ren, Zuofan Wu, Yuan Zhou, and Jian Peng. Off-policy reinforcement learning with delayed rewards. International Conference on Machine Learning, pages -380-8303. PMLR, 2022.
[30] Josiah Hanna, Scott Niekum, and Peter Stone. Importance sampling policy evaluation with an estimated behavior policy. In International Conference on Machine Learning, pages 2605-2613. PMLR, 2019.
[31] Botao Hao, Xiang Ji, Yaqi Duan, Hao Lu, Csaba Szepesvari, and Mengdi Wang. Bootstrapping fitted q-evaluation for off-policy inference. In International Conference on Machine Learning. pages 4074 4084. PMLR, 2021.
[32] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(3):1735-1788 1997.
[33] Yan Jia, John Burden, Tom Lawton, and Ibrahim Habli. Safe reinforcement learning for sepsis treatment. 2020 IEEE International conference on healthcare informatics (ICHI) pages 1-7. IEEE.
[34] Nan Jiang and Lihong Li. Doubly robust off-policy value evaluation for reinforcement learning. In International Conference on Machine Learning, pages 652-661. PMLR, 2016.
[35] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint urXiv:131..60 14, 2013.
[36] Iya Kostrikov and Ofir Nachum. Statistical bootstrapping for uncertainty estimation in off- policy evaluation. arXiv preprint arXiv:2007.13609 2020.
[37] A.A. Kühn, A. Kupsch, GH. Schneider, and P Brown. Reduction in subthalamic 8-35 hz oscillatory activity correlates with clinical improvement in parkinson's 's disease. Euro. J. of Neuroscience, 1957919956, 2006.
[38] Solomon Kullback and Richard A Leibler. On information and sufficiency. The annals of mathematical statistics, 22(1):79-86, 1951.
[39] Alexis M Kuncel and Warren M Grill. Selection of stimulus parameters for deep brain stimula- tion. Clinical neurophysiology, 115(11):2431
2004.
[40] Hoang Le, Cameron Voloshin, and Yisong Yue. Batch policy learning under constraints. In International Conference on Machine Learning, pages 3703-3712. PMLR, 2019.
[41] Pierre L L'Ecuyer and Bruno Tuffin. Approximate zero- simulation. In 2008 Winter Simulation Conference, pages 170-181. IEEE, 2008
[42] Alex Lee, Anusha Nagabandi, Pieter Abbeel, and Sergey Levine. Stochastic latent actor-critic: Deep reinforcement learning with a latent variable model. Advances in Neural Information Processing Systems, 33, 2020.
[43] Lihong Li, Wei Chu, John Langford, and Xuanhui Wang. Unbiased offline evaluation of bannttttuul-ba-ii----ee
news article recommendation algorithms. In Proceedings of the fourth ACM International Conference on Web Search and Data Mining, pages 297-306, 2011.
[44] Christopher G Lis, Kamal Patel, and Digant Gupta. The relationship between patient satisfaction with service quality and survival in non-small cell lung cancer-is self-rated health potential confounder? PloS one, 10(7):e0134617, 2015.
[45] Evan Liu, Moritz Stephan, Allen Nie, Chris Piech, Emma Brunskill, and Chelsea Finn. Giv- ing feedback on interactive student programs with meta-exploration, Advances in Neural Information Processing Systems, 35:36282-36294 2022.
13

---

[46] Yao Liu, Pierre-Luc Bacon, and Emma Brunskill. off ofllyy evaluation via conditional importance sampling. In International Conference on Machine Learning, pages 6184-618 6193. PMLR, 2020.
Understanding the curse of horizon in
[47] James MacGlashan, Mark K Ho, Robert Loftin, Bei Peng, Guan Wang, David L Roberts, Matthew E Taylor, and Michael L Littman. Interactive learning from policy-dependent human feedback. In International Conference on Machine Learning, pages 2285-2294, PMLR, 2017.
[48] Travis Mandel, Yun-En Liu, Emma Brunskill, and Zoran Popović. Where to add actions in human-in-the-loop reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
[49] Travis Mandel, Yun-En Liu, Sergey Levine, Emma Brunskill, and Zoran Popovic, Offline policy evaluation across representations with applications educational games. In AAMAS, volume 1077, 2014.
[50] Rishabh Mehrotra, James McInerney, Hugues Bouchard, Mounia Lalmas, and Fernando Diaz. Towards a fair marketplace: Counterfactual evaluation of the trade-off between relevance, fair- ness & satisfaction in recommendation systems. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management, pages 224--221,, 2018.
[51] Ofir Nachum, Yinlam Chow, Bo Dai, and Lihong Li. Dualdice: Behavior-agnostic estimation of discounted stationary distribution corrections. Advances in Neural Information Processing Systems, 32:2318-222 22188222222222 2019.
[52] Vivek Nagaraj, Andrew Lamperski, and Theoden I Netoff. Seizure control in computational model using a reinforcement learning stimulation paradigm. International Journal of Neural Systems, 27(07): 2017.
[53] Hongseok Namkoong, Ramtin Keramati, Steve Yadlowsky, and Emma Brunskill. Off-policy policy evaluation for sequential decisions under unobserved confounding. Advances in Neural Information Processing Systems, 19-331, 2020.
[54] Allen Nic, Yannis Flet-Berliac, Deon Jordan, William Steenbergen, and Emma Brunskill. Data- efficient pipeline for offline reinforcement learning with limited data. Advances in Neural Information Processing Systems, 35: 155:148001444 2022.
[55] Michael S Okun. Deep-brain stimulation for parkinson's 's disease. New England Journal of Medicine, 367((6):1529-1555 1538, 2012.
[56] Michael S Okun. Drap- stimulation for parkinson's 's disease. New England Journal of Medicine, 367(16):129-15388 2012.
[57] Bahram Parvinian, Christopher Scully, Hanniebey Wiyor, Allison Kumar, and Sandy Weininger. Regulatory considerations for physiological closed-loop controlled medical devices used for automated critical care: food and drug administration workshop discussion topics. Anesthesia and analgesia, 126(6):1916, 2018.
[58] Vihang P Patil, Markus Hofmarcher, Marius-Constantin Dinu, Matthias Dorfer, Patrick M Blies, Johannes Brandstetter, Jose A Arjona-Medina, and Sepp Hochreiter. Align-rudder: Learning from few demonstrations by reward redistribution. arXiv preprint arXiv:2009. /4108, 2020.
[59] Pineau, A. Guez, Robert Vincent, Gabriella Panuccio, and Massimo Avoli. Treating epilepsy via adaptive neurostimulation: reinforcement learning approach. Int. J. of Neural Sys., 19(04):227-240, 2009.
[60] Rob Powers, Maryam Etezadi-Amoli, Edith M Arnold, Sara Kianian, Irida Mance, Maxsim Gibiansky, Dan Trietsch, Alexander Singh Alvarado, James D Kretlow, Todd M Herrington, et al. Smartwatch inertial sensors continuously monitor real-world motor fluctuations in parkinson's 's disease. Science translational medicine, x3(579)) 2021.
[61] Doina Precup. Eligibility traces for off-policy policy evaluation. Computer Science Department Faculty Publication Series, page 80, 2000.
14

---

[62] Rafael Rafailov, Tianhe Yu, Aravind Rajeswaran, and Chelsea Finn. Offline reinforcement learning from images with latent space models. In Learning for Dynamics and Control, pages 1154-1166 168. PMIR, 2021.
[63] Claudia Ramaker, Johan Marinus, Anne Margarethe Stiggelbout, and Bob Johannes Van Hilten. Systematic evaluation of rating scales for impairment and disability in parkinson's 's disease. Movement disorders, 17()):677-876 2002.
[64] Sherry Ruan, Allen Nie, William Steenbergen, Jiayu He, JQ Zhang, Meng Guo, Yao Liu, Kyle Dang Nguyen, Catherine Wang. Rui Ying, et al. Reinforcement learning tutor better supported lower performers in math task. arXiv preprint arXiv:2304.04933 2023.
[65] Oleh Rybkin, Chuning Zhu, Anusha Nagabandi, Kostas Daniilidis, Igor Mordatch, and Sergey Levine. Model-based reinforcement learning via latent-space collocation. In International Conference on Machine Learning, pages 9190-9201. PMLR, 2021.
[66] Charlie Snell, Iya Kostrikov, Yi Su, Mengjiao Yang. and Sergey Levine. Offline rl for natural language generation with implicit language q learning. arXiv preprint arXiv:2206. 111022 1871, 2022.
[67] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
[68] Shengpu Tang and Jenna Wiens. Model selection for offline reinforcement learning: Practical considerations for healthcare settings. In Machine Learning for Healthcure Conference, pages 2-35. PMLR, 2021.
[69] Ziyang Tang. Yihao Feng. Lihong Li, Dengyong Zhou, and Qiang Liu. Doubly robust bias reduction in infinite horizon off- aff-policy estimation. In International Conference Learning Representations, 2019.
[70] Miguel Tejedor, Ashenafi Zehene Woldaregay, and Fred Godtliebsen. Reinforcement learning application in diabetes blood glucose control: A systematic review. Artificial intelligence in medicine, 104: 101836, 2020.
[71] Philip Thomas and Emma Brunskill. Data-efficient off off-pooliy policy evaluation for reinforce- ment learning. In International Conference on Machine Learning, pages 2139-2148. PMLR, 2016.
[72] Philip S Thomas. Safe reinforcement learning. 2015.
[73] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, (11),, 2008
[74] Mel Vecerik, Todd Hester, Jonathan Scholz, Fumin Wang, Olivier Pietquin, Bilal Piot, Nicolas Heess, Thomas Rothörl, Thomas Lampe, and Martin Riedmiller. Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards. arXiv preprint arXiv:1707.08817 2017.
[75] Martin Wattenberg, Fernanda Viégas, and Ian Johnson. How t-sne effectively. Distill, 1(10)ee,, 2016
[76] Joshua K Wong, Günther Deuschl, Robin Wolke, Hagai Bergman, Muthuraman Muthuraman, Sergiu Groppa, Sameer A Sheth, Helen M Bronte-Stewart, Kevin B Wilkins, Matthew N Petrucci, et al. Proc. the 9th annual deep brain stimulation think tank: Advances in cutting edge technologies, artificial intelligence, neuromodulation, neurocthics, pain, interventional psychiatry, epilepsy, and traumatic brain injury. Frontiers in Human Neuroscience, page 25, 2022.
[77] Mengjiao Yang, Bo Dai, Ofrr Nachum, George Tucker, and Dale Schuurmans. Offline policy selection under uncertainty. In Deep RL Workshop NeurIPS 2021, 2021.
[78] Mengjiao Yang, Ofir Nachum, Bo Dai, Lihong Li, and Dale Schuurmans. Off-policy evaluation via the regularized lagrangian. Advances in Neural Information Processing Systems, 33:6551- 6561, 2020.
15

---

[79] Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offline model-based policy optimization. Advances neural information processing systems, 011289944-8966 2021.
[80] Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. Advances in Neural Information Processing Systems,
14122444, 2020.
[81] Marvin Zhang, Sharad Vikram, Laura Smith, Pieter Abbeel, Matthew Johnson, and Sergey Levine. Solar: Deep structured representations for model-based reinforcement learning. In International Conference on Machine Learning, pages 7444-7453. PMLR, 2019.
[82] Michael Zhang, Thomas Paine, Ofir Nachum, Cosmin Paduraru, George Tucker, Mohammad Norouzi, et al. Autoregressive dynamics models for offline policy evaluation and optimization. In International Conference on Learning Representations, 2020.
[83] Ruiyi Zhang, Dai, Lihong Li, and Dale Schuurmans. Gendice: Generalized offline estimation of stationary values. In International Conference on Learning Representations, 2020.
[84] Shangtong Zhang, Bo Liu, and Shimon Whiteson. Gradientdice: Rethinking generalized offline estimation of stationary values. In International Conference on Machine Learning, pages 11194-11208 PMLR, 2020.
[85] Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. 2019.
arXiv preprint
16

---

A Proofs for Section 2.2
Recall that the per-decision importance sampling (PDIS)
61
estimator follows G Gppis
N
1 N-1 En y o H(1); here, w/ ( (o) (o(0)
B(a; ")
and henceforth
(8)
II =(a 1 (1))
are the PDIS weights for offline trajectory 7(0) To simplify our notation, we consider the PDIS estimator defined over single trajectory, L.e,
T-1
GTDIS = Σ Awaat?
(6)
1=0
as the results can be carried to N trajectories by multiplying with a factor 1/N, We omit the superscript-ed (0's in the rest of the proof also for conciseness.
A.1
Proof of Proposition I
Proof. We start with lemma from [46].
Lemma 46 ). Given X,, and Yt as two sequences of random variables. Then
[[ 2 Σ
[[XXXXX s (x) v Σ E(Ys)x.).
(7)
tck
where V() refer the variances.
Now, if we set Y; woer and Xx = Toe which is segment (from
t) of an offline
trajectory т, it is sufficient to prove Proposition | by proving that for any 1 - + 1 S k: T.
B 24 wo:twul < Biy?" 2 F Worr 1
(8)
since
Elr Wo.lwon
E [.][Y/XX)
(9)
<E|Y,YH
(10)
(11)
Applying the law of total expectations to (8) i.e.
E [IE
H4 Th Wo. I B[[[rr]
H Th
(12)
then it sufficient to show
Eir H wo.
I Bir 2 -
(13)
We start by showing that
E Wor roaa
u Flrr Too [[ F wi w14
(14)
= Blr To:: [
H wh 11417-111 1 F 177a)
(15)
The transition above follows from the fact that the likelihood ratio wort martingale shown in [41 (6). i.e., E[worr-1 To:]] = woee which implies E[w4+1--1] = and therefore
E E 24 w [oxe]
(16)
E I H. 1
Toa]
(17)
=E 2f - Toa].
(18)
Note that we keep using the notation E/ in (4p and (5), due to the setting of HMDP that r is variable sampled from the distribution R
(-]se, ae) given (Se, az); see Section 2.1
17

---

Since To. is given, r'k 24 and w1+1:T-1 equivalent to r k-1-1 --2 for some other trajectory (i)
())
and ())
-(0) p². Also given the statement that -0-
and r---- 10: are positively correlated, it
(1) (j)
follows from [5 that
F6 H. Elre r4 woo
(19)
Wi Elr [T: H
(20)
Swo Ely To:t [R] 1 [
(21)
=10 [r H wo 140: - [oa]
wo.T
(22)
which justifies
113 and completes the proof.
B Derivation of the ELBO 2
Note that unlike the ELBO in 18 the VLM-H includes an additional component that estimates the human return G of each trajectory, i.e.,
log Pel (Su.T, ro: G
(23)
=
log
Po (S 21:T, 10:1-1, (
(24)
ZITEZ
= -log
(0 (0:T-1) Vo (0
(25)
#1,TEZ
BB [logp(z))
+ llooo 21.7, ( + log (G (T)) log (2.0Tll0 4g.7
1)]
(26)
-Ego logg(zo) + log po + IG. + Σ T
1=1
log (2gl so)
(2
(27)
-Ego
logp(zo) - log (20 -btp
log (G (2T)
log 12002120-1,0011113
1=1
log
(28)
log p (G (2T) + Σ 1 log logpo(5e120) + Σ logpo(ri-1lz))
=E =Ego
(K((g(tooss))ll(eo)) Σn T
(29)
Note that simplify our presentation, omit s/ p part of the conditional terms in the joint likelihoods. The transition from 255) to (26) follows Jensen's inequality.
C Brief Overview of Existing OPE methods
C.1 Importance Sampling (IS) and per-decision IS (PDIS)
IS refers to statistical technique that calculate the expectation of function f(x) w.r.. an unknown distribution p(z) using a given distribution (Ix) through re-weighting, i.e.,
f())p(a) g(x))
[,]) E,
(30)
This technique can be applied in the context of OPE by setting fz) as the accumulated return Go p as the trajectory distribution p" over the target policy æ, and q as the trajectory distribution n over the behavioral policy 3.
18

---

According to [6], the vanilla IS estimator follows
N N-1
G% N
(4)
G
(31)
0:7
m(eelss) where WO:T t== B(a,]s)) refers to the IS weight, and Go is the return for the i-th (out of N)
(6)
offline trajectory. On the other hand, the PDIS estimator follows
N-1T-1 Σ
OPDIS N
Σ
(32)
rla being the PDIS weight, (0 the environmental reward obtained the with w I1 Play eg)
t-th step of the i-th offline trajectory, and y is the discounting factor.
C.2 Doubly Robust (DR)
DR attempts to reduce the variance of IS estimation by introducing the value function approximation, which trades off estimation variance by making the estimation biased. The non-recursive definition of DR estimation is provided in [71]. i.e.
N-1T-1 Σ
N-1T-1 ΣΣ
G
1 (2( Q
(1) V=
Woa
(33)
here, (*( and V*(1 (.) are the Q-function and value function, respectively, over the target policy, while S and a, are the state and action taken at the f-th step of the i-th offline trajectory.
C.3 DIstributional Correction Estimation (DICE)
DICE aims to estimate the propensity the target policy to visit particular state-action pairs relative to their likelihood of appearing in the offline trajectories, i.e.,
ds (s,
(34)
(s.
(x.a) where (s.0) is the distribution correction ratio. Existing DICE variants
83
78
51
seek
to approximate the ratio without knowledge of or d8, while recent work has summarized that
all existing variants can be formulated as regularized Lagrangians of the same linear program (78)
unifying the choice regularizations and constraints used their original objectives.
C.4 Fitted Q-Evaluation (FQE)
According to 40), FQE approximates the Q-function over the target policy by minimizing the loss
min (rr (Q( az k)) ye) y1)²}
(35)
s.t. Vi = + (00e441,(55444))) (88(11): k)):
(36)
here, the approximated Q-function parameterized by н. Note that different from the objective
for classic Q-learning, or fitted Q-iteration, by using (T(++1) instead of max, Q( a; K) in the
target l/t.
D Additional Experimental Details and Results
In this section provide additional details that are supplement to the discussions in Section
19

---

D.1 Definition of the OPE Metrics
Mean Absolute error (MAE). MAE is defined as the absolute difference between the actual return and estimated return of a policy: MAE = IV V= here, V is the actual value of the policy x. and V7 is the estimated value of x.
Rank correlation. Rank correlation measures the Spearman's 's rank correlation coefficient between the ordinal rankings of the estimated returns and actual returns across policies, i.e., (oe(rank p.).k.a where (o
is the ordinal rankings of the actual returns, and rank Vi.p)) the ordinal rankings of the OPE-estimated returns.
Regret@ I is the (normalized) difference between the value of the actual best policy, against the value of the policy associated with the best OPE-estimated return, which is defined as (max V max jebest((())))) VJ")/ max4e::P Vi", where best(1 P) denotes the index of the best policy over the set of P policies measured by estimated values 1"
Regret@1 1.
D.2 Implementations and Hyper-parameters
VLM-H. Both encoder (2 94-1, s,) and decoder Pp Pp P=(24|2-11,q-1) of VLM-H are con- figured be LSTMs with 64 nodes for the hidden states, followed by two fully connected layers with 128 and 64 nodes each. All other parts of the encoder and decoder, including go(2o] Ps ( ( (r), P (s: Po are multi-layered perceptrons (MLPs) with two fully
connected layers with 128 and 64 nodes each. The regularization term introduced in [18] is added to the ELBO ) with a scaling factor C" balancing the scale of the two terms, which is applied over the LSTM states between Uu (=x/=// (, and (= ; (t--). 1). The scale of this regularization is selected from C" =/ee004, 5e-03. Ie-03, 5c-02, 1c-02, 0.1, I., 2., 5. Learning rate is tuned by grid search from (0.003, 0.001, 0,0007, 0.0005, 0.0003, 0.0001, 0.00005 J. Exponential decay is applied to the learning rate, which decays the learning rate by 0.997 every iteration. The total number of training epoches is set to be 20 and minibatch size set to 64. Adam optimizer is used to perform gradient descent. L2 weight decay with coefficient 0.001 and batch normalization are applied to all hidden fully connected layers. The top-10 models/checkpoints achieving the highest ELBOs proceed to the RILR step.
Downstream OPE Estimators. The implementations of downstream OPE estimators are built on top of [36 8
which provides the functionalities for using per-decision IS with behavioral policy estimation [30 DR (7I) DICE (78) or FQE 40 process the trajectories with reconstructed IHRs and estimate human returns.
Code Availabilities and Computational Resources. Code implementations for all three compo- nents above can be found in the supplementary materials attached. All experimental workloads are distributed among 4 Nvidia RTX A5000 24GB and 3 Nvidia Quadro RTX 6000 24GB graphics cards.
D.3 More on the Adaptive Neurostimulation Experiment - Deep Brain Stimulation (DBS)
DBS Hardware and Implementation. The internal pulse generator (IPG) is part of the Medtronic's Summit RC+S system. Moreover, Summit software APIs are provided by Medtronic to be leveraged to implement the RL policies to be tested in the clinic. The overall pipeline is shown in Figure 5 Specifically, each control cycle (an MDP step) takes 2 seconds, where the IPG collects local field potentials (LFPs) from the basal ganglia (BG) region of the participant's 's brain at a sampling rate of 500 Hz. Then, the LFPs are sent to the research tablet for signal processing, after which the beta-band power spectral densities (beta powers) are calculated and used to constitute the MDP state and reward signals. The beta powers caleulated by applying fast Fourior transform (FFT), with 512 bin width, to the L.FPs collected in the past 2 seconds. The beta-band usually refers to the frequency band of 13-35 Hz, however, clinicians and electrical engineers had customized the range for each specific
'Original implementation obtained from Nommhttps::/// ottp://coooooooooeeeeeeooooooooo
tree/mmaster/pollcyycoo following the Apache License v2.0
20

---

Adaptive Program
Stimulation
Amplitudes
Amp.
Protovoll
RL Controller
Com
Wireless
Telemetry
LFPs
Beta-power Calculation
Local Field
Research
Potentials
Internal Pulse
Tablet
(LFPs)
Generator (IPG)
Figure 5: Implementation of RL control the research tablet over the API provided by the ufacturer of the DBS device. During clinical testing, the research tablet evaluates the RI, policies which determine the stimulation amplitudes to be used in each step (every 2 seconds). Then, by communicating with the internal pulse generator (IPG) through wireless telemetry, the IPG adjust the stimulus accordingly and sends back the local field potentials (LFPs) which used to determine the MDP state and reward for the next step.
Electrode
GPe GPI
STN
Contact #1
Contact #2
Contact
Pulse Generator
Contact #4
Figure 6: An illustration towards the IPG electrodes are placed in the basal ganglia (BG) region of the brain 4-contact electrodes placed in the subthalmic nucleus (STN) and globus pallidus (GP).
patient, in order to allow beta powers to better capture the irregular neuronal behaviors. The specific range for each patient can be found in the paragraph below (i.e., patient characteristics). According to the latest LFP and beta power signals, the RL policy adjusts the amplitude of the electrical stimulus be used for the cycle, which is then sent back the IPG through wireless telemetry. In the BG region of participant's brain, as illustrated in Fig. 6 4-contact electrodes are placed in the subthalmic nucleus (STN) and globus pallidus (GP) regions. Monopolar stimulation was delivered on a single contact on each lead (with the case serving as counter-electroder The two contacts surrounding the stimulation contact were used for sensing LFPs (i.e., sandwich sensing).
The IPG device was already approved for clinical research use by the FDA. The changes made to the software program also approved by the FDA via an investigational device exemption (IDE). Corresponding IRB approvals were obtained from Duke University Health System. All participants were compensated per institutional and administrative guidelines.
Patient Characteristics. As discussed in Section 3.1 all PD participants have unique characteristics toward varied PD severity and symptoms, which can be found in (22].
The MDP Setup. The state and action space, as well as rewards are briefly introduced in Section 3.1 Here we provide the details that are omitted in the main paper. Each discrete step MDP corresponds to control cycle which lasts 2 seconds; see the DBS hardware and implementation paragraph above. Each episode lasts least T 300 steps (i.e., the horizon), or more than 10 minutes in the real world. Specifically, states are constituted by a historical sequence of beta powers obtained from past 10 steps, following similar setup as 20 (22 The actions a E AA are [g. 100%] C RR 0% Ç < 100%, which represent the percentage over the maximum amplitude of the stimulus the IPG can employ (determined by clinicians). Specifically, & 40% for patient #0 and 60% for patients #1 #1-#3, to ensure safety and minimum efficacy of DBS during clinical testing. The environmental rewards
21

---

Table 4: Raw results of the adaptive neurostimulation experiment from each patient (Patient #0).
IS
DR
Ablation
OPEHF (our)
OPEHF
Fusion
Rescale
Fusion
Rescale
VLM-H
(our)
MAE Rank
0.96+0.0
0.74+0.0
0.17#0.01
0.21
165=#,007
0.09=0.01
0.97=0.02
0.65+0.01 0.05 0.0
0.65+0.01
0.74+0.07
0.43+0.05
0.63+0.08 0.02 + 0.07
0.73+0.09 0.58+0.1
Regret@1
0.05 # 0.0
0.04 0.01
0.03 0.38
0.03 # 0.01
0.02=0.02
DICE
FQE
OPEHF
OPEHF
Fusion
Rescale
Fusion
Rescale
(our)
(our)
MAE Rank
4.59+0.1 17
4.0+0.2
0.98:0.01
1.6420.03
0.63+0.0
0.27=0.0
0.52+0.05
0.5640.08
0.62#0.09
0.48=0.07
0.49#0.02 0.05 + 0.0
0.58=0.07
Regret@1
0.05 = 0.17
0.02 = 0.2
0.03 0.01
0.02 + 0.03
0.03 = 0.0
Table 5: Raw results the adaptive neurostimulation experiment from each patient (Patient #1).
IS
DR
Ablation
OPEHF
OPEHF
Fusion
Rescale
Fusion
Rescale
VLM-H
(our)
(our)
MAE Rank
0.5910.00
0.44.10.0
0.43.10.01
2.1110.14
1.03
0.27 1 0.03
0.98+0.01
0..01
0.52.10.08
0.6610.08
0.74.10.07
0.6.10.55
0,87 0.02
0.58+0.03
Regret@
0.21
0.02.10.0
0.02.10.01
0.0110.14
0.0
0.03
0.03+0.01
DICE
FQE
OPEHF
OPEHF
Fusion
Rescale
Fusion
Rescale
(our)
(our)
MAE Rank
0.47:10.01
0.45=00.1
0.87=1001
2.53+0.07
1.05
0.53
0.73+0.05
0.72=0.09
0.9::0.33
0.48
0.76
0.74:10.06
Regret
0.03:1001
0.03:0.01
0.03:1001
0.01
0.04
0.02:10002
defined following, i.e.,
1 0.2a if latest beta power is above a threshold, 0.2a if latest beta power is below threshold;
R(s,
(37)
here, -0 -0.2a is the penalty for stimulating with higher-than-needed amplitudes, to preserve the runtime of the IPG device (powered by a rechargeable battery) between recharges, and an additional -1 is given if stimulating with the amplitude determined by the policy cannot reduce beta power to below threshold specific each patient. The thresholds above set to be the lower 20% quantile of the beta powers observed from initial exploration (introduced below). The human return at the end of each episode quantified as 50%-25%2554 weighted sum over the patient rating, hand grasp speed captured from performing the maneuver (rapid and full extension and close of all fingers) for the bradykinesia test, as well as the proportional length of the session where the patient displays tremor (as captured by wearable accelerometry).
Collection of Offline Trajectories. Mixed types of controllers are used for initial exploration of the environment (for obtaining target policies introduced below), and the resulting trajectories are used as the offline training data for OPEHF. Specifically, a random controller that uniformly sample the stimulation amplitudes from [s. 100%), and proportional-integral (PI) controller tuned by clinicians and electrical engineers, are used at the initial phase of clinical testing. They help identify the state-action regions that can lead to non-trivial returns. Then, such data are used to train RL policies- - following deep deterministic policy gradient (DDPG), the deterministic counterpart of the soft actor-critic (SAC) for initial exploration, most of which can only obtain up to 60% returns as obtained by the target policies below on average. This makes OPEHF challenging considering the limited exploration coverage provided by the offline trajectories.
Target Policies. A total of target policies, to be evaluated by OPEHE, are considered for each patient. One of them is an open-loop policy that always stimulate with the lowest possible amplitude
22

---

Table 6: Raw results the adaptive neurostimulation experiment from each patient (Patient #2).
IS
DR
Ablation
OPEHF
OPEHF
Fusion
Rescale
Fusion
Rescale
VLM-H
(our)
(our)
MAE Rank
0.63+-0.03
0.7+-0.02
0.33+-0.01
2.17+-0.3
1.110.116
0.3+-0.01
0.97+0.02
0.65+0011
0.71+-0.05
0.94-0.03
0.52+00.09
0.59+00.04
0.84+00.3
0.6+0.0
Regret@1
0.05=10.03
0.0810002
0.08
0.0
0.08
0.1:0.01
0+0.02
DICE
FQE
OPEHF
OPEHF
Fusion
Rescale
Fusion
Rescale
(our)
(our)
MAE Rank
5.44+0.55
3.52=0.12
0.66+-0.02
1.6+0.066
0.81=0.02
0.38
0.66+0.09
0.48+0.08
0.85
0.6+0.05
0.67+0.04
0.93++0.01
Regret@
0.11
0.08=0012
0.1+00.02
0.08+0006
0.08=
0.08+
Table 7: Raw results the adaptive neurostimulation experiment from each patient (Patient #3).
IS Rescale
DR
Ablation
OPEHF
OPEHF
Fusion
Fusion
Rescale
VLM-H
(our)
(our)
MAE Rank
0.94+00011
0.76++0.01
0.24+10.0
1.69+00.51
0.69-00.06
0.17+10011
0.94+0.01
0.42++0.11
0.16+00.3
0.71++0.05
0.48++0.05
0.42=0011
0.62+0.14
0.45+0.06
Regret@1
0.21+0001
0.01
0.04+0000
0.35+0.51
0.1=0.06
0.04+00.01
0.03+0.01
DICE
FQE
OPEHF
OPEHF
Fusion
Rescale
Fusion
Rescale
(our)
(our)
MAE
0.41=00.22
1.15=0.07
1.03=1.04
1.56+0.09
0.53#0.02
0.19:1001
Rank
0.28+0.05
0.22+0003
0.6620.07
0.48+0.05
0.29=0.02
0.52+10001
Regret@)
0.02:1000
0.03+0.07
0.03:10.04
0.1
0.05=0002
0.03+0001
(E%), benchmarking the worst-case control performance. The other 5 target DBS policies are all trained using DDPG. Specifically, each is trained over a growing experience buffer containing the data collected from the latest trials, in addition to the fixed set of offline trajectories above; this would lead to a set of target policies with varied performance, as the policies obtained in the later stage tend to be more optimal since the environment has been explored the policies obtained at earlier stages. Moreover, the policies have to demonstrate at least moderate control efficacy (as determined by clinicians), in order to be considered as target policies; since the ground-truth return of each target policy is determined by averaging the human returns over more than 10 sessions of clinical testing (> 100 minutes in total). The raw human returns of all target policies over each patient can be found in Tables to I1 as well as in the legends of Figure 4 Note that, for each patient, there may exist few policies that lead close returns, due the testing limitations introduced above, i.e., the use of & as well as the safety protocol for long-term testing (introduced above).
broader extent, compared to
Correlation between beta power and patient feedback. As discussed in Section 3.1 existing research [66 37 76] have found inconsistency between the beta power and patient feedback, due to confounders related to patient's ' specific characteristics as well as varied symptoms caused by PD. Figure also shows that only low-to-moderate correlations can be found between beta power and the HF signals that are used to define the human returns in general. This makes OPEHF more challenging the environmental signals may contain limited information useful for extrapolating human returns.
D.4 More the Intelligent Tutoring Experiment
Experimental Setup. All the students follow a 4-step procedure, i.e., (i) study with textbook, (ii) pre-test, (i7) study with the intelligent tutoring system, (iv) post-test. The RL policies are deployed only in step (iif) following the MDP setup introduced in Section 3.2 The human returns are quantified
23

---

Right Hand Grasp Speed [Hz]]
P. P0,80003
Left Hand Grasp Speed [Hz]
6 6.13
0000
2.0-
20
64
-1.0
-55 0.
15 2.0 2.5
-05 D.
34 14
2.5
1.
Beta Amplitude [In[uV)]
Beta Amplitude [In[uV)]
(a) Low correlations are found between beta power and the left hand grasp speeds as result the bradykinesia test.
(b) Low correlations found between beta power and the right hand grasp speeds as result the bradykinesia test.
1/2
Time Proportion with Tremor (%)
0.00
10-
-0.8-002
PL, -087
0.34
P2 R54
-2.6
0
0.75
Rating
24
-1.0
-25
5.
25
10
15
2.0
255
3.0
-1.0 -05
00 25 15
20 25
3.0
Beta Amplitude [In(uV)]
Beta Amplitude [In(uV)]
(c) Low-to-moderator correlations over patients, strong correlation over patient, are found be- tween heta power and the tremor severity quantified as the total time (proportional to the horizon) when patients experience tremor (captured by wearable accelcrometry) each testing session.
(d) Low-to to-meerattorr correlations and strong correlation over patient, are found be- tween heta power and the rating (1-10) provided by the patients at the end each testing session.
patients,
Figure 7: Correlation between the beta power and HF provided by patients from all cinical sessions.
as the normalized gain over the difference between the scores from post-test and pre-test, i.e.,
GH
scorEpost scorepre
(38)
scorepre
Specifically, pre-test consists of total of 14 single- and prrnciplp problems. No feedback are given following the pre-test, nor are they allowed to re-visit any questions from the test. After working with the intelligent tutoring system, the students take the post-test which contain 20 problems, 14 of which are isomorphic
pre-test and the remaining ones are non-isomopphic multiple-principle problems. Both tests were auto outo-graded following a specific rubric.
More on the MDP States. introduced in Section 3.2 the states are constituted by 140 expert- selected features; each fall into one of the five categories including (i) autonomy (10 features),

---

Table 8: Environmental and human returns of all the target policies pertaining Patient #0, from the adaptive neurostimulation experiment.
Target Policy
Environmental Returns
Human Returns (x10)
#0
-64.18
182.12
#1
-128.96
187.36
#2
-131.37
174.78
-134.84
182.23
#4
-129.82
158.04
-80.69
104.8
Table 9: Environmental and human returns of all the target policies pertaining to Patient #1, from the adaptive neurostimulation experiment.
Target Policy
Environmental Returns
Human Returns (x10)
-144.51
240.47
#1
-155.0
234.51
#2
-156.53
233.78
#3
-135.75
229.01
-47.19
229.38
#5
-86.13
186.45
capturing the effort spent on solving problems in the intelligent tutoring system, such as the number of times the student revisit problem, (ii) temporal information (29 features), including the time (35 features), such as the difficulty of each problem,
spent on each problem etc., (iii) problem-specin (iv) performance (57 features), such the grade students received on each problem, (v) hint-related (11 features), including the total amount of hints requested, if applicable.
Target Policies. total of four target policies are considered. One the policies randomly selects an action at each state, which benchmarks the worst-case performance. The other 3 policies are trained using deep Q-network (DQN) with different learning rates, from (le-3, le-4, le-5).
E A Challenging Simulation Environment: Visual Q&A Dialogue
We also consider the visual Q&A dialogue environment 11. following the recent framework [66] that generalizes offline RL towards language generation tasks. In this environment, language generation agents are trained to ask questions over the image captions that are given (without showing the actual images), in order to gain as much information from the responses (returned by environment) that are useful to characterize the the key elements contained the underlying image.
Overall Setup. Though this environment does not involve actually HF, the policies' performance intrinsically pertain to two types of returns as per the setup from [66]. Specifically, the immediate rewards, used to train the language generation policy (captured by generative pre-trained transformer version 2, or GPT2) to asks questions, are discrete, i.e., a -1 reward is given at each step until most key elements can be inferred from the dialogue, and additional -1 reward given if the questions asked by the agent lead to responses that only contain trivial information towards extrapolating the image characteristics (such the responses that only contain 'yes/no") ) In contrast, a separate pre-trained referee model is used to determine when sufficient elements in the underlying images can be captured from the dialogue history and terminates the episode, which maps the dialogue history to embeddings that represent the features pertaining to the image elements that can be extrapolated from the dialogue, followed by scoring the dialogue with relative percentile ranking of the ground truth image's ' distance from the predicted embedding among set of images taken from the evaluation set provided by [66. 10,, i.e., an episode terminated if the percentile ranking improved by at least. 50% compared to the percentile ranking before the dialogue begins,
PT) 2 55 where pr the ground truth image's percentile rank at dialogue's 's last turn and Po is the ground truth image's 's percentile rank at the beginning of the dialogue, when only the image caption is observed. As
Po),
25

---

Table 10: Environmental and human returns of all the target policies pertaining Patient #2, from the adaptive neurostimulation experiment.
Target Policy
Environmental Returns
Human Returns (x10)
#0
142.20
160.42
#1
-79.18
163.55
#2
-155.11
158.53
-152.65
165.62
#4
-154.66
161.03
-80.63
69.42
Table 11: Environmental and human returns of all the target policies pertaining to Patient #3, from the adaptive neurostimulation experiment.
Target Policy
Environmental Returns
Human Returns (x10)
#0
-60.78
154.89
#1
-83.83
66.20
#2
-126.97
142.45
#3
-154.83
129.57
-147.44
142.29
#5
-165.74
143.39
result, the former type of rewards considered the environmental rewards per the OPEHF setup, and the latter one is closely related to the human returns as it is only provided at the end of each episode and obtained following a different schema. Specifically,
consider Po - Pr the synthetic human return, which quantifies the improvement over the percentile ranking throughout the dialogue. More details the environmental reward function and the percentile ranking generated by the referee model can be found in [66]
MDP Formulation. Consequently, in this experiment compare OPEHF's performance against baselines, toward predicting the percentile rankings corresponding to the embeddings predicted from the referee model the end of episodes. The MDP states captured by the embeddings output by the attention layer from the encoder of the GPT2, by feeding in the dialogue history up to the current step. The action space is captured as Vi, where V is the vocabulary considered and 7 is the maximum number of words the agent is allowed to place in the questions in each step. The definition of environmental rewards and synthetic human returns are introduced above.
Offiine Trajectories and Target Policies. Five behavioral policies are used to collect offline 4 types of RI. algorithms following the setup introduced in [66]. i.e., 1 policy follows conservative Q-learning (CQL., follow implicit Q-learning (IQL), follows behavioral clone (BC) and the last follows decision transformer (DT), All the behavioral policies are trained to obtain only 10%-50% of the best synthetic human returns that can be achieved, which are considered sub-optimal. Eight different policies (also obtained from 4 types of RL algorithms) constitute the set of target policies whose performance will be estimated by OPEHF and the baselines. Their performance spread roughly equally between 0% 100% of the optimal return that could be achieved; details can be found in Table [3] It can be found that synthetic human returns can be negative the dialogue provides trivial or sometimes misleading, information that are not helpful for capturing the underlying image. In practice one can shift the returns to become all positive order to comply with the assumption Proposition [ Moderate correlations are found between the environmental and synthetic human returns in this experiment, i.e., Pearson's 's correlation coefficient 0.606 and Spearman's 's rank correlation coefficient is 0.69.
trajectories, which are obtained from training
Results and Discussion. The results are summarized in 12 It can be observed that the OPEHF in general outperforms the baselines, with significantly lower MAES and higher ranks IS, DR or FQE are selected as the downstream estimator. We also find that if DICE is selected as the downstream estimator, it in general leads lower MAEs and regrets for both the OPEHF and rescale baseline, but significantly low rank correlations. Our hypothesize would be that considering DICE directly
26

---

Table 12: Results from the visual Q&A environment.
IS
DR
OPEHF (our)
OPEHF
Fusion
Rescale
Fusion
Rescale
(our)
MAE Rank
1.13#0.41
0.40=0.27
0.56:1011
1.27=0.33
0.72:10.21
0.63=1.08
-0.21=0.55
0.19=0.15
0.6410.113
0.11=0.45
0.08
0.59
Regret@1
0.76++-033
0.08
0+-0.05
0.43++0.29
0.25++0.09
0.01
DICE
FQE
OPEHF
OPEHF
Fusion
Rescale
Fusion
Rescale
(our)
(our)
MAE
0.99=t0.12
0.25
0.11+0002
0.97
1.13#0.14
0.89=10.03
Rank
0.02
-0.13:0.1
0.06+1.32
0.1500.05
0.12::0.52
0.55=10.28 0.03+10.03
Regret
0.01
0.07=0.0
0.02++0.03
0.01
0.01+000
Table 13:
Environmental and synthetic human returns of the target policies, as well as the algorithms used to obtain them, considered in the Q&A dialogue experiment.
Target Policy
Environmental Returns
Synthetic Human Returns
Algorithm
#0
-5.46
-8.15
CQL
#1
-11.77
-3.20
IQL
#2
-10.86
1.09
CQL
#3
-13.39
2.24
CQL
#4
-13.50
2.36
BC
#5
-11.33
3.09
CQL
#6
-6.05
3.23
IQL
#7
-9.77
3.66
DT
estimate the discrepancy between target and behavioral policy's propensity of visiting particular state-action pairs across the offline trajectories, it may benefit from leveraging the information (over the acteonctionn space) captured by language encoder that place roughly equal attention to each step of the dialogue, as opposed to the GPT2's encoder which is not fine-tuned specifically for such a case.
27


Reviews:
[{"review_id": "EZQnerfI4q", "reviewer": "Reviewer_ochq", "summary": "The paper introduces a framework for Off-Policy Evaluation for Human Feedback (OPEHF). Off-policy evaluation allows the evaluation of learned policies on offline recorded data, which is especially valuable in scenarios where online deployment to evaluate policies is dangerous or expensive. Especially in scenarios such as healthcare, human feedback is very valuable and apparently often not very correlated with environmental rewards, making the combination of these two aspects very interesting. The paper considers the challenging case where human feedback is only available at the end of an episode without any per-step immediate human reward (IHR). To utilize existing OPE approaches, the per episode human reward is first reconstructed to a sequence of IHR by optimizing the sequence such that the cumulative discounted sum equals the human rewards, while additionally regularizing the reconstructed IHRs to align well with a latent representation of a state-action pair sequence. The resulting approach is evaluated on 2 real world scenarios, showing interesting results.\n", "questions": "- In Proposition 1 you assume that the expected value of the rewards is positive. This is obvious in the considered scenarios, but would this approach generalize to other scenarios where human feedback could indeed be negative?\n- To what extend does the VLM-H hyperparameter tuning play a role here? How robust is the overall approach to less optimal setups here?\n- If you could discuss further evaluation venues for the proposed method, that would be interesting, but honestly I quite like the overall paper and find it to be an interesting read. If no further bigger issues are brought up by the other reviewers, I am willing to up my rating, I would just like to keep it a little conservative given this is very much outside of my expertise.", "limitations": "Some limitations are discussed and no other obvious limitations or major negative societal impacts come to mind.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper tackles a problem with realistic applications and proposes an interesting approach to do so.\n- It's well written with an extensive supplementary. Even though this is sadly not my area of expertise, I felt the paper was made to be understandable.\n- Results look favorable compared to simpler baselines and the method does not rely on environmental rewards being correlated with the human feedback, for which the paper gives multiple interesting example use cases.\n- Code is provided.\n\n", "weaknesses": "- While I very much like the actual longer term studies with real humans, overall the empirical evidence still is not extremely convincing. I guess this is an inherent problem with human feedback being expensive to collect. Nevertheless, some additional experiments would of course be very valuable.\n ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate the reviewer's thoughtful questions and constructive feedback. Please find our responses below. We hope they addressed the reviewer's outstanding concerns sufficiently, and we would be more than happy to further respond to any follow-ups the reviewer may have.\n\nQ1: While I very much like the actual longer term studies with real humans, overall the empirical evidence still is not extremely convincing. I guess this is an inherent problem with human feedback being expensive to collect. Nevertheless, some additional experiments would of course be very valuable\n\nR1: We greatly appreciate the reviewer's understanding toward the complexity of conducting experiments with human participants, as well as collecting feedback from them, over years of effort. We tried to focus on real-human experiments as currently there lacks a standard testing bench toward OPEHF's objective. However, we had found that recently some natural language Q&A machines [1] employ a similar evaluation fashion as our problem setup -- policies are trained over discrete environmental rewards, but are evaluated by a separate referee model that produces dense end-of-episode returns. This experiment is briefly mentioned in lines 223-225 and details can be found in Appendix E. Only moderate-correlations are found between the two types of rewards as well, and results show that our method still rigorously outperforms the baselines.\n\nQ2: In Proposition 1 you assume that the expected value of the rewards is positive. This is obvious in the considered scenarios, but would this approach generalize to other scenarios where human feedback could indeed be negative?\n\nR2: Thank you for this thoughtful question that points to one of the future directions of our work. The assumption is mainly used in the proof (Appendix A, equations (20) and (21)) that the expectation of rewards appears on both sides of the inequality. With some initial exploration, we might be able to relax this assumption in the future by using a different theoretical framework for the proof, but with the price of obtaining an asymptotic bound, instead of the strict variance reduction property as shown in proposition 1. On the other hand, if the lower and upper limit of rewards are given *a priori*, one may apply transformations over the rewards so that they satisfy the assumption. In the meantime, please note that several downstream OPE methods may intrinsically require the rewards to follow specific properties. For example, some FQE methods [2, 3] assume that rewards are in $[0,1]\\subset\\mathbb{R}$, and a doubly-robust (DR) method [4] assumes that the lower and upper limit of the rewards are known.\n\nQ3: To what extent does the VLM-H hyperparameter tuning play a role here? How robust is the overall approach to less optimal setups here?\n\nR3: Good question. We have done further analyses to this end -- please see Figure 1 in the PDF uploaded for rebuttal. The **VLM-H is mainly used to facilitate the regularization** (eq.(3)), where the state-action pairs that lead to similar embeddings should get similar (reconstructed) immediate human rewards (IHRs). So we find that the overall performance is not significantly bounded by VLM-H. Specifically, in Figure 1 we visualize the latent space of multiple sub-optimal VLM-H's trained with different hyper-parameter sets. The latent space in general demonstrates decent clustering behavior, if the hyper-parameters are not too off, which can be guaranteed by inspecting the visualization of a holdout validation set. Sub-optimality in VLM-H would only lead to less than 5% fluctuation in performance in general, which would be considered insignificant as our method outperforms baselines generally with great margins. \n\nQ4: If you could discuss further evaluation venues for the proposed method, that would be interesting, but honestly I quite like the overall paper and find it to be an interesting read. If no further bigger issues are brought up by the other reviewers, I am willing to up my rating.\n\nR4: What makes the evaluation of OPEHF challenging, would be the fact that the goal is to estimate participants' feedback toward their own experience (e.g., patient's satisfaction after treatment). Unlike frameworks like RLHF for language model training [5], where one can hire 3rd-party raters to provide feedback simply by following a predefined rubric (without the need of domain expertise); in human-centric domains like healthcare or education, professionals that are thoroughly trained (e.g., doctors, professors, instructors) over years may only be eligible to rate, other than the participants themselves. As a result, we currently also invest significant effort toward building a platform that provides de-identified and privacy-protected dataset (e.g., the one we used in this paper) through API access, as well as allow verified professionals to provide feedback/ratings. Moreover, once the platform scales, we can train a referee model (similar to the one used in the automated Q&A machine), to capture how professionals would examine new trajectories and provide ratings directly. Hopefully, in the near future, researchers can leverage the platform to evaluate OPEHF-related works in an standardized manner.\n\n[1] Snell et al. \"Offline RL for Natural Language Generation with Implicit Language Q Learning.\" ICLR'22.\n\n[2] Zhang et al. \"Off-policy fitted q-evaluation with differentiable function approximators: Z-estimation and inference theory.\" ICML'22.\n\n[3] Hao et al. \"Bootstrapping fitted q-evaluation for off-policy inference.\" ICML'21.\n\n[4] Thomas et al. \"Data-efficient off-policy policy evaluation for reinforcement learning.\" ICML'16.\n\n[5] Ziegler et al. \"Fine-tuning language models from human preferences.\" arXiv:1909.08593 (2019).\n"}, {"review_id": "b140U54PMh", "reviewer": "Reviewer_KHGy", "summary": "This paper discusses a framework that learns to decompose human feedback reward, which is usually sparse and only given at the end of the episode, to intermediate step reward. They formulate a reward decomposition objective that considers original environment trajectory information using a variational auto-encoding representation learning objective as an implicit constraint. The authors show that their framework can reduce MAE for multiple OPE algorithms on a few domains.\n\nThe core contribution of this paper is composed of a few known techniques, but the final result is an enhanced OPE with a lower MAE. The real-world experiments are the highlight of this paper.", "questions": "1. FQE is an OPE algorithm that tries to do reward decomposition. However, the experiment still showed that using the decomposed reward produced from OPEHF, FQE-OPEHF can have a lower MAE than just using FQE alone. Can the authors comment on why this would be the case? \n2. The objective is on the sum of discounted rewards being the same, but there is no constraint on reward from each timestep — this means “theoretically” we can have a neural network predict -10000 on one timestep and 9999 on the next step, and still have the sum of reward to be -1. Can the authors comment on why this degenerate case hasn’t happened, and can authors provide some analysis/insight/summary data on what the decomposed intermediate human feedback rewards look like? ", "limitations": "The authors addressed the limitations of their approach.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper has offered two very interesting real-world experiments to demonstrate the effectiveness of the proposed framework and show how it can enhance existing OPE algorithms. Indeed, a lot of existing OPE algorithms often work well with dense rewards, and these algorithms do not have an additional component that can infer intermediate rewards.\n\nI find the real-world experiments very refreshing and quite convincing.", "weaknesses": "The methodology proposed in this paper is not very novel — the authors assembled a few known techniques in the AI/ML community. Also, I’m not entirely sure some of the content in the paper is necessary. The authors can correct me if I’m wrong or misunderstood the contribution, but some parts of the paper do not seem like original contributions but simply restatements of known results.\n\nLine 101: Problem 1. The authors indicated one of the issues is “the total number of offline trajectories AND $r_t^H$ being unknown.” However, in the later sections, I think only $r_t^H$ was solved. It doesn’t seem like the proposed framework would change much if “the total number of offline trajectories” is known or unknown. Can authors provide more insight on how this affects their framework design? Specifically, how would they change the algorithm if the number of trajectories is known? I sincerely suggest dropping this if this is not a serious factor that affects your algorithm design.\n\nLine 121 Proposition 1: The rescaled OPE estimator has a lower variance than the original OPE estimator. I think this is a known result. For example, weighted importance sampling has a lower variance than importance sampling. Similarly, weighted PDIS has a lower variance than PDIS. See Thomas [1] Chapter 3 for consistent weighted PDIS. Can the authors explain and clarify how they made a novel contribution beyond what’s provided in [1]? Otherwise, I suggest the authors cite the previous work so readers will not get confused.\n\n[1] Thomas, P. S. Safe Reinforcement Learning. PhD thesis, University of Massachusetts Amherst, 2015b.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate the reviewer's thoughtful questions and constructive feedback. Please find our responses below. We would be more than happy to further respond to any follow-ups the reviewer may have.\n\nQ1: The methodology proposed in this paper is not very novel — the authors assembled a few known techniques in the AI/ML community. \n\nR1: To the best of our knowledge, our work is the first to introduce a practical framework that allows OPE to accurately estimate human returns that are only provided at the end of each episode, which has been a long-standing challenge in OPE as the reviewer pointed out in the Strengths section of the review. Our methodology is designed to be straightforward, allowing the maximum compatibility with existing down-stream OPE methods, without reinventing the wheels.\n\nThe main **technical novelty** lies in formulating the surrogate objective for capturing the immediate human rewards (IHRs) in eq.(1), and solving it by regularizing the reconstructed rewards over state-action pairs that achieved similar embeddings in the latent space (eq.(3) and (4)). The VLM-H and bi-directional LSTM models were used to capture environmental dynamics and reconstructing IHRs respectively, however, the bi-directional LSTM leverages the latent information encoded by VLM-H to map end-of-episode human returns to IHRs. Consequently, each component is introduced toward a specific sub-goal, and they are all closely tied with each other in order to achieve the overall objective.\n\nMoreover, we have shared an **important empirical finding** that could be useful for future works to leverage, when shaping the problem setup along the line of OPEHF. Since only weak-to-moderate correlations are found between human returns and environmental rewards (as a result of the two real-world experiments), two separate reward functions would be needed when formulating the MDP setup.\n\nQ2: Line 101: Problem 1. The authors indicated one of the issues is “the total number of offline trajectories AND $r_t^\\mathcal{H}$ being unknown.” \n\nR2:  We thank the reviewer for pointing this out. We meant to only consider the IHRs, $r_t^\\mathcal{H}$, to be unknown. This sentence is rather long and the writeup there may cause this confusion here -- we will correct it if the paper gets accepted. \n\nQ3: Line 121 Proposition 1: The rescaled OPE estimator has a lower variance than the original OPE estimator. I think this is a known result. \n\nR3: We thank the reviewer for sharing this relevant literature and we will cite it in the camera-ready, if our paper is accepted. Proposition 1 is used to illustrate the motivation why in our methodology it needs to map the end-of-episode human returns to each step. The discussion there is specific to our problem setup (section 2.1) under the human reward function considered within the MDP, where $r_t^\\mathcal{H}$ is considered a random variable sampled from a distribution. This is due to the fact that human rewards can be affected by multiple underlying factors [1,2], which could be non-deterministic even if the $(s_t,a_t)$ pair is already observed. The analysis then follows from here, whereas in the literature provided by the reviewer, it mainly assumes that rewards are scalar once $(s_t,a_t)$ is observed.\n\nQ4: The experiment still showed that using the decomposed reward produced from OPEHF, FQE-OPEHF can have a lower MAE than just using FQE alone. \n\nR4: Thank you for this thoughtful question that leads to additional findings. Existing works have found that performance of temporal difference (TD) methods (e.g., FQE) may suffer from vanishing information [3], as well as exponential error amplification in the horizon [4]. In contrast, the immediate human rewards reconstructed by the RILR technique (section 2.3) can help resolve this problem, as it leverages latent representations learned by VLM-H to help populate reward information through the regularization (eq.(3)). Such a finding also aligns with [5], showing that by introducing a representation learning component, it can improve FQE's performance.\n\n\nQ5: The objective is on the sum of discounted rewards being the same, but there is no constraint on reward from each timestep.\n\nR5: Thank you for this thoughtful question. Table 1 of the PDF uploaded for rebuttal we report the 95% confidence interval (CI) of the reconstructed IHRs, which lie within a relatively compact range. We believe the regularization term in eq.(3) is helpful for this -- it constrained the state-action pairs that have similar latent encodings to be assigned with similar (reconstructed) IHRs, which implicitly regularize the overall scale of the reconstruction outputs and help ensure that the extreme cases would not be likely to appear. In table 1 we also listed the min and max reconstructed IHR over all trajectories per patient, and there do exist some outliers that stay further away from the 95% CI. The reason could be that either there exist (rare) state-action pairs that are critical for resulting in relatively higher/lower returns, or they are the outlier while sampling $\\hat r_t^\\mathcal{H}$ 's from the learned distribution $f_\\theta$ (which we use Gaussian to represent). Figure 2 in the PDF further visualizes the distribution of reconstructed IHRs over the consolidated state and action space. \n\n\n[1] Namkoong et al. \"Off-policy policy evaluation for sequential decisions under unobserved confounding.\" NeurIPS'20.\n\n[2] Chesnaye et al. \"An introduction to inverse probability of treatment weighting in observational research.\" Clinical Kidney Journal 2022.\n\n[3] Arjona-Medina et al. \"Rudder: Return decomposition for delayed rewards.\" NeurIPS'19.\n\n[4] Wang et al. \"What are the Statistical Limits of Offline RL with Linear Function Approximation?.\" ICLR'20.\n\n[5] Chang et al. \"Learning bellman complete representations for offline policy evaluation.\" ICML'22.\n"}, {"review_id": "QPuqgO6i6x", "reviewer": "Reviewer_1oyy", "summary": "The authors examine an off-policy evaluation scenario where we have an environment with an intrinsic environment reward $r$, and an extrinsic human reward $r^H$. A further assumption is made that the human reward is only observed at the end of the episode, and comes as an rating on the total return of the episode. This is done to line up with cases where human ratings may only be clear after the fact.\n\nThe goal of the paper is to do off-policy evaluation in this modified class of MDPs. Two baseline approaches for doing this are considered. The first is to treat the human return as a reward that is achieved at the last timestep of the episode, with $r = 0$ at all earlier steps. The second is to set the human reward $r^H = r$, where $r$ is the original known environment reward, then add an offset $G^H - G$ term to the last timestep, where $G^H$ = total human return and $G$ = total environment return.\n\nThis paper proposes a method to learn timestep level $r^H$ directly. First it argues that given accurate timestep level $r^H$, the variance of the policy return estimate is equal or smaller than the estimate in the sparse MDP where all human reward is allocated to the final timestep. The reward learning method is then done based on two objective terms. The total reward must equal the human label of return for each trajectory, and the reward for individual $(s,a)$ should be similar to reward for other $(s',a')$ that have similar embeddings.\n\nThis then leaves determining the embeddings. This is done via a VAE to learn a latent space, and the the nearest K-neighbors of a given $(s,a)$ embed are treated as similar examples.\n\nOnce a reward estimate is learned, any off the shelf OPE method can be used.", "questions": "Are there examples of ways the original environment reward can be used, aside from the fusion baseline mentioned? I am trying to understand if it makes sense to even view this as a new class of MDPs, vs treating it like an MDP with one reward function that is only observed at the last timestep. It seems like the method proposed here does not use original env reward in any way besides training the encoder-decoder in the VAE?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper makes an important observation that human feedback tends to be given at the end of an episode or after the fact, and is usually not given at the per-timestep level. It is clear that if per-timestep labels can be learned, then using them to estimate policy performance should give better results than estimating just total episode return (at least assuming learning per-timestep labels is a well-formed problem)", "weaknesses": "I don't like that the paper proposes an \"OPEHF framework\", yet the bulk of the paper is really about learning a reward function / learning to distribute a return estimate over past trajectories. Nothing about the reward function learning is inherently tied to off-policy evaluation, it could theoretically be done online too, it's just something that can be done off-policy as well.\n\nThe idea of return decomposition in the paper seems very similar to the RUDDER: Return Decomposition for Delayed Rewards paper, albeit with a different use case (in that paper, the environment naturally has a large sparse reward, which is decomposed to provide partial reward signal for an RL agent instead of needing to learn to reachieve the large sparse reward seen before.) It seems very relevant, in that the RUDDER paper proposes a number of return decomposition methods, none of which are considered in this paper.\n\nThe embedding of $(s,a)$ is said to come from a bi-directional LSTM, in order to leverage information from both previous and subsequent steps. Doesn't this break the Markov assumption of MDPs? Since the rewards $r(s,a)$ now may depend on future states?\n\nHow does the method handle cases where $r(s,a)$ may not be well defined? For example, two human raters have differing opinions on the score of a given trajectory.\n\nOverall I do not feel the paper is benchmarked sufficiently to be accepted.\n\nEdit: see rebuttal chain, have decided to increase score based on discussion of concerns.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Please find our responses below. We hope they addressed the reviewer's outstanding concerns sufficiently, and we would be more than happy to respond to any follow-ups the reviewer may have.\n\nQ1:The bulk of the paper is really about learning a reward function. It could theoretically be done online too.\n\nR1: We respectfully disagree that our work is more of a reward shaping work, nor can be adapted to the online learning setup, from two perspectives. \n\n* The motivation for decomposing human returns in our case, follows from the analysis in proposition 1 that proper decomposition can help reduce variance and lead to **more accurate OPE estimation over policies that lead to varied performance**. This is rather different than in existing reward shaping works that try to smooth over sparse rewards, and mainly *focus on improving the convergence and/or optimality properties of the optimal policy* that can be learned out of the shaped reward functions. \n\n* The focus of our work is to estimate the feedback/returns that are directly provided by human participants, toward the cases where the performance pre-existing RL policies need to be demonstrated **before they get deployed to human participants**, e.g., as required in healthcare systems. So the problem setup of OPEHF cannot be changed to online. Moreover, the use of bi-directional LSTM requires all trajectories to be fully observed *a priori*, and cannot be used online as well (see more in R3 below).\n\n\nQ2: The idea of return decomposition in the paper seems very similar to the RUDDER paper, albeit with a different use case.\n\nR2: We thank the reviewer for sharing this paper. After reading it, we found that RUDDER and our work have different targets, and it would not be very straightforward to adapt the setup considered in one into another. Specifically, for two reasons, i.e.,\n\n* RUDDER is a policy optimization work that focuses on designing the rewards such that the high-return state-action pairs can be quickly identified and frequently re-visited, by allowing the value function to follow shapes like step functions. RUDDER's framework only guarantees that the optimal policy would not change under the MDP before and after reward shaping. However, it does not discuss/prove the rank of returns over sub-optimal policies will be preserved, where in OPE it is important that the returns of policies that lead to varied performance can be estimated accurately [1, 2].\n\n* It seems that RUDDER requires online updates where new data is constantly being added to the buffer during training, which does not align with the OPE where the agent only has access to a **fixed** set of **offline trajectories** (no new data can be added).\n\nNevertheless, it was a very interesting read and we can cite this work and discuss how the ideas introduced in RUDDER could potentially facilitate future works down the line of OPEHF, if our paper is accepted.\n\nQ3: The embedding of $(s,a)$ is said to come from a bi-directional LSTM. Doesn't this break the Markov assumption of MDPs? \n\nR3: The bi-directional LSTM is used to map the end-of-episode human return back to immediate human rewards (IHRs) over the entire horizon, for all the *offline trajectories* being considered. Specifically, **it is used only after these trajectories are already observed** (i.e., no online deployment nor adding new trajectories). Such a case falls under the typical sequence modeling setup where RNNs/LSTMs are mainly used. As an analogy, the use case here is somewhat reminiscent of how decision transformers [3, 4] are used to model the decision process.\n\nQ4: How does the method handle cases where $r(s,a)$ may not be well defined? For example, two human raters have differing opinions on the score of a given trajectory.\n\nR4: In this work we considered that each trajectory only receives a single human return **provided by the participants themselves** at the last step (see section 2.1). For example, pertaining to the physiological and/or subjective feedback in our DBS experiments. So there would not be conflict cases. Please refer to our response to reviewer ochq's Q4 for why we could not follow RLHF-styled evaluations (by hiring 3-rd party labellers). Similar setups are adopted in other OPE works [5, 6], and are considered more challenging than evaluating labeller's ratings, since human returns/feedback can be intrinsically affected by multiple underlying factors that are not very easy to be captured.\n\nQ5: Overall I do not feel the paper is benchmarked sufficiently to be accepted.\n\nR5: To evaluate our method one needs access to records that both log the trajectory and participant ratings, where we agree that currently it lacks a standard testing framework for this. However, both real-world experiments we presented are backed with substantial data collected from extensive testing (over years), where results demonstrated statistical significance. Please also see our general response above, as well as response to reviewer ochq's Q1 for details on the **additional natural language Q&A experiment we had in Appendix E**.\n\nQ6: Are there examples of ways the original environment reward can be used, aside from the fusion baseline mentioned? It seems like the method proposed here does not use original env reward in any way besides training the encoder-decoder in the VAE?\n\nR6: The experiments (Tables 1 and 3) have shown that the immediate environmental rewards may not correlate very well with the human returns (i.e., mostly weak-to-moderate correlations), due to the nature of human feedback [5, 6]. So in our methodology the environmental rewards have only been used as part of supplementary information to train the VLM-H to better capture the underlying dynamics. We would leave for future works to explore how to further leverage the environmental rewards. We appreciate this thoughtful question that expands the future avenues following our work."}, {"review_id": "D3mvgGk8Yk", "reviewer": "Reviewer_cuXN", "summary": "This paper proposes an off-policy evaluation method that utilizes sparse human feedback. Experiments are conducted on two real-world environments and one simulation environment.", "questions": "1.\tAre 4 patients enough for validating the method? I’m wondering how many participants other similar methods adopt. \n2.\tThe proposed method additionally requires offline human feedback data. Are these comparing baselines also using the extra human data? If not, what did you do to make fair comparisons?\n", "limitations": "Please see the questions.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The idea of incorporating human feedback into policy evaluation is interesting and significant. The proposed framework is compatible with existing methods. The experiments are comprehensive. Especially, real-world experiments are appreciated.", "weaknesses": "My major concerns are described in the Questions section. It would be appreciated if the authors address them in detail and I’m happy to raise my score.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate the reviewer's thoughtful questions and constructive feedback. Please find our responses below. We hope they addressed the reviewer's outstanding concerns sufficiently, and we would be more than happy to further respond to any follow-ups the reviewer may have.\n\nQ1: Are 4 patients enough for validating the method? I’m wondering how many participants other similar methods adopt.\n\nR1: Good question. The scale of our cohort is limited in the sense that a special programmable DBS pulse generator is needed to be implanted to the patient's chest (see Figure 6 in Appendix), in order to deploy RL policies that can adapt stimulation amplitudes in real-time. As a result, it was challenging to find patients who are willing to participate in the study, starting with implanting a non-ordinary device followed by experimenting with relatively new control strategies in DBS. In contrast, most existing devices only simply support turning on/off stimulations, and some are not even adjustable in any cases [1, 2]. Under the topic of amplitude-adaptive DBS, we had found one existing work that tested a proportional-integral controller over 5 patients [3], and another one that tested over 13 patients but only 8 hours per patient [4]. On the other hand, though a limited number of patients were available, each one had been followed up for at least 2 years, leading to nearly hundred-hour of logged data for OPEHF to be used for training and validation. Consequently, the performance comparison results are considered statistically significant over each patient. Please refer to our response to reviewer ochq's Q4, on how we plan to dedicate our future efforts on creating more robust and significant evaluation procedures along the line of OPEHF.\n\nQ2: The proposed method additionally requires offline human feedback data. Are these comparing baselines also using the extra human data? If not, what did you do to make fair comparisons?\n\nR2: Yes. The baseline called `rescale` directly assigned the offline human feedback received at the end of each episode as the reward for the last step, and divided it by $\\gamma^T$ ($T$ is the trajectory horizon) to ensure that the discounted sum of rewards equals the feedback received. The other baseline called `fusion` additionally includes the immediate environmental rewards at each step, and then add in the human feedback to the reward at the last step, still with a correction applied such that the cumulative sum of discounter rewards in each trajectory is equal to the human feedback provided. We discussed this in between lines 226 and 236, and we would be more than happy to clarify any parts that are unclear there in the camera ready, if our paper is accepted. We thank the reviewer for paying attention to details.\n\n[1] Beudel, M., and P. Brown. \"Adaptive deep brain stimulation in Parkinson's disease.\" Parkinsonism & related disorders 22 (2016): S123-S126.\n\n[2] Little, Simon, et al. \"Adaptive deep brain stimulation for Parkinson's disease demonstrates reduced speech side effects compared to conventional stimulation in the acute setting.\" Journal of Neurology, Neurosurgery & Psychiatry 87.12 (2016): 1388-1389.\n\n[3] Malekmohammadi M, Herron J, Velisar A, et al. Kinematic adaptive deep brain stimulation for resting tremor in Parkinson's disease. Mov Disord 2016;31:426-428\n\n[4] Arlotti M, Marceglia S, Foffani G, et al. Eight-hours adaptive deep brain stimulation in patients with Parkinson disease. Neurology 2018;90:e971-e976\n"}]

Author rebuttal:
# General Response

We sincerely thank the reviewers for the time and efforts dedicated to the review process. 

We feel that currently the main discrepancy centered around the evaluation of OPEHF. While reviewers KHGy and cuXN pointed that our real-world experiments are convincing and comprehensive, reviewer 1oyy seemed to prefer more evaluation studies. Reviewer ochq stayed neutral while conceded that obtaining data with participant's feedback would be intrinsically challenging.

We agree that currently there may lack a standard testing bench for OPEHF across the community, as (to the best of our knowledge) we are the first that introduces the framework that revives existing OPE methods toward estimating the human feedback/returns that are only provided at the end of each episode (that are not correlate well with environmental rewards); our method addresses a long-standing challenge in the community of OPE (as pointed by KHGy, ochq and cuXN). However, please note that our two real-world experiments are backed with substantial data collected from years of follow-ups, with results showing our method significantly outperforms over baselines. **We did also supplement these experiments with a simulation study on natural language Q&A machine** in Appendix E, which employs similar schema that policies are trained on discrete environmental rewards, but evaluated against dense returns issued by a separate referee model. Our method significantly outperform the baselines in this setting as well.

We had also discussed, in section 4, that OPEHF is mainly evaluated over *direct preference* provided by participants themselves, which is different than the *ranked preference* used in RLHF, hence made it intractable to evaluate over RLHF's setup -- the participants cannot revisit the same procedure multiple times (patients may not undergo the same surgeries several times and rank the experiences). However, on the other hand, we discussed in the response to reviewer ochq's Q4 in regards to facilitating future venues for evaluating OPEHF-related works, where we are currently building a platform that allows verified 3rd party professionals (doctors and professors) to provide direct feedback to logged trajectories, to further consolidate the standardized evaluation process for future works along this line.

Again, we greatly appreciate the reviewers' and chairs' efforts toward the review and rebuttal processes, allowing the work to be thoroughly discussed and evaluated. 

------------------------------------------------------------------------------------------------------------------------------------

At last, please find in attached the PDF that contains figures and tables, corresponding to the additional empirical analyses used to address some of the individual comments below. 

Meta-review: This paper studies the problem of offline policy evaluation with human feedback. Under this setting, human feedback as a reward is given only at the end of an episode. It results in a sparse reward environment whose data is difficult to collect in practice such as in healthcare. This paper proposes to first learn a surrogate per-step immediate human reward (IHR) that decomposes the final human feedback, and then apply existing OPE methods on the learned reward.

This paper makes a good progress on this challenging and realistic problem. Multiple reviewers consider its two real-world experiments a strength. Some reviewer is worried that the dataset is too small for thorough evaluation but the author explains it's due to the lack of data in real problems that takes years to collect.

The main concern comes from the fact that the main method can be considered as a type of reward shaping. It is not compared to prior works along this perspective, such as RUDDER. While authors argue RUDDER was devised for policy optimization rather than OPE and is not applicable, the reviewer consider the reward decomposition idea of RUDDER still relevant and its variants should be detailed discussed in this work.

The reviewers also made multiple insightful comments including the design choice of the proposed bi-directional LSTM, the MDP formulation and Markovian assumption, the limitation on handling conflicting human feedbacks. While these comments does not undermine the contribution of this paper to tackling the OPE-HF problem, the authors should take them seriously. Incorporate those comments in the revision and providing a clear discussion on the limitations would definitely improve the work.
Decision: Accept (poster)