Title: Off-Policy Evaluation for Human Feedback
Venue: NeurIPS 2023
Accepted: yes
Keywords: Off-policy evaluation (OPE)

Variational latent model for trajectory representation learning

Reinforcement learning and OPE for adaptive neurostimulation
Abstract: Off-policy evaluation (OPE) is important for closing the gap between offline training and evaluation of reinforcement learning (RL), by estimating performance and/or rank of target (evaluation) policies using offline trajectories only. It can improve the safety and efficiency of data collection and policy testing procedures in situations where online deployments are expensive, such as healthcare. However, existing OPE methods fall short in estimating human feedback (HF) signals, as HF may be conditioned over multiple underlying factors and are only sparsely available; as opposed to the agent-defined environmental rewards (used in policy optimization), which are usually determined over parametric functions or distributions. Consequently, the nature of HF signals makes extrapolating accurate OPE estimations to be challenging. To resolve this, we introduce an OPE for HF (OPEHF) framework that revives existing OPE methods in order to accurately evaluate the HF signals. Specifically, we develop an immediate human reward (IHR) reconstruction approach, regularized by environmental knowledge distilled in a latent space that captures the underlying dynamics of state transitions as well as issuing HF signals. Our approach has been tested over *two real-world experiments*, adaptive *in-vivo* neurostimulation and intelligent tutoring, and a simulation environment (visual Q&A). Results show that our approach significantly improves the performance toward estimating HF signals accurately, compared to directly applying (variants of) existing OPE methods.

Off-Policy Evaluation for Human Feedback

Abstract

Off-policy evaluation (OPE) is important for closing the gap between offline training and evaluation of reinforcement learning (RL), by estimating performance and/or rank of target (evaluation) policies using offline trajectories only. It can improve the safety and efficiency of data collection and policy testing procedures in situations where online deployments are expensive, such as healthcare. However, existing OPE methods fall short in estimating human feedback (HF) signals, as HF may be conditioned over multiple underlying factors and is only sparsely available; as opposed to the agent-defined environmental rewards (used in policy optimization), which are usually determined over parametric functions or distributions. Consequently, the nature of HF signals makes extrapolating accurate OPE estimations to be challenging. To resolve this, we introduce an OPE for HF (OPEHF) framework that revives existing OPE methods in order to accurately evaluate the HF signals. Specifically, we develop an immediate human reward (IHR) reconstruction approach, regularized by environmental knowledge distilled in a latent space that captures the underlying dynamics of state transitions as well as issuing HF signals. Our approach has been tested over two real-world experiments , adaptive in-vivo neurostimulation and intelligent tutoring, as well as in a simulation environment (visual Q&A). Results show that our approach significantly improves the performance toward estimating HF signals accurately, compared to directly applying (variants of) existing OPE methods.

1 Introduction

Off-policy evaluation (OPE) aims to estimate the performance of reinforcement learning (RL) policies using only a fixed set of offline trajectories [61], i.e. , without online deployments. It is considered to be a critical step in closing the gap between offline RL training and evaluation, for environments and systems where online data collection is expensive or unsafe. Specifically, OPE facilitates not only offline evaluation of the safety and efficacy of the policies ahead of online deployment, but policy selection as well; this allows one to maximize the efficiency when online data collection is possible, by identifying and deploying the policies that are more likely to result in higher returns. OPE has been used in various application domains including healthcare [68, 53, 23, 22], robotics [15, 18, 24], intelligent tutoring [64, 45, 17], recommendation systems [50, 43].

The majority of existing OPE methods focus on evaluating the policies' performance defined over the environmental reward functions which are mainly designed for use in policy optimization (training). However, as an increasing number of offline RL frameworks are developed for human-involved systems [64, 45, 1, 48, 16], existing OPE methods lack the ability to estimate how human users would evaluate the policies, e.g. , ratings provided by patients (on a scale of 1-10) over the procedure facilitated by automated surgical robots; as human feedback (HF) can be noisy and conditioned over various confounders that could be difficult to be captured explicitly [53, 7, 44]. For example, patient satisfaction over a specific diabetes therapy may vary across the cohort, depending on many subjective

factors, such as personal preferences and activity level of the day, while participating in the therapy, in addition to the physiological signals ( e.g. , blood sugar level, body weight) that are more commonly used as the sources for determining environmental rewards toward policy optimization [70, 33, 21, 19]. Moreover, the environmental rewards are sometimes discrete to ensure optimality of the learned policies [67], which further reduces its correlation against HF signals.

In this work, we introduce the OPE for human feedback (OPEHF) framework that revives existing OPE approaches in the context of evaluating HF from offline data. Specifically, we consider the challenging scenario where the HF signal is only provided at the end of each episode i.e. , no per-step HF signals, referred to as immediate human rewards (IHRs) below, are provided - benchmarking the common real-world situations where the participants are allowed to rate the procedures only at the end of the study. The goal is set to estimate the end-of-episode HF signals, also referred to as human returns , over the target (evaluation) policies, using a fixed set of offline trajectories collected over some behavioral policies. To facilitate OPEHF, we introduce an approach that first maps the human return back to the sequence of IHRs, over the horizon, for each trajectory. Specifically, this follows from optimizing over an objective that consists of a necessary condition where the cumulative discounted sum of IHRs should equal the human return, as well as a regularization term that limits the discrepancy of the reconstructed IHRs over state-action pairs that are determined similar over a latent representation space into which environmental transitions and rewards are encoded. At last, this allows for the use of any existing OPE methods to process the offline trajectories with reconstructed IHRs and estimate human returns under target policies.

Our main contributions are tri-fold. ( i ) We introduce a novel OPEHF framework that revives existing OPE methods toward accurately estimating highly sparse HF signals (provided only at the end of each episode) from offline trajectories, through IHRs reconstruction. ( ii ) Our approach does not require the environmental rewards and the HF signals to be strongly correlated, benefiting from the design where both signals are encoded to a latent space regularizing the objective for reconstructions of IHRs, which is justified empirically over real-world experiments. ( iii ) Two real-world experiments , i.e ., adaptive in-vivo neurostimulation for the treatment of Parkinson's disease and intelligent tutoring for computer science students in colleges, as well as one simulation environment ( i.e. , visual Q&A), facilitated the thorough evaluation of our approach; various degrees of correlations between the environment rewards and HF signals existed across the environments, as well as the varied coverage of the state-action space provided by offline data over sub-optimal behavioral policies, imposing different levels of challenges for OPEHF.

2 Off-Policy Evaluation for Human Feedback (OPEHF)

In this section, we introduce an OPEHF framework that allows for the use of existing OPE methods to estimate the human returns that are available only at the end of each episode, with IHRs remaining unknown. This is in contrast to the goal of classic OPE that only estimates the environmental returns following the user-defined reward function used in the policy optimization phase. A brief overview of existing OPE methods can be found in Appendix C.

2.1 Problem Formulation

Wefirst formulate the human-involved MDP (HMDP), which is a tuple M = ( S , A , P , R, R H , s 0 , γ ) , where S is the set of states, A the set of actions, P : S × A → S is the transition distribution usually captured by probabilities p ( s t | s t -1 , a t -1 ) , R : S × A → R is the environmental reward function, R H ( r H | s, a ) is the human reward distribution from which the IHR r H t ∼ R H ( ·| s t , a t ) are sampled, s 0 is the initial state sampled from the initial state distribution p ( s 0 ) , and γ ∈ [0 , 1) is the discounting factor. Note that we set the IHRs to be determined probabilistically, as opposed to the environmental rewards r t = R ( s t , a t ) that are deterministic; this is due to the fact that many underlying factors may affect the feedback provided by humans [53, 7, 44], as we have also observed while performing human-involved experiments (see Appendix D). Finally, the agent interacts with the MDP following some policy π ( a | s ) that defines the probabilities of taking action a at state s .

In this work, we make the following assumption over R and R H .

Assumption 1 (Unknown IHRs) . We assume that the immediate environmental reward function R is known and R ( s, a ) can be obtained for any state-action pairs in S×A . Moreover, the IHR distribution R H is assumed to be unknown, i.e., r H ∼ R H ( ·| s, a ) are unobservable, for all ( s, a ) ∈ S × A .

Instead, the cumulative human return G H 0: T , defined over R H , is given at the end of each trajectory, i.e., G H 0: T = ∑ T t =0 γ t r H t , with T being the horizon and r H t ∼ R H ( ·| s t , a t ) .

The assumption above follows the fact that human feedback (HF) is not available until the end of each episode, as opposed to immediate rewards that can be defined over the environment and evaluated for any ( s t , a t ) pairs at any time. This is especially true in environments such as healthcare where the clinical treatment outcome is not foreseeable until a therapeutic cycle is completed, or in intelligent tutoring where the overall gain from students over a semester is mostly reflected by the final grades. Note that although the setup can be generalized to the scenario where HF can be sparsely obtained over the horizon, we believe that issuing the HF only at the end of each trajectory leads to a more challenging setup for OPE. Consequently, the goal of OPEHF can be formulated as follows.

Problem 1 (Objective of OPEHF) . Given offline trajectories collected by some behavioral policy β , ρ β = { τ (0) , τ (1) , . . . , τ ( N -1) | a t ∼ β ( a t | s t ) } , with τ ( i ) = [( s ( i ) 0 , a ( i ) 0 , r ( i ) 0 , r H ( i ) 0 , s ( i ) 1 ) , . . . , ( s ( i ) T -1 , a ( i ) T -1 , r ( i ) T -1 , r H ( i ) T -1 , s ( i ) T ) , G H ( i ) 0: T ] being a single trajectory, N the total number of offline trajectories, and r H t 's being unknown , the objective is to estimate the expected total human return over the unknown state-action visitation distribution ρ π of the target (evaluation) policy π , i.e., E ( s,a ) ∼ ρ π ,r H ∼ R H [ ∑ T t =0 γ t r H t ] .

2.2 Reconstrunction of IHRs for OPEHF

We emphasize that the human returns are only issued at the end of each episode, with IHRs remaining unknown. One can set all IHRs from t = 0 to t = T -2 to be zeros ( i.e. , r H 0: T -2 = 0 ), and rescale the cumulative human return to be the IHR at the last step ( i.e. , r H T -1 = G H 0: T /γ T -1 ), to allow the use of existing OPE methods toward OPEHF (Problem 1). However, the sparsity over r H 's here may impose difficulties for OPE to estimate the human returns accurately over the target policies. For OPEHF, we start by showing that for the per-decision importance sampling (PDIS) method - a variance-reduction variant of the importance sample (IS) family of OPE methods [61] - if IHRs were to be available , they could reduce the variance in the estimation compared to the rescale approach above.

Recall that the PDIS estimator follows ˆ G π PDIS = 1 N ∑ N -1 i =1 ∑ T -1 t =0 γ t ω ( i ) 0: t r H ( ⟩ ) t , where ω ( i ) 0: t = ∏ t k =0 π ( a ( i ) k | s ( i ) k ) β ( a ( i ) k | s ( i ) k ) are the PDIS weights for offline trajectory τ ( i ) . Moreover, the estimator of the rescale approach 3 above is ˆ G π Rescale = 1 N ∑ N -1 i =1 ω ( i ) 0: T -1 G H ( i ) 0: T , which is equivalent to the vanilla IS estimator [61, 72]. We now show the variance reduction property of ˆ G π PDIS in the context of OPEHF.

Proposition 1. Assume that (i) E [ r H t ] ≥ 0 , and (ii) given the horizon T , consider any 1 ≤ t +1 ≤ k ≤ T of any offline trajectory τ , ω 0: k and r H t ω 0: k are positively correlated. Then, V ( ˆ G π PDIS ) ≤ V ( ˆ G π Rescale ) , with V ( · ) representing the variance.

The proof can be found in Appendix A. Assumption ( i ) can be easily satisfied in the real world, as HF signals are usually quantified as positive values, e.g. , ratings (1-10) provided by participants. Assumption ( ii ) is most likely to be satisfied when the target policies do not visit low-return regions substantially [46], which is a pre-requisite for testing RL policies in human-involved environments as initial screening are usually required to filter the ones that could potentially pose risks to participants [57].

Besides IS, doubly robust (DR) [71, 34, 69, 12] and fitted Q-evaluation (FQE) [40] methods require learning value functions. Sparsity of rewards (following the rescale approach above) in the offline dataset may lead to poorly learned value functions [74], considering that the offline data in OPE is usually fixed ( i.e. , no new samples can be added), and are often generated by behavioral policies that are sub-optimal, which results in limited coverage of the state-action space. Limited availabilities of environment-policy interactions ( e.g. , clinical trials) further reduce the scale of the exploration and therefore limit the information that can be leveraged toward obtaining accurate value function approximations.

Reconstruction of IHRs. To address this challenge, our approach aims to project the end-of-episode human returns back to each environmental step, i.e. , to learn a mapping f θ ( τ, G H 0: T ) : ( S × A ) T ×

!

!

"

!

#

!

#

"

!

"

$

"

#

#

. . .

. . .

!

#

"

"

#

$

!

!"#

"

!"$

!

$

"

!"#

Trajectory Inference (Encoding)

Trajectory Generation (Decoding)

"

%&'

$

!

$

$

!

!"#

Offline Trajectories

w/ Reconstructed IHRs

Ep. 1

Ep. N   . . . . . .

(&

!

, (

!

)

(&

$

, (

$

)

…

(&

"#$

, (

"#$

)

* 0

! ℋ

(

reconstruct

ed

)

* 0

$ ℋ

(

reconstruct

ed

)

…

* 1

"#$

ℋ

(

reconstruct

ed

)

Target

!

Downstream OPE

Estimator

…,(+, ,, #

%

ℋ

) , …

! (

!:$

ℋ

(under

.)

Step

.

.

.

Episode 1

Episode N

. . . . . .

(&

!

, (

!

)

…

(&

"#$

, (

"#$

)

-

*

!

…

*

"#$

-

*

! ℋ

(

unknown

)

…

*

"#$

ℋ

(

unknown

)

+

!:"

ℋ

(

under

,

)

Offline Trajectories

!

!

"

!

#

!

#

"

!

"

$

"

#

#

. . .

. . .

!

#

"

"

#

$

!

!"#

"

!"$

!

$

"

!"#

Trajectory Inference (Encoding)

Trajectory Generation (Decoding)

"

%&'

$

!

$

$

!

!"#

VLM-H

Train

IHR

Reconstruction

!

!:$

ℋ

(

under

"

)

[% #

! ℋ

, #

%

" ℋ

, … ,(

%

$/"

ℋ

]

(

reconstructed

)

Latent Representations

Learnt by VLM-H

Step

Step

Regularize

.

.

.

23.33

24.49

25.06

25.83

26.12

26.59

27.24

3.1

12.47

19.81

2

3

R → R T , parameterized by θ , that maximizes the sum of log-likelihood of the estimated IHRs, [ˆ r H 0 , . . . , ˆ r H T -1 ] ⊺ ∼ f θ ( τ, G H 0: T ) , following max θ 1 N ∑ N -1 i =0 ∑ T -1 t =0 log p (ˆ r H t = r H ( i ) t | θ, τ ( i ) , G H ( i ) 0: T ) , where G H ( i ) 0: T and r H ( i ) t 's are respectivelly the human return and IHRs (unknown) of the i -th trajectory in the offline dataset ρ β , and N is the total number of trajectories in ρ β . Given that the objective above is intractable due to unknown r H ( i ) t 's, we introduce a surrogate objective

Here, the first term is a necessary condition for ˆ r H t 's to be valid for estimating r H t 's, as they should sum to G H 0: T . Since many solutions may exist if one only optimizes over the first term, the second term L regu serves as a regularization that imposes constraints on r H t 's to follow the properties specific to their corresponding state-action pairs; e.g. , ( s, a ) pairs that are similar to each other in a representation space, defined over the state-action visitation space, tend to yield similar immediate rewards [18].

The detailed regularization technique is introduced in sub-section below. Practically, we choose f θ to be a bi-directional long-short term memory (LSTM) [32], since the reconstruction of IHRs can leverage information from both previous and subsequent steps as provided in the offline trajectories.

2.3 Reconstruction of IHRs over Latent Representations (RILR) for OPEHF

Now, we introduce the regularization technique for the reconstruction of IHRs, i.e. , reconstructing IHRs over latent representations (RILR). Specifically, we leverage the representations captured by variational auto-encoders (V AEs) [35], learned over ρ β , to regularize the reconstructed IHRs, ˆ r H t .

VAEs have been adapted toward learning a compact latent space over offline state-action visitations, facilitating both offline policy optimization [42, 81, 65, 27, 26, 28] and OPE [18]. In this work, we specifically consider building on the variational latent model (VLM) proposed in [18] since it is originally proposed to facilitate OPE, as opposed to others that mainly use knowledge captured in the latent space to improve sample efficiency for policy optimization. Moreover, the VLM has shown to be effective for learning an expressive representation space, where the encoded state-action pairs are clustered well in the latent space, as measured by the difference over the returns of the policies from which the state-action pairs are sampled; see Figure 1 (mid) which uses t -SNE to visualize the encoded state-action pairs in trajectories collected from a visual Q&A environment (Appendix E).

Note that VLM originally does not account for HF signals (neither r H t 's nor G H 0: T 's), so we introduce the variational latent model with human returns (VLM-H) below, building on the architecture introduced in [18]. VLM-H consists of a prior p ( z ) over the latent variables z ∈ Z ⊂ R L , with Z representing the latent space and L the dimension, along with a variational encoder q ψ ( z t | z t -1 , a t -1 , s t ) , a decoder p ϕ ( z t , s t , r t -1 | z t -1 , a t -1 ) for generating per-step transitions (over both state-action and latent space), and a separate decoder p ϕ ( G H 0: T | z T ) for the reconstruction of the human returns at the end of each episode. Note that encoders and decoders are parameterized by ψ and ϕ respectively. The overall architecture is illustrated in Figure 1 (left).

Trajectory inference (encoding). VLM-H's encoder approximates the intractable posterior p ( z t | z t -1 , a t -1 , s t ) = p ( z t -1 ,a t -1 ,z t ,s t ) ∫ z t ∈Z p ( z t -1 ,a t -1 ,z t ,s t ) dz t , by avoiding to integrate over the unknown latent space a priori . The inference (or encoding) process can be decomposed as, i.e. , q ψ ( z 0: T | s 0: T , a 0: T -1 ) = q ψ ( z 0 | s 0 ) ∏ T t =1 q ψ ( z t | z t -1 , a t -1 , s t ); here, q ψ ( z 0 | s 0 ) encodes initial states s 0 into latent variables z 0 , and q ψ ( z t | z t -1 , a t -1 , s t ) captures all subsequent environmental transitions in the latent space over z t 's. In general, both q ψ 's are represented as diagonal Gaussian distributions 4 with mean and variance determined by neural network ψ , as in [18, 42, 27, 26, 28].

Trajectory generation (decoding). The generative (or decoding) process follows, i.e. , p ϕ ( z 1: T , s 0: T , r 0: T -1 , G H 0: T | z 0 , π ) = p ϕ ( G H 0: T | z T ) · ∏ T t =1 p ϕ ( z t | z t -1 , a t -1 ) p ϕ ( r t -1 | z t ) · ∏ T t =0 p ϕ ( s t | z t ); here, p ϕ ( z t | z t -1 , a t -1 ) enforces the transition of latent variables z t over time, p ϕ ( s t | z t ) and p ϕ ( r t -1 | z t ) are used to sample the states and immediate environmental rewards, while p ϕ ( G H 0: T | z T ) generates the human return issued at the end of each episode. Note that here we still use the VLM-H to capture environmental rewards, allowing the VLM-H to formulate a latent space that captures as much information about the dynamics underlying the environment as possible. All p ϕ 's are represented as diagonal Gaussians 5 with parameters determined by network ϕ .

To train ϕ and ψ , one can maximize the evidence lower bound (ELBO) of the joint log-likelihood log p ϕ ( s 0: T , r 0: T -1 , G H 0: T | ϕ, ψ, ρ β ) , i.e. ,

the first three terms are the log-likelihoods of reconstructing the human return, states, and environmental rewards, and the two terms that follow are Kullback-Leibler (KL) divergence [38] regularizing the inferred posterior q ψ . Derivation of the ELBO can be found in Appendix B. In practice, if ϕ and ψ are chosen to be recurrent networks, one can also regularize the hidden states of ϕ, ψ by including the additional regularization term introduced in [18].

Regularizing the reconstruction of IHRs. Existing works have shown that the latent space not only facilitates the generation of synthetic trajectories but demonstrated that the latent encodings of state-action pairs form clusters, over some measures in the latent space [73], if they are rolled out from policies that lead to similar returns [42, 18]. As a result, we regularize ˆ r H t following

for each step t ; here, ( s ( i ) 0: t , a ( i ) 0: t -1 ) ∈ τ ( i ) ∼ ρ β , J = { j 0 , . . . , j K -1 } are the indices of offline trajectories that correspond to the latent encodings { z ( j k ) t ′ ∼ q ψ ( ·| s ( j k ) 0: t ′ , a ( j k ) 0: t ′ -1 ) | j k ∈ J , t ′ ∈ [0 , T -1] } that are K -neighbours of the latent encoding z ( i ) t pertaining to ( s ( i ) 0: t , a ( i ) 0: t -1 ) , defined over some similarity/distance function d ( ·||· ) , following, i.e. ,

In practice, we choose d ( ·||· ) to follow stochastic neighbor embedding (SNE) similarities [73], as it has been shown effective for capturing Euclidean distances in high-dimensional space [75].

Overall objective of RILR for OPEHF. As a result, by following (1) and leveraging the L regu from (3) above, the objective for reconstructing the IHRs is set to be, i.e. ,

Move from RILR to OPEHF. In what follows, one can leverage any existing OPE methods to take as inputs the offline trajectories, with the immediate environmental rewards r t 's replaced by the reconstructed IHRs ˆ r H t 's, to achieve the OPEHF's objective (Problem 1). Moreover, our method does not require the IHRs to be correlated with the environmental rewards, as the VLM-H learns to reconstruct both by sampling from two independent distributions, p ϕ ( r t -1 | z t ) and p ϕ ( G H 0: T | z T ) respectively, following (2); this is also illustrated empirically over the experiments introduced below (Sections 3), where exceedingly low correlations are found in specific scenarios.

The overall pipeline summarizing our method is shown in Figure 1 (right).

3 Real-World Experiments with Human Participants

In this section, we validate the OPEHF framework introduced above over two real-world experiments, adaptive neurostimulation, and intelligent tutoring. Specifically, we consider four types of OPE methods to be used as the downstream estimator following the RILR step (Section 2.3), including perdecision importance sampling (IS) with behavioral policy estimation [30], doubly robust (DR) [71], distribution correction estimation (DICE) [78] and fitted Q-evaluation (FQE) [40]. A brief overview of these methods can be found in Appendix C, and the specific implementations we use are documented in Appendix D. In Appendix E, we have also tested our method within a visual Q&A environment [10, 66], which follows similar mechanisms as in the two real-world experiments, i.e. , two types of return signals are considered though no human participants are involved.

Baselines and Ablations. The baselines include two variants for each of the OPE methods above, i.e. , ( i ) the rescale approach discussed in Section 2.2, and ( ii ) another variant that sets all the IHRs to be equal to the environmental rewards at corresponding steps, r H t = r t ∀ t ∈ [0 , T -2] , and then let r H T -1 = r T -1 +( G H 0: T -G 0: T ) /γ T -1 with G 0: T = ∑ t γ t r t being the environmental return, which is referred to as fusion below -this baseline may perform better when strong correlations existed between environmental and human rewards, as it intrinsically decomposes the human returns into IHRs. Consequently, in each experiment below, we compare the performance of the OPEHF framework extending all four types of OPE methods above, <IS/DR/DICE/FQE>-OPEHF , against the corresponding baselines,

Historical Trajectories

Episode 1

Episode 2

.

.

.

t=1

t=2

t=3

t=4 …

t=T

Rating

.

.

.

.

.

.

.

.

.

Tremor

.

.

.

t=T

t=T

t=T

Stimulation

Amplitudes

Sensing

Stimulation

Test Results

Beta Pwr. &

Stim Amps

Rating

Internal Pulse

Generator (IPG)

Tablet

running RL

policies

Tremor

Detection

Tremor

Severity

Wearable

Devices

Patient Feedback

(end-of-episode)

Environmental/Physiological

Feedback

(available every 2s)

<IS/DR/DICE/FQE>-<Fusion/Rescale> . We also include the VLM-H as an ablation baseline, as if it is a model-based approach standalone; this is achieved by sampling the estimate returns from the decoder, ˆ G H 0: T ∼ p ϕ ( G H 0: T | z T ) .

Metrics. Following a recent OPE benchmark [15], three metrics are considered to validate the performance of each method, including mean absolute error (MAE), rank correlation, and regret@1. Mathematical definitions can be found in Appendix D. Also, following [15], each method is evaluated over 3 random seeds, and the mean performance (with standard errors) is reported.

3.1 Adaptive Neurostimulation: Deep Brain Stimulation

Adaptive neurostimulation facilitates treatments for a variety of neurological disorders [4, 11, 13, 55]. Deep brain stimulation (DBS) is a type of neurostimulation used specifically toward Parkinson's disease (PD), where an internal pulse generator (IPG), implanted under the collarbone, sends electrical stimulus to the basal ganglia (BG) area of the brain through invasive electrodes; Figure 2 illustrates

VLM-H

IS-Fusion

IS-Rescale

IS-OPEHF

DR-Fusion

DR-Rescale

DR-OPEHF

DICE-Fusion

DICE-Rescale

DICE-OPEHF

FQE-Fusion

FQE-Rescale

FQE-OPEHF

MAE

↓

VLM-H

IS-Fusion

IS-Rescale

IS-OPEHF

DR-Fusion

DR-Rescale

DR-OPEHF

DICE-Fusion

DICE-Rescale

DICE-OPEHF

FQE-Fusion

FQE-Rescale

FQE-OPEHF

Rank

↑

VLM-H

IS-Fusion

IS-Rescale

IS-OPEHF

DR-Fusion

DR-Rescale

DR-OPEHF

DICE-Fusion

DICE-Rescale

DICE-OPEHF

FQE-Fusion

FQE-Rescale

FQE-OPEHF

Regret

↓

VLM-H

Ablation

Baselines

OPEHF

(Our)

the setup. Adaptive DBS aims to adjust the strength (amplitude) of the stimulus in real-time, to respond to irregular neuronal activities caused by PD, leveraging the local field potentials (LFPs) as the immediate feedback signals, i.e. , the environmental rewards. Existing works have leveraged RL for adaptive DBS over computational BG models [25, 20, 52, 59], using rewards defined over a physiological signal - beta-band power spectral density of LFPs ( i.e. , the beta power) since physiologically PD could lead to increased beta power due to the irregular neuronal activations it causes [39]. However, in clinical practice, the correlation between beta power and the level of satisfaction reported by the patients varies depending on the specific characteristics of each person, as PD can cause different types of symptoms over a wide range of severity [56, 37, 5, 76]. Such findings further justify the significance of evaluating HF/human returns in the real world using OPEHF.

In this experiment, we leverage OPEHF to estimate the feedback provided by 4 PD patients who participate in monthly clinical testings of RL policies trained to adapt amplitudes of the stimulus toward reducing their PD symptoms, i.e. , bradykinesia and tremor. A mixture of behavioral policies is used to collect the offline trajectories ρ β . Specifically, in every step, the state s t is a historical sequence of LFPs capturing neuronal activities, and the action a t up-

dates the amplitude of the stimulus to be sent 6 . Then, an environmental reward r t = R ( s t , a t ) gives a penalty if the beta power computed from the latest LFPs is greater than some threshold (to promote treatment efficacy) as well as a penalty proportional to the amplitudes of the stimulus being sent (to improve battery life of the IPG). At the end of each episode, the human returns G H 0: T are determined from three sources (weighted by 50%, 25%, 25%, respectively), i.e. , ( i ) a satisfaction rating (between 1-10) provided by the patient, ( ii ) hand grasp speed as a result of the bradykinesia test [63], and ( iii ) level of tremorcalculated over the data from a wearable accelerometry [60, 6]. Each session lasts more than 10 minutes, and each discrete step above corresponds to 2 seconds in the real world; thus, the horizon T ≥ 300 (more details are provided in Appendix D). Approval of an Institutional Review Board (IRB) is obtained from Duke University Health System, as well as the exceptional use of the DBS system by the US Food and Drug Administration (FDA).

For each patient, OPEHF and the baselines are used to estimate the human returns of 6 target policies with varied performance. The ground-truth human return for each target policy is obtained as a result of extensive clinical testing following the same schema above, over more than 100 minutes. Table 1 shows the Pearson's and Spearman's correlation coefficients [14], measuring the linear and rank correlations between the environmental returns G 0: T and the human returns G H 0: T over all the target DBS policies considered for each patient. Pearson's coefficients are all negative since the environmental reward function only issues penalties, while human returns are all captured by positive values. It can be observed that only weak-to-moderate degrees of linear correlations exist for all four patients, while ranks between G 0: T 's and G H 0: T 's are not preserved across patients; thus, it highlights

Patient #0

Patient #0

Patient #1

Patient #2

Patient #3

234.51

240.47

186.45

229.01

229.38

233.78

69.42

158.53

160.42

161.03

163.55

165.62

129.57

142.29

142.45

66.2

143.39

154.89

104.8

158.04

174.78

182.12

182.23

187.36

±

±

±

±

±

±

the need for leveraging OPEHF to estimate human returns, which is different than the classic OPE that focus on estimating environmental returns.

The overall performance averaged across the 4-patient cohort, is reported in Fig. 3. Raw performance over every single patient can be found in Appendix D. It can be observed that our OPEHF framework significantly improves MAEs and ranks compared to the two baselines, for all 4 types of downstream OPE methods we considered (IS, DR, DICE, and FQE). Moreover, our method also significantly outperforms the ablation VLM-H in terms of these two metrics, as the VLM-H's performance is mainly determined by how well it could capture the underlying dynamics and returns. In contrast, our OPEHF framework not only leverages the latent representations learnt by the VLM-H (for regularizing RILR), it also inherits the advantages intrinsically associated with the downstream estimators; e.g. , low-bias nature of IS, or low-variance provided by DR. Moreover, the fusion baseline in general performs worse than the rescale baseline as expected, since no strong correlations between environmental and human returns are found, as reported in Table 1.

Note that the majority of the methods lead to similar (relatively low) regrets, as there exist a few policies that lead to human returns that are close over some patients (see the raw statistics in Appendix D). The reason is that all the policies to be extensively tested in clinics are subject to initial screening, where clinicians ensure they would not lead to undesired outcomes or pose significant risks to the patients; thus, the performance of some target policies tends to be close. Nonetheless, low MAEs and high ranks achieved by our method show that it can effectively capture the subtle differences in returns resulting from other HF signals, i.e. , levels of bradykinesia and tremor. Moreover, Figure 4 visualizes the VLM-H encodings over the trajectories collected from the 6 target DBS policies for each participant and shows that encoded pairs associated with the policies that lead to similar returns are in general clustered together, which justifies the importance of leveraging the similarities over latent representations to regularize the reconstruction of IHRs as in the RILR objective (5).

3.2 Intelligent Tutoring

Intelligent tutoring refers to a system where students can actively interact with an autonomous tutoring agent that can customize the learning content, tests, etc., to improve engagement and learning

outcomes [2, 64, 45]. OPEHF is important in such a setup for directly estimating the potential outcomes that could be obtained by students, as opposed to environmental rewards that are mostly discrete; see detailed setup below. Existing works have explored this topic over classic OPE setting in simulations [49, 54].

The system is deployed in an undergraduate-level introduction to probability and statistics course over 5 academic years at North Carolina State University, where the interaction logs obtained from 1,288 students who voluntarily opted-in for this experiment are recorded. 7 Specifically, each episode refers to a student working on a set of 12 problems ( i.e. , horizon T = 12 ), where the agent suggests the student approach each problem through independent work, working with the hints provided, or directly providing the full solution (for studying purposes) - these options constitute the action space of the agent. The states are characterized by 140 features extracted from the logs, designed by domain experts; they include, for example, the time spent on each problem, and the correctness of the solution provided. In each step, an immediate environmental reward of +1 is issued if the answer submitted by students, for the current problem, is at least 80% correct (auto-graded following pre-defined rubrics). A reward of 0 is issued if the grade is less than 80% or the agent chooses the action that directly displays the full solution. Moreover, students are instructed to complete two exams, one before working on any problems and another after finishing all the problems. The normalized difference between the grades of two exams constitutes the human return for each episode. More details are provided in Appendix D.

The intelligent tutoring agent follows different policies across academic years, where the data collected from the first 4 years (1148 students total) constitutes the offline trajectories ρ β (as a result of a mixture of behavioral policies). The 4 policies deployed in the 5th year (140 students total) serve as the target policies, whose groundtruth performance is determined by averaging over the human returns of the episodes that are

associated with each policy respectively. Table 3 documents the Pearson's and Spearman's correlation coefficients between the environmental and human returns from data collected over each academic year, showing weak linear and rank correlations across all 5 years. Such low correlations are due to the fact that the environmental rewards are discrete and do not distinguish among the agent's choices, i.e. , a +1 reward can be obtained either if the student works out a solution independently or by following hints, and a 0 reward is issued every time the agent chooses to display the solution even if the student could have solved the problem. As a result, such a setup makes OPEHF to be more challenging; because human returns are only available at the end of each episode, and the immediate environmental rewards do not carry substantial information toward extrapolating IHRs.

Table 2 documents the performance of OPEHF and the baselines toward estimating the human returns of the target policies. It can be observed that our OPEHF framework achieves state-of-the-art performance, over all types of downstream OPE estimators considered. This result echos the design of the VLM-H where both environmental information (state transitions and rewards) and human returns are encoded into the latent space, which helps formulate a compact and expressive latent space for regularizing the downstream RILR objective (5). Moreover, it is important to use the latent information to guide the reconstruction of IHRs (as regularizations in RILR), as opposed to using the VLM-H to predict human returns standalone; since limited convergence guarantees/error bounds can be provided for VAE-based latent models, which is illustrated in both Figure 3 and Table 2 where OPEHF largely outperforms the VLM-H ablation over MAE and rank.

4 Related Works

OPE. Majority of existing model-free OPE methods can be categorized into one of the four types, i.e. , IS, DR, DICE, and FQE. Recently, variants of IS and DR methods have been proposed for variance or bias reduction [34, 71, 12, 69], as well as adaptations toward unknown behavioral policies [30].

DICE methods are intrinsically designed to work with offline trajectories rolled out from a mixture of behavioral policies, and existing works have introduced the DICE variants toward specific environmental setups [84, 83, 77, 78, 51, 9]. FQE extrapolates policy returns from the approximated Q-values [31, 40, 36]. There also exist model-based OPE methods [82, 18] that first captures the dynamics underlying the environment, and estimate policy performance by rolling out trajectories under the target policies. A more detailed review of existing OPE methods can be found in Appendix C. Note that these OPE methods have been designed for estimating the environmental returns. In contrast, the objective for OPEHF is to estimate the human returns which may not be strongly correlated with the environmental returns, as they are usually determined under different schemas.

VAEs for OPE and offline RL. There exists a long line of research developing latent models to capture the dynamics underlying environments in the context of offline RL as well as OPE. Specifically, PlaNet [27] uses recurrent neural networks to capture the transitions of latent variables over time. Latent representations learned by such V AE architectures have been used to augment the state space in offline policy optimization to improve sample efficiency, e.g. , in Dreamers [26, 28], SOLAR [81] and SLAC [42]. On the other hand, LatCo [65] attempts to improve sample efficiency by searching in the latent space which allows by-passing physical constraints. Also, MOPO [80], COMBO [79], and LOMPO [62] train latent models to quantify the confidence of the environmental transitions as learned from offline data and prevent the policies from following transitions over uncertain regions during policy training. Given that such models are mostly designed for improving sample efficiency in policy optimization/training, we choose to leverage the architecture from [18] for RILR as it is the first work that adapts latent models to the OPE setup.

Reinforcement learning from human feedback (RLHF). Recently, the concept of RLHF has been widely used in guiding RL policy optimization with the HF signals deemed more informative than the environmental rewards [8, 85, 47]. Specifically, they leverage the ranked preference provided by labelers to train a reward model, captured by feed-forward neural networks, that is fused with the environmental rewards to guide policy optimization. However, in this work, we focus on estimating the HF signals that serve as direct evaluation of the RL policies used in human-involved experiments, such as the level of satisfaction ( e.g. , on a scale 1-10) and the treatment outcome. The reason is that in many scenarios the participants cannot revisit the same procedure multiple times, e.g. , patients may not undergo the same surgeries several times and rank the experiences. More importantly, OPEHF's setup is critical when online testing of RL policies may be even prohibited, without sufficient justifications over safety and efficacy upfront, as illustrated by the experiments above.

Reward shaping. Although reward shaping methods [3, 58, 29] pursue similar ideas of decomposing the delayed and/or sparse rewards ( e.g. , the human return) into immediate rewards, they fundamentally rely on transforming the MDP to such that the value functions can be smoothly captured and high-return state-action pairs can be quickly identified and frequently re-visited. For example, RUDDER [3] leverages the transformed MDP that has expected future rewards equal to zero. Though the optimization objective is consistent between pre- and post-transformed MDPs, this approach likely would not converge to an optimal policy in practice. On the other hand, the performance ( i.e., returns) of sub-optimal policies is not preserved across the two MDPs. This significantly limits its use cases toward OPE which requires the returns resulted by sub-optimal policies to be estimated accurately. As a result, such methods are not directly applicable to the OPEHF problem we consider.

5 Conclusion and Future Works

Existing OPE methods fall short in estimating HF signals, as HF can be dependent upon various confounders. Thus, in this work, we introduced the OPEHF framework that revived existing OPE methods for estimating human returns, through RILR. The framework was validated over two real-world experiments and one simulation environment, outperforming the baselines in all setups. Although in the future it could be possible to extend OPEHF to facilitate estimating the HF signals needed for updating the policies similar to RLHF, we focused on policy evaluation which helped to isolate the source of improvements; as policy optimization's performance may depend on multiple factors, such as the exploration techniques used as well as the objective/optimizer chosen for updating the policy. Moreover, this work mainly focuses on the scenarios where the human returns are directly provided by the participants. So under the condition where the HF signals are provided by 3-rd parties ( e.g, clinicians), non-trivial adaptations over this work may be needed to consider special cases such as conflicting HF signals provided by different sources.

References

Saminda Wishwajith Abeyruwan, Laura Graesser, David B D'Ambrosio, Avi Singh, Anish Shankar, Alex Bewley, Deepali Jain, Krzysztof Marcin Choromanski, and Pannag R Sanketi. i-sim2real: Reinforcement learning of robotic policies in tight human-robot interaction loops. In Conference on Robot Learning , pages 212-224. PMLR, 2023.

John R Anderson, C Franklin Boyle, and Brian J Reiser. Intelligent tutoring systems. Science , 228(4698):456-462, 1985.

Jose A Arjona-Medina, Michael Gillhofer, Michael Widrich, Thomas Unterthiner, Johannes Brandstetter, and Sepp Hochreiter. Rudder: Return decomposition for delayed rewards. Advances in Neural Information Processing Systems , 32, 2019.

Alim Louis Benabid. Deep brain stimulation for parkinson's disease. Current opinion in neurobiology , 13(6):696-706, 2003.

Peter Brown, Antonio Oliviero, Paolo Mazzone, Angelo Insola, Pietro Tonali, and Vincenzo Di Lazzaro. Dopamine dependency of oscillations between subthalamic nucleus and pallidum in parkinson's disease. Journal of Neuroscience , 21(3):1033-1038, 2001.

Witney Chen, Lowry Kirkby, Miro Kotzev, Patrick Song, Ro'ee Gilron, and Brian Pepin. The role of large-scale data infrastructure in developing next-generation deep brain stimulation therapies. Frontiers in Human Neuroscience , 15:717401, 2021.

Nicholas C Chesnaye, Vianda S Stel, Giovanni Tripepi, Friedo W Dekker, Edouard L Fu, Carmine Zoccali, and Kitty J Jager. An introduction to inverse probability of treatment weighting in observational research. Clinical Kidney Journal , 15(1):14-20, 2022.

Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems , 30, 2017.

Bo Dai, Ofir Nachum, Yinlam Chow, Lihong Li, Csaba Szepesvári, and Dale Schuurmans. Coindice: Off-policy confidence interval estimation. Advances in Neural Information Processing Systems , 33:9398-9411, 2020.

Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, José M.F. Moura, Devi Parikh, and Dhruv Batra. Visual Dialog. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2017.

Günther Deuschl, Carmen Schade-Brittinger, Paul Krack, Jens Volkmann, Helmut Schäfer, Kai Bötzel, Christine Daniels, Angela Deutschländer, Ulrich Dillmann, Wilhelm Eisner, et al. A randomized trial of deep-brain stimulation for parkinson's disease. New England Journal of Medicine , 355(9):896-908, 2006.

Mehrdad Farajtabar, Yinlam Chow, and Mohammad Ghavamzadeh. More robust doubly robust off-policy evaluation. In International Conference on Machine Learning , pages 1447-1456. PMLR, 2018.

Kenneth A Follett, Frances M Weaver, Matthew Stern, Kwan Hur, Crystal L Harris, Ping Luo, William J Marks Jr, Johannes Rothlind, Oren Sagher, Claudia Moy, et al. Pallidal versus subthalamic deep-brain stimulation for parkinson's disease. New England Journal of Medicine , 362(22):2077-2091, 2010.

David Freedman, Robert Pisani, and Roger Purves. Statistics (international student edition). Pisani, R. Purves, 4th edn. WW Norton & Company, New York , 2007.

Justin Fu, Mohammad Norouzi, Ofir Nachum, George Tucker, Alexander Novikov, Mengjiao Yang, Michael R Zhang, Yutian Chen, Aviral Kumar, Cosmin Paduraru, et al. Benchmarks for deep off-policy evaluation. In International Conference on Learning Representations , 2020.

Ge Gao, Qitong Gao, Xi Yang, Miroslav Pajic, and Min Chi. A reinforcement learning-informed pattern mining framework for multivariate time series classification. In International Joint Conference on Artificial Intelligence (IJCAI) , 2022.

Ge Gao, Song Ju, Markel Sanz Ausin, and Min Chi. Hope: Human-centric off-policy evaluation for e-learning and healthcare. In International Conference on Autonomous Agents and Multiagent Systems (AAMAS) , 2023.

Qitong Gao, Ge Gao, Min Chi, and Miroslav Pajic. Variational latent branching model for off-policy evaluation. In The Eleventh International Conference on Learning Representations (ICLR) , 2023.

Qitong Gao, Davood Hajinezhad, Yan Zhang, Yiannis Kantaros, and Michael M Zavlanos. Reduced variance deep reinforcement learning with temporal logic specifications. In Proceedings of the 10th ACM/IEEE International Conference on Cyber-Physical Systems , pages 237-248, 2019.

Qitong Gao, Michael Naumann, Ilija Jovanov, Vuk Lesi, Karthik Kamaravelu, Warren M Grill, and Miroslav Pajic. Model-based design of closed loop deep brain stimulation controller using reinforcement learning. In 2020 ACM/IEEE 11th International Conference on Cyber-Physical Systems (ICCPS) , pages 108-118. IEEE, 2020.

Qitong Gao, Miroslav Pajic, and Michael M Zavlanos. Deep imitative reinforcement learning for temporal logic robot motion planning with noisy semantic observations. In 2020 IEEE International Conference on Robotics and Automation (ICRA) , pages 8490-8496. IEEE, 2020.

Qitong Gao, Stephen L Schmidt, Afsana Chowdhury, Guangyu Feng, Jennifer J Peters, Katherine Genty, Warren M Grill, Dennis A Turner, and Miroslav Pajic. Offline learning of closed-loop deep brain stimulation controllers for parkinson disease treatment. In Proceedings of the ACM/IEEE 14th International Conference on Cyber-Physical Systems (with CPS-IoT Week 2023) , pages 44-55, 2023.

Qitong Gao, Stephen L Schmidt, Karthik Kamaravelu, Dennis A Turner, Warren M Grill, and Miroslav Pajic. Offline policy evaluation for learning-based deep brain stimulation controllers. In 2022 ACM/IEEE 13th International Conference on Cyber-Physical Systems (ICCPS) , pages 80-91. IEEE, 2022.

Qitong Gao, Dong Wang, Joshua D Amason, Siyang Yuan, Chenyang Tao, Ricardo Henao, Majda Hadziahmetovic, Lawrence Carin, and Miroslav Pajic. Gradient importance learning for incomplete observations. International Conference on Learning Representations , 2022.

A. Guez, R. D. Vincent, M. Avoli, and J. Pineau. Adaptive treatment of epilepsy via batch-mode reinforcement learning. In AAAI , pages 1671-1678, 2008.

Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In International Conference on Learning Representations , 2020.

Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International conference on machine learning , pages 2555-2565. PMLR, 2019.

Danijar Hafner, Timothy P Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. In International Conference on Learning Representations , 2020.

Beining Han, Zhizhou Ren, Zuofan Wu, Yuan Zhou, and Jian Peng. Off-policy reinforcement learning with delayed rewards. In International Conference on Machine Learning , pages 8280-8303. PMLR, 2022.

Josiah Hanna, Scott Niekum, and Peter Stone. Importance sampling policy evaluation with an estimated behavior policy. In International Conference on Machine Learning , pages 2605-2613. PMLR, 2019.

Botao Hao, Xiang Ji, Yaqi Duan, Hao Lu, Csaba Szepesvari, and Mengdi Wang. Bootstrapping fitted q-evaluation for off-policy inference. In International Conference on Machine Learning , pages 4074-4084. PMLR, 2021.

Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation , 9(8):1735-1780, 1997.

Yan Jia, John Burden, Tom Lawton, and Ibrahim Habli. Safe reinforcement learning for sepsis treatment. In 2020 IEEE International conference on healthcare informatics (ICHI) , pages 1-7. IEEE.

Nan Jiang and Lihong Li. Doubly robust off-policy value evaluation for reinforcement learning. In International Conference on Machine Learning , pages 652-661. PMLR, 2016.

Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 , 2013.

Ilya Kostrikov and Ofir Nachum. Statistical bootstrapping for uncertainty estimation in offpolicy evaluation. arXiv preprint arXiv:2007.13609 , 2020.

A.A. Kühn, A. Kupsch, GH. Schneider, and P Brown. Reduction in subthalamic 8-35 hz oscillatory activity correlates with clinical improvement in parkinson's disease. Euro. J. of Neuroscience , 23(7):1956-1960, 2006.

Solomon Kullback and Richard A Leibler. On information and sufficiency. The annals of mathematical statistics , 22(1):79-86, 1951.

Alexis M Kuncel and Warren M Grill. Selection of stimulus parameters for deep brain stimulation. Clinical neurophysiology , 115(11):2431-2441, 2004.

Hoang Le, Cameron Voloshin, and Yisong Yue. Batch policy learning under constraints. In International Conference on Machine Learning , pages 3703-3712. PMLR, 2019.

Pierre L'Ecuyer and Bruno Tuffin. Approximate zero-variance simulation. In 2008 Winter Simulation Conference , pages 170-181. IEEE, 2008.

Alex Lee, Anusha Nagabandi, Pieter Abbeel, and Sergey Levine. Stochastic latent actor-critic: Deep reinforcement learning with a latent variable model. Advances in Neural Information Processing Systems , 33, 2020.

Lihong Li, Wei Chu, John Langford, and Xuanhui Wang. Unbiased offline evaluation of contextual-bandit-based news article recommendation algorithms. In Proceedings of the fourth ACM International Conference on Web Search and Data Mining , pages 297-306, 2011.

Christopher G Lis, Kamal Patel, and Digant Gupta. The relationship between patient satisfaction with service quality and survival in non-small cell lung cancer-is self-rated health a potential confounder? PloS one , 10(7):e0134617, 2015.

Evan Liu, Moritz Stephan, Allen Nie, Chris Piech, Emma Brunskill, and Chelsea Finn. Giving feedback on interactive student programs with meta-exploration. Advances in Neural Information Processing Systems , 35:36282-36294, 2022.

Yao Liu, Pierre-Luc Bacon, and Emma Brunskill. Understanding the curse of horizon in off-policy evaluation via conditional importance sampling. In International Conference on Machine Learning , pages 6184-6193. PMLR, 2020.

James MacGlashan, Mark K Ho, Robert Loftin, Bei Peng, Guan Wang, David L Roberts, Matthew E Taylor, and Michael L Littman. Interactive learning from policy-dependent human feedback. In International Conference on Machine Learning , pages 2285-2294. PMLR, 2017.

Travis Mandel, Yun-En Liu, Emma Brunskill, and Zoran Popovi´ c. Where to add actions in human-in-the-loop reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 31, 2017.

Travis Mandel, Yun-En Liu, Sergey Levine, Emma Brunskill, and Zoran Popovic. Offline policy evaluation across representations with applications to educational games. In AAMAS , volume 1077, 2014.

Rishabh Mehrotra, James McInerney, Hugues Bouchard, Mounia Lalmas, and Fernando Diaz. Towards a fair marketplace: Counterfactual evaluation of the trade-off between relevance, fairness & satisfaction in recommendation systems. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management , pages 2243-2251, 2018.

Ofir Nachum, Yinlam Chow, Bo Dai, and Lihong Li. Dualdice: Behavior-agnostic estimation of discounted stationary distribution corrections. Advances in Neural Information Processing Systems , 32:2318-2328, 2019.

Vivek Nagaraj, Andrew Lamperski, and Theoden I Netoff. Seizure control in a computational model using a reinforcement learning stimulation paradigm. International Journal of Neural Systems , 27(07):1750012, 2017.

Hongseok Namkoong, Ramtin Keramati, Steve Yadlowsky, and Emma Brunskill. Off-policy policy evaluation for sequential decisions under unobserved confounding. Advances in Neural Information Processing Systems , 33:18819-18831, 2020.

Allen Nie, Yannis Flet-Berliac, Deon Jordan, William Steenbergen, and Emma Brunskill. Dataefficient pipeline for offline reinforcement learning with limited data. Advances in Neural Information Processing Systems , 35:14810-14823, 2022.

Michael S Okun. Deep-brain stimulation for parkinson's disease. New England Journal of Medicine , 367(16):1529-1538, 2012.

Michael S Okun. Deep-brain stimulation for parkinson's disease. New England Journal of Medicine , 367(16):1529-1538, 2012.

Bahram Parvinian, Christopher Scully, Hanniebey Wiyor, Allison Kumar, and Sandy Weininger. Regulatory considerations for physiological closed-loop controlled medical devices used for automated critical care: food and drug administration workshop discussion topics. Anesthesia and analgesia , 126(6):1916, 2018.

Vihang P Patil, Markus Hofmarcher, Marius-Constantin Dinu, Matthias Dorfer, Patrick M Blies, Johannes Brandstetter, Jose A Arjona-Medina, and Sepp Hochreiter. Align-rudder: Learning from few demonstrations by reward redistribution. arXiv preprint arXiv:2009.14108 , 2020.

J. Pineau, A. Guez, Robert Vincent, Gabriella Panuccio, and Massimo Avoli. Treating epilepsy via adaptive neurostimulation: a reinforcement learning approach. Int. J. of Neural Sys. , 19(04):227-240, 2009.

Rob Powers, Maryam Etezadi-Amoli, Edith M Arnold, Sara Kianian, Irida Mance, Maxsim Gibiansky, Dan Trietsch, Alexander Singh Alvarado, James D Kretlow, Todd M Herrington, et al. Smartwatch inertial sensors continuously monitor real-world motor fluctuations in parkinson's disease. Science translational medicine , 13(579):eabd7865, 2021.

Doina Precup. Eligibility traces for off-policy policy evaluation. Computer Science Department Faculty Publication Series , page 80, 2000.

Rafael Rafailov, Tianhe Yu, Aravind Rajeswaran, and Chelsea Finn. Offline reinforcement learning from images with latent space models. In Learning for Dynamics and Control , pages 1154-1168. PMLR, 2021.

Claudia Ramaker, Johan Marinus, Anne Margarethe Stiggelbout, and Bob Johannes Van Hilten. Systematic evaluation of rating scales for impairment and disability in parkinson's disease. Movement disorders , 17(5):867-876, 2002.

Sherry Ruan, Allen Nie, William Steenbergen, Jiayu He, JQ Zhang, Meng Guo, Yao Liu, Kyle Dang Nguyen, Catherine Y Wang, Rui Ying, et al. Reinforcement learning tutor better supported lower performers in a math task. arXiv preprint arXiv:2304.04933 , 2023.

Oleh Rybkin, Chuning Zhu, Anusha Nagabandi, Kostas Daniilidis, Igor Mordatch, and Sergey Levine. Model-based reinforcement learning via latent-space collocation. In International Conference on Machine Learning , pages 9190-9201. PMLR, 2021.

Charlie Snell, Ilya Kostrikov, Yi Su, Mengjiao Yang, and Sergey Levine. Offline rl for natural language generation with implicit language q learning. arXiv preprint arXiv:2206.11871 , 2022.

Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction . MIT press, 2018.

Shengpu Tang and Jenna Wiens. Model selection for offline reinforcement learning: Practical considerations for healthcare settings. In Machine Learning for Healthcare Conference , pages 2-35. PMLR, 2021.

Ziyang Tang, Yihao Feng, Lihong Li, Dengyong Zhou, and Qiang Liu. Doubly robust bias reduction in infinite horizon off-policy estimation. In International Conference on Learning Representations , 2019.

Miguel Tejedor, Ashenafi Zebene Woldaregay, and Fred Godtliebsen. Reinforcement learning application in diabetes blood glucose control: A systematic review. Artificial intelligence in medicine , 104:101836, 2020.

Philip Thomas and Emma Brunskill. Data-efficient off-policy policy evaluation for reinforcement learning. In International Conference on Machine Learning , pages 2139-2148. PMLR, 2016.

Philip S Thomas. Safe reinforcement learning. 2015.

Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research , 9(11), 2008.

Mel Vecerik, Todd Hester, Jonathan Scholz, Fumin Wang, Olivier Pietquin, Bilal Piot, Nicolas Heess, Thomas Rothörl, Thomas Lampe, and Martin Riedmiller. Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards. arXiv preprint arXiv:1707.08817 , 2017.

Martin Wattenberg, Fernanda Viégas, and Ian Johnson. How to use t-sne effectively. Distill , 1(10):e2, 2016.

Joshua K Wong, Günther Deuschl, Robin Wolke, Hagai Bergman, Muthuraman Muthuraman, Sergiu Groppa, Sameer A Sheth, Helen M Bronte-Stewart, Kevin B Wilkins, Matthew N Petrucci, et al. Proc. the 9th annual deep brain stimulation think tank: Advances in cutting edge technologies, artificial intelligence, neuromodulation, neuroethics, pain, interventional psychiatry, epilepsy, and traumatic brain injury. Frontiers in Human Neuroscience , page 25, 2022.

Mengjiao Yang, Bo Dai, Ofir Nachum, George Tucker, and Dale Schuurmans. Offline policy selection under uncertainty. In Deep RL Workshop NeurIPS 2021 , 2021.

Mengjiao Yang, Ofir Nachum, Bo Dai, Lihong Li, and Dale Schuurmans. Off-policy evaluation via the regularized lagrangian. Advances in Neural Information Processing Systems , 33:65516561, 2020.

Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offline model-based policy optimization. Advances in neural information processing systems , 34:28954-28967, 2021.

Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. Advances in Neural Information Processing Systems , 33:14129-14142, 2020.

Marvin Zhang, Sharad Vikram, Laura Smith, Pieter Abbeel, Matthew Johnson, and Sergey Levine. Solar: Deep structured representations for model-based reinforcement learning. In International Conference on Machine Learning , pages 7444-7453. PMLR, 2019.

Michael R Zhang, Thomas Paine, Ofir Nachum, Cosmin Paduraru, George Tucker, Mohammad Norouzi, et al. Autoregressive dynamics models for offline policy evaluation and optimization. In International Conference on Learning Representations , 2020.

Ruiyi Zhang, Bo Dai, Lihong Li, and Dale Schuurmans. Gendice: Generalized offline estimation of stationary values. In International Conference on Learning Representations , 2020.

Shangtong Zhang, Bo Liu, and Shimon Whiteson. Gradientdice: Rethinking generalized offline estimation of stationary values. In International Conference on Machine Learning , pages 11194-11203. PMLR, 2020.

Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593 , 2019.

A Proofs for Section 2.2

Recall that the per-decision importance sampling (PDIS) [61] estimator follows ˆ G π PDIS = 1 N ∑ N -1 i =1 ∑ T -1 t =0 γ t ω ( i ) 0: t r H ( i ) t ; here, ω ( i ) t = π ( a ( i ) t | s ( i ) t ) ˆ β ( a ( i ) t | s ( i ) t ) , and henceforth ω ( i ) 0: t = ∏ t k =0 π ( a ( i ) k | s ( i ) k ) ˆ β ( a ( i ) k | s ( i ) k ) are the PDIS weights for offline trajectory τ ( i ) . To simplify our notation, we consider the PDIS estimator defined over a single trajectory, i.e. ,

as the results can be carried to N trajectories by multiplying with a factor 1 /N . We omit the superscript-ed ( i ) 's in the rest of the proof also for conciseness.

A.1 Proof of Proposition 1

Proof. We start with a lemma from [46].

Lemma 1 ([46]) . Given X t and Y t as two sequences of random variables. Then

where V ( · ) refer to the variances.

Now, if we set Y t = r H t ω 0: T and X t = τ 0: t which is a segment (from t = 0 to t = t ) of an offline trajectory τ , it is sufficient to prove Proposition 1 by proving that for any 1 ≤ t +1 ≤ k ≤ T ,

Applying the law of total expectations to (8), i.e. ,

then it is sufficient to show

We start by showing that

The transition above follows from the fact that the likelihood ratio ω 0: t is a martingale as shown in [41, 46], i.e. , E [ ω 0: T -1 | τ 0: t ] = ω 0: t , which implies E [ ω t +1: T -1 ] = 1 and therefore

Note that we keep using the notation E [ r H t | τ 0: t ] in (14) and (15), due to the setting of HMDP that r H t is a variable sampled from the distribution R H ( ·| s t , a t ) given ( s t , a t ) ; see Section 2.1.

since

Since τ 0: t is given, r H k and ω t +1: T -1 are equivalent to r ( j ) k -t -1 and ω ( j ) 0: T -t -2 for some other trajectory τ ( j ) ∼ ρ β . Also given the statement that ω ( j ) 0: T -t -2 and r ( j ) k -t -1 ω ( j ) 0: T -t -2 are positively correlated, it follows from (15) that

which justifies (13) and completes the proof.

B Derivation of the ELBO (2)

Note that unlike the ELBO in [18], the VLM-H includes an additional component that estimates the human return G H 0: T of each trajectory, i.e. ,

Note that to simplify our presentation, we omit ϕ, ψ, ρ β as part of the conditional terms in the joint likelihoods. The transition from (25) to (26) follows Jensen's inequality.

C Brief Overview of Existing OPE methods

C.1 Importance Sampling (IS) and per-decision IS (PDIS)

IS refers to a statistical technique that can calculate the expectation of a function f ( x ) w.r.t. an unknown distribution p ( x ) using a given distribution q ( x ) through re-weighting, i.e. ,

This technique can be applied in the context of OPE by setting f ( x ) as the accumulated return G 0: T , p as the trajectory distribution ρ π over the target policy π , and q as the trajectory distribution ρ β over the behavioral policy β .

According to [61], the vanilla IS estimator follows

where ω ( i ) 0: T -1 = ∏ T -1 t =0 π ( a t | s t ) β ( a t | s t ) refers to the IS weight, and G ( i ) 0: T is the return for the i -th (out of N ) offline trajectory. On the other hand, the PDIS estimator follows

with ω ( i ) 0: t = ∏ t k =0 π ( a k | s k ) β ( a k | s k ) being the PDIS weight, r ( i ) t is the environmental reward obtained at the t -th step of the i -th offline trajectory, and γ is the discounting factor.

C.2 Doubly Robust (DR)

DR attempts to reduce the variance of IS estimation by introducing the value function approximation, which trades off estimation variance by making the estimation biased. The non-recursive definition of DR estimation is provided in [71], i.e. ,

here, Q π ( · , · ) and V π ( · ) are the Q-function and value function, respectively, over the target policy, while s ( i ) t and a ( i ) t are the state and action taken at the t -th step of the i -th offline trajectory.

C.3 DIstributional Correction Estimation (DICE)

DICE aims to estimate the propensity of the target policy to visit particular state-action pairs relative to their likelihood of appearing in the offline trajectories, i.e. ,

where d π ( s,a ) d β ( s,a ) is the distribution correction ratio. Existing DICE variants [84, 83, 78, 51, 9] seek to approximate the ratio without knowledge of d π or d β , while a recent work has summarized that all existing variants can be formulated as regularized Lagrangians of the same linear program [78], unifying the choice of regularizations and constraints used in their original objectives.

C.4 Fitted Q-Evaluation (FQE)

According to [40], FQE approximates the Q-function over the target policy by minimizing the loss

here, the approximated Q-function is parameterized by κ . Note that it is different from the objective for classic Q-learning, or fitted Q-iteration, by using π ( s t +1 ) instead of max a Q ( s t +1 , a ; κ ) in the target y t .

D Additional Experimental Details and Results

In this section we provide additional details that are supplement to the discussions in Section 3.

D.1 Definition of the OPE Metrics

Mean Absolute error (MAE). MAE is defined as the absolute difference between the actual return and estimated return of a policy: MAE = | V π -ˆ V π | ; here, V π is the actual value of the policy π , and ˆ V π is the estimated value of π .

Rank correlation. Rank correlation measures the Spearman's rank correlation coefficient between the ordinal rankings of the estimated returns and actual returns across policies, i.e. ,

ρ = Cov ( rank ( V π 1: P ) , rank ( ˆ V π 1: P )) σ ( rank ( V π 1: P )) σ ( rank ( ˆ V π 1: P )) , where rank ( V π 1: P ) is the ordinal rankings of the actual returns, and rank ( ˆ V π 1: P ) is the ordinal rankings of the OPE-estimated returns.

Regret@1. Regret@1 is the (normalized) difference between the value of the actual best policy, against the value of the policy associated with the best OPE-estimated return, which is defined as (max i ∈ 1: P V π i -max j ∈ best (1: P ) V π j ) / max i ∈ 1: P V π i , where best (1 : P ) denotes the index of the best policy over the set of P policies as measured by estimated values ˆ V π .

D.2 Implementations and Hyper-parameters

VLM-H. Both encoder q ψ ( z t | z t -1 , a t -1 , s t ) and decoder p ϕ ( z t | z t -1 , a t -1 ) of VLM-H are configured to be LSTMs with 64 nodes for the hidden states, followed by two fully connected layers with 128 and 64 nodes each. All other parts of the encoder and decoder, including q ψ ( z 0 | s 0 ) , p ϕ ( G H 0: T | z T ) , p ϕ ( s t | z t ) , p ϕ ( r t -1 | z t ) , are multi-layered perceptrons (MLPs) with two fully connected layers with 128 and 64 nodes each. The regularization term introduced in [18] is added to the ELBO (2) with a scaling factor C ′ balancing the scale of the two terms, which is applied over the LSTM states between q ψ ( z t | z t -1 , a t -1 , s t ) and p ϕ ( z t | z t -1 , a t -1 ) . The scale of this regularization is selected from C ′ = {1e-04, 5e-03, 1e-03, 5e-02, 1e-02, 0.1, 1., 2., 5.}. Learning rate is tuned by grid search from {0.003, 0.001, 0.0007, 0.0005, 0.0003, 0.0001, 0.00005}. Exponential decay is applied to the learning rate, which decays the learning rate by 0.997 every iteration. The total number of training epoches is set to be 20 and minibatch size set to 64. Adam optimizer is used to perform gradient descent. L 2 weight decay with coefficient 0.001 and batch normalization are applied to all hidden fully connected layers. The top-10 models/checkpoints achieving the highest ELBOs proceed to the RILR step.

Downstream OPE Estimators. The implementations of downstream OPE estimators are built on top of [36] 8 , which provides the functionalities for using per-decision IS with behavioral policy estimation [30], DR [71], DICE [78] or FQE [40] to process the trajectories with reconstructed IHRs and estimate human returns.

Code Availabilities and Computational Resources. Code implementations for all three components above can be found in the supplementary materials attached. All experimental workloads are distributed among 4 Nvidia RTX A5000 24GB and 3 Nvidia Quadro RTX 6000 24GB graphics cards.

D.3 More on the Adaptive Neurostimulation Experiment - Deep Brain Stimulation (DBS)

DBS Hardware and Implementation. The internal pulse generator (IPG) is part of the Medtronic's Summit RC+S system. Moreover, Summit software APIs are provided by Medtronic to be leveraged to implement the RL policies to be tested in the clinic. The overall pipeline is shown in Figure 5. Specifically, each control cycle (an MDP step) takes 2 seconds, where the IPG collects local field potentials (LFPs) from the basal ganglia (BG) region of the participant's brain at a sampling rate of 500 Hz. Then, the LFPs are sent to the research tablet for signal processing, after which the beta-band power spectral densities (beta powers) are calculated and used to constitute the MDP state and reward signals. The beta powers are calculated by applying fast Fourior transform (FFT), with 512 bin width, to the LFPs collected in the past 2 seconds. The beta-band usually refers to the frequency band of 13-35 Hz, however, clinicians and electrical engineers had customized the range for each specific

Internal Pulse

Generator

(IPG)

Research

Tablet

Adaptive DBS Program

RDK

Com.

Protocol

Local Field

Potentials

Stimulation

Amplitudes

Beta-power

Calculation

Stim.

Wireless

Telemetry

(p)

Electrode

GPe

Thalamus

GPi-

STN

Nigra

Substantia

Contact #1

Contact #2

Contact #3

Pulse

Contact#4

Generator

patient, in order to allow beta powers to better capture the irregular neuronal behaviors. The specific range for each patient can be found in the paragraph below ( i.e. , patient characteristics). According to the latest LFP and beta power signals, the RL policy adjusts the amplitude of the electrical stimulus to be used for the next cycle, which is then sent back to the IPG through wireless telemetry. In the BG region of participant's brain, as illustrated in Fig. 6, 4-contact electrodes are placed in the subthalmic nucleus (STN) and globus pallidus (GP) regions. Monopolar stimulation was delivered on a single contact on each lead (with the case serving as counter-electrode). The two contacts surrounding the stimulation contact were used for sensing LFPs ( i.e. , sandwich sensing).

The IPG device was already approved for clinical research use by the FDA. The changes made to the software program were also approved by the FDA via an investigational device exemption (IDE). Corresponding IRB approvals were obtained from Duke University Health System. All participants were compensated as per institutional and administrative guidelines.

Patient Characteristics. As discussed in Section 3.1, all PD participants have unique characteristics toward varied PD severity and symptoms, which can be found in [22].

The MDP Setup. The state and action space, as well as rewards are briefly introduced in Section 3.1. Here we provide the details that are omitted in the main paper. Each discrete step in MDP corresponds to a control cycle which lasts 2 seconds; see the DBS hardware and implementation paragraph above. Each episode lasts at least T = 300 steps ( i.e. , the horizon), or more than 10 minutes in the real world. Specifically, states are constituted by a historical sequence of beta powers obtained from past 10 steps, following a similar setup as in [20, 52]. The actions a ∈ A are in [ ξ, 100%] ⊂ R , 0% ≤ ξ < 100% , which represent the percentage over the maximum amplitude of the stimulus the IPG can employ (determined by clinicians). Specifically, ξ = 40% for patient #0 and ξ = 60% for patients #1-#3, to ensure safety and minimum efficacy of DBS during clinical testing. The environmental rewards are

±

defined following, i.e. ,

here, -0 . 2 a is the penalty for stimulating with higher-than-needed amplitudes, to preserve the runtime of the IPG device (powered by a rechargeable battery) between recharges, and an additional -1 is given if stimulating with the amplitude determined by the policy cannot reduce beta power to below a threshold specific to each patient. The thresholds above are set to be the lower 20% quantile of the beta powers observed from initial exploration (introduced below). The human return at the end of each episode is quantified as a 50%-25%-25% weighted sum over the patient rating, hand grasp speed captured from performing the maneuver (rapid and full extension and close of all fingers) for the bradykinesia test, as well as the proportional length of the session where the patient displays tremor (as captured by a wearable accelerometry).

Collection of Offline Trajectories. Mixed types of controllers are used for initial exploration of the environment (for obtaining target policies introduced below), and the resulting trajectories are used as the offline training data for OPEHF. Specifically, a random controller that uniformly sample the stimulation amplitudes from [ ξ, 100%] , and a proportional-integral (PI) controller tuned by clinicians and electrical engineers, are used at the initial phase of clinical testing. They help identify the state-action regions that can lead to non-trivial returns. Then, such data are used to train RL policies - following deep deterministic policy gradient (DDPG), the deterministic counterpart of the soft actor-critic (SAC) - for initial exploration, most of which can only obtain up to 60% returns as obtained by the target policies below on average . This makes OPEHF challenging considering the limited exploration coverage provided by the offline trajectories.

Target Policies. A total of 6 target policies, to be evaluated by OPEHF, are considered for each patient. One of them is an open-loop policy that always stimulate with the lowest possible amplitude

±

±

±

±

±

±

±

±

±

±

±

±

±

±

±

±

±

( ξ % ), benchmarking the worst-case control performance. The other 5 target DBS policies are all trained using DDPG. Specifically, each is trained over a growing experience buffer containing the data collected from the latest trials, in addition to the fixed set of offline trajectories above; this would lead to a set of target policies with varied performance, as the policies obtained in the later stage tend to be more optimal since the environment has been explored to a broader extent, compared to the policies obtained at earlier stages. Moreover, the policies have to demonstrate at least moderate control efficacy (as determined by clinicians), in order to be considered as target policies; since the ground-truth return of each target policy is determined by averaging the human returns over more than 10 sessions of clinical testing ( > 100 minutes in total). The raw human returns of all target policies over each patient can be found in Tables 8 to 11, as well as in the legends of Figure 4. Note that, for each patient, there may exist a few policies that lead to close returns, due to the testing limitations introduced above, i.e. , the use of ξ as well as the safety protocol for long-term testing (introduced above).

Correlation between beta power and patient feedback. As discussed in Section 3.1, existing research [56, 37, 5, 76] have found inconsistency between the beta power and patient feedback, due to confounders related to patient's specific characteristics as well as varied symptoms caused by PD. Figure 7 also shows that only low-to-moderate correlations can be found between beta power and the HF signals that are used to define the human returns in general. This makes OPEHF more challenging as the environmental signals may contain limited information useful for extrapolating human returns.

D.4 More on the Intelligent Tutoring Experiment

Experimental Setup. All the students follow a 4-step procedure, i.e. , ( i ) study with textbook, ( ii ) pre-test, ( iii ) study with the intelligent tutoring system, ( iv ) post-test. The RL policies are deployed only in step ( iii ) following the MDP setup introduced in Section 3.2. The human returns are quantified

PO, R2 = 0.29

[ZH]

P1, R2 = 0.13

4.0

P2, R2 = 0.34

1 Grasp Speed [

P3, R2 = 0.26

3.5

3.0

2.5

Left Hand

2.0

1.5

1.0

0.5

-1.0

-0.50.0

0.5

1.0

1.5

2.0

2.5

3.0

Beta Amplitude[In(μV)]

Time Proportion with Tremor [%]

PO, R2 =0.08

P1, R² = 0.87

P2, R2 = 0.54

0.8

P3, R2 = 0.57

0.6

0.4

0.2

0.0

1.0

-0.50.0

0.5

1.0

1.5

2.0

2.5

3.0

Beta Amplitude [In(μV)]

[ZH]

PO, R2 = 0.3

P1, R2 = 0.01

4.0

P2, R2 = 0.31

Speed

P3, R2= 0.07

3.5

3.0

Grasp

2.5

Right Hand (

2.0

1.5

1.0

0.5

1.0

0.5

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Beta Amplitude[In(uV)]

10

PO, R2 = 0.23

P1, R2 = 0.14

P2, R2 = 0.36

P3, R2 = 0.75

8

Rating

g

4

1.0

0.5

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Beta Amplitude [ln(μV)]

as the normalized gain over the difference between the scores from post-test and pre-test, i.e. ,

Specifically, pre-test consists of a total of 14 single- and multiple-principle problems. No feedback are given following the pre-test, nor are they allowed to re-visit any questions from the test. After working with the intelligent tutoring system, the students take the post-test which contain 20 problems, 14 of which are isomorphic to the pre-test and the remaining ones are non-isomorphic multiple-principle problems. Both tests were auto-graded following a specific rubric.

More on the MDP States. As introduced in Section 3.2, the states are constituted by 140 expertselected features; each fall into one of the five categories including ( i ) autonomy (10 features),

capturing the effort spent on solving problems in the intelligent tutoring system, such as the number of times the student revisit a problem, ( ii ) temporal information (29 features), including the time spent on each problem etc., ( iii ) problem-specific (35 features), such as the difficulty of each problem, ( iv ) performance (57 features), such as the grade students received on each problem, ( v ) hint-related (11 features), including the total amount of hints requested, if applicable.

Target Policies. A total of four target policies are considered. One of the policies randomly selects an action at each state, which benchmarks the worst-case performance. The other 3 policies are trained using deep Q-network (DQN) with different learning rates, from {1e-3, 1e-4, 1e-5}.

E A Challenging Simulation Environment: Visual Q&A Dialogue

Wealso consider the visual Q&A dialogue environment [10], following the recent framework [66] that generalizes offline RL towards language generation tasks. In this environment, language generation agents are trained to ask questions over the image captions that are given (without showing the actual images), in order to gain as much information from the responses (returned by environment) that are useful to characterize the the key elements contained in the underlying image.

Overall Setup. Though this environment does not involve actually HF, the policies' performance intrinsically pertain to two types of returns as per the setup from [66]. Specifically, the immediate rewards, used to train the language generation policy (captured by generative pre-trained transformer version 2, or GPT2) to asks questions, are discrete, i.e. , a -1 reward is given at each step until most key elements can be inferred from the dialogue, and an additional -1 reward is given if the questions asked by the agent lead to responses that only contain trivial information towards extrapolating the image characteristics (such as the responses that only contain a 'yes/no'). In contrast, a separate pre-trained referee model is used to determine when sufficient elements in the underlying images can be captured from the dialogue history and terminates the episode, which maps the dialogue history to embeddings that represent the features pertaining to the image elements that can be extrapolated from the dialogue, followed by scoring the dialogue with relative percentile ranking of the ground truth image's distance from the predicted embedding among a set of images taken from the evaluation set provided by [66, 10], i.e. , an episode is terminated if the percentile ranking is improved by at least 50% compared to the percentile ranking before the dialogue begins, (1 -p T ) ≥ . 5 · (1 -p 0 ) , where p T is the ground truth image's percentile rank at dialogue's last turn and p 0 is the ground truth image's percentile rank at the beginning of the dialogue, when only the image caption is observed. As

a result, the former type of rewards are considered as the environmental rewards as per the OPEHF setup, and the latter one is closely related to the human returns as it is only provided at the end of each episode and are obtained following a different schema. Specifically, we consider p 0 -p T as the synthetic human return, which quantifies the improvement over the percentile ranking throughout the dialogue. More details on the environmental reward function and the percentile ranking generated by the referee model can be found in [66].

MDP Formulation. Consequently, in this experiment we compare OPEHF's performance against baselines, toward predicting the percentile rankings corresponding to the embeddings predicted from the referee model at the end of episodes. The MDP states are captured by the embeddings output by the attention layer from the encoder of the GPT2, by feeding in the dialogue history up to the current step. The action space is captured as V l , where V is the vocabulary considered and l is the maximum number of words the agent is allowed to place in the questions in each step. The definition of environmental rewards and synthetic human returns are introduced above.

Offline Trajectories and Target Policies. Five behavioral policies are used to collect offline trajectories, which are obtained from training over 4 types of RL algorithms following the setup introduced in [66], i.e. , 1 policy follows conservative Q-learning (CQL), 2 follow implicit Q-learning (IQL), 1 follows behavioral clone (BC) and the last follows decision transformer (DT). All the behavioral policies are trained to obtain only 10%-50% of the best synthetic human returns that can be achieved, which are considered sub-optimal. Eight different policies (also obtained from 4 types of RL algorithms) constitute the set of target policies whose performance will be estimated by OPEHF and the baselines. Their performance spread roughly equally between 0%-100% of the optimal return that could be achieved; details can be found in Table 13. It can be found that synthetic human returns can be negative if the dialogue provides trivial, or sometimes misleading, information that are not helpful for capturing the underlying image. In practice one can shift the returns to become all positive in order to comply with the assumption in Proposition 1. Moderate correlations are found between the environmental and synthetic human returns in this experiment, i.e. , Pearson's correlation coefficient is 0.606 and Spearman's rank correlation coefficient is 0.69.

Results and Discussion. The results are summarized in Table 12. It can be observed that the OPEHF in general outperforms the baselines, with significantly lower MAEs and higher ranks if IS, DR or FQE are selected as the downstream estimator. We also find that if DICE is selected as the downstream estimator, it in general leads to lower MAEs and regrets for both the OPEHF and rescale baseline, but significantly low rank correlations. Our hypothesize would be that considering DICE directly

±

±

±

±

±

±

estimate the discrepancy between target and behavioral policy's propensity of visiting particular state-action pairs across the offline trajectories, it may benefit from leveraging the information (over the state-action space) captured by a language encoder that place roughly equal attention to each step of the dialogue, as opposed to the GPT2's encoder which is not fine-tuned specifically for such a case.

Peer reviews:
comments: SUMMARY:
This paper proposes an off-policy evaluation method that utilizes sparse human feedback. Experiments are conducted on two real-world environments and one simulation environment.

STRENGTHS:
The idea of incorporating human feedback into policy evaluation is interesting and significant. The proposed framework is compatible with existing methods. The experiments are comprehensive. Especially, real-world experiments are appreciated.

WEAKNESSES:
My major concerns are described in the Questions section. It would be appreciated if the authors address them in detail and I’m happy to raise my score.

QUESTIONS:
1.	Are 4 patients enough for validating the method? I’m wondering how many participants other similar methods adopt. 
2.	The proposed method additionally requires offline human feedback data. Are these comparing baselines also using the extra human data? If not, what did you do to make fair comparisons?

LIMITATIONS:
Please see the questions.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-28

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors examine an off-policy evaluation scenario where we have an environment with an intrinsic environment reward $r$, and an extrinsic human reward $r^H$. A further assumption is made that the human reward is only observed at the end of the episode, and comes as an rating on the total return of the episode. This is done to line up with cases where human ratings may only be clear after the fact.\n\nThe goal of the paper is to do off-policy evaluation in this modified class of MDPs. Two baseline approaches for doing this are considered. The first is to treat the human return as a reward that is achieved at the last timestep of the episode, with $r = 0$ at all earlier steps. The second is to set the human reward $r^H = r$, where $r$ is the original known environment reward, then add an offset $G^H - G$ term to the last timestep, where $G^H$ = total human return and $G$ = total environment return.\n\nThis paper proposes a method to learn timestep level $r^H$ directly. First it argues that given accurate timestep level $r^H$, the variance of the policy return estimate is equal or smaller than the estimate in the sparse MDP where all human reward is allocated to the final timestep. The reward learning method is then done based on two objective terms. The total reward must equal the human label of return for each trajectory, and the reward for individual $(s,a)$ should be similar to reward for other $(s',a')$ that have similar embeddings.\n\nThis then leaves determining the embeddings. This is done via a VAE to learn a latent space, and the the nearest K-neighbors of a given $(s,a)$ embed are treated as similar examples.\n\nOnce a reward estimate is learned, any off the shelf OPE method can be used.\n\nSTRENGTHS:\nThe paper makes an important observation that human feedback tends to be given at the end of an episode or after the fact, and is usually not given at the per-timestep level. It is clear that if per-timestep labels can be learned, then using them to estimate policy performance should give better results than estimating just total episode return (at least assuming learning per-timestep labels is a well-formed problem)\n\nWEAKNESSES:\nI don't like that the paper proposes an \"OPEHF framework\", yet the bulk of the paper is really about learning a reward function / learning to distribute a return estimate over past trajectories. Nothing about the reward function learning is inherently tied to off-policy evaluation, it could theoretically be done online too, it's just something that can be done off-policy as well.\n\nThe idea of return decomposition in the paper seems very similar to the RUDDER: Return Decomposition for Delayed Rewards paper, albeit with a different use case (in that paper, the environment naturally has a large sparse reward, which is decomposed to provide partial reward signal for an RL agent instead of needing to learn to reachieve the large sparse reward seen before.) It seems very relevant, in that the RUDDER paper proposes a number of return decomposition methods, none of which are considered in this paper.\n\nThe embedding of $(s,a)$ is said to come from a bi-directional LSTM, in order to leverage information from both previous and subsequent steps. Doesn't this break the Markov assumption of MDPs? Since the rewards $r(s,a)$ now may depend on future states?\n\nHow does the method handle cases where $r(s,a)$ may not be well defined? For example, two human raters have differing opinions on the score of a given trajectory.\n\nOverall I do not feel the paper is benchmarked sufficiently to be accepted.\n\nEdit: see rebuttal chain, have decided to increase score based on discussion of concerns.\n\nQUESTIONS:\nAre there examples of ways the original environment reward can be used, aside from the fusion baseline mentioned? I am trying to understand if it makes sense to even view this as a new class of MDPs, vs treating it like an MDP with one reward function that is only observed at the last timestep. It seems like the method proposed here does not use original env reward in any way besides training the encoder-decoder in the VAE?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper discusses a framework that learns to decompose human feedback reward, which is usually sparse and only given at the end of the episode, to intermediate step reward. They formulate a reward decomposition objective that considers original environment trajectory information using a variational auto-encoding representation learning objective as an implicit constraint. The authors show that their framework can reduce MAE for multiple OPE algorithms on a few domains.\n\nThe core contribution of this paper is composed of a few known techniques, but the final result is an enhanced OPE with a lower MAE. The real-world experiments are the highlight of this paper.\n\nSTRENGTHS:\nThe paper has offered two very interesting real-world experiments to demonstrate the effectiveness of the proposed framework and show how it can enhance existing OPE algorithms. Indeed, a lot of existing OPE algorithms often work well with dense rewards, and these algorithms do not have an additional component that can infer intermediate rewards.\n\nI find the real-world experiments very refreshing and quite convincing.\n\nWEAKNESSES:\nThe methodology proposed in this paper is not very novel — the authors assembled a few known techniques in the AI/ML community. Also, I’m not entirely sure some of the content in the paper is necessary. The authors can correct me if I’m wrong or misunderstood the contribution, but some parts of the paper do not seem like original contributions but simply restatements of known results.\n\nLine 101: Problem 1. The authors indicated one of the issues is “the total number of offline trajectories AND $r_t^H$ being unknown.” However, in the later sections, I think only $r_t^H$ was solved. It doesn’t seem like the proposed framework would change much if “the total number of offline trajectories” is known or unknown. Can authors provide more insight on how this affects their framework design? Specifically, how would they change the algorithm if the number of trajectories is known? I sincerely suggest dropping this if this is not a serious factor that affects your algorithm design.\n\nLine 121 Proposition 1: The rescaled OPE estimator has a lower variance than the original OPE estimator. I think this is a known result. For example, weighted importance sampling has a lower variance than importance sampling. Similarly, weighted PDIS has a lower variance than PDIS. See Thomas [1] Chapter 3 for consistent weighted PDIS. Can the authors explain and clarify how they made a novel contribution beyond what’s provided in [1]? Otherwise, I suggest the authors cite the previous work so readers will not get confused.\n\n[1] Thomas, P. S. Safe Reinforcement Learning. PhD thesis, University of Massachusetts Amherst, 2015b.\n\nQUESTIONS:\n1. FQE is an OPE algorithm that tries to do reward decomposition. However, the experiment still showed that using the decomposed reward produced from OPEHF, FQE-OPEHF can have a lower MAE than just using FQE alone. Can the authors comment on why this would be the case? \n2. The objective is on the sum of discounted rewards being the same, but there is no constraint on reward from each timestep — this means “theoretically” we can have a neural network predict -10000 on one timestep and 9999 on the next step, and still have the sum of reward to be -1. Can the authors comment on why this degenerate case hasn’t happened, and can authors provide some analysis/insight/summary data on what the decomposed intermediate human feedback rewards look like?\n\nLIMITATIONS:\nThe authors addressed the limitations of their approach.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a framework for Off-Policy Evaluation for Human Feedback (OPEHF). Off-policy evaluation allows the evaluation of learned policies on offline recorded data, which is especially valuable in scenarios where online deployment to evaluate policies is dangerous or expensive. Especially in scenarios such as healthcare, human feedback is very valuable and apparently often not very correlated with environmental rewards, making the combination of these two aspects very interesting. The paper considers the challenging case where human feedback is only available at the end of an episode without any per-step immediate human reward (IHR). To utilize existing OPE approaches, the per episode human reward is first reconstructed to a sequence of IHR by optimizing the sequence such that the cumulative discounted sum equals the human rewards, while additionally regularizing the reconstructed IHRs to align well with a latent representation of a state-action pair sequence. The resulting approach is evaluated on 2 real world scenarios, showing interesting results.\n\nSTRENGTHS:\n- The paper tackles a problem with realistic applications and proposes an interesting approach to do so.\n- It's well written with an extensive supplementary. Even though this is sadly not my area of expertise, I felt the paper was made to be understandable.\n- Results look favorable compared to simpler baselines and the method does not rely on environmental rewards being correlated with the human feedback, for which the paper gives multiple interesting example use cases.\n- Code is provided.\n\nWEAKNESSES:\n- While I very much like the actual longer term studies with real humans, overall the empirical evidence still is not extremely convincing. I guess this is an inherent problem with human feedback being expensive to collect. Nevertheless, some additional experiments would of course be very valuable.\n\nQUESTIONS:\n- In Proposition 1 you assume that the expected value of the rewards is positive. This is obvious in the considered scenarios, but would this approach generalize to other scenarios where human feedback could indeed be negative?\n- To what extend does the VLM-H hyperparameter tuning play a role here? How robust is the overall approach to less optimal setups here?\n- If you could discuss further evaluation venues for the proposed method, that would be interesting, but honestly I quite like the overall paper and find it to be an interesting read. If no further bigger issues are brought up by the other reviewers, I am willing to up my rating, I would just like to keep it a little conservative given this is very much outside of my expertise.\n\nLIMITATIONS:\nSome limitations are discussed and no other obvious limitations or major negative societal impacts come to mind.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-27"}