SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Overall objective of RILR for OPEHF. As a result, by following (1) and leveraging the  $\mathcal{L}_{regu}$  from (3) above, the objective for reconstructing the IHRs is set to be, i.e.,
$$
\max  _ {\theta} \frac {1}{N} \sum_ {i = 0} ^ {N - 1} \left[ \log p \left(\sum_ {t = 0} ^ {T - 1} \gamma^ {t} \hat {r} _ {t} ^ {\mathcal {H}} = G _ {0: T} ^ {\mathcal {H} (i)} \mid \theta , \tau^ {(i)}, G _ {0: T} ^ {\mathcal {H} (i)}\right) - C \cdot \sum_ {t = 0} ^ {T - 1} \mathcal {L} _ {r e g u} \left(\hat {r} _ {t} ^ {\mathcal {H}} \mid \theta , \psi , s _ {0: t} ^ {(i)}, a _ {0: t - 1} ^ {(i)}, G _ {0: T} ^ {\mathcal {H} (i)}\right) \right]. \tag {5}
$$
This helps facilitate an orthogonal basis of the latent space, which would improve the expressiveness of the model.
5If needed, one can project the states over to the orthogonal basis, to ensure that they follow a diagonal covariance.
Move from RILR to OPEHF. In what follows, one can leverage any existing OPE methods to take as inputs the offline trajectories, with the immediate environmental rewards  $r_t$ 's replaced by the reconstructed IHRs  $\hat{r}_t^{\mathcal{H},\mathrm{s}}$ , to achieve the OPEHF's objective (Problem 1). Moreover, our method does not require the IHRs to be correlated with the environmental rewards, as the VLM-H learns to reconstruct both by sampling from two independent distributions,  $p_{\phi}(r_{t - 1}|z_t)$  and  $p_{\phi}(G_{0:T}^{\mathcal{H}}|z_T)$  respectively, following (2); this is also illustrated empirically over the experiments introduced below (Sections 3), where exceedingly low correlations are found in specific scenarios.
The overall pipeline summarizing our method is shown in Figure 1 (right).
3 Real-World Experiments with Human Participants
In this section, we validate the OPEHF framework introduced above over two real-world experiments, adaptive neurostimulation, and intelligent tutoring. Specifically, we consider four types of OPE methods to be used as the downstream estimator following the RILR step (Section 2.3), including per-decision importance sampling (IS) with behavioral policy estimation [30], doubly robust (DR) [71], distribution correction estimation (DICE) [78] and fitted Q-evaluation (FQE) [40]. A brief overview of these methods can be found in Appendix C, and the specific implementations we use are documented in Appendix D. In Appendix E, we have also tested our method within a visual Q&A environment [10, 66], which follows similar mechanisms as in the two real-world experiments, i.e., two types of return signals are considered though no human participants are involved.
Baselines and Ablations. The baselines include two variants for each of the OPE methods above, i.e., (i) the rescale approach discussed in Section 2.2, and (ii) another variant that sets all the IHRs to be equal to the environmental rewards at corresponding steps,  $r_t^{\mathcal{H}} = r_t\forall t\in [0,T - 2]$ , and then let  $r_{T - 1}^{\mathcal{H}} = r_{T - 1} + (G_{0:T}^{\mathcal{H}} - G_{0:T}) / \gamma^{T - 1}$  with  $G_{0:T} = \sum_{t}\gamma^{t}r_{t}$  being the environmental return, which is referred to as fusion below - this baseline may perform better when strong correlations existed between environmental and human rewards, as it intrinsically decomposes the human returns into IHRs. Consequently, in each experiment below, we compare the performance of the OPEHF framework extending all four types of OPE methods above, <IS/DR/DICE/FQE>-OPEHF, against the corresponding baselines,
<IS/DR/DICE/FQE>--<Fusion/Rescale>. We also include the VLM-H as an ablation baseline, as if it is a model-based approach standalone; this is achieved by sampling the estimate returns from the decoder,  $\hat{G}_{0:T}^{\mathcal{H}}\sim p_{\phi}(G_{0:T}^{\mathcal{H}}|z_T)$
Metrics. Following a recent OPE benchmark [15], three metrics are considered to validate the performance of each method, including mean absolute error (MAE), rank correlation, and regret@1. Mathematical definitions can be found in Appendix D. Also, following [15], each method is evaluated over 3 random seeds, and the mean performance (with standard errors) is reported.
3.1 Adaptive Neurostimulation: Deep Brain Stimulation
Adaptive neurostimulation facilitates treatments for a variety of neurological disorders [4, 11, 13, 55]. Deep brain stimulation (DBS) is a type of neurostimulation used specifically toward Parkinson's disease (PD), where an internal pulse generator (IPG), implanted under the collarbone, sends electrical stimulus to the basal ganglia (BG) area of the brain through invasive electrodes; Figure 2 illustrates
Figure 2: Setup of the neurostimulation experiments, as well as the formulation of offline trajectories. Environmental rewards and human returns are captured in streams 1 and 2-3 respectively.
Figure 3: Results from the adaptive neurostimulation experiment, i.e., deep brain stimulation (DBS). Each method is evaluated over the data collected from each patient, toward corresponding target policies, respectively. The performance shown above are averaged over all 4 human participants affected by Parkinson's disease (PD). Raw performance over each patient can be found in Appendix D.
the setup. Adaptive DBS aims to adjust the strength (amplitude) of the stimulus in real-time, to respond to irregular neuronal activities caused by PD, leveraging the local field potentials (LFPs) as the immediate feedback signals, i.e., the environmental rewards. Existing works have leveraged RL for adaptive DBS over computational BG models [25, 20, 52, 59], using rewards defined over a physiological signal - beta-band power spectral density of LFPs (i.e., the beta power) since physiologically PD could lead to increased beta power due to the irregular neuronal activations it causes [39]. However, in clinical practice, the correlation between beta power and the level of satisfaction reported by the patients varies depending on the specific characteristics of each person, as PD can cause different types of symptoms over a wide range of severity [56, 37, 5, 76]. Such findings further justify the significance of evaluating HF/human returns in the real world using OPEHF.
In this experiment, we leverage OPEHF to estimate the feedback provided by 4 PD patients who participate in monthly clinical testings of RL policies trained to adapt amplitudes of the stimulus toward reducing their PD symptoms, i.e., bradykinesia and tremor. A mixture of behavioral policies is used to collect the offline trajectories  $\rho^{\beta}$ . Specifically, in every step, the state  $s_t$  is a historical sequence of LFPs capturing neuronal activities, and the action  $a_t$  up
Table 1: Correlations between the environmental and human returns of the 6 target DBS policies associated with each PD patient.
Patient #	0	1	2	3
Pearson's	-0.396	-0.477	-0.599	-0.275
Spearman's	-0.2	-0.6	0.086	0.086
dates the amplitude of the stimulus to be sent<sup>6</sup>. Then, an environmental reward  $r_t = R(s_t, a_t)$  gives a penalty if the beta power computed from the latest LFPs is greater than some threshold (to promote treatment efficacy) as well as a penalty proportional to the amplitudes of the stimulus being sent (to improve battery life of the IPG). At the end of each episode, the human returns  $G_{0:T}^{\mathcal{H}}$  are determined from three sources (weighted by 50%, 25%, 25%, respectively), i.e., (i) a satisfaction rating (between 1-10) provided by the patient, (ii) hand grasp speed as a result of the bradykinesia test [63], and (iii) level of tremorcalculated over the data from a wearable accelerometer [60, 6]. Each session lasts more than 10 minutes, and each discrete step above corresponds to 2 seconds in the real world; thus, the horizon  $T \geq 300$  (more details are provided in Appendix D). Approval of an Institutional Review Board (IRB) is obtained from Duke University Health System, as well as the exceptional use of the DBS system by the US Food and Drug Administration (FDA).
For each patient, OPEHF and the baselines are used to estimate the human returns of 6 target policies with varied performance. The ground-truth human return for each target policy is obtained as a result of extensive clinical testing following the same schema above, over more than 100 minutes. Table 1 shows the Pearson's and Spearman's correlation coefficients [14], measuring the linear and rank correlations between the environmental returns  $G_{0:T}$  and the human returns  $G_{0:T}^{\mathcal{H}}$  over all the target DBS policies considered for each patient. Pearson's coefficients are all negative since the environmental reward function only issues penalties, while human returns are all captured by positive values. It can be observed that only weak-to-moderate degrees of linear correlations exist for all four patients, while ranks between  $G_{0:T}$ 's and  $G_{0:T}^{\mathcal{H}}$ 's are not preserved across patients; thus, it highlights
$^{6}$ RL policies only adapt the stimulation amplitudes within a safe range as determined by neurologists/neurosurgeons, making sure they will not lead to negative effects to participants.
Patient #0
Patient #1
Figure 4:  $t$ -SNE visualizing the VLM-H encodings of the state-action pairs rolled out over DBS policies with different human returns (shown in the legend). It can be observed that distances among the encoded pairs associated with the policies that lead to similar returns are in general smaller, justifying the RILR objective (5).
Patient #2
Patient #3
Table 2: Results from the intelligent tutoring experiment, i.e., performance achieved by our OPEHF framework compared to the ablation and baselines over all four types of downstream OPE estimators.
Fusion	IS Rescale	OPEHF (our)	Fusion	DR Rescale	OPEHF (our)	Ablation VLM-H
MAE	0.7±0.14	0.77±0.08	0.57±0.09	1.03±0.07	1.03±0.25	0.86±0.04	1.00±0.01
Rank	0.47±0.11	0.4±0.09	0.8±0.09	0.33±0.05	0.4±0.0	0.53±0.2	0.41±0.25
Regret@1	0.36±0.16	0.36±0.16	0.41±0.04	0.41±0.0	0.41±0.0	0.41±0.0	0.28±0.19
Fusion	DICE Rescale	OPEHF (our)	Fusion	FQE Rescale	OPEHF (our)
MAE	3.19±0.57	2.33±0.59	1.01±0.01	0.74±0.07	0.98±0.1	0.59±0.1
Rank	0.47±0.2	0.33±0.2	0.53±0.22	0.27±0.14	0.4±0.0	0.47±0.05
Regret@1	0.55±0.06	0.45±0.18	0.37±0.15	0.36±0.16	0.41±0.0	0.41±0.0
the need for leveraging OPEHF to estimate human returns, which is different than the classic OPE that focus on estimating environmental returns.
The overall performance averaged across the 4-patient cohort, is reported in Fig. 3. Raw performance over every single patient can be found in Appendix D. It can be observed that our OPEHF framework significantly improves MAEs and ranks compared to the two baselines, for all 4 types of downstream OPE methods we considered (IS, DR, DICE, and FQE). Moreover, our method also significantly outperforms the ablation VLM-H in terms of these two metrics, as the VLM-H's performance is mainly determined by how well it could capture the underlying dynamics and returns. In contrast, our OPEHF framework not only leverages the latent representations learnt by the VLM-H (for regularizing RILR), it also inherits the advantages intrinsically associated with the downstream estimators; e.g., low-bias nature of IS, or low-variance provided by DR. Moreover, the fusion baseline in general performs worse than the rescale baseline as expected, since no strong correlations between environmental and human returns are found, as reported in Table 1.
Note that the majority of the methods lead to similar (relatively low) regrets, as there exist a few policies that lead to human returns that are close over some patients (see the raw statistics in Appendix D). The reason is that all the policies to be extensively tested in clinics are subject to initial screening, where clinicians ensure they would not lead to undesired outcomes or pose significant risks to the patients; thus, the performance of some target policies tends to be close. Nonetheless, low MAEs and high ranks achieved by our method show that it can effectively capture the subtle differences in returns resulting from other HF signals, i.e., levels of bradykinesia and tremor. Moreover, Figure 4 visualizes the VLM-H encodings over the trajectories collected from the 6 target DBS policies for each participant and shows that encoded pairs associated with the policies that lead to similar returns are in general clustered together, which justifies the importance of leveraging the similarities over latent representations to regularize the reconstruction of IHRs as in the RILR objective (5).
3.2 Intelligent Tutoring
Intelligent tutoring refers to a system where students can actively interact with an autonomous tutoring agent that can customize the learning content, tests, etc., to improve engagement and learning
outcomes [2, 64, 45]. OPEHF is important in such a setup for directly estimating the potential outcomes that could be obtained by students, as opposed to environmental rewards that are mostly discrete; see detailed setup below. Existing works have explored this topic over classic OPE setting in simulations [49, 54].
The system is deployed in an undergraduate-level introduction to probability and statistics course over 5 academic years at North Carolina State University, where the interaction logs obtained from 1,288 students who voluntarily opted-in for this experiment are recorded. Specifically, each episode refers to a student working on a set of 12 problems (i.e., horizon  $T = 12$ ), where the agent suggests the student approach each problem through independent work, working with the hints provided, or directly providing the full solution (for studying purposes) – these options constitute the action space of the agent. The states are characterized by 140 features extracted from the logs, designed by domain experts; they include, for example, the time spent on each problem, and the correctness of the solution provided. In each step, an immediate environmental reward of +1 is issued if the answer submitted by students, for the current problem, is at least 80% correct (auto-graded following pre-defined rubrics). A reward of 0 is issued if the grade is less than 80% or the agent chooses the action that directly displays the full solution. Moreover, students are instructed to complete two exams, one before working on any problems and another after finishing all the problems. The normalized difference between the grades of two exams constitutes the human return for each episode. More details are provided in Appendix D.
The intelligent tutoring agent follows different policies across academic years, where the data collected from the first 4 years (1148 students total) constitutes the offline trajectories  $\rho^{\beta}$  (as a result of a mixture of behavioral policies). The 4 policies deployed in the 5th year (140 students total) serve as the target policies, whose ground-truth performance is determined by averaging over the human returns of the episodes that are
Table 3: Correlations between the environmental and human returns from data collected over each academic year.
Year #	0	1	2	3	4
Pearson's	0.033	0.176	0.089	0.154	0.183
Spearman's	0.082	0.156	0.130	0.161	0.103
associated with each policy respectively. Table 3 documents the Pearson's and Spearman's correlation coefficients between the environmental and human returns from data collected over each academic year, showing weak linear and rank correlations across all 5 years. Such low correlations are due to the fact that the environmental rewards are discrete and do not distinguish among the agent's choices, i.e., a  $+1$  reward can be obtained either if the student works out a solution independently or by following hints, and a 0 reward is issued every time the agent chooses to display the solution even if the student could have solved the problem. As a result, such a setup makes OPEHF to be more challenging; because human returns are only available at the end of each episode, and the immediate environmental rewards do not carry substantial information toward extrapolating IHRs.
Table 2 documents the performance of OPEHF and the baselines toward estimating the human returns of the target policies. It can be observed that our OPEHF framework achieves state-of-the-art performance, over all types of downstream OPE estimators considered. This result echos the design of the VLM-H where both environmental information (state transitions and rewards) and human returns are encoded into the latent space, which helps formulate a compact and expressive latent space for regularizing the downstream RILR objective (5). Moreover, it is important to use the latent information to guide the reconstruction of IHRs (as regularizations in RILR), as opposed to using the VLM-H to predict human returns standalone; since limited convergence guarantees/errror bounds can be provided for VAE-based latent models, which is illustrated in both Figure 3 and Table 2 where OPEHF largely outperforms the VLM-H ablation over MAE and rank.
OPE. Majority of existing model-free OPE methods can be categorized into one of the four types, i.e., IS, DR, DICE, and FQE. Recently, variants of IS and DR methods have been proposed for variance or bias reduction [34, 71, 12, 69], as well as adaptations toward unknown behavioral policies [30].
<sup>7</sup>An IRB approval is obtained from North Carolina State University. The use/test of the intelligent tutoring system is overseen by a departmental committee, ensuring it does not risk the academic performance and privacy of the participants.
DICE methods are intrinsically designed to work with offline trajectories rolled out from a mixture of behavioral policies, and existing works have introduced the DICE variants toward specific environmental setups [84, 83, 77, 78, 51, 9]. FQE extrapolates policy returns from the approximated Q-values [31, 40, 36]. There also exist model-based OPE methods [82, 18] that first captures the dynamics underlying the environment, and estimate policy performance by rolling out trajectories under the target policies. A more detailed review of existing OPE methods can be found in Appendix C. Note that these OPE methods have been designed for estimating the environmental returns. In contrast, the objective for OPEHF is to estimate the human returns which may not be strongly correlated with the environmental returns, as they are usually determined under different schemas.
VAEs for OPE and offline RL. There exists a long line of research developing latent models to capture the dynamics underlying environments in the context of offline RL as well as OPE. Specifically, PlaNet [27] uses recurrent neural networks to capture the transitions of latent variables over time. Latent representations learned by such VAE architectures have been used to augment the state space in offline policy optimization to improve sample efficiency, e.g., in Dreamers [26, 28], SOLAR [81] and SLAC [42]. On the other hand, LatCo [65] attempts to improve sample efficiency by searching in the latent space which allows by-passing physical constraints. Also, MOPO [80], COMBO [79], and LOMPO [62] train latent models to quantify the confidence of the environmental transitions as learned from offline data and prevent the policies from following transitions over uncertain regions during policy training. Given that such models are mostly designed for improving sample efficiency in policy optimization/training, we choose to leverage the architecture from [18] for RILR as it is the first work that adapts latent models to the OPE setup.
Reinforcement learning from human feedback (RLHF). Recently, the concept of RLHF has been widely used in guiding RL policy optimization with the HF signals deemed more informative than the environmental rewards [8, 85, 47]. Specifically, they leverage the ranked preference provided by labelers to train a reward model, captured by feed-forward neural networks, that is fused with the environmental rewards to guide policy optimization. However, in this work, we focus on estimating the HF signals that serve as direct evaluation of the RL policies used in human-involved experiments, such as the level of satisfaction (e.g., on a scale 1-10) and the treatment outcome. The reason is that in many scenarios the participants cannot revisit the same procedure multiple times, e.g., patients may not undergo the same surgeries several times and rank the experiences. More importantly, OPEHF's setup is critical when online testing of RL policies may be even prohibited, without sufficient justifications over safety and efficacy upfront, as illustrated by the experiments above.
Reward shaping. Although reward shaping methods [3, 58, 29] pursue similar ideas of decomposing the delayed and/or sparse rewards (e.g., the human return) into immediate rewards, they fundamentally rely on transforming the MDP to such that the value functions can be smoothly captured and high-return state-action pairs can be quickly identified and frequently re-visited. For example, RUDDER [3] leverages the transformed MDP that has expected future rewards equal to zero. Though the optimization objective is consistent between pre- and post-transformed MDPs, this approach likely would not converge to an optimal policy in practice. On the other hand, the performance (i.e., returns) of sub-optimal policies is not preserved across the two MDPs. This significantly limits its use cases toward OPE which requires the returns resulted by sub-optimal policies to be estimated accurately. As a result, such methods are not directly applicable to the OPEHF problem we consider.
Existing OPE methods fall short in estimating HF signals, as HF can be dependent upon various confounders. Thus, in this work, we introduced the OPEHF framework that revived existing OPE methods for estimating human returns, through RILR. The framework was validated over two real-world experiments and one simulation environment, outperforming the baselines in all setups. Although in the future it could be possible to extend OPEHF to facilitate estimating the HF signals needed for updating the policies similar to RLHF, we focused on policy evaluation which helped to isolate the source of improvements; as policy optimization's performance may depend on multiple factors, such as the exploration techniques used as well as the objective/optimizer chosen for updating the policy. Moreover, this work mainly focuses on the scenarios where the human returns are directly provided by the participants. So under the condition where the HF signals are provided by 3-rd parties (e.g., clinicians), non-trivial adaptations over this work may be needed to consider special cases such as conflicting HF signals provided by different sources.
This work is sponsored in part by the AFOSR under award number FA9550-19-1-0169, by the NIH UH3 NS103468 award, and by the NSF CNS-1652544, DUE-1726550, IIS-1651909 and DUE-2013502 awards, as well as the National AI Institute for Edge Computing Leveraging Next Generation Wireless Networks, Grant CNS-2112562. Investigational Summit RC+S systems and technical support provided by Medtronic PLC. Apple Watches were provided by Rune Labs. We thank Stephen L. Schmidt and Jennifer J. Peters from Duke University Department of Biomedical Engineering, as well as Katherine Genty from Duke University Department of Neurosurgery, for the efforts overseeing DBS experiments in the clinic.