SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[1] Saminda Wishwajith Abeyruwan, Laura Graesser, David B D'Ambrosio, Avi Singh, Anish Shankar, Alex Bewley, Deepali Jain, Krzysztof Marcin Choromanski, and Pannag R Sanketi. i-sim2real: Reinforcement learning of robotic policies in tight human-robot interaction loops. In Conference on Robot Learning, pages 212–224. PMLR, 2023.
[2] John R Anderson, C Franklin Boyle, and Brian J Reiser. Intelligent tutoring systems. Science, 228(4698):456-462, 1985.
[3] Jose A Arjona-Medina, Michael Gillhofer, Michael Widrich, Thomas Unterthiner, Johannes Brandstetter, and Sepp Hochreiter. Rudder: Return decomposition for delayed rewards. Advances in Neural Information Processing Systems, 32, 2019.
[4] Alim Louis Benabid. Deep brain stimulation for parkinson's disease. Current opinion in neurobiology, 13(6):696-706, 2003.
[5] Peter Brown, Antonio Oliviero, Paolo Mazzone, Angelo Insola, Pietro Tonali, and Vincenzo Di Lazzaro. Dopamine dependency of oscillations between subthalamic nucleus and pallidum in parkinson's disease. Journal of Neuroscience, 21(3):1033-1038, 2001.
[6] Witney Chen, Lowry Kirkby, Miro Kotzev, Patrick Song, Ro'ee Gilron, and Brian Pepin. The role of large-scale data infrastructure in developing next-generation deep brain stimulation therapies. Frontiers in Human Neuroscience, 15:717401, 2021.
[7] Nicholas C Chesnaye, Vianda S Stel, Giovanni Tripepi, Friedo W Dekker, Edouard L Fu, Carmine Zoccali, and Kitty Jager. An introduction to inverse probability of treatment weighting in observational research. Clinical Kidney Journal, 15(1):14-20, 2022.
[8] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
[9] Bo Dai, Ofir Nachum, Yinlam Chow, Lihong Li, Csaba Szepesvári, and Dale Schuurmans. Coindex: Off-policy confidence interval estimation. Advances in Neural Information Processing Systems, 33:9398-9411, 2020.
[10] Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, José M.F. Moura, Devi Parikh, and Dhruv Batra. Visual Dialog. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
[11] Gunther Deuschl, Carmen Schade-Brittinger, Paul Krack, Jens Volkmann, Helmut Schafer, Kai Bötzel, Christine Daniels, Angela Deutschländer, Ulrich Dillmann, Wilhelm Eisner, et al. A randomized trial of deep-brain stimulation for parkinson's disease. New England Journal of Medicine, 355(9):896-908, 2006.
[12] Mehrdad Farajtabar, Yinlam Chow, and Mohammad Ghavamzadeh. More robust doubly robust off-policy evaluation. In International Conference on Machine Learning, pages 1447-1456. PMLR, 2018.
[79] Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offline model-based policy optimization. Advances in neural information processing systems, 34:28954-28967, 2021.
[80] Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. Advances in Neural Information Processing Systems, 33:14129-14142, 2020.
[81] Marvin Zhang, Sharad Vikram, Laura Smith, Pieter Abbeel, Matthew Johnson, and Sergey Levine. Solar: Deep structured representations for model-based reinforcement learning. In International Conference on Machine Learning, pages 7444-7453. PMLR, 2019.
[82] Michael R Zhang, Thomas Paine, Ofir Nachum, Cosmin Paduraru, George Tucker, Mohammad Norouzi, et al. Autoregressive dynamics models for offline policy evaluation and optimization. In International Conference on Learning Representations, 2020.
[83] Ruiyi Zhang, Bo Dai, Lihong Li, and Dale Schuurmans. Gendice: Generalized offline estimation of stationary values. In International Conference on Learning Representations, 2020.
[84] Shangtong Zhang, Bo Liu, and Shimon Whiteson. Gradientdice: Rethinking generalized offline estimation of stationary values. In International Conference on Machine Learning, pages 11194-11203. PMLR, 2020.
[85] Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019.
A Proofs for Section 2.2
Recall that the per-decision importance sampling (PDIS) [61] estimator follows  $\hat{G}_{PDIS}^{\pi} = \frac{1}{N}\sum_{i=1}^{N-1}\sum_{t=0}^{T-1}\gamma^{t}\omega_{0:t}^{(i)}r_{t}^{\mathcal{H}(i)}$ ; here,  $\omega_{t}^{(i)} = \frac{\pi(a_{t}^{(i)}|s_{t}^{(i)})}{\hat{\beta}(a_{t}^{(i)}|s_{t}^{(i)})}$ , and henceforth  $\omega_{0:t}^{(i)} = \prod_{k=0}^{t}\frac{\pi(a_{k}^{(i)}|s_{k}^{(i)})}{\hat{\beta}(a_{k}^{(i)}|s_{k}^{(i)})}$  are the PDIS weights for offline trajectory  $\tau^{(i)}$ . To simplify our notation, we consider the PDIS estimator defined over a single trajectory, i.e.,
$$
\hat {G} _ {P D I S} ^ {\pi} = \sum_ {t = 0} ^ {T - 1} \gamma^ {t} \omega_ {0: t} r _ {t} ^ {\mathcal {H}}, \tag {6}
$$
as the results can be carried to  $N$  trajectories by multiplying with a factor  $1 / N$ . We omit the superscript-ed  $(i)$ 's in the rest of the proof also for conciseness.
Proof. We start with a lemma from [46].
Lemma 1 ([46]). Given  $X_{t}$  and  $Y_{t}$  as two sequences of random variables. Then
$$
2 \sum_ {t <   k} \mathbb {E} \left[ Y _ {t} Y _ {k} \right] - 2 \sum_ {t <   k} \mathbb {E} \left[ \mathbb {E} \left[ Y _ {t} \mid X _ {t} \right] \mathbb {E} \left[ Y _ {k} \mid X _ {k} \right] \right] \leq \mathbb {V} \left(\sum_ {t} Y _ {t}\right) - \mathbb {V} \left(\sum_ {t} \mathbb {E} \left[ Y _ {t} \mid X _ {t} \right]\right), \tag {7}
$$
where  $\mathbb{V}(\cdot)$  refers to the variances.
Now, if we set  $Y_{t} = r_{t}^{\mathcal{H}}\omega_{0:T}$  and  $X_{t} = \tau_{0:t}$  which is a segment (from  $t = 0$  to  $t = t$ ) of an offline trajectory  $\tau$ , it is sufficient to prove Proposition 1 by proving that for any  $1 \leq t + 1 \leq k \leq T$ ,
$$
\mathbb {E} \left[ r _ {t} ^ {\mathcal {H}} r _ {k} ^ {\mathcal {H}} \omega_ {0: t} \omega_ {0: k} \right] \leq \mathbb {E} \left[ r _ {t} ^ {\mathcal {H}} r _ {k} ^ {\mathcal {H}} \omega_ {0: T - 1} \omega_ {0: T - 1} \right]; \tag {8}
$$
$$
\begin{array}{l} \mathbb {E} \left[ r _ {t} ^ {\mathcal {H}} r _ {k} ^ {\mathcal {H}} \omega_ {0: t} \omega_ {0: k} \right] = \mathbb {E} \left[ \mathbb {E} \left[ Y _ {t} \mid X _ {t} \right] \mathbb {E} \left[ Y _ {k} \mid X _ {k} \right] \right] (9) \\ \leq \mathbb {E} \left[ Y _ {t} Y _ {k} \right] (10) \\ = \mathbb {E} \left[ r _ {t} ^ {\mathcal {H}} r _ {k} ^ {\mathcal {H}} \omega_ {0: T - 1} \omega_ {0: T - 1} \right]. (11) \\ \end{array}
$$
Applying the law of total expectations to (8), i.e.,
$$
\mathbb {E} \left[ \mathbb {E} \left[ r _ {t} ^ {\mathcal {H}} r _ {k} ^ {\mathcal {H}} \omega_ {0: t} \omega_ {0: k} | \tau_ {0: t} \right] \right] \leq \mathbb {E} \left[ \mathbb {E} \left[ r _ {t} ^ {\mathcal {H}} r _ {k} ^ {\mathcal {H}} \omega_ {0: T - 1} \omega_ {0: T - 1} | \tau_ {0: t} \right] \right]; \tag {12}
$$
then it is sufficient to show
$$
\mathbb {E} \left[ r _ {t} ^ {\mathcal {H}} r _ {k} ^ {\mathcal {H}} \omega_ {0: t} \omega_ {0: k} \mid \tau_ {0: t} \right] \leq \mathbb {E} \left[ r _ {t} ^ {\mathcal {H}} r _ {k} ^ {\mathcal {H}} \omega_ {0: T - 1} \omega_ {0: T - 1} \mid \tau_ {0: t} \right]. \tag {13}
$$
We start by showing that
$$
\begin{array}{l} \mathbb {E} \left[ r _ {t} ^ {\mathcal {H}} r _ {k} ^ {\mathcal {H}} \omega_ {0: t} \omega_ {0: k} | \tau_ {0: t} \right] = \omega_ {0: t} ^ {2} \mathbb {E} \left[ r _ {t} ^ {\mathcal {H}} | \tau_ {0: t} \right] \mathbb {E} \left[ r _ {k} ^ {\mathcal {H}} \omega_ {t + 1: k} | \tau_ {0: t} \right] (14) \\ = \omega_ {0: t} ^ {2} \mathbb {E} \left[ r _ {t} ^ {\mathcal {H}} \mid \tau_ {0: t} \right] \mathbb {E} \left[ r _ {k} ^ {\mathcal {H}} \omega_ {t + 1: T - 1} \mid \tau_ {0: t} \right] \mathbb {E} \left[ \omega_ {t + 1: T - 1} \mid \tau_ {0: t} \right]. (15) \\ \end{array}
$$
The transition above follows from the fact that the likelihood ratio  $\omega_{0:t}$  is a martingale as shown in [41, 46], i.e.,  $\mathbb{E}[\omega_{0:T-1}|\tau_{0:t}] = \omega_{0:t}$ , which implies  $\mathbb{E}[\omega_{t+1:T-1}] = 1$  and therefore
$$
\begin{array}{l} \mathbb {E} \left[ r _ {k} ^ {\mathcal {H}} \omega_ {t + 1: k} \mid \tau_ {0: t} \right] (16) \\ = \mathbb {E} \left[ r _ {k} ^ {\mathcal {H}} \omega_ {t + 1: k} \mid \tau_ {0: t} \right] \mathbb {E} \left[ \omega_ {k + 1: T - 1} \mid \tau_ {0: t} \right] (17) \\ = \mathbb {E} \left[ r _ {k} ^ {\mathcal {H}} \omega_ {t + 1: T - 1} \mid \tau_ {0: t} \right]. (18) \\ \end{array}
$$
Note that we keep using the notation  $\mathbb{E}[r_t^{\mathcal{H}}|\tau_{0:t}]$  in (14) and (15), due to the setting of HMDP that  $r_t^{\mathcal{H}}$  is a variable sampled from the distribution  $R^{\mathcal{H}}(\cdot |s_t,a_t)$  given  $(s_t,a_t)$ ; see Section 2.1.
Since  $\tau_{0:t}$  is given,  $r_k^{\mathcal{H}}$  and  $\omega_{t+1:T-1}$  are equivalent to  $r_{k-t-1}^{(j)}$  and  $\omega_{0:T-t-2}^{(j)}$  for some other trajectory  $\tau^{(j)} \sim \rho^\beta$ . Also given the statement that  $\omega_{0:T-t-2}^{(j)}$  and  $r_{k-t-1}^{(j)}\omega_{0:T-t-2}^{(j)}$  are positively correlated, it follows from (15) that
$$
\mathbb {E} \left[ r _ {t} ^ {\mathcal {H}} r _ {k} ^ {\mathcal {H}} \omega_ {0: t} \omega_ {0: k} \mid \tau_ {0: t} \right] \tag {19}
$$
$$
= \omega_ {0: t} ^ {2} \mathbb {E} \left[ r _ {t} ^ {\mathcal {H}} \mid \tau_ {0: t} \right] \mathbb {E} \left[ r _ {k} ^ {\mathcal {H}} \omega_ {t + 1: T - 1} \mid \tau_ {0: t} \right] \mathbb {E} \left[ \omega_ {t + 1: T - 1} \mid \tau_ {0: t} \right] \tag {20}
$$
$$
\leq \omega_ {0: t} ^ {2} \mathbb {E} \left[ r _ {t} ^ {\mathcal {H}} \mid \tau_ {0: t} \right] \mathbb {E} \left[ r _ {k} ^ {\mathcal {H}} \omega_ {t + 1: T - 1} \omega_ {t + 1: T - 1} \mid \tau_ {0: t} \right] \tag {21}
$$
$$
= \mathbb {E} \left[ r _ {t} ^ {\mathcal {H}} r _ {k} ^ {\mathcal {H}} \omega_ {0: T - 1} \omega_ {0: T - 1} \mid \tau_ {0: t} \right], \tag {22}
$$
which justifies (13) and completes the proof.
B Derivation of the ELBO (2)
Note that unlike the ELBO in [18], the VLM-H includes an additional component that estimates the human return  $G_{0:T}^{\mathcal{H}}$  of each trajectory, i.e.,
$$
\begin{array}{l} \log p _ {\phi} \left(s _ {0: T}, r _ {0: T - 1}, G _ {0: T} ^ {\mathcal {H}}\right) (23) \\ = \log \int_ {z _ {1: T} \in \mathcal {Z}} p _ {\phi} \left(s _ {0: T}, z _ {1: T}, r _ {0: T - 1}, G _ {0: T} ^ {\mathcal {H}}\right) d z (24) \\ = \log \int_ {z _ {1: T} \in \mathcal {Z}} \frac {p _ {\phi} \left(s _ {0 : T} , z _ {1 : T} , r _ {0 : T - 1} , G _ {0 : T} ^ {\mathcal {H}}\right)}{q _ {\psi} \left(z _ {0: T} \mid s _ {0: T} , a _ {0: T - 1}\right)} q _ {\psi} \left(z _ {0: T} \mid s _ {0: T}, a _ {0: T - 1}\right) d z (25) \\ \geq \mathbb {E} _ {q _ {\psi}} \left[ \log p \left(z _ {0}\right) + \log p _ {\phi} \left(s _ {0: T}, z _ {1: T}, r _ {0: T - 1} \mid z _ {0}\right) + \log p _ {\phi} \left(G _ {0: T} ^ {\mathcal {H}} \mid z _ {T}\right) - \log q _ {\psi} \left(z _ {0: T} \mid s _ {0: T}, a _ {0: T - 1}\right) \right] (26) \\ = \mathbb {E} _ {q _ {\psi}} \left[ \log p (z _ {0}) + \log p _ {\phi} (s _ {0} | z _ {0}) + \log p _ {\phi} \left(G _ {0: T} ^ {\mathcal {H}} | z _ {T}\right) + \sum_ {t = 1} ^ {T} \log p _ {\phi} \left(s _ {t}, z _ {t}, r _ {t - 1} | z _ {t - 1}, a _ {t - 1}\right) \right. \\ \left. - \log q _ {\psi} \left(z _ {0} \mid s _ {0}\right) - \sum_ {t = 1} ^ {T} \log q _ {\psi} \left(z _ {t} \mid z _ {t - 1}, a _ {t - 1}, s _ {t}\right) \right] (27) \\ = \mathbb {E} _ {q _ {\psi}} \left[ \log p (z _ {0}) - \log q _ {\psi} (z _ {0} | s _ {0}) + \log p _ {\phi} (s _ {0} | z _ {0}) + \log p _ {\phi} \left(G _ {0: T} ^ {\mathcal {H}} \mid z _ {T}\right) - \sum_ {t = 1} ^ {T} \log q _ {\psi} \left(z _ {t} \mid z _ {t - 1}, a _ {t - 1}, s _ {t}\right) \right. \\ \left. + \sum_ {t = 1} ^ {T} \log \left(p _ {\phi} \left(s _ {t} \mid z _ {t}\right) p _ {\phi} \left(r _ {t - 1} \mid z _ {t}\right) p _ {\phi} \left(z _ {t} \mid z _ {t - 1}, a _ {t - 1}\right)\right) \right] (28) \\ = \mathbb {E} _ {q _ {\psi}} \left[ \log p _ {\phi} \left(G _ {0: T} ^ {\mathcal {H}} \mid z _ {T}\right) + \sum_ {t = 0} ^ {T} \log p _ {\phi} \left(s _ {t} \mid z _ {t}\right) + \sum_ {t = 1} ^ {T} \log p _ {\phi} \left(r _ {t - 1} \mid z _ {t}\right) \right. \\ \left. - K L \left(q _ {\psi} \left(z _ {0} \mid s _ {0}\right) \mid p \left(z _ {0}\right)\right) - \sum_ {t = 1} ^ {T} K L \left(q _ {\psi} \left(z _ {t} \mid z _ {t - 1}, a _ {t - 1}, s _ {t}\right) \mid p _ {\phi} \left(z _ {t} \mid z _ {t - 1}, a _ {t - 1}\right)\right) \right]. (29) \\ \end{array}
$$
Note that to simplify our presentation, we omit  $\phi, \psi, \rho^{\beta}$  as part of the conditional terms in the joint likelihoods. The transition from (25) to (26) follows Jensen's inequality.
C Brief Overview of Existing OPE methods
C.1 Importance Sampling (IS) and per-decision IS (PDIS)
IS refers to a statistical technique that can calculate the expectation of a function  $f(x)$  w.r.t. an unknown distribution  $p(x)$  using a given distribution  $q(x)$  through re-weighting, i.e.,
$$
\mathbb {E} _ {p} [ f (x) ] = \mathbb {E} _ {q} \left[ \frac {f (x) p (x)}{q (x)} \right]. \tag {30}
$$
This technique can be applied in the context of OPE by setting  $f(x)$  as the accumulated return  $G_{0:T}$ ,  $p$  as the trajectory distribution  $\rho^{\pi}$  over the target policy  $\pi$ , and  $q$  as the trajectory distribution  $\rho^{\beta}$  over the behavioral policy  $\beta$ .
According to [61], the vanilla IS estimator follows
$$
\hat {G} _ {I S} ^ {\pi} = \frac {1}{N} \sum_ {i = 1} ^ {N - 1} \omega_ {0: T - 1} ^ {(i)} G _ {0: T} ^ {(i)}, \tag {31}
$$
where  $\omega_{0:T-1}^{(i)} = \prod_{t=0}^{T-1} \frac{\pi(a_t|s_t)}{\beta(a_t|s_t)}$  refers to the IS weight, and  $G_{0:T}^{(i)}$  is the return for the  $i$ -th (out of  $N$ ) offline trajectory. On the other hand, the PDIS estimator follows
$$
\hat {G} _ {P D I S} ^ {\pi} = \frac {1}{N} \sum_ {i = 1} ^ {N - 1} \sum_ {t = 0} ^ {T - 1} \gamma^ {t} \omega_ {0: t} ^ {(i)} r _ {t} ^ {(i)}, \tag {32}
$$
with  $\omega_{0:t}^{(i)} = \prod_{k=0}^{t} \frac{\pi(a_k|s_k)}{\beta(a_k|s_k)}$  being the PDIS weight,  $r_t^{(i)}$  is the environmental reward obtained at the  $t$ -th step of the  $i$ -th offline trajectory, and  $\gamma$  is the discounting factor.
C.2 Doubly Robust (DR)
DR attempts to reduce the variance of IS estimation by introducing the value function approximation, which trades off estimation variance by making the estimation biased. The non-recursive definition of DR estimation is provided in [71], i.e.,
$$
\hat {G} _ {D R} ^ {\pi} = \frac {1}{N} \sum_ {i = 1} ^ {N - 1} \sum_ {t = 0} ^ {T - 1} \gamma^ {t} \omega_ {0: t} ^ {(i)} r _ {t} ^ {(i)} - \frac {1}{N} \sum_ {i = 1} ^ {N - 1} \sum_ {t = 0} ^ {T - 1} \left(\gamma^ {t} \omega_ {0: t} ^ {(i)} r _ {t} ^ {(i)} Q ^ {\pi} \left(s _ {t} ^ {(i)}, a _ {t} ^ {(i)}\right) - \omega_ {0: t - 1} ^ {(i)} V ^ {\pi} \left(s _ {t} ^ {(i)}\right)\right); \tag {33}
$$
here,  $Q^{\pi}(\cdot ,\cdot)$  and  $V^{\pi}(\cdot)$  are the Q-function and value function, respectively, over the target policy, while  $s_t^{(i)}$  and  $a_{t}^{(i)}$  are the state and action taken at the  $t$ -th step of the  $i$ -th offline trajectory.
C.3 Distributional Correction Estimation (DICE)
DICE aims to estimate the propensity of the target policy to visit particular state-action pairs relative to their likelihood of appearing in the offline trajectories, i.e.,
$$
\hat {G} _ {D I C E} ^ {\pi} = \mathbb {E} _ {(s, a, r) \sim \rho^ {\beta}} \left[ \frac {d ^ {\pi} (s , a)}{d ^ {\beta} (s , a)} \cdot r \right], \tag {34}
$$
where  $\frac{d^{\pi}(s,a)}{d^{\beta}(s,a)}$  is the distribution correction ratio. Existing DICE variants [84, 83, 78, 51, 9] seek to approximate the ratio without knowledge of  $d^{\pi}$  or  $d^{\beta}$ , while a recent work has summarized that all existing variants can be formulated as regularized Lagrangians of the same linear program [78], unifying the choice of regularizations and constraints used in their original objectives.
C.4 Fitted Q-Evaluation (FQE)
According to [40], FQE approximates the Q-function over the target policy by minimizing the loss
$$
\min  _ {\kappa} \mathbb {E} _ {\left(s _ {t}, a _ {t}, r _ {t}\right) \sim \rho^ {\beta}} \left[ \left(Q \left(s _ {t}, a _ {t}; \kappa\right) - y _ {t}\right) ^ {2} \right], \tag {35}
$$
$$
\text {s . t .} \quad y _ {t} = r _ {t} + \gamma Q \left(s _ {t + 1}, \pi \left(s _ {t + 1}\right); \kappa\right); \tag {36}
$$
here, the approximated Q-function is parameterized by  $\kappa$ . Note that it is different from the objective for classic Q-learning, or fitted Q-iteration, by using  $\pi(s_{t+1})$  instead of  $\max_a Q(s_{t+1}, a; \kappa)$  in the target  $y_t$ .
D Additional Experimental Details and Results
In this section we provide additional details that are supplement to the discussions in Section 3.
D.1 Definition of the OPE Metrics
Mean Absolute error (MAE). MAE is defined as the absolute difference between the actual return and estimated return of a policy:  $MAE = |V^{\pi} - \hat{V}^{\pi}|$ ; here,  $V^{\pi}$  is the actual value of the policy  $\pi$ , and  $\hat{V}^{\pi}$  is the estimated value of  $\pi$ .
Rank correlation. Rank correlation measures the Spearman's rank correlation coefficient between the ordinal rankings of the estimated returns and actual returns across policies, i.e.,
$\rho = \frac{Cov(\mathrm{rank}(V_{1:P}^{\pi}),\mathrm{rank}(\hat{V}_{1:P}^{\pi}))}{\sigma(\mathrm{rank}(V_{1:P}^{\pi}))\sigma(\mathrm{rank}(\hat{V}_{1:P}^{\pi}))}$ , where  $\mathrm{rank}(V_{1:P}^{\pi})$  is the ordinal rankings of the actual returns, and  $\mathrm{rank}(\hat{V}_{1:P}^{\pi})$  is the ordinal rankings of the OPE-estimated returns.
Regret@1. Regret@1 is the (normalized) difference between the value of the actual best policy, against the value of the policy associated with the best OPE-estimated return, which is defined as  $(\max_{i\in 1:P}V_i^\pi -\max_{j\in \mathrm{best}(1:P)}V_j^\pi) / \max_{i\in 1:P}V_i^\pi$ , where  $\mathrm{best}(1:P)$  denotes the index of the best policy over the set of  $P$  policies as measured by estimated values  $\hat{V}^{\pi}$ .
D.2 Implementations and Hyper-parameters
VLM-H. Both encoder  $q_{\psi}(z_t|z_{t-1}, a_{t-1}, s_t)$  and decoder  $p_{\phi}(z_t|z_{t-1}, a_{t-1})$  of VLM-H are configured to be LSTMs with 64 nodes for the hidden states, followed by two fully connected layers with 128 and 64 nodes each. All other parts of the encoder and decoder, including  $q_{\psi}(z_0|s_0), p_{\phi}(G_{0:T}^{\mathcal{H}}|z_T), p_{\phi}(s_t|z_t), p_{\phi}(r_{t-1}|z_t)$ , are multi-layered perceptrons (MLPs) with two fully connected layers with 128 and 64 nodes each. The regularization term introduced in [18] is added to the ELBO (2) with a scaling factor  $C'$  balancing the scale of the two terms, which is applied over the LSTM states between  $q_{\psi}(z_t|z_{t-1}, a_{t-1}, s_t)$  and  $p_{\phi}(z_t|z_{t-1}, a_{t-1})$ . The scale of this regularization is selected from  $C' = \{1e-04, 5e-03, 1e-03, 5e-02, 1e-02, 0.1, 1., 2., 5.\}$ . Learning rate is tuned by grid search from \{0.003, 0.001, 0.0007, 0.0005, 0.0003, 0.0001, 0.00005\}. Exponential decay is applied to the learning rate, which decays the learning rate by 0.997 every iteration. The total number of training epochs is set to be 20 and minibatch size set to 64. Adam optimizer is used to perform gradient descent.  $L_2$  weight decay with coefficient 0.001 and batch normalization are applied to all hidden fully connected layers. The top-10 models/checkpoints achieving the highest ELBOs proceed to the RILR step.
Downstream OPE Estimators. The implementations of downstream OPE estimators are built on top of  $[36]^8$ , which provides the functionalities for using per-decision IS with behavioral policy estimation [30], DR [71], DICE [78] or FQE [40] to process the trajectories with reconstructed IHRs and estimate human returns.
Code Availability and Computational Resources. Code implementations for all three components above can be found in the supplementary materials attached. All experimental workloads are distributed among 4 Nvidia RTX A5000 24GB and 3 Nvidia Quadro RTX 6000 24GB graphics cards.
D.3 More on the Adaptive Neurostimulation Experiment - Deep Brain Stimulation (DBS)
DBS Hardware and Implementation. The internal pulse generator (IPG) is part of the Medtronic's Summit RC+S system. Moreover, Summit software APIs are provided by Medtronic to be leveraged to implement the RL policies to be tested in the clinic. The overall pipeline is shown in Figure 5. Specifically, each control cycle (an MDP step) takes 2 seconds, where the IPG collects local field potentials (LFPs) from the basal ganglia (BG) region of the participant's brain at a sampling rate of  $500\mathrm{Hz}$ . Then, the LFPs are sent to the research tablet for signal processing, after which the beta-band power spectral densities (beta powers) are calculated and used to constitute the MDP state and reward signals. The beta powers are calculated by applying fast Fourier transform (FFT), with 512 bin width, to the LFPs collected in the past 2 seconds. The beta-band usually refers to the frequency band of  $13 - 35\mathrm{Hz}$ , however, clinicians and electrical engineers had customized the range for each specific
$^{8}$ Original implementation obtained from https://github.com/google-research/google-research/tree/master/policy_eval, following the Apache License v2.0
Figure 5: Implementation of RL control on the research tablet over the API provided by the manufacturer of the DBS device. During clinical testing, the research tablet evaluates the RL policies which determine the stimulation amplitudes to be used in each step (every 2 seconds). Then, by communicating with the internal pulse generator (IPG) through wireless telemetry, the IPG adjust the stimulus accordingly and sends back the local field potentials (LFPs) which are used to determine the MDP state and reward for the next step.
Figure 6: An illustration towards the IPG electrodes are placed in the basal ganglia (BG) region of the brain - 4-contact electrodes are placed in the subthalmic nucleus (STN) and globus pallidus (GP).
patient, in order to allow beta powers to better capture the irregular neuronal behaviors. The specific range for each patient can be found in the paragraph below (i.e., patient characteristics). According to the latest LFP and beta power signals, the RL policy adjusts the amplitude of the electrical stimulus to be used for the next cycle, which is then sent back to the IPG through wireless telemetry. In the BG region of participant's brain, as illustrated in Fig. 6, 4-contact electrodes are placed in the subthalmic nucleus (STN) and globus pallidus (GP) regions. Monopolar stimulation was delivered on a single contact on each lead (with the case serving as counter-electrode). The two contacts surrounding the stimulation contact were used for sensing LFPs (i.e., sandwich sensing).
The IPG device was already approved for clinical research use by the FDA. The changes made to the software program were also approved by the FDA via an investigational device exemption (IDE). Corresponding IRB approvals were obtained from Duke University Health System. All participants were compensated as per institutional and administrative guidelines.
Patient Characteristics. As discussed in Section 3.1, all PD participants have unique characteristics toward varied PD severity and symptoms, which can be found in [22].