SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The MDP Setup. The state and action space, as well as rewards are briefly introduced in Section 3.1. Here we provide the details that are omitted in the main paper. Each discrete step in MDP corresponds to a control cycle which lasts 2 seconds; see the DBS hardware and implementation paragraph above. Each episode lasts at least  $T = 300$  steps (i.e., the horizon), or more than 10 minutes in the real world. Specifically, states are constituted by a historical sequence of beta powers obtained from past 10 steps, following a similar setup as in [20, 52]. The actions  $a \in \mathcal{A}$  are in  $[\xi, 100\%] \subset \mathbb{R}, 0\% \leq \xi < 100\%$ , which represent the percentage over the maximum amplitude of the stimulus the IPG can employ (determined by clinicians). Specifically,  $\xi = 40\%$  for patient #0 and  $\xi = 60\%$  for patients #1-#3, to ensure safety and minimum efficacy of DBS during clinical testing. The environmental rewards are
Table 4: Raw results of the adaptive neurostimulation experiment from each patient (Patient #0).
Fusion	IS Rescale	OPEHF (our)	Fusion	DR Rescale	OPEHF (our)	Ablation VLM-H
MAE	0.96±0.0	0.74±0.0	0.17±0.01	2.21±0.38	1.65±0.07	0.09±0.01	0.97±0.02
Rank	0.65±0.01	0.65±0.01	0.74±0.07	0.43±0.05	0.63±0.08	0.58±0.1	0.73±0.09
Regret@1	0.05 ± 0.0	0.05 ± 0.0	0.04 ± 0.01	0.03 ± 0.38	0.02 ± 0.07	0.03 ± 0.01	0.02±0.02
Fusion	DICE Rescale	OPEHF (our)	Fusion	FQE Rescale	OPEHF (our)
MAE	4.59±0.17	4.0±0.2	0.98±0.01	1.64±0.03	0.63±0.0	0.27±0.0
Rank	0.52±0.05	0.56±0.08	0.62±0.09	0.48±0.07	0.49±0.02	0.58±0.07
Regret@1	0.05 ± 0.17	0.02 ± 0.2	0.03 ± 0.01	0.02 ± 0.03	0.05 ± 0.0	0.03 ± 0.0
Table 5: Raw results of the adaptive neurostimulation experiment from each patient (Patient #1).
Fusion	IS Rescale	OPEHF (our)	Fusion	DR Rescale	OPEHF (our)	Ablation VLM-H
MAE	0.59±0.0	0.44±0.0	0.43±0.01	2.11±0.14	1.03±0.08	0.27±0.03	0.98±0.01
Rank	0.58±0.06	0.52±0.08	0.66±0.08	0.74±0.07	0.6±0.05	0.87±0.02	0.58±0.03
Regret@1	0.21±0.0	0.02±0.0	0.02±0.01	0.01±0.14	0.0±0.08	0.02±0.03	0.03±0.01
Fusion	DICE Rescale	OPEHF (our)	Fusion	FQE Rescale	OPEHF (our)
MAE	0.47±0.01	0.45±0.01	0.87±0.01	2.53±0.07	1.05±0.09	0.53±0.02
Rank	0.73±0.05	0.72±0.09	0.9±0.03	0.48±0.12	0.76±0.01	0.74±0.06
Regret@1	0.03±0.01	0.03±0.01	0.03±0.01	0.01±0.07	0.04±0.09	0.02±0.02
defined following, i.e.,
$$
R \left(s, a, s ^ {\prime}\right) = \left\{ \begin{array}{l l} - 1 - 0. 2 a & \text {i f l a t e s t b e t a p o w e r i s a b o v e a t h r e s h o l d ,} \\ - 0. 2 a & \text {i f l a t e s t b e t a p o w e r i s b e l o w a t h r e s h o l d ;} \end{array} \right. \tag {37}
$$
here,  $-0.2a$  is the penalty for stimulating with higher-than-needed amplitudes, to preserve the runtime of the IPG device (powered by a rechargeable battery) between recharges, and an additional  $-1$  is given if stimulating with the amplitude determined by the policy cannot reduce beta power to below a threshold specific to each patient. The thresholds above are set to be the lower  $20\%$  quantile of the beta powers observed from initial exploration (introduced below). The human return at the end of each episode is quantified as a  $50\%-25\%-25\%$  weighted sum over the patient rating, hand grasp speed captured from performing the maneuver (rapid and full extension and close of all fingers) for the bradykinesia test, as well as the proportional length of the session where the patient displays tremor (as captured by a wearable accelerometer).
Collection of Offline Trajectories. Mixed types of controllers are used for initial exploration of the environment (for obtaining target policies introduced below), and the resulting trajectories are used as the offline training data for OPEHF. Specifically, a random controller that uniformly sample the stimulation amplitudes from  $[\xi, 100\%]$ , and a proportional-integral (PI) controller tuned by clinicians and electrical engineers, are used at the initial phase of clinical testing. They help identify the state-action regions that can lead to non-trivial returns. Then, such data are used to train RL policies – following deep deterministic policy gradient (DDPG), the deterministic counterpart of the soft actor-critic (SAC) – for initial exploration, most of which can only obtain up to  $60\%$  returns as obtained by the target policies below on average. This makes OPEHF challenging considering the limited exploration coverage provided by the offline trajectories.
Target Policies. A total of 6 target policies, to be evaluated by OPEHF, are considered for each patient. One of them is an open-loop policy that always stimulate with the lowest possible amplitude
Table 6: Raw results of the adaptive neurostimulation experiment from each patient (Patient #2).
Fusion	IS Rescale	OPEHF (our)	Fusion	DR Rescale	OPEHF (our)	Ablation VLM-H
MAE	0.63±0.03	0.7±0.02	0.33±0.01	2.17±0.3	1.11±0.16	0.3±0.01	0.97±0.02
Rank	0.65±0.11	0.71±0.05	0.9±0.03	0.52±0.09	0.59±0.04	0.84±0.03	0.6±0.0
Regret@1	0.05±0.03	0.08±0.02	0.08±0.01	0.03±0.3	0.08±0.16	0.1±0.01	0±0.02
Fusion	DICE Rescale	OPEHF (our)	Fusion	FQE Rescale	OPEHF (our)
MAE	5.44±0.55	3.52±0.12	0.66±0.02	1.6±0.06	0.81±0.02	0.38±0.01
Rank	0.66±0.09	0.48±0.08	0.85±0.05	0.6±0.05	0.67±0.04	0.93±0.01
Regret@1	0.11±0.55	0.08±0.12	0.1±0.02	0.08±0.06	0.08±0.02	0.08±0.01
Table 7: Raw results of the adaptive neurostimulation experiment from each patient (Patient #3).
Fusion	IS Rescale	OPEHF (our)	Fusion	DR Rescale	OPEHF (our)	Ablation VLM-H
MAE	0.94±0.01	0.76±0.01	0.24±0.0	1.69±0.51	0.69±0.06	0.17±0.01	0.94±0.01
Rank	0.42±0.11	0.16±0.03	0.71±0.05	0.48±0.05	0.42±0.11	0.62±0.14	0.45±0.06
Regret@1	0.21±0.01	0.01±0.01	0.04±0.0	0.35±0.51	0.1±0.06	0.04±0.01	0.03±0.01
Fusion	DICE Rescale	OPEHF (our)	Fusion	FQE Rescale	OPEHF (our)
MAE	0.41±0.02	1.15±0.07	1.03±0.04	1.56±0.09	0.53±0.02	0.19±0.01
Rank	0.28±0.05	0.22±0.03	0.66±0.07	0.48±0.05	0.29±0.02	0.52±0.01
Regret@1	0.02±0.02	0.03±0.07	0.03±0.04	0.1±0.09	0.05±0.02	0.03±0.01
$(\xi \%)$ , benchmarking the worst-case control performance. The other 5 target DBS policies are all trained using DDPG. Specifically, each is trained over a growing experience buffer containing the data collected from the latest trials, in addition to the fixed set of offline trajectories above; this would lead to a set of target policies with varied performance, as the policies obtained in the later stage tend to be more optimal since the environment has been explored to a broader extent, compared to the policies obtained at earlier stages. Moreover, the policies have to demonstrate at least moderate control efficacy (as determined by clinicians), in order to be considered as target policies; since the ground-truth return of each target policy is determined by averaging the human returns over more than 10 sessions of clinical testing ( $>100$  minutes in total). The raw human returns of all target policies over each patient can be found in Tables 8 to 11, as well as in the legends of Figure 4. Note that, for each patient, there may exist a few policies that lead to close returns, due to the testing limitations introduced above, i.e., the use of  $\xi$  as well as the safety protocol for long-term testing (introduced above).
Correlation between beta power and patient feedback. As discussed in Section 3.1, existing research [56, 37, 5, 76] have found inconsistency between the beta power and patient feedback, due to confounders related to patient's specific characteristics as well as varied symptoms caused by PD. Figure 7 also shows that only low-to-moderate correlations can be found between beta power and the HF signals that are used to define the human returns in general. This makes OPEHF more challenging as the environmental signals may contain limited information useful for extrapolating human returns.
D.4 More on the Intelligent Tutoring Experiment
Experimental Setup. All the students follow a 4-step procedure, i.e., (i) study with textbook, (ii) pre-test, (iii) study with the intelligent tutoring system, (iv) post-test. The RL policies are deployed only in step (iii) following the MDP setup introduced in Section 3.2. The human returns are quantified
(a) Low correlations are found between beta power and the left hand grasp speeds as result of the bradykinesia test.
(b) Low correlations are found between beta power and the right hand grasp speeds as result of the bradykinesia test.
(c) Low-to-moderator correlations over 3 patients, and strong correlation over 1 patient, are found between beta power and the tremor severity quantified as the total time (proportional to the horizon) when patients experience tremor (captured by wearable accelerometer) in each testing session.
Figure 7: Correlation between the beta power and HF provided by patients from all clinical sessions.
(d) Low-to-moderator correlations over 3 patients, and strong correlation over 1 patient, are found between beta power and the rating (1-10) provided by the patients at the end of each testing session.
as the normalized gain over the difference between the scores from post-test and pre-test, i.e.,
$$
G _ {0: T} ^ {\mathcal {H}} = \frac {\text {s c o r e} _ {\text {p o s t}} - \text {s c o r e} _ {\text {p r e}}}{\sqrt {1 - \text {s c o r e} _ {\text {p r e}}}}. \tag {38}
$$
Specifically, pre-test consists of a total of 14 single- and multiple-principle problems. No feedback are given following the pre-test, nor are they allowed to re-visit any questions from the test. After working with the intelligent tutoring system, the students take the post-test which contain 20 problems, 14 of which are isomorphic to the pre-test and the remaining ones are non-isomorphic multiple-principle problems. Both tests were auto-graded following a specific rubric.
More on the MDP States. As introduced in Section 3.2, the states are constituted by 140 expert-selected features; each fall into one of the five categories including  $(i)$  autonomy (10 features),
Table 8: Environmental and human returns of all the target policies pertaining to Patient #0, from the adaptive neurostimulation experiment.
Target Policy	Environmental Returns	Human Returns (x10)
#0	-64.18	182.12
#1	-128.96	187.36
#2	-131.37	174.78
#3	-134.84	182.23
#4	-129.82	158.04
#5	-80.69	104.8
Table 9: Environmental and human returns of all the target policies pertaining to Patient #1, from the adaptive neurostimulation experiment.
Target Policy	Environmental Returns	Human Returns (x10)
#0	-144.51	240.47
#1	-155.0	234.51
#2	-156.53	233.78
#3	-135.75	229.01
#4	-47.19	229.38
#5	-86.13	186.45
capturing the effort spent on solving problems in the intelligent tutoring system, such as the number of times the student revisit a problem, (ii) temporal information (29 features), including the time spent on each problem etc., (iii) problem-specific (35 features), such as the difficulty of each problem, (iv) performance (57 features), such as the grade students received on each problem, (v) hint-related (11 features), including the total amount of hints requested, if applicable.
Target Policies. A total of four target policies are considered. One of the policies randomly selects an action at each state, which benchmarks the worst-case performance. The other 3 policies are trained using deep Q-network (DQN) with different learning rates, from  $\{1\mathrm{e} - 3,1\mathrm{e} - 4,1\mathrm{e} - 5\}$ .
E A Challenging Simulation Environment: Visual Q&A Dialogue
We also consider the visual Q&A dialogue environment [10], following the recent framework [66] that generalizes offline RL towards language generation tasks. In this environment, language generation agents are trained to ask questions over the image captions that are given (without showing the actual images), in order to gain as much information from the responses (returned by environment) that are useful to characterize the key elements contained in the underlying image.
Overall Setup. Though this environment does not involve actually HF, the policies' performance intrinsically pertain to two types of returns as per the setup from [66]. Specifically, the immediate rewards, used to train the language generation policy (captured by generative pre-trained transformer version 2, or GPT2) to asks questions, are discrete, i.e., a  $-1$  reward is given at each step until most key elements can be inferred from the dialogue, and an additional  $-1$  reward is given if the questions asked by the agent lead to responses that only contain trivial information towards extrapolating the image characteristics (such as the responses that only contain a 'yes/no'). In contrast, a separate pre-trained referee model is used to determine when sufficient elements in the underlying images can be captured from the dialogue history and terminates the episode, which maps the dialogue history to embeddings that represent the features pertaining to the image elements that can be extrapolated from the dialogue, followed by scoring the dialogue with relative percentile ranking of the ground truth image's distance from the predicted embedding among a set of images taken from the evaluation set provided by [66, 10], i.e., an episode is terminated if the percentile ranking is improved by at least  $50\%$  compared to the percentile ranking before the dialogue begins,  $(1 - p_T) \geq .5 \cdot (1 - p_0)$ , where  $p_T$  is the ground truth image's percentile rank at dialogue's last turn and  $p_0$  is the ground truth image's percentile rank at the beginning of the dialogue, when only the image caption is observed. As
Table 10: Environmental and human returns of all the target policies pertaining to Patient #2, from the adaptive neurostimulation experiment.
Target Policy	Environmental Returns	Human Returns (x10)
#0	-142.20	160.42
#1	-79.18	163.55
#2	-155.11	158.53
#3	-152.65	165.62
#4	-154.66	161.03
#5	-80.63	69.42
Table 11: Environmental and human returns of all the target policies pertaining to Patient #3, from the adaptive neurostimulation experiment.
Target Policy	Environmental Returns	Human Returns (x10)
#0	-60.78	154.89
#1	-83.83	66.20
#2	-126.97	142.45
#3	-154.83	129.57
#4	-147.44	142.29
#5	-165.74	143.39
a result, the former type of rewards are considered as the environmental rewards as per the OPEHF setup, and the latter one is closely related to the human returns as it is only provided at the end of each episode and are obtained following a different schema. Specifically, we consider  $p_0 - p_T$  as the synthetic human return, which quantifies the improvement over the percentile ranking throughout the dialogue. More details on the environmental reward function and the percentile ranking generated by the referee model can be found in [66].
MDP Formulation. Consequently, in this experiment we compare OPEHF's performance against baselines, toward predicting the percentile rankings corresponding to the embeddings predicted from the referee model at the end of episodes. The MDP states are captured by the embeddings output by the attention layer from the encoder of the GPT2, by feeding in the dialogue history up to the current step. The action space is captured as  $\mathcal{V}^l$ , where  $\mathcal{V}$  is the vocabulary considered and  $l$  is the maximum number of words the agent is allowed to place in the questions in each step. The definition of environmental rewards and synthetic human returns are introduced above.
Offline Trajectories and Target Policies. Five behavioral policies are used to collect offline trajectories, which are obtained from training over 4 types of RL algorithms following the setup introduced in [66], i.e., 1 policy follows conservative Q-learning (CQL), 2 follow implicit Q-learning (IQL), 1 follows behavioral clone (BC) and the last follows decision transformer (DT). All the behavioral policies are trained to obtain only  $10\% -50\%$  of the best synthetic human returns that can be achieved, which are considered sub-optimal. Eight different policies (also obtained from 4 types of RL algorithms) constitute the set of target policies whose performance will be estimated by OPEHF and the baselines. Their performance spread roughly equally between  $0\% -100\%$  of the optimal return that could be achieved; details can be found in Table 13. It can be found that synthetic human returns can be negative if the dialogue provides trivial, or sometimes misleading, information that are not helpful for capturing the underlying image. In practice one can shift the returns to become all positive in order to comply with the assumption in Proposition 1. Moderate correlations are found between the environmental and synthetic human returns in this experiment, i.e., Pearson's correlation coefficient is 0.606 and Spearman's rank correlation coefficient is 0.69.
Results and Discussion. The results are summarized in Table 12. It can be observed that the OPEHF in general outperforms the baselines, with significantly lower MAEs and higher ranks if IS, DR or FQE are selected as the downstream estimator. We also find that if DICE is selected as the downstream estimator, it in general leads to lower MAEs and regrets for both the OPEHF and rescale baseline, but significantly low rank correlations. Our hypothesize would be that considering DICE directly
Table 12: Results from the visual Q&A environment.
Fusion	IS
Rescale	OPEHF
(our)	Fusion	DR
Rescale	OPEHF
(our)
MAE	1.13±0.41	0.40±0.27	0.56±0.12	1.27±0.33	0.72±0.21	0.63±0.08
Rank	-0.21±0.55	0.19±0.15	0.64±0.13	0.11±0.45	0.08±0.27	0.59±0.18
Regret@1	0.76±0.34	0.08±0.09	0±0.05	0.43±0.29	0.25±0.09	0.01±0.0
Fusion	DICE
Rescale	OPEHF
(our)	Fusion	FQE
Rescale	OPEHF
(our)
MAE	0.99±0.12	0.25±0.03	0.11±0.02	0.97±0.08	1.13±0.14	0.89±0.03
Rank	0.02±0.24	-0.13±0.1	0.06±0.32	0.1±0.05	0.12±0.52	0.55±0.28
Regret@1	0.01±0.01	0.07±0.0	0.02±0.03	0.01±0.0	0.03±0.03	0.01±0.0
Table 13: Environmental and synthetic human returns of the target policies, as well as the algorithms used to obtain them, considered in the Q&A dialogue experiment.
Target Policy	Environmental Returns	Synthetic Human Returns	Algorithm
#0	-15.46	-18.15	CQL
#1	-11.77	-3.20	IQL
#2	-10.86	1.09	CQL
#3	-13.39	2.24	CQL
#4	-13.50	2.36	BC
#5	-11.33	3.09	CQL
#6	-6.05	3.23	IQL
#7	-9.77	3.66	DT
estimate the discrepancy between target and behavioral policy's propensity of visiting particular state-action pairs across the offline trajectories, it may benefit from leveraging the information (over the state-action space) captured by a language encoder that place roughly equal attention to each step of the dialogue, as opposed to the GPT2's encoder which is not fine-tuned specifically for such a case.