SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\pi_ {\mathrm {A O C}} \left(o _ {t}, h _ {t}\right) = \arg \max  _ {a _ {t} \in \left\{a _ {t} ^ {(k)} \right\}} \sum_ {c = 1} ^ {C} w ^ {c} v ^ {c}, \quad \text {s . t .} \quad r _ {\mathcal {C}} (\boldsymbol {b} \left(o _ {t}, a _ {t}, h _ {t}\right)) \leq \epsilon \tag {11}
$$
The complete optimization procedure is detailed in Algorithm 1.
Table 1: AOC is distinct as it satisfies 5 desired properties mentioned earlier: (P1) Conservation: AOC seeks the best potential action in the belief convex hull, such that the estimations of decision outcomes are interpolated, avoiding aggressive extrapolation that is harmful in offline control [8]; (P2) Accountability: the decision-making process is supported by a corpus subset from the offline dataset, hence all the decisions are traceable; (P3) Low-Data Requirement: it works in the low-data regime; (P4) Adaptivity: the control behavior of AOC can be adjusted according to additional constraints as clinical guidelines without modification or re-training; (P5) Reward-Free: AOC can be extended to the strictly offline imitation setting where rewards are unavailable.
Method / Property	Conservation	Accountable	Low-Data	Adaptive	Reward-Free	Examples
Q-Learning	✓	✓	✘	[8–16]
Episodic Control	✘	✘	✓	✘	[17–19]
Nearest Neighbor	✘	✓	✘	✓	✓	[20, 21]
Model-Based RL	✘	✘	✘	✓	✘	[22–24]
Behavior Clone	✘	✘	✘	✘	✓	[25]
AOC	✓	✓	✓	✓	✓	Ours
We review related literature and contrast their difference in Table 1. Due to constraints on space, we have included comprehensive discussions on related work in Appendix B. Our proposed method stands out due to its unique attributes: It incorporates properties of accountability, built-in conservation for offline learning, adaptation to preferences or guidelines, is suitable for the low-data regime, and is compatible with the strictly offline imitation learning setting where the reward signal is not recorded in the dataset.
**Environments and Set-ups** In this section, we present empirical evidence of the properties of our proposed method in a variety of environments. These environments include (1) control benchmarks that simulate POMDP and heterogeneous outcomes in healthcare, (2) a continuous navigation task that allows for the visualization of properties, and (3) a real-world healthcare dataset from Ward [26] that showcases the potential of AOC for deployment in high-stakes, real-world tasks.
In Section 5.1, we benchmark the performance of AOC on the classic control benchmark, contrasting it with other algorithms and highlighting (P1-P3). In Section 5.2, Section 5.3, and Section 5.4 we individually highlight properties (P1), (P2), (P4). Subsequently, in Section 5.5, we apply AOC to a real-world dataset, highlighting its ability to perform accountable, high-quality decision-making under the strictly offline imitation setting (P1-P5). Finally, Section 5.6 provides additional empirical evidence supporting our intuitions and extensive stress tests of AOC.
5.1 (P1-P3): Accountable Offline Control in the Low-Data Regime
Experiment Setting Our initial experiment employs a synthetic task emulating the healthcare setting, simultaneously allowing for flexibility in stress testing the algorithms.
We adapt our environment, Pendulum-Het, from the classic Pendulum control task involving system stabilization. Heterogeneous outcomes in healthcare contexts are portrayed through varied dynamics governing the data generation process. Data generation process details can be found in Appendix D.4. Taking heterogeneous potential outcomes and partial observability into consideration, the Pendulum-Het task is significantly more challenging than the classical counterpart.
We compare AOC with the nearest-neighbor controller (1NN) [20] and its variant that using  $k$  neighbors (kNN), model-based RL with Mode Predictive Control (MPC) [24], model-free RL (MFRL) [27], and behavior clone (BC) [25]. Our implementation for those baseline algorithms is based on open-sourced codebases, with details elaborated in Appendix D.6. We change the size of the dataset to showcase the performance difference of various methods under different settings. The Low-Data denotes settings with 100000 transition steps, Mid-Data denotes settings with 300000 transition steps, Rich-Data denotes settings with 600000 transition steps.
Results We compare AOC against baseline methods in Table 2. AOC achieves the highest average cumulative score among the accountable methods and is comparable with the best black-box controller. In experiments, we find increasing the number of offline training examples does not improve the performance of MFRL due to stability issues. Detailed discussions can be found in Appendix D.7.
Table 2: Results on the Heterogeneous Pendulum dataset. The cumulative reward of each method is reported. Experiments are repeated with 8 seeds. Higher is better.
Task	Low-Data	Mid-Data	Rich-Data
AOC	-1.39 ± 1.39	-1.25 ± 0.40	-0.6 ± 0.08
kNN	-849.45 ± 91.23	-670.51 ± 321.09	-645.72 ± 220.33
INN	-557.07 ± 256.64	-690.49 ± 152.59	-512.71 ± 131.2
BC	-422.77 ± 409.51	-225.32 ± 340.83	-126.74 ± 280.73
MFRL	-4.1 ± 2.76	-11.95 ± 4.68	-15.27 ± 6.46
MPC	-1.5 ± 0.43	-1.34 ± 0.15	-1.41 ± 0.26
Data-Avg-Return	-307.81 ± 387.53	-245.54 ± 338.65	-208.81 ± 272.84
Take-Away: As an accountable algorithm, AOC performs on par with state-of-the-art black-box learning algorithms. Particularly in the POMDP task, wherein data originates from heterogeneous systems, AOC demonstrates robustness across diverse data availability conditions.
5.2 (P1): Avoid Aggressive Extrapolation with the Minimal Hull and Minimized Residual
Experiment Setting In Equation (11), our arg-max operator includes two adjustable hyperparameters: the number of uniformly sampled actions and the threshold. In this section, we demonstrate how these hyperparameters can be unified and easily determined.
The consideration of sample size primarily represents a balance between optimization precision and computational cost — although such cost can be minimized by working with modern computational platforms through parallelization. On the other hand, selecting an appropriate value for  $\epsilon$  might initially seem complex, as the residual's magnitude should depend on the learned belief space.
In our implementation, we employ a quantile number to filter out sampled actions whose corresponding corpus residuals exceed a certain population quantile. For instance, when  $\epsilon = 0.3$  with a sample size of 100, only the top  $30\%$  of actions with the smallest residuals are considered in the arg-max operator in Equation (11). The remaining number of actions is referred to as the effective action size. This section presents results for the Pendulum-Het environment, while additional qualitative and quantitative results on the conservation are available in Appendix F.2. Intuitively, using a larger sampling size will improve the performance, at a sub-linear cost of computational expenses, and using a smaller quantile number will make the behavior more conservative. That said, using a small quantile number may not be always good, as it also decreases the effective action size. The golden rule is to use larger sampling size and smaller quantile threshold.
Results Table 3 exhibits the results. Our experiments highlight the significance of this filtering mechanism: removal or weakening of this mechanism, through a larger quantile number that filters fewer actions, leads to aggressive extrapolation and subsequently, markedly deteriorated performance. This is because the decisions made by AOC in those cases cannot be well supported by the minimal convex hull, introducing substantial epistemic uncertainty in value estimation.
Table 3: The  $\epsilon$  in Equation (11) controls the conservation tendency of the control time behaviors.
ε (quantile)	Performance
0.3	-2.05 ± 0.45
0.5	-1.25 ± 0.40
0.9	-503.39 ± 301.60
1.0	-853.29 ± 138.43
Conversely, using a smaller quantile number yields a reduced
effective action size, which subsequently decreases optimization accuracy and leads to a minor performance drop. Hence when computational resources allow, utilizing a larger sample size and a smaller quantile number can stimulate more conservative behavior from AOC. In all experiments reported in Section 5.1, we find a sample size of 100 and  $\epsilon = 0.5$  generally work well.
Take-Away: AOC is designed for offline control tasks by adhering to the principle of conservative estimation. The  $\epsilon$  hyperparameter, introduced for conservative estimation, can be conveniently determined by incorporating it into the choice of an effective action size.
5.3 (P2): Accountable Offline Control by Tracking Reference Examples
Experiment Setting In healthcare, the treatments suggested by the clinician during different phases can be based on different patients, and those treatments may even be given by different clinicians according to their expertise. In this section, we highlight the accountability of AOC within a Continuous Maze task. Such a task involves the synthesis of decisions from multiple experts to achieve targeted goals. It resembles the healthcare example above and is ideal for visualizing the concept of accountability.
Figure 2: Left: the Maze; Right: trajectories generated by 4 experts for different tasks.
Figure 2 displays the map, tasks, and expert trajectories within a 16x16 Maze environment. This environment accommodates continuous state and action spaces. We gathered trajectories from 4 agents, each of which is near-optimal for a distinct navigation task:
$①$  agent  $\pi_1$  starts at position  $(0,0)$  and targets position (8, 16). Trajectories of  $\pi_1$  are marked in blue.
$②$  agent  $\pi_2$  starts at position  $(0,0)$  and targets position (8, 8). Trajectories of  $\pi_2$  are marked in orange.
$③$  agent  $\pi_3$  starts at position (8, 16) and aims position (16, 0). Trajectories of  $\pi_3$  are marked in green.
$④$  agent  $\pi_4$  starts at position (8, 8) and targets position (16, 0). Trajectories of  $\pi_4$  are marked in red.
We generate 250 trajectories for each of the 4 agents, creating the offline dataset. The control task initiates at position  $(0,0)$ , with the goal situated at  $(16,0)$  — requires a fusion of expertise in the dataset to finish. To complete the task, an agent could potentially learn to replicate  $\pi_1$ , followed by adopting  $\pi_3$ 's strategy. Alternatively, it could replicate  $\pi_2$  before adopting  $\pi_4$ 's approach.
Results In this experiment, we employ AOC to solve the task and visualize the composition of the corpus subset at each decision step in Figure 3. In the initial stages, the corpus subset may comprise a blend of trajectories from  $\pi_1$  and  $\pi_2$ . Subsequently, AOC selects one of the two potential solutions at random, as reflected in the chosen reference corpus subsets. Upon crossing the central gates on the x-axis, AOC's decisions become reliant on either  $\pi_3$  or  $\pi_4$ , persisting until the final steps, when the corpus subsets can again be a mixture of the overlapped trajectories.
Figure 3: Visualize Accountability. The upper colors display the components of the corpus subset at different x-positions.
Take-Away: AOC's decision-making process exhibits strong accountability, as the corpus subset can be tracked at every decision-making step. This is evidenced by AOC's successful completion of a multi-stage Maze task, wherein it effectively learns from mixed trajectories generated by multiple policies at differing stages.
5.4 (P4): Flexible Offline Control under User Specification
Experiment Setting Given the tractable nature of AOC's decision-making process, the agent's behavior can be finely tuned. One practical way of modifying these decisions is to adjust the reference dataset, which potentially contributes to the construction of the corpus subset. In this section, we perform experiments in the Continuous Maze environment with the same offline dataset described in Section 5.3. We modify the sampling rate of trajectories generated by  $\pi_1$  and experiment with both re-sampling and sub-sampling strategies, using sample rates ranging from  $[ \times 4, \times 3, \times 2, \times 1, \times 0.75, \times 0.5, \times 0.25 ]$ . In the re-sampling experiments (sample rates larger than 1), we repeat trajectories produced by  $\pi_1$  to augment their likelihood of selection as part of the corpus subset. In contrast, the sub-sampling experiments partially omit trajectories from  $\pi_1$  to reduce their probability of selection. In each setting, 100 control trajectories are generated.
Results Figure 4 shows the results: the success rate (Succ. Rate) quantifies the quality of control time behaviors, while the remaining two curves denote the percentage of control time trajectories that accomplish the task by either following either  $\pi_1 - \pi_3$  (passing the upper gate) or  $\pi_2 - \pi_4$  (passing the lower gate). The behavior of AOC is influenced by the sampling rate. Utilizing a high sampling rate for the trajectories generated by  $\pi_1$  often leads to the selection of the solution that traverses the upper gate. As the sampling rate diminishes, AOC's behavior begins to be dominated by the alternative strategy. In both re-sampling
Figure 4: The behavior of AOC can be controlled by re-sampling or sub-sampling the offline dataset.
and sub-sampling strategies, the success rate of attaining the goal remains consistently high. We provide further visualizations of the learned policies under various sampling rates in Appendix E.
Take-Away: The control time behavior of AOC can be manipulated by adjusting the proportions of behaviors within the offline dataset. Both re-sampling and sub-sampling strategies prove equally effective in our experiments where multiple high-performing solutions are available.
5.5 (P1-P5): Real-World Dataset: The Strictly Batch Imitation Setting in Healthcare
Experiment Setting We experiment with the Ward dataset [26] that is publicly available with [28]. The dataset contains treatment information for more than 6000 patients who received care on the general medicine floor at a large medical center in California. These patients were diagnosed with a range of ailments, such as pneumonia, sepsis, and fevers, and were generally in stable condition. The measurements recorded included standard vital signs, like blood pressure and pulse, as well as lab test results. In total, there were 8 static observations and 35 temporal observations that captured the dynamics of the patient's conditions. The action space was restricted to a choice of a binary
action pertaining to the use of an oxygen therapy device. In the healthcare setting, treatment decisions are given by domain experts and the objective of policy learning is to mimic those expert behaviors through imitation. However, as interaction with the environment is strictly prohibited, the learning and evaluation of policies should be conducted in a strictly offline manner. The dataset is split into  $10:1$  training and testing sets and the test set accuracy is used as the performance metric.
We benchmark the variant of AOC tailored for the strictly batch imitation setting, dubbed as ABC — as an abbreviation for Accountable Behavior Clone — against three baselines: (1) behavior cloning with a linear model (BC-LR), which serves as a transparent baseline in the offline imitation setting; (2) behavior cloning with a neural network policy class (BC-MLP), which represents the level of performance achievable by black-box models in such contexts; and (3) the k-nearest neighbor controller (kNN) as an additional accountable baseline.
Results The results of kNN with various choices of  $k$  in the experiments, alongside a comparison with ABC using the same number of examples in the corpus subset, are presented in Figure 5. Shaded areas in the figure denote the variance across 10 runs.
Operating as an accountable policy under the offline imitation setting, ABC achieves performance similar to that of behavior cloning methods utilizing black-box models. Meanwhile, other baselines face challenges in striking a balance between transparency and performance.
Furthermore, we visualize the corpus subset supporting control time decisions in a 2-dimensional belief space to illustrate the accountability. We demonstrate the differences in the decision boundary where predictions are more challenging. In such cases, the decision supports of kNN rely on extrapolation and have no idea of the potential risk of erroneous prediction, while the heterogeneity in ABC's minimal convex hull reflects the risk. We provide a more detailed discussion in Appendix E.2.
Take-Away: The variant of AOC, dubbed as ABC, is well-suited for the strictly batch imitation setting, enabling accountable policy learning. While other baseline methods in this context struggle to balance transparency and policy performance, ABC achieves comparable performance with black-box models while maintaining accountability.
5.6 Additional Empirical Studies
Figure 5: Results on the Ward dataset. ABC is able to perform accountable imitation under the strict offline setting with high performance comparable with the black-box models.
To further verify our intuitions and stress test AOC, we
provide additional empirical analysis in Appendix F. Specifically, we analyze the trade-off between accountability and performance in Appendix F.1; we visualize the controllable conservation in Appendix F.2; we benchmark AOC on another benchmark for the general interests of the RL community in Appendix F.3; and show how to combine AOC with black-box sampler in Appendix F.4; Finally, we show AOC can detect control time OOD examples in Appendix F.5.
In conclusion, this study presents the Accountable Offline Controller (AOC) as a solution for offline control in responsibility-sensitive applications. AOC possesses five essential properties that make it particularly suitable for tasks where traceability and trust are paramount: ensures accountability through offline decision data, thrives in low-data environments, exhibits conservative behavior in offline contexts, can be easily adapted to user specifications, and demonstrates flexibility in strictly offline imitation learning settings. Results from multiple offline control tasks, including a real-world healthcare dataset, underscore the potential of AOC in enhancing trust and accountability in control systems, paving the way for its widespread adoption in real-world applications.
HS acknowledges and thanks the funding from the Office of Naval Research (ONR). HS would thank Jonathan Crabbe for his inspiration and encouragement in exploring this idea in a group meeting. We thank Jonathan Crabbe for reviewing our manuscript and providing insightful comments and suggestions for improving the paper. We thank all anonymous reviewers, ACs, SACs, and PCs for their efforts and time in the reviewing process and in improving our paper. This work is done with the warm support from the van der Schaar Lab members.
[1] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643, 2020.
[2] Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, et al. Scalable deep reinforcement learning for vision-based robotic manipulation. In Conference on Robot Learning, pages 651-673. PMLR, 2018.
[3] Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. Way off-policy batch deep reinforcement learning of implicit human preferences in dialog. arXiv preprint arXiv:1907.00456, 2019.
[4] Gregory Kahn, Pieter Abbeel, and Sergey Levine. Badgr: An autonomous self-supervised learning-based navigation system. IEEE Robotics and Automation Letters, 6(2):1312-1319, 2021.
[5] Samuel Holt, Alihan Hüyük, and Mihaela van der Schaar. Active observing in continuous-time control. In Conference on Neural Information Processing Systems, 2023.
[6] Mark T Keane and Eoin M Kenny. How case-based reasoning explains neural networks: A theoretical analysis of xai using post-hoc explanation-by-example from a survey of ann-cbr twin-systems. In Case-Based Reasoning Research and Development: 27th International Conference, ICCBR 2019, Otzenhausen, Germany, September 8–12, 2019, Proceedings 27, pages 155–171. Springer, 2019.
[7] Jonathan Crabbé, Zhaozhi Qian, Fergus Imrie, and Mihaela van der Schaar. Explaining latent representations with a corpus of examples. Advances in Neural Information Processing Systems, 34:12154-12166, 2021.
[8] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International conference on machine learning, pages 2052-2062. PMLR, 2019.
[9] Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. Advances in Neural Information Processing Systems, 32, 2019.
[10] Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00177, 2019.
[11] Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. In International Conference on Machine Learning, pages 104-114. PMLR, 2020.
[12] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33: 1179-1191, 2020.
[13] Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty-based offline reinforcement learning with diversified q-ensemble. Advances in neural information processing systems, 34:7436-7447, 2021.