SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To manifest the potential heterogeneous outcome in healthcare and generalize the study into POMDP, we consider a heterogeneous variant of the original task. There are two contradictory Pendulum systems: the normal one and the converse one. While in the first system, adding torque will lead to a dynamical change according to the original physical design, in the converse system, the torque inputs will be sent to the system with a negation.
We train TD3 policies in each system and merge the collected dataset together as the offline dataset. In the Low-Data settings, 50000 transitions of each environment are collected, hence the dataset contains in total of 100000 transitions; in the Mid-Data regime, 150000 transitions of each environment are collected, hence the dataset contains in total of 300000 transitions; in the Rich-Data regime, 300000 transitions of each environment are collected, hence the dataset contains in total of 600000 transitions.
In order to achieve high performance, the agent must be able to identify the decision corpora that are collected from the same system dynamics as in the control time.
D.5 Hardware and Running Time
We experiment on a machine with 2 TITAN X GPUs and 32 Intel(R) E5-2640 CPUs. In general, the computational expense of model training in AOC is cheap, as the neural networks used in AOC are in general shallow and of small scale. Learning the belief space requires half the calculation of building a world model. However, we acknowledge the exact calculation of the minimal convex hull in a large-data regime can be computationally expensive. And we have discussed our practical solution (Appendix D.3). With our proposed solution, the convex hull decomposition takes less than 10 seconds with a uniform sampler that samples 100 actions randomly for every time step. Increasing the dimension of belief space size will lead to a sub-linear increase in computational time with parallelization.
D.6 Baseline Implementations
Benchmark Algorithms Except for standardized components that we will introduce below, we use the publicly available source code when constructing the benchmark algorithms; references are in the following:
- kNN and 1NN: https://scikit-learn.org/../neighbors, Reference: [20].
- BC: Implementation is straightforward using supervised learning.
- MFRL: https://github.com/sfujim/TD3, Reference: [27].
- MPC: https://github.com/UM-ARM-Lab/pytorch-mppi/../mppi.py, Reference: [90].
Neural Network Backbones In all baseline methods and our implementation for AOC, we use the same network structure: 3-layer MLP with a recurrent model that encodes the historical trajectory information, which is called as the Context Variable in the literature [91-94]. Our implementation of the recurrent model is based on the open-sourced code of https://github.com/amazon-science/metaq-learning. In all experiments, we use the same neural network architecture and match the hyperparameters for a fair comparison, except stated otherwise (e.g., the Q-learning baseline requires a larger batch size to guarantee convergence and boost stability, which will be elaborated in the following section.).
D.7 The Model-Free RL Baseline and Its Stability Issue
Our implementation of the Q-learning baseline leverages the twin-delayed techniques [27] to stabilize training. We find that Q-learning requires a large batch size (i.e., 10240) and many optimization epochs to converge. In the experiment settings with more offline data, the convergence becomes even harder: the rich-data regime containing  $6 \times$  more data takes  $6 \times$  more training epochs to converge. And the converged performance is always with high vibration, leading to worse performance.
Learning curves are shown in Figure 6, experiments are done with 8 seeds and both averaged learning curves and individual curves are plotted.
Figure 6: Q-Learning learning curves. When the size of offline data increases, more training epochs are needed for the convergence, and the stability of performance at convergence is reduced, leading to a larger variance and poorer performance in the rich-data regime.
E Additional Qualitative and Quantitative Results
E.1 Adaptivity: Visualizing the Decisions of AOC and Quantitative Performance
We plot the control time trajectories in experiments of Section 5.4 in Figure 7. It is clearly shown that with a decreasing sampling rate of  $\pi_1$ 's trajectories (passing through the upper gate) in the offline dataset, the control time behaviors tend to choose more actions following the behaviors of  $\pi_3$  (passing through the lower gate).
Quantitatively, Table 5 shows that while changing the sampling rate of trajectories from  $\pi_1$ , the success rate does not change much while the proportions of strategies chosen by the control policy vary accordingly.
Table 5: Quantitative results in the re-sampling and sub-sampling experiments. In all settings, 100 trajectories in total are generated using the proposed method. The success rate of reaching the goal and choices of solutions are presented in the table.
Re/Sub-Sample	×4	×3	×2	×1	×0.75	×0.5	×0.25
Success Rate	0.91	0.97	0.90	0.92	0.92	0.93	0.92
Passing Upper Gate	76	69	61	42	31	24	17
Passing Lower Gate	15	28	29	50	61	69	75
E.2 More Visualization Results and Extended Discussion on the Healthcare Dataset
We provide more qualitative results in Figure 8. In those figures, the colors yellow and blue indicate different treatments in the training data, separately.
Figure 7: Visualization of control behaviors. The first two plots show the task and collected trajectories in the offline dataset. This is the same environment we have used in Section 5.3. The control time preferences can be controlled by changing the sub-sampling or re-sampling rate.
In the realm of healthcare, treatment decisions involving atypical patients — those with rare or non-standard characteristics that do not closely resemble the majority of training examples — present a significant challenge. These outliers often reside on the boundaries of existing treatment records of patients, rendering the optimal course of action ambiguous even for domain experts.
Our proposed method AOC provides a critical advantage in these complex scenarios. AOC constructs a minimal convex hull around such patients, unearthing potential risks associated with incorrect treatment approaches. In cases where the decision corpus spans disparate treatment groups, AOC reveals the impossibility of establishing a more representative convex hull that both contains the patient and exclusively includes members of the same treatment class.
Conversely, the k-nearest neighbors (kNN) approach fails to identify these high-risk patients. The nearest neighbors for such boundary cases can all belong to the same treatment group, thus offering no warning signals that the patient might be atypical and warrant special attention.
Therefore, AOC outperforms the kNN method not merely in terms of higher accuracy in suggesting treatments, but also in its ability to flag patients whose decision basis exhibits heterogeneity at the test time. In these instances, doctors or other human experts can give particular consideration, thereby enhancing the trustworthiness of the decision system.
Figure 8: Visualization of the control time decision supports in the healthcare dataset. The two colors of the scatter plots denote different training time decisions. Test data is denoted by red cross marks, and the identified belief corpus subsets are marked with black circles.
F Additional Experiments
F.1 Trade-Off between Accountability and Performance
Experiment Setting In principle, the minimal convex hull in the  $d$ -dimensional belief space contains  $d + 1$  examples. However, searching for such a minimal convex hull is a combinatorial optimization problem and can be extremely hard in high-dimensional space with many samples. In our implementation, we leverage a heuristic search method that constrains the combinatorial search inside the  $k$ -nearest neighbors of a given control time belief state.
We change such a hyper-parameter — the maximal number of examples in the corpus subsets can be used in building the minimal belief space convex hull. We experiment with  $k = 1, 10, 20, 100$  separately.
Results Table 6 shows the results. To make AOC work, the choice of  $k$  should at least roughly match the dimension of the latent space. Otherwise, the aggressive extrapolation will hinder the performance of AOC, including using only the nearest neighbor as an approximation. While such a choice naturally trades off between explainability and decision quality, our empirical study shows using a redundant set of the corpus will hinder the performance — this demonstrates the necessity of using the minimal convex hull in AOC.
Take-Away: The minimal convex hull design in AOC is crucial for performance. We recommend the choice of size in constructing a minimal convex hull should match the dimension of the belief space.
Table 6: The cardinality of the corpus subset is a hyper-parameter that trades off between accountability and performance. While in general using a larger number of corpus examples can better support the control decisions, it also has a risk of increasing the corpus residual in synthesizing the control time decision, hence hindering the performance of the control policy.
K	Performance
1	-670.51 ± 321.09
10	-510.39 ± 311.24
20	-1.25 ± 0.4
100	-2.06 ± 0.55
F.2 Visualizing Conservation of AOC
Experiment Setting To better illustrate the conservative behaviors introduced by the hyperparameter  $\epsilon$ , we conduct experiments on a Two Gates Maze environment to visualize the differences of using different choices of  $\epsilon$ 's.
In the Two Gates Maze task, an agent needs to navigate to a goal located at (16, 8), the middle point of the right side wall, starting from (0, 8), the middle point of the left side wall. Two gates open in a middle wall located at  $x = 8$ , hence the agent needs to pass one of those gates to reach the goal.
We generate an offline dataset from two behavior policies, each of which selects one of the two gates to pass the middle wall. The first two plots in Figure 9 show the map as well as the behavior trajectories as the dataset.
We then experiment with different choices of  $\epsilon = [0.1, 0.3, 0.5, 0.7]$  in AOC to perform offline control.
Results Results are shown in the last 4 plots in Figure 9 (Purple lines denote the control time trajectories, ideal conservative policies' behaviors should be bounded by the behavior trajectories). In each setting, we roll out with 100 trajectories by AOC and visualize the behaviors, and report the averaged performance in Table 7. When a small  $\epsilon$  is used, the control time behaviors show little extrapolation: trajectories are in general surrounded by the dataset behaviors. When  $\epsilon$  becomes larger, more aggressive extrapolations emerge in control time behaviors, leading to poorer performances.
Table 7: The  $\epsilon$  in Equation (11) contributes to the conservative behaviors of AOC.
€	Performance
0.1	6.97 ± 2.57
0.3	2.58 ± 5.13
0.5	-3.10 ± 1.00
0.7	-3.20 ± 0.00
Take-Away: Using a small  $\epsilon$  leads to more conservative behaviors in AOC. AOC performs offline control avoiding aggressive extrapolations by constructing the Minimal Hull and performing control with the decision corpora that have minimized residuals.
F.3 Additional Environment: LunarLanderContinuous
Experiment Setting We additionally experiment on the LunarLanderContinuous-v2 environment [95] for the general interests of the RL community. The LunarLanderContinuous-v2 environment is a physics-based simulation game in which the agent must control a lunar lander to successfully land on a designated landing pad. It has a 2-dim action space and a 8-dim state space, both of which are continuous.
We compare AOC with the nearest-neighbor controller (1NN) [20] and its variant that using  $k$  neighbors (kNN), model-based RL with Mode Predictive Control (MPC) [24], model-free RL (MFRL) [27], and behavior clone (BC) [25]. We change the size of the dataset to showcase the
Figure 9: Visualizing the conservative behaviors controlled by the threshold controlled by a quantile number  $\epsilon$ . Using a smaller  $\epsilon$  leads to more conservative behaviors hence benefiting the offline control problems where aggressive extrapolations can be dangerous.
performance difference of various methods under different settings. We experiment with different offline data availability, varying from 0.1M to 1M.
Table 8: Performance comparison on the LunarLanderContinuous-v2 environment under different settings (availability of offline data). The episodic cumulative reward of each method is reported. The last row (Data) reports the performance of trajectories in the offline dataset. Higher is better.
0.1M	0.2M	0.3M	0.5M	1M
AOC	100.77 ± 174.5	184.83 ± 88.28	240.81 ± 44.47	252.38 ± 45.66	253.71 ± 25.11
kNN	-86.07 ± 172.57	-43.87 ± 69.1	-3.04 ± 116.76	-59.63 ± 143.33	-31.47 ± 152.72
1NN	-42.2 ± 31.36	-98.27 ± 89.35	-14.93 ± 115.75	-57.37 ± 59.1	-64.41 ± 69.71
BC	-130.23 ± 33.23	-118.61 ± 42.18	-36.91 ± 57.32	18.08 ± 35.4	54.46 ± 69.16
MFRL	92.0 ± 82.05	165.11 ± 54.48	221.3 ± 19.63	219.55 ± 38.97	254.55 ± 24.42
MPC	-31.42 ± 32.09	-35.52 ± 43.09	-50.96 ± 22.96	-67.88 ± 55.49	-96.22 ± 85.01
Data	149.99 ± 133.45	207.92 ± 58.34	171.19 ± 100.43	169.42 ± 96.02	184.92 ± 69.59
Results We present the results in Table 8. In all settings, we find AOC is able to achieve an on-par performance of black-box decision-making algorithms, and outperforms the accountable baselines. In this environment, we find the model-based approach is not able to learn a well-performing reward function approximator. On the other hand, different from the Pendulum-Het settings where the stability of the balanced state is essential in achieving high performance, the stability of the model-free method in this environment is not an issue.
Take-Away: The results on the LunarLanderContinuous environment again demonstrate the desired properties of AOC: while being accountable, it achieves similar performance as the black-box learning algorithms and is robust under different data availability.
F.4 Black-Box Policy as More Efficient Sampler
Experiment Setting In the main text, we introduce the uniform sampling approach for control time execution. While such an approach is simple and effective in our accountability-sensitive control benchmark tasks, it can be inefficient in high-dimensional continuous control tasks. For the interest of the general continuous control community, we experiment with a higher dimensional task in this section to stress test the capability of AOC in more challenging tasks.
We experiment on the BipedalWalker-v3 environment that has 24-dim observational space and 4-dimensional action space. Different from previous experiments where a uniform sampler is applied, in this section, we use black-box models as the more efficient sampler and leverage AOC as a post-hoc interpreter for decision accountability. Specifically, we compare AOC with a uniform sampler (AOC-Uniform) and AOC with Behavior Clone policy as a black-box sampler (AOC-BC) against the same baselines as previously, i.e., the nearest-neighbor controller (1NN) [20] and its variant that using  $k$  neighbors (kNN), model-based RL with Mode Predictive Control (MPC) [24], model-free RL (MFRL) [27], and behavior clone (BC) [25]. To improve the black-box controller's performance and hence isolate the source of gain, we use the offline dataset of size 1M.
Results Results are reported in Table 9. In this high-dimensional control task, the uniform sampler is inefficient and fails to converge to a well-performing policy in control time. Among all black-box methods, BC achieves the best performance. And AOC with BC as its sampler achieves improved performance, while at the same time being accountable.
Table 9: Performance on the BipedalWalker-v3 environment. The episodic cumulative reward of each method is reported. Higher is better.
Method	Performance
kNN	-109.72 ± 5.85
1NN	-111.95 ± 8.25
AOC-Uniform	-90.44 ± 14.06
AOC-BC	276.98 ± 82.78
BC	208.72 ± 95.72
MFRL	18.51 ± 111.53
MPC	-96.82 ± 24.7
Data-Avg-Return	202.25 ± 102.61
Take-Away: AOC can work both in isolation or combined with black-box policies. AOC can be used as a plug-in to add accountability to black-box controllers in a post-hoc manner. In high-dimensional control tasks, uniform sampling can be inefficient and black-box samplers can alleviate such a difficulty.
F.5 Identify Control Time OOD Examples with AOC
Experiment Setting In this section, we show that AOC can be applied to OOD example identification [96] during control time. Specifically, we conduct the experiment with the BipedalWalker environment. During control time, a black-box controller described in the last section is used, and we start to inject Gaussian noise into the control actions as disturbance after the 500-th timestep to create an OOD scenario.
Results We repeat the above process for 10 times and report the averaged step-wise instant reward curve and corpus residual curve during rollouts in Figure 10. We note that in the figure in the beginning steps, the walker starts from a static state to walk, thus leading to an increase in the instant reward curve. We then label the control steps between the 100-th and the 500-th step as the stable walking phase, during which the walker receives a nearly constant instant reward (around 0.4 per step), and the corpus residual during this period is stable and close to 0. The Gaussian noise is injected after the 500-th timestep, leading to a clear increase in the corpus residual curve and a sudden decrease in the instant reward curve. According to the corpus residual values' sudden increase, the
Figure 10: AOC can be used for OOD detection in control time. The corpus residual values can be used to detect large epistemic uncertainty on OOD examples. Shaded areas in the figure denote the 3-sigma interval.
control time OOD examples can be identified according to the 3-sigma rule of thumb. (The 3-sigma intervals are marked with shaded areas in the figure).
Take-Away: Control time OOD examples can be identified by AOC according to the sudden increases in the corpus residual value.
Limitations and Future Work
In this study, our exploration of accountable offline control primarily targets low-dimensional control tasks, drawing inspiration from healthcare applications where treatments are typically of low dimension. However, the uniform sampling approach we propose may not perform as well in high-dimensional control systems, which can be of interest in the field of robotics study. We do offer an initial exploration of using black-box samplers in AOC within our appendix, but there remains ample room for further work in enhancing the efficiency of the sampling process. Additionally, there's potential for expanding AOC into online control settings by combining it with optimism in the face of uncertainty (OFU) explorers, to pursue accountable online control. This, however, lies beyond the scope of our current paper.
In this study, we examine the offline control problem, which holds significant potential for applications in costly, safety-sensitive, and critical domains such as healthcare and finance. While previous works have primarily focused on efficient learning in offline settings, the accountability of offline decisions remains largely unexplored despite its importance.
In critical domains like healthcare, it's vital that decisions are based on supportive evidence. For instance, when a patient is treated in a certain manner, it should be based on the successful outcomes of previous patients with comparable conditions who received the same treatment. The ability to trace the supportive basis of decisions enhances the process of policy reasoning and debugging, thereby improving the trustworthiness of decision-making systems.
However, we bring to light the potential risk associated with applying AOC to critical real-world decision-making systems. This risk stems from potential heterogeneous outcomes, i.e., the aleatoric uncertainty associated with decision outcomes. For example, similar patients undergoing the same treatment may experience different results. Therefore, when the variance of outcomes in the corpus subset is high, users should exercise caution regarding the potential heterogeneous outcomes of decisions.