Accountability in Offline Reinforcement Learning: Explaining Decisions with a Corpus of Examples
Authors: Hao Sun
Alihan Hüyük
Daniel Jarrett
Mihaela van der Schaar
Venue: NeurIPS 2023 poster (2023)
Keywords: Accountability
Reinforcement Learning
Batched Control
Accountable Decision-Making
Offline RL
Interpretability in RL
TL;DR: We introduce the Accountable Offline Controller that employs the offline dataset as the Decision Corpus and performs accountable offline control.

Abstract: Learning controllers with offline data in decision-making systems is an essential area of research due to its potential to reduce the risk of applications in real-world systems. However, in responsibility-sensitive settings such as healthcare, decision accountability is of paramount importance, yet has not been adequately addressed by the literature.
This paper introduces the Accountable Offline Controller (AOC) that employs the offline dataset as the Decision Corpus and performs accountable control based on a tailored selection of examples, referred to as the Corpus Subset. AOC operates effectively in low-data scenarios, can be extended to the strictly offline imitation setting, and displays qualities of both conservation and adaptability.
We assess AOC's performance in both simulated and real-world healthcare scenarios, emphasizing its capability to manage offline control tasks with high levels of performance while maintaining accountability.

Accountability in Offline Reinforcement Learning: Explaining Decisions with a Corpus of Examples
Hao Sun, Alihan Hüyük, Daniel Jarrett, Mihaela van der Schaar Department of Applied Mathematics and Theoretical Physics University of Cambridge
Abstract
Learning controllers with offline data in decision- making systems is an essential area of research due to its potential to reduce the risk of applications in real-world systems. However, in responsibility-seniiii settings such as healthcare, decision accountability is of paramount importance, yet has not been adequately addressed by the literature. This paper introduces the Accountable Offline Controller (AOC) that employs the offline dataset as the Decision Corpus and performs accountable control based on tailored selection of examples, referred to as the Corpus Subset. AOC operates effectively in low-data scenarios, can be extended to the strictly offline imitation setting, and displays qualities of both conservation and adaptabil- ity. We assess AOC's s performance in both simulated and real-world healthcare scenarios, emphasizing its capability to manage offline control tasks with high levels of performance while maintaining accountability.
Introduction
In recent years, offline control that uses pre-collected data to generate control policies has gained attention due to its potential to reduce the costs and risks associated with applying control algorithms in real-world systems M,, which are especially advantageous situations where real-time feedback is challenging or expensive obtain 52.5.. However, existing literature has primarily focused on enhancing learning performance, leaving the need for accountable and reliable control policies in offline settings largely unaddressed, particularly for high- responsibility- sensitive applications.
However, many critical real-world applications such as healthcare, the challenge beyond enhancing policy performance. It requires the decisions made by learned policies to be transparent, traceable, and justifiable. Yet those essential properties, summarized as Accountability, are left largely unaddressed by existing literature.
In our context, we use Accountability to indicate the existence of a supportive basis for decision- making. For instance. in tumor treatment using high-risk options like radiotherapy and chemotherapy. the treatment decisions should be based on the successful outcomes experienced by previous patients who share similar conditions and were given the same medication. Another concrete illustrative example the allocation decisions of ventilator machines. The decision to allocate a ventilator should be accountable, in the way that it juxtaposes the potential consequences of both utilization and non-utilization and provides reasonable decision on those bases. In those examples, the ability to refer to existing cases that support current decisions can enhance reliability and facilitate reasoning or debugging of the policy. To advance offline control towards real-world responsibility applications, five properties are desirable: (P1) controllable conservation: this ensures the policy learning performance by avoiding aggressive extrapolation. (P2) accountability: as underscored by prior examples, there is need for a clear basis upon which decisions made. (P3) suitability
hhx789 @aaam.camcuuu
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

for low-data regimes: given the frequent scarcity of high-stake decision data, it's essential to have methods that perform well with limited data. (P4) adaptability to user specification: this ensures the policy can adjust to changes, like evolving clinical guidelines, allowing for tailored solutions. (P5) flexibility Strictly Offfine Imitation: this property ensures broader applicability across various scenarios and data availability.
To embody all these properties, we need venture beyond the current scope of literature focused on conservative offline learning. In our work:
1. Methodologically, we introduce the formal definitions and necessary concepts in accountable decision-making. We propose the Accountable Offline Controller (AOC), which makes decisions according decomposition on the basis of the representative existing decision examples.
2. Theoretically, we prove the existence and uniqueness of the decomposition under mild conditions, guiding the design of our algorithms.
3. Practically, we introduce an efficient algorithm that takes all the aforementioned desired properties into consideration and circumvented the computational difficulty.
4. Empirically, we verify and highlight the desired properties of AOC on a variety of offline control tasks, including five simulated continuous control tasks and one real-world healthcare dataset.
Preliminaries
POMDP We develop our work under the general Partially Observable Markov Decision Process (POMDP) setting, denoted as a tuple (X, w. O, A, R, Ys Po). where X' S IR denotes the under- lying d, dimensional state space, X' O is the emission function that maps the underlying state into a d, dimensional observation O C Rdo; A C Ie is a da dimensional action space, transition dynamics T X' A X controls the underlying transition between states given actions; and the reward function R: + R maps state into a reward scalar; we use y to denote the discount factor and Po the initial state distribution. Note that the MDP is the case when is identical mapping.
T,
Offline Control Specifically, our work considers the offline learning problem: a logged dataset D = fot of, ri, of o 1. containing N trajectories with length T is collected by rolling out some behavior policies in the POMDP, where za Pu sampled from the initial state distribution and o = w(r)). af oi sampled from unknown behavior policies Ti.. We denote the observational transition history until time t as h4, such that f12 = (o<i,aci,
H. C +d. +1)-tt1),
Learning Objective A policy ग O H + A function of observational transition history. The learning objective is to find π that maximizes the expected cumulative return in the POMDP:
max *
max [r're]
(I)
where 7 Po: n, T means the trajectories is generated by sampling initial state 3o from Poo and action a from n, such that: - = { On = w(ro), ag = T (oo, ho = Φ), = T(xo, (z)) ro = R...)...... learned belief variable by E Rdd can be introduced to represent the underlying state It.
Accountable Control with Decision Corpus
3.1 Method Sketh: A Roadmap
The high-level core idea of our work is to introduce an example-based accountable framework for offline decision-making such that the decision basis can be clear and transparent.
To achieve this, a naive approach would be leverage the insights of Nearest Neighbors: for each action, this involves finding the most similar transitions in the offline dataset, and estimating the corresponding outcomes. Nonetheless, pivotal hurdle arises in defining similarity, particularly when taking into account the intricate nature of trajectories, given both the observation space heterogeneity and the inherent temporal structure in decision-making Compounding such challenge, another difficulty arises in identifying the most representative examples and integrating the pivotal principle

---

of conservation, which is widely acknowledged to be essential for the success of offline policy learning.
Our proposed method seeks to address those challenges. We start by introducing the basic definitions to support formal discussion of accountability: in Definition 3.1 we introduce the concept of
Decision Corpus.
To address the similarity challenge, showcase that nice linear property (Property 3.2
generally exists (Remarks 3.3 & (.4) when working in the belief space (Definition 3.5 This subsequently
leads to a theoretical bound for estimation error (Proposition 3.8
To address the representative challenge while obeying the principle of conservation, we underscore those examples that span the convex hull (Definition (3.6 and introduce the related optimization objective (Definition 3.7 & 3.9 nutshell, the intuition is use minimal set of representative training examples to encapsulate test-time decisions. Under mild conditions, we show the solution would exist and be unique (Proposition 3.10
3.2 Understanding Decision-Making with a Subset of Offline Data
In order to perform accountable decision-making construct our approach upon the foundation of the example-hased explanation framework [6] which motivates us to define the offline decision dataset DD as the Decision Corpus, and introduce the concept of Corpus Subset that is composed of representative subset of the decision corpus. These corpus subsets will be utilized for comprehending control decisions. Formally:
Definition 3.1 (Corpus Subset). A Corpus Subser C is defined as subset of the decision corpus D, indexed by [C] == (1, C} - the natural numbers between and C.
c = {( (0 a hi v E (0
h,, In the equation above, the transition tuple (of, a 16, vd) can be obtained from the decision corpus.
We absorb both superscripts (i) and subscripts (t) to the index (c) for the conciseness of notions. Later in our work, we use subscripts / to denote the control time step. The variable of E V C I represents a performance metric that is defined based on the user-specified decision objective. e.g.. of may correspond to ongoing cumulative return, immediate reward,
risk-sensitive measure.
understand the decision decision-making process for control time rollouts with the decision corpus. To do this, naive approach is to reconstruct the control time examples with the corpus subset (ot, ag he) Σg (" a", h"), where w [o. 1] and D w" 1. Then the corresponding performance measures of can be used to estimate 2y [a, the control time return of given decision ag Nonetheless, there are critical issues associated with this straightforward method:
Our objective
1. Defining distance metric for the joint observation- actionn--toon-n-nn historr space presents a challenge. This is not only due to the fact that various historical observations are defined over different spaces but also because the Euclidean distance fails to capture the structural information effectively.
2. The conversion from the observation- paattiionaationniittoo space to the performance measure does not necessarily correspond to linear mapping: v + Ee wfe..
To address those difficulties, we propose an alternative approach called Accountable Offline Controller (AOC) that executes traceable offline control leveraging certain examples as decision corpus.
3.3 Linear Belief Learning for Accountable Offline Control
To alleviate the above problems, we first introduce a specific type of belief function b: Ax H + C RRd, which maps a joint observation-action--iiiti input onto a diddimmnsiona belief space.
Property 3.2 (Belief Space Linearity). The performance measure function V : O x A H + can always be decomposed as V = 1 b, where b : O x A x H + B C Rd. maps the joint observation--aciion-iiite space d, dimensional belief variable by = b(o,, az, ht) and 1 B is a linear function that maps the belief b, to output l(b,) e V.
3

---

Remark 3.3. This property is often the case with prevailing neural network approximators, where the belief state is the last activated layer before the final linear output layer.
Remark 3.4. The belief mapping function maps a de + da + (d, du + 1) (t 1) dimensional varied-length input into a fixed-length de dimensional output. Usually, we have = de < d, < d, + dg + (d, + d. + 1) (t 1). is important to highlight that calculating the distance in the belief space is feasible, whereas it is intractable in the joint original space.
Based on Property 3.2 the linear relationship between the belief variable and the performance measure makes beliet space a better choice for example-based decision understanding - this is because linearly decomposing the belief space also breaks down the performance measure, hence can guide action selection towards optimizing the performance metric. To be specific, we have
b(ar lu) = o b(or, ae; hi) = Σ w ao Σ w b a, a Σ w
(3)
Although we have formally defined of the Corpus Subset, its composition has not been explicitly specified. We will now present the fundamental principles and techniques employed to create practical Corpus Subset.
3.4 Selection of Corpus Subset
As suggested by Equation (3), if belief state can be expressed linear combination of examples from the offline dataset, the same weights applied to the linear decomposition of the belief space can be utilized to decompose and synthesize the outcome. More formally, we define the convex hull spanned by the image of the Corpus Subset under the mapping function b:
Definition 3.5 (Belief Corpus Subset). A Belief Corpus Subset b(C) is defined by applying the belief function b to the corpus subset C, with the corresponding set of values denoted as v(C):
) e a ( EC CBV
(C)) = bD = b(o",a", ,h),
(4)
Definition 3.6 (Belief Corpus Convex Hull). The Belief Corpus Convex Hull spanned by a corpus subset C with the belief corpus b(C) is the convex set
C Σ w =1 Gm
CB(C) = Σ wo w E [0.1].00 E ECCC) VC [C),
(5)
Note that an accurate belief corpus subset decomposition for a specific control time belief value b may be unattainable when by & CB(C). In such situations, the optimal course of action is to identify the element b, within CB(C) that provides the closest approximation to 6, Denoting the norm in the belief space as ||- ggg this equates to minimizing a corpus residual that is defined as the distance from the control time belief to the nearest point within the belief corpus convex hull:
Definition 3.7 (Belief Corpus Residual). The Belief Corpus Residual given a control time belief' variable 6, and a belief corpus subset C is given
re(b:)
min b. ECB))())
|bb b.1lB
In summary, when b, resides within CB(C). it can be decomposed using the belief corpus subset according bt Σe w Here, the weights can be understood representing the similarity and importance in reconstructing the control time belief variable through the belief corpus subset; otherwise, the optimizer of Equation (6) can be employed a proxy for b,. Under such circumstances, the belief corpus residual acts as a gauge for the degree of extrapolation. Given those definitions, use the following proposition to inform the choice of appropriate corpus subset.
Proposition 3.8 (Estimation Error Bound for v((t), of. Crabbé et al. (7]). Consider the belief variable by = b(or, ae h4) and b the optimizer of Equation 66, the estimated value residual between l(b:) and z(b) is controlled by the corpus residual:
[[((b) - l(bb)llv I [|4 : TC(())
where ||. Ily is a norm on and ||lloo = inf () E IR+ [|z(b)]lv S A|lblss is the operator norm for the linear mapping.

---

b
Tele
6/2 D (o a ..)
We illustrate the concepts of Corpus Residual and Minimal Hull in a 2-dim example. The belief function b maps trajectories in the offline dataset (colors of blue and green denote that they may come from different behavior policies) onto belief space. The control trajectory with different sampled actions at time can also mapped the belief space. Some those sampled action candidates (e.g.. a.") (2)) can be well-supported by a corpus subset, while in another case (o the corpus residual manifests extrapolation Among the three plotted belief's generated by candidate actions, 6,5) is out of the convex hulls that can be spanned by any decision corpus, 6(2) has the minimal corpus (green shaded area), 6,t) .b(²) are also in larger corpus hull (blue shaded area) but not ideal.
Figure I:
Proposition 3.8 indicates that minimizing the belief corpus residual also minimizes the estimation error. In order to approximate the belief variables in control time accurately, we propose the corpus subset that spans the minimum convex hull in the belief space.
Definition 3.9 (Minimal Hull and Minimal Corpus Subset). Denoting the corpus subsets whose belief convex hull contains belief variable by by
C(b.) = c
Among those subsets, the one that contains maximally dp + examples and has the smallest hyper- volume forms the minimal hull is called the minimal corpus subset (w.rt. by), denoted by Č(b,):
C(b.) arg min { dV
CBB(C)
Proposition 3.10 (Existence and Uniqueness). Consider the belief variable b, = blo,, at fe)), if re(b,) & 0 holds for = D, then Č(b)) exists and the decomposition on the corresponding minimal convex hull exists and is unique.
In this work, represent the control time belief variable bt Those concepts are illustrated in Figure] [
the corpus subset that constructs the minimal (belief corpus convex) hull to
3.5 Accountable Offline Control with Belief' Corpus
Given the linear relationship between the variables e and b, for specific action dz, we can estimate the corresponding i(a.) z(δ) Ec w"r", where o and h4 the control time observation and transition history respectively. Applying an arg-max type of operator to the above value estimator, we able select the action with the highest value that is supported by the belief corpus:
TAOC(or, h4) arg max i(at),
(8)
Flexibility of the Measurement Function: Plug-and-Play, As discussed earlier, the choice of measurement function V depends on the objective of policy learning. For the low-dimensional tasks, we can instantiate V using the Monte Carlo estimation of the cumulative return: v = ΣT at-ter as the most straightforward choice in control tasks; for higher- dimensooii tasks, the measurement metric can be the learned Q-value function; for other applications like when -anstr-constraints are also taken into consideration, the measurement function can be adapted accordingly.
3.6 Optimization Procedure
The optimization process in our proposed AOC method involves three main steps: optimizing the belief function and linear mapping, reconstructing the belief during control time rollout, and
5

---

Algorithm The Accountable Offline Controller
Input
Bached Dataset = {ol: af. ré, of--)== I, control time observation or, control history ht. Hyper- parametens: K for the number of candidate actions, for the threshold.
Output
Accountable action for control.
# Optimize belief function and linear mapping according to Equation
#
2. Sample K candidate actions uniformly in the action space: at ~ U ('A), E [K].
# Find minimal hulls C(be) for each action, based on b and be b(oe, of he). 4. Minimize corpus residuals for each action candidate at according to Equation 10
# Estimate the value of action a; by i(as jo,, hT) = -_n= wº
6. Outputs the candidate action that has the highest estimated value according to Equation 11
performing sampling- based optimization on the action space. In this section, we detail each of these steps along with corresponding optimization objectives. First, the belief function b and the linear mapping I are learned by minimizing the difference between the predicted performance measure and the actual value from the dataset D:
b, I = min Eio,a.b,eje (v to b(o, a, h))²
(9)
6,
Next, with the observation o and trace he, any candidate action de corresponds to belief b, b(or, at he), which be reconstructed by the minimal corpus subset C(b)). according to:
20 arg min 1010
w b(o , Nr) bell. s.t. Σ w =1,
E(b))
(10)
Then the value of action a, can be estimated by ((, h4) Ees wfe Finally, we perform sampling-bas based optimization on the action space, with the corpus residual taken into consideration for the pursuance of conservation. For each control step, we uniformly sample K actions in the action space (A) (A) and then search for the action that corresponds to the maximal performance measure while constraining the belief corpus residual to be smaller than a threshold 2
Tooc(or, h.) arg max Σ w"," s.t. re(b(o,, ai hc)) G.E(a
(11)
c=
The complete optimization procedure detailed in Algorithm
Related Work
Table AOC distinct it satisfies desired properties mentioned earlier: (P1) Conservation: AOC seeks the best potential action in the belief convex hull, such that the estimations of decision outcomes are interpolated,
avoiding aggressive extrapolation that is harmful in offline control [8] (P2) Accountability: the decision-making process is supported by corpus subset from the offline dataset, hence all the decisions traceable; (P3)
Low-Data Requirement: it works in the low-data regime; (P4) Adaptivity: the control behavior AOC can be adjusted according to additional constraints as clinical guidelines without modification or re- te-rrinning: (P5)
Reward-Free: AOC can be extended to the strictly offline imitation setting where rewards are unavailable.
Method/ Property
Conservation
Accountable
Adaptive
Reward-Free
Examples
Q-Learning Episodic Control
8-16
17419
Nearest Neighhor
X
X
20 21
Model-Based RL
X
X
X
/
Behavior Clone
X
AOC
/
Ours
We review related literature and contrast their difference in Table Due to constraints on space, we have included comprehensive discussions on related work in Appendix[B] Our proposed method stands out due to its unique attributes: incorporates properties of accountability, built-in conservation for offline learning, adaptation to preferences or guidelines, is suitable for the low-data regime, and is compatible with the strictly offline imitation learning setting where the reward signal not recorded in the datasct.

---

Experiments
Environments and Set-ups In this section, we present empirical evidence of the properties of our
proposed method in variety of environments. These environments include (1) control benchmarks that simulate POMDP and heterogeneous outcomes in healthcare, (2) a continuous navigation task that allows for the visualization of properties, and (3) real-world healthcare dataset from Ward [26 that showcases the potential of AOC for deployment in high-stakes, real-world tasks.
In Section 5.1 we benchmark the performance of AOC on the classic control benchmark, contrasting it with other algorithms and highlighting (PI-P3). In Section 5.2, Section [.3] and Section 554 we
individually highlight properties (PI),(P2),(P4) Subsequently, in Section 5.5 we apply AOC to
real-world dataset, highlighting its ability to perform accountable, high-quality decision-making
under the strictly offline imitation setting (P1-P5). Finally, Section 5.6 provides additional empirical evidence supporting our intuitions and extensive stress tests of AOC.
5.1 (P1-P3): Accountable Offline Control in the Low-Data Regime
Experiment Setting Our initial experiment employs a synthetic task emulating the healthcare setting, simultaneously allowing for flexibility in stress testing the algorithms.
We adapt our environment, Pendulum-Het, from the classic Pendulum control task involving system stabilization. Heterogeneous outcomes in healthcare contexts are portrayed through varied dynamics governing the data generation process. Data generation process details can be found in Appendix [D.4 Taking heterogeneous potential outcomes and partial observability into consideration, the Pendulum- Het task is significantly more challenging than the classical counterpart.
We compare AOC with the nearest-neighbor controller (1NN) 20 and its variant that using k neighbors (kNN), model-based RL with Mode Predictive Control (MPC) (24), model-free RL (MFRL) [27]. and behavior clone (BC) [25]. Our implementation for those baseline algorithms is based on open-sourced codebases, with details elaborated Appendix [.6] We change the size of the dataset to showcase the performance difference of various methods under different settings. The Low-Data denotes settings with 100000 transition steps, Mid-Data denotes settings with 300000 transition steps, Rich-Data denotes settings with 600000 transition steps.
We compare AOC against baseline methods Table 2 cumulative score among the accountable methods and is comparable with the best black-box controller. In experiments, we find increasing the number offline training examples does not improve the performance of MFRL due to stability issues. Detailed discussions can be found in Appendix [D77
Results
AOC achieves the highest average
Table 2: Results on the Heterogeneous Pendulum dataset. The cumulative reward of each method is reported.
Experiments are repeated with 8 seeds. Higher is better.
Task
Low-Data
Mid-Data
Rich-Data
AOC
-1.39 + 1.39
-1.25 0.40
-0.6 0.08 -645.72 + 220.33
KNN
-849.45 + 91.23
-670.51
321.09
INN
555..07 1 256.64
-690.49 1 152.59
-512.71 1 131.2
BC
422.77 1: 409.51
225.32
340.83
126.74 1: 280.73
MFRL
-4.1 2.76
-11.95
4.68
-15.27 + 6.46
MPC
-1.5 + 0.43
-1.34 0.15
-1.41 0.26
Data-Avg Dala-Avg-Retum Return
3077881 4: 387.53
-245.54
338.65
-208.81 + 272.84
Take-Away: As an accountable algorithm,, AOC performs on par with state-of-the-ar black-box learning algorithms. Particularly in the POMDP task, wherein data originates from heterogeneous systems, AOC demonstrates robustness across diverse data availability conditions.
5.2 (P1): Avoid Aggressive Extrapolation with the Minimal Hull and Minimized Residual
Experiment Setting In Equation [1] ters: the number of uniformly sampled actions and the threshold. this section, we demonstrate how these hyperparameters can be unified and casily determined.
our arg-max operator includes two adjustable hyperparame-
7

---

The consideration of sample size primarily represents balance between optimization precision and computational cost post--alionn although such cost can be minimized by working with modern computational platforms through parallelization. On the other hand, selecting an appropriate value for e might initially seem complex, as the residual's magnitude should depend on the learned belief space.
In our implementation, we employ a quantile number to filter out sampled actions whose correspond- ing corpus residuals exceed certain population quantile, For instance, when e 0.3 with a sample size of 100, only the top 30% of actions with the smallest residuals are considered in the arg-max operator in Equation [T]. The remaining number of actions is referred as the effective action size. This section presents results for the Pendulum-He environment, while additional qualitative and quantitative results on the conservation are available in Appendix F22 Intuitively, using larger sampling size will improve the performance, at sub-linear cost of computational expenses, and using a smaller quantile number will make the behavior more conservative. That said, using a small quantile number may not be always good, as it also decreases the effective action size. The golden rule is use larger sampling size and smaller quantile threshold.
Results Table 3 exhibits the results. Our experiments high- light the significance of this filtering mechanism: removal or weakening of this mechanism, through larger quantile number that filters fewer actions, leads to aggressive extrapolation and subsequently, markedly deteriorated performance. This is be- cause the decisions made by AOC in those cases cannot be well supported by the minimal convex hull, introducing substantial epistemic uncertainty in value estimation.
Table 3: The in Equation [1 con-
trols the conservation tendency of the control time hehaviors.
(quantile)
Performance
0.3
-2.5 + 0.45
0.5
-1.25 + 0.40
0.9
503.39 + 301.60
1.0
853.29 + 138.43
Conversely, using smaller quantile number yields reduced effective action size, which subsequently decreases optimization accuracy and leads to a minor performance drop. Hence when computational resources allow, utilizing a larger sample size and a smaller quantile number can stimulate more conservative behavior from AOC. In all experiments reported in Section 5.1 we find sample size of 100 and 0.5 generally work well.
Twke-Away: AOC is designed for offline control tasks by adhering to the principle of conservative estimation. The hyperparameter. introduced for conservative estimation, can be conveniently determined by incorporating it into the choice of an effective action size.
5.3 (P2): Accountable Offline Control by Tracking Reference Examples
Experiment Setting In healthcare, the treatments suggested by the clinician during different phases can be based different patients, and those treatments may even be given by different clinicians according to their expertise. In this section, we highlight the account- ability of AOC within a Continuous Maze task. Such task involves the synthesis of decisions from multi- ple experts to achieve targeted goals. It resembles the healthcare example above and is ideal for visualizing the concept of accountability.
Map and Sub-Tasks
Decision Corpus Trajectoriss
16
M
13
Coal
12
10
Sattss
10
25
160 125
25 30 Ro
Figure 2: Left: the Maze; Right: trajectorics generated by 4 experts for different tasks.
Figure 2 displays the map, tasks, and expert trajectories within a 16x16 Maze environment. This environment accommodates continuous state and action spaces. We gathered trajectories from 4 agents, each of which is near-optimal for distinct navigation task:
@ agent TT starts at position (0, 0) and targets position (8, 16). Trajectories of m are marked in blue.
agent ø2 starts at position (0, 0) and targets position (8, 8). Trajectories of m2 marked in orange.
agent T3 starts at position (8, 16) and aims position (16, 0). Trajectories of I3 are marked in green.
agent T4 starts at position (8,8) 8) and targets position (16, 0). Trajectories of n4 are marked in red.
We generate 250 trajectories for each of the 4 agents, creating the offline dataset. The control task initiates at position (0. 0), with the goal situated at (16, 0) requires fusion of expertise in the dataset to finish. To complete the task, an agent could potentially learn to replicate T1, followed by adopting Ft's 's strategy. Alternatively, it could replicate Ta before adopting n.'s 's approach.
8

---

Results In this experiment, we employ AOC solve the task and visualize the composition of the corpus subset at each decision step in Figure 3 In the initial stages, the corpus subset may comprise a blend of trajectories from #i and T2. Sub- sequently, AOC selects one of the two po- tential solutions at random, as reflected the chosen reference corpus subsets. Upon crossing the central gates on the x-axis, AOC's decisions become reliant on either T3 or 74, persisting until the final steps, when the corpus subsets again be mixture of the overlapped trajectories.
Percent Decision Corpus
..Prrent Decision Corpus
00
16
26
23
22
14
Figure 3: Visualize Accountability. The upper colors display the components of the corpus subset at different x-positions.
Take-Away: AOC's decision-making process exhibits strong accountability, the corpus subset can be tracked at every decision-making step. This is evidenced by AOC's successful completion of multi-stage Maze task, wherein it effectively learns from mixed trajectories generated by multiple policies at differing stages.
5.4 (P4): Flexible Offline Control under User Specification
Experiment Setting Given the tractable nature of AOC's decision -making process, the agent's behavior can finely tuned. One practical way modifying these decisions to adjust the reference dataset, which potentially contributes to the construction of the corpus subset. In this section, we perform experiments in the Continuous Maze environment with the same offline dataset described in Section 5.3 We modify the sampling rate of trajectories generated by and experiment with both samplng and sub-sampling strategies, using sample rates ranging from [x4, x3, x2, x1, x0.75, x0.5, x0.25]. In the re-sampling experiments (sample rates larger than 1), we repeat trajectories produced by क to augment their likelihood selection as part of the corpus subset. In contrast, the sub-sam sampling experiments partially omit trajectories from π to reduce their probability of selection. each setting, 100 control trajectories are generated.
5.3]
Results Figure shows the results: the success rate (Succ. Rate) quantifies the quality of control time behav- iors, while the remaining two curves denote the percentage
1.00
Percentage
of control time trajectories that accomplish the task by ther following either Ti W3 (passing the upper gate) T2 TA (passing the lower gate). The behavior of AOC is influenced by the sampling rate. Utilizing high sampling rate for the trajectories generated by πi often leads to the selection of the solution that traverses the upper gate. As the sampling rate diminishes, AOC's behavior begins to be dominated by the alternative strategy. In both ra-sampling and sub-sampling strategies, the success rate of attaining the goal remains consistently high. We provide further visualizations of the learned policies under various sampling rates in Appendix
0.50
Succ. Rate
x1 x6
75
x0.5 x0.25
Trajectory Sample Rate of m Figure 4: The behavior of AOC be controlled by re-sampling sub bub-sampling the offfine dataset.
Take-Away: The control time behavior of AOC can be manipulated by adjusting the proportions of behaviors within the offline dataset. Both re-sampling and sub- sab-sampling strategies prove equally effective in our experiments where multiple high togh-peerooning solutions are available.
5.5
(P1-P5): Real-World Dataset: The Strietly Batch Imitation Setting in Healthcare
Experiment Setting We experiment with the Ward dataset 26 that is publicly available with
(8). The dataset contains treatment information for more than 6000 patients who received care on the general medicine floor at a large medical center in California. These patients were diagnosed with a range of ailments, such as pneumonia, sepsis, and fevers, and were generally in stable condition. The measurements recorded included standard vital signs, like blood pressure and pulse, as well as lab test results. In total, there were static observations and 35 temporal observations that captured the dynamics of the patient's conditions. The action space was restricted to choice of binary

---

action pertaining to the use of an oxygen therapy device. In the healthcare setting, treatment decisions are given by domain experts and the objective of policy learning is to mimic those expert behaviors through imitation. However, as interaction with the environment is strictly prohibited, the learning and evaluation of policies should be conducted a strictly offline manner. The dataset is split into 10: 1 training and testing sets and the test set accuracy is used as the performance metric.
We benchmark the variant of AOC tailored for the strictly batch imitation setting, dubbed as ABC as an abbreviation for Accountable Behavior Clone - against three baselines: (1) behavior cloning with linear model (BC-LR), which serves as transparent baseline in the offline imitation setting; (2) behavior cloning with neural network policy class (BC-MLP), which represents the level of performance achievable by black-box models such contexts; and (3) the k-nearest neighbor controller (kNN) as an additional accountable baseline.
Ward Domain Test Set Performance
Results The results of KNN with various choices of & in the experiments, alongside comparison with ABC using
0.98
Test Set Accuracy
the same number of examples in the corpus subset, are presented in Figure 5 Shaded areas in the figure denote
0.96
the variance across 10 runs.
0.94
ABC
KNN
Operating as an accountable policy under the offline im- itation setting, ABC achieves performance similar to that of behavior cloning methods utilizing black-box models. Meanwhile, other baselines face challenges in striking a balance between transparency and performance.
0.92
BC-LR
BC-MLP
0.90
20
40
60
8o
100
Number of Decision Corpus/ in KNN
Yakning Date of Corpus
Thaning Cate
Furthermore, we visualize the corpus subset supporting control time decisions in 2-dimensional belief space to illustrate the accountability. We demonstrate the dif- ferences in the decision boundary where predictions are more challenging. In such cases, the decision supports of KNN rely extrapolation and have no idea of the potential risk of erroncous prediction, while the hetero- geneity in ABC's minimal convex hull reflects the risk. We provide a more detailed discussion in Appendix E.2
Compes
heeiet Da menoia
so Dissencoon
Eellet MiC
Za bnd Corpue duu
Yaknnng
thaning Cata
Hed ol Corpus
Coppus
The variant of AOC, dubbed as ABC, is well- suited for the strictly batch imitation setting. enabling ac- countable policy learning. While other baseline methods in this context struggle to balance transparency and pol- icy performance, ABC achieves comparable performance with black-box models while maintaining accountability.
Take-Away:
Results the Ward dataset. ABC is able to perform accountable imitation under the strict offline setting with high performance comparable with the black-box models.
Figure 5:
5.6 Additional Empirical Studies
To further verify our intuitions and stress test AOC,
we
provide additional empirical analysis in Appendix F Specifically, we analyze the trade-off between
accountability and performance in Appendix F.I we visualize the controllable conservation in Appendix F.2) we benchmark AOC on another benchmark for the general interests of the RL com- munity in Appendix F.3 and show how to combine AOC with black-box sampler in Appendix F.4 Finally, we show AOC can detect control time OOD examples in Appendix F.5
6 Conclusion
In conclusion, this study presents the Accountablc Offline Controller (AOC) as a solution for offline control in responsibliiyyseesiiii
applications. AOC possesses five essential properties that make it particularly suitable for tasks where traceability and trust are paramount: ensures accountability through offline decision data, thrives in low-data environments, exhibits conservative behavior in offline contexts, can be easily adapted to user specifications, and demonstrates flexibility strictly offline imitation learning settings. Results from multiple offline control tasks, including real-world healthcare dataset, underscore the potential of AOC in enhancing trust and accountability in control systems, paving the way for its widespread adoption in real-world applications.
10

---

Acknoledgement
HS acknowledges and thanks the funding from the Office of Naval Research (ONR). HS would thank Jonathan Crabbe for his inspiration and encouragement in exploring this idea in group meeting. We thank Jonathan Crabbe for reviewing our manuscript and providing insightful comments and suggestions for improving the paper. We thank all anonymous reviewers, ACs, SACs, and PCs for their efforts and time in the reviewing process and in improving our paper. This work done with the warm support from the van der Schaar Lab members.
References
[1] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01644 2020.
[2] Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, et al. Scalable deep reinforcement learning for vision-based robotic manipulation. In Conference on Robot Learning, pages 651-673. PMLR, 2018.
[3] Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. Way off-policy batch deep reinforcement learning of implicit human preferences dialog. arXiv preprint arXiv: /90X:v9:70../. 007400556 2019.
[4] Gregory Kahn, Pieter Abbeel, and Sergey Levine. Badgr: An autonomous self-supervised learning-based navigation system. IEEE Robolics and Automation Letters, 6(2):1312-1319, 2021.
[5] Samuel Holt, Alihan Hüyük,, and Mihaela van der Schaar. Active observing in continuous-time control. In Conference on Neural Information Processing Systems, 2023.
[6] Mark Keane and Eoin M Kenny. How case- based reasoning explains neural networks: A theoretical analysis of xai using post-hoc explanation- exxmmee from survey of ann-cbr twin- systems. In Case-Based Reasoning Research and Development: 27th International Conference, ICCBR 2019, Olzenhausen, Germany, September 8-12, 2019, Proceedings 27, pages 155-171. Springer, 2019.
[7] Jonathan Crabbé, Zhaozhi Qian, Fergus Imrie, and Mihaela van der Schaar. Explaining latent representations with corpus of examples, Advances in Neural Information Processing Systems, 34:12154-12166, 2021.
[8] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International conference on machine learning, pages 2052-2062. PMLR, 2019.
[9] Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off- policy q-learming via bootstrapping error reduction. Advances in Neural Information Processing Systems, 32, 2019.
[10] Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advintage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00177 2019.
[11] Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. In International Conference on Machine Learning, pages 104-114, PMLR, 2020.
[12] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33: 1179-1191, 2020.
[13] Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty-base offline reinforcement learning with diversified q-ensemble. Advances in neural information processing systems, 34:7436-7447, 2021.

---

[14] Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances neural information processing systems, 34:20132-20145, 2021.
[15] Rui Yang, Chenjia Bai, Xiaoteng Ma, Zhaoran Wang, Chongjic Zhang, and Lei Han. Rorl: Robust offline reinforcement learning via conservative smoothing. arXiv preprint arXiv:2206.0m 02829, 2022.
[16] Alexander Nikulin, Vladislav Kurenkov, Denis Tarasov, and Sergey Kolesnikov. Anti- exploration by random network distillation. arXiv preprint arXiv:2301.13616, 2023.
[17] Charles Blundell, Benigno Uria, Alexander Pritzel, Yazhe Li, Avraham Ruderman, Joel Z Leibo, Jack Rae, Daan Wierstra, and Demis Hassabis. Model-free episodic control. arXiv preprini arXiv: 16060446000 2016.
[18] Alexander Pritzel, Benigno Uria, Sriram Srinivasan, Adria Puigdomenech Badia, Oriol Vinyals, Demis Hassabis, Daan Wierstra, and Charles Blundell. Neural episodic control. In International conference on machine learning, pages 2827-2836. PMLR, 2017.
[19] Zichuan Lin, Tianqi Zhao, Guangwen Yang, and Lintao Zhang. Episodic memory deep q- networks. arXiv preprint arXiv: 18050068, 2018.
[20] Devavrat Shah and Qiaomin Xie. Q-lcarning with nearest neighbors. Advances in Neural Information Processing Systems, 31, 2018.
[21] Elman Mansimov and Kyunghyun Cho. Simple nearest neighbor policy method for continuous control tasks. 2018.
[22] Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model-based offline reinforcement learning. Advances in neural information processing systems, 13:21810-22 11-1000-222231 823, 2020.
[23] Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. Advances in Neural Information Processing Systems, 33:14129-14142 2020.
[24] Grady Williams, Paul Drews, Brian Goldfain, James M Rehg. and Evangelos A Theodorou. Aggressive driving with model predictive path integral control. In 2016 IEEE International Conference on Robotics and Automation (ICRA), pages 1433-1440. IEEE, 2016
[25] Dean A Pomerleau. Alvinn: An autonomous land vehicle in a neural network. Advances in neural information processing systems, 1, 1988.
[26] Ahmed M Alaa, Jinsung Yoon, Scott Hu, and Mihaela Van der Schaar. Personalized risk scoring for critical care progmosis using mixtures of gaussian processes. IEEE Transactions on Biomedical Engineering, 65(1)2207-218, 2017.
[27] Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pages 1587-1596. PMLR, 2018
[28] Alex J Chan, Ioana Bica, Alihan Huyuk, Daniel Jarrett, and Mihaela van der Schaar. The ledkit-leam (ing) environment: Medical decision modelling through simulation. arXiv preprint Stth:21000042400 2021.
[29] Rui Yang, Yiming Lu, Wenzhe Li, Hao Sun, Meng Fang, Yali Du, Xiu Li, Lei Han, and Chongjic Zhang. Rethinking goal-conditioned supervised learning and its connection to offline rl. arXiv preprint 20X1v:2202.04478, 2022.
[30] Zhihan Liu, Yuteng Zhang, Zuyue Fu, Zhuoran Yang, and Zhaoran Wang. Learning from demon- stration: Provably efficient adversarial policy imitation with linear function approximation. In International Conference on Machine Learning, pages 14094-14138. PMLR, 2022.
[31] Daniel Jarrett, Ioana Bica, and Mihaela van der Schaar. Strictly batch imitation learning by energy-base distribution matching. Advances Neural Information Processing Systems, 33: 7354-7365, 2020.
12

---

[32] Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv:1911.11611, 2019.
[33] Hao Sun, Lei Han, Rui Yang, Xiaoteng Ma, Jian Guo, and Bolei Zhou. Exploit reward shifting in value-based deep-rl: Optimistic curiosity-based exploration and conservative exploitation via linear reward shaping. Advances in Neural Information Processing Systems, 35:37719-37734, 2022.
[34] Ching-An An Cheng, Tengyang Xie, Nan Jiang, and Alekh Agarwal. Adversarially trained actor critic for offline reinforcement learning. In International Conference on Machine Learning, pages 3852-3878. PMLR, 2022.
[35] S Holt, A Hüyük, Z Qian, H Sun, and M van der Schaar. Neural Laplace control for continuous- time delayed systems. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2023.
[36] S Holt, A Hüyük, and M van der Schaar. Active observing continuous-time control. In Conference on Neural Information Processing Systems (NeurIPS), 2023.
[37] Máté Lengyel and Peter Dayan. Hippocampal contributions to control: the third way. Advances in nearal information processing systems, 20, 2007.
[38] Hao Hu, Jianing Ye, Guangxiang Zhu, Zhizhou Ren, and Chongjie Zhang. Generalizable episodic memory for deep reinforcement learning. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Pro- ceedings of Machine Learning Research, pages 4380 4390. PMLR, 18-24 24 Jul 2021. URL hattps://proceee
mlr prross/v139/hu2dd.rr html
[39] Xiaoteng Ma, Yiqin Yang. Hao Hu, Qihan Liu, Jun Yang. Chongjie Zhang, Qianchuan Zhao, and Bin Liang. Offline reinforcement learning with value-basee episodic memory. arXiv preprint arXiv:21 10.09966, 2021.
[40] Zhuo Li, Derui Zhu, Yujing Hu, Xiaofei Xie, Lei Ma, Yan Zheng, Yan Song, Yingfeng Chen, and Jianjun Zhao. Neural episodic control with state abstraction. arXiv preprint arXiv:2301.114900 2023.
[41] Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning, 8:279-292, 1992.
[42] Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529-55 533, 2015.
[43] Gerald Tesauro al. Temporal difference learning and td id-gammon. Communications of the ACM, 38(3):58-46) 68, 1995.
[44] Richard Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
[45] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pages 1889-1897. PMLR, 2015.
[46] David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In International conference on machine learning. pages 387-395. Pmir, 2014.
[47] Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv: 1509002971 2015.
[48] Qing Wang, Jiechao Xiong, Lei Han, Han Liu, Tong Zhang, et al. Exponentially weighted imitation learning for batched historical data. Advances Neural Information Processing Systems, 31, 2018.
13

---

[49] Hao Sun, Zhizhong Li, Xiaotong Liu, Bolei Zhou, and Dahua Lin. Policy continuation with hindsight inverse dynamics. Advances in Neural Information Processing Systems, 32, 2019.
[50] Hao Sun, Ziping Xu, Taiyi Wang, Meng Fang, and Bolei Zhou. Supervised q-learning for continuous control. In Deep Reinforcement Learning Workshop NeurlPS 2022, 2022.
[51] Jiankai Sun, Hao Sun, Tian Han, and Bolei Zhou. Neuro-symbolli program search for au- tonomous driving decision module design. In Conference on Robot Learning, pages 21-30. PMLR, 2021.
[52] Qianggang Ding, Sifan Wu, Hao Sun, Jiadong Guo, and Jian Guo. Hierarchical multi-scale gaussian transformer for stock movement prediction. In ISCAI, pages 4640 4640- 2020.
[53] Arthur Charpentier, Romuald Elie, and Carl Remlinger. Reinforcement learning in economics and finance. Computational Economics, pages 1-38, 2021.
[54] Omer Gottesman, Fredrik Johansson, Matthieu Komorowski, Aldo Faisal, David Sontag, Finale Doshi-Veles and Leo Anthony Celi. Guidelines for reinforcement learning in healthcare. Nature medicine, 25(1): 2019.
[55] Hüyük, D Jarrett, C Tekin, and M van der Schaar. Explaining by imitating: Understanding decisions by interpretable policy learning. In International Conference on Learning Representa- tions (ICLR), 2021.
[56] Jarrett, A Hüyük, and M van der Schaar. Inverse decision modeling: Learning interpretable representations of behavior. In International Conference Machine Learning (ICML), 2021.
[57] Y Qin, F Imrie, A Hüyük, D Jarrett, A E Gimson, and M van der Schaar. Closing the loop in medical decision support by understanding clinical decision- making:: case study on organ transplantation. In Conference on Neural Information Processing Systems (NeurIPS), 2021.
[58] A Hüyük, W R Zame, and M van der Schaar. Inferring lexicographically-ordeee rewards from preferences. In AAAI Conference on Artificial Intelligence. 2022.
[59] A Hüyük, D Jarrett, and M van der Schaar. Inverse contextual bandits: Learning how behavior evolves over time. In International Conference on Machine Learning (ICML), 2022.
[60] Stephanie Milani, Nicholay Topin, Manuela Veloso, and Fei Fang. A survey of explainable reinforcement learning. arXiv preprint arXiv:2202.08434. 2022.
[61] Daniel Hein, Steffen Udluft, and Thomas A Runkler. Interpretable policies for reinforcement learning by genetic programming. Engineering Applications of Artificial Intelligence, 76: 158-169, 2018
[62] Andrew Silva, Matthew Gombolay, Taylor Killian, Ivan Jimenez, and Sung- Son. Op- timization methods for interpretable differentiable decision trees applied to reinforcement learning. In International conference artificial intelligence and statistics, pages 1855-1865. PMLR, 2020.
[63] Li Zhang, Xin Li, Mingzhong Wang, and Andong Tian. Off-policy differentiable logic reinforce- ment learning. In Machine Learning and Knowledge Discovery in Databases. Research Track: European Conference, ECML PKDD 2021, Bilbao, Spain, September 13-17, 2021, Proceedings, Part I1 21, pages 617-632. Springer, 2021.
[64] Abhinav Verma, Vijayaraghavan Murali, Rishabh Singh, Pushmeet Kohli, and Swarat Chaudhuri. Programmatically interpretable reinforcement learning. In International Conference on Machine Learning, pages
PMLR, 2018.
[65] Aman Jhunjhunwala. Policy extraction via online value distillation. Master's thesis, University of Waterloo, 2019.
[66] Guiliang Liu, Oliver Schulte, Wang Zhu, and Qingean Li. Toward interpretable deep reinforce- ment learning with linear model u-trees. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML. PKDD 2018, Dublin, Ireland, September 10-14, 2018, Proceedings, Part II 18, pages 414-422 429.. Springer, 2019.
14

---

[67] Hengzhe Zhang, Aimin Zhou, and Xin Lin. Interpretable policy derivation for reinforcement learning based on evolutionary feature synthesis. Complex & Intelligent Systems, 6:741-753, 2020.
[68] Tom Bewley and Jonathan Lawry. Tripletree: A versatile interpretable representation of black box agents and their environments. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 11415-11422, 2021.
[69] Bradley Hayes and Julie A Shah. Improving robot controller transparency through autonomous policy explanation. In Proceedings of the 2017 ACM/IEEE international conference on human- robot interaction, pages 303-312, 2017.
[70] Xinzhi Wang, Shengcheng Yuan, Hui Zhang, Michael Lewis, and Katia Sycara. Verbal explana- tions for deep reinforcement learning neural networks with attention on extracted features. In 2019 28th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN), pages 1-7. IEEE, 2019.
[71] Upol Ehsan and Mark O Riedl. Human-centered explainable ai: Towards a reflective sociotechni- cal approach. In HCI International 2020-Late Breaking Papers: Multimodality and Intelligence: 22nd HCI International Conference, HICI 2020, Copenhagen, Denmark, July 19-24, 2020, Proceedings 22, pages 449-466. Springer, 2020.
[72] Alexander Mott, Daniel Zoran, Mike Chrzanowski, Daan Wierstra, and Danilo Jimenez Rezende. Towards interpretable reinforcement learning using attention augmented agents. Advances in neural information processing systems, 32, 2019.
[73] Akanksha Atrey, Kaleigh Clary, and David Jensen. Exploratory not explanatory: Counterfactual analysis of saliency maps for deep reinforcement learning. arXiv preprint wrXiv:1912.05744 2019.
[74] Yujin Tang, Duong Nguyen, and David Ha. Neuroevolution of self-interpretable agents. In Proceedings of the 2020 Genetic and Evolutionary Computation Conference, pages 414-424, 2020.
[75] Christian Rupprecht, Cyril Ibrahim, and Christopher Pal. Finding and visualizing weaknesses of deep reinforcement learning agents. arXiv preprint arXiv:1904.01331. 2019.
[76] Prashan Madumal, Tim Miller, Liz Sonenberg, and Frank Vetere. Explainable reinforcement learning through a causal lens. Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 2493-2500, 2020.
[77] Ioana Bica, Daniel Jarrett, Alihan Hüyük, and Mihaela van der Schaar. Learning" " what-if" explanations for sequential decision-making. arXiv preprint 20Xi: 2020.
[78] Andrew Anderson, Jonathan Dodge, Amrita Sadarangani, Zoe Juozapaitis, Evan Newman, Jed Irvine, Souti Chattopadhyay, Alan Fern, and Margaret Burnett. Explaining reinforcement learning to mere mortals: An empirical study. arXiv preprint arXiv:1903.099 2019.
[79] Benjamin Beyret, Ali Shafti, and A Aldo Faisal. Dot-to-dot: Explainable hierarchical rein- forcement learning for robotic manipulation. In 2019 JEEE/RSJ International Conference on intelligent robots and systems (IROS), pages 5014-5019. IEEF, 2019.
[80] Wenbo Guo, Xian Wu, Usmann Khan, and Xinyu Xing. Edge: Explaining deep reinforcement learning policies. Advances in Neural Information Processing Systems, 34:12222- 2021.
[81] James MacGlashan, Evan Archer, Alisa Devlic, Takuma Seno, Craig Sherstan, Peter Wurman, and Peter Stone. Value function decomposition for iterative design of reinforcement learning agents. arXiv preprint arXiv:2206. / 13901, 2022.
[82] Giang Dao, Indrajeet Mishra, and Minwoo Lee. Deep reinforcement learning monitor for snapshot recording. In 2018 17th IEEE International Conference on Machine Learning and Applications (ICMLA), pages 591-598, 2018. doi: 10.1109/ 18.900995
15

---

[83] Omer Gottesman, Joseph D. Futoma, Yao Liu, Soanli Parbhoo, Leo Anthony Celi, Emma Brunskill, and Finale Doshi-Velez. Interpretable off pooley evaluation in reinforcement learning by highlighting influential transitions. In International Conference on Machine Learning, 2020.
[84] Nicholay Topin and Manuela Veloso. Generation of policy-level explanations for reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 2514-25211 2019.
[85] Sarath Sreedharan, Siddharth Srivastava, and Subbarao Kambhampati. Tldr: Policy summariza- tion for factored ssp problems using temporal abstractions. In Proceedings of the International Conference on Aulomated Planning and Scheduling, volume 30, pages 272-280, 2020.
[86] Anton Matsson and Fredrik D Johansson. Case-based off-policy evaluation using prototype learning. In Uncertainty in Artificial Intelligence, pages 1339-1349 PMLR, 2022.
[87] Claire Glanois, Paul Weng, Matthieu Zimmer, Dong Li, Tianpei Yang. Jianye Hao, and Wulong Liu. A survey on interpretable reinforcement learning. arXiv preprint arXiv:2112.11 12, 2021.
[88] George A Vouros. Explainable deep reinforcement learning: state of the art and challenges. ACM Computing Surveys, 55(5):1-39, 2022.
[89] Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv: 1412:355, 2014.
[90] Grady Williams, Nolan Wagener, Brian Goldfain, Paul Drews, James M Rehg, Byron Boots, and Evangelos A Theodorou. Information theoretic mpe for model-based reinforcement learning. In 2017 IEEE International Conference on Roboties and Automation (ICRA), pages 1714-1721. IEEE, 2017.
[91] Rasool Fakoor, Pratik Chaudhari, Stefano Soatto, and Alexander J. Smola. Meta-q-leamning, In International Conference on Learning Representations, 2020. URL htttpss//openrrviei
[92] Hao Sun, Ziping Xu, Meng Fang. Zhenghao Peng, Jiadong Guo, Bo Dai, and Bolei Zhou. Safe exploration by solving early terminated mdp. arXiv preprint arXiv:2107.042000 2021.
[93] Tianwei Ni, Benjamin Eysenbach, and Ruslan Salakhutdinov. Recurrent model-free RL can be a strong baseline for many POMDPs. In Kamalika Chaudhuri, Stefanic Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 16691-16723, PMLR, 17-23 Jul 2022. URL https: v162/ni22a.. html
https::/prrceeedngss pttp:/dinccce...c/co/ mlr www.mmess/
[94] Hao Sun, Ziping Xu, Zhenghao Peng. Meng Fang, Taiyi Wang, Bo Dai, and Bolei Zhou. Constrained mdps can be solved by cearly-termination with recurrent models. In NeurIPS 2022 Foundation Models for Decision Making Workshop, 2022.
[95] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016.
[96] Hao Sun, Boris van Breugel, Jonathan Crabbe, Nabeel Seedat, and Mihaela van der Schaar. Daux: a bass approach for uncertainty explanations. arXiv preprint arXiv:2207001661, 2022.
16

---

Accountability in Offline Reinforcement Learning: Explaining Decisions with a Corpus of Examples
Appendix: Table of Contents
Missing Proofs
18
A.I Proof of the Estimation Error Bound for v((,))
18
A.2 Proof of the Existence and Uniqueness
18
Extended Related Work
19
J Offline RL
19
B.2 Episodic Control and Nearest Neighbor Control
20
B.3 Explainable RL
The Strictly Batch(Offline) Imitation Setting
21
Further Implementation and Experiment Details
22
D.I
Reproduceability: Code
22
D.2
Learning the Belief Space
22
D.3
Addressing the Scalability Issue: Finding the Minimal Convex Hull Elficiently
22
D.4
Data Generation Process: Heterogeneous Pendulum
23
D.5
Hardware and Running Time
23
D.6
Baseline Implementations
23
D.7
The Model-Free RL Baseline and Its Stability Issue
24
Additional Qualitative and Quantitative Results
24
Adaptivity: Visualizing the Decisions of AOC and Quantitative Performance)
24
More Visualization Results and Extended Discussion on the Healthcare Dataset
24
Additional Experiments
26
Trade-Off between Accountability and Performance
26
Visualizing Conservation of AOC
27
Additional Environment: Lunar anderr
27
Black-Box Policy as More Efficient Sampler
29
F.S
Identify Control Time OOD Examples with AOC
29

---

A Missing Proofs
A.1 Proof of Proposition 3.8 (See page
Proposition 3.8 (Estimation Error Bound for v(ae). cf Crabbé et al. (7). Consider the belief variable
by b(or, ae hx) and b the optimizer of Equation 6 the estimated value residual between I(b;) and
1(6) is controlled by the corpus residual:
[b 1(b,)llv & |0p re((b)
where ||. Ily is norm on V and ||4|pp inf {) E g+ ||((6)]b S {|ll[la is the operator norm for the lineur mapping.
Proof. Leveraging the linearity of operator 1, the definition of the operator norm || llop. and Definition 3.7, we have:
[1(b) - l(b))llv ||0(b b.)llv
< |14|| lop ||6 belly
(12)
|| lppp (b.b))
A.2 Proof of Proposition 3.10
3.10 (See page
Lemma A.1 (Affine Independence of C(b))). The elements in the belief corpus built on top of (bb)) as the corpus subset: b E b(((bb)) c B must be affinely independent, that is
C
X =0 Σ = Σ
(13)
Proof. The proof is based on contradiction. Note that by definition, there are de + elements in the belief corpus set. those elements are not affinely independent, it basically means that b, can be expressed in a lower dimensional space. In this case, the composition of the convex set is redundant. Without loss of generality, we use 6² to denote the redundant element
bー
(14)
X
indicating that without 8', we still have b, CBB(C), where C' = =C h2)))) This contradicts that C(b:) is the minimal hull that contains b..
Proposition 3.10 (Existence and Uniqueness). Consider the belief variable b. = b(or, ax, he), if rc(b)) < O holds for = D, then C(be) exists and the decomposition on the corresponding minimal convex hull exists and unique.
Proof. Following the assumption, there is at least one trivial corpus subset that exists for b the offline dataset D the corpus subset such that zero belief corpus residual can be achieved. finite number. The existence of
The existence of C(b,) follows the fact the cardinality of the decomposition on C(b.) is then a consequence of the definition of the minimal hull.
Based on Lemma A.I the elements in C(b,) construct the minimal hull and are affinely independent. The uniqueness of the decomposition can be shown by contradiction:
Assume that there are two different decompositions for composing the belief state by:
b,
We.
(15)
*otherwise, it falls into out-of distriibbtti prediction problem and is beyond the scope of this work. Nonetheless, note that AOC able to perform OOD detection. See section ES
18

---

where w", E 10, E w = =1 w 1. In this case, we have
Σ
W Σ
==1
Σ
-w-- -60)
(16)
Σ
Xb6 where N -
However, Σ =c= x== Ec - = w Ee t 1 contradicts with the
fact that the elements affinely independent. Hence the proof is completed.
B
Extended Related Work
B.1
Offfine RL
15 has gained increasing attention in recent years due its potential for solving practical problems, such as robotics control and game playing, where collecting new data can be expensive or time-consuming, However, also presents several challenges, such as distribution shift and overfitting, which can lead to poor performance when deploying the learned policy to the actual environment. There are several model-free approaches to addressing these challenges Offline RL. Such as distributional matching 8, 31 regularization techniques to prevent overfitting (32). conservation [T2 33 adding noise [[5] to the policy or using adversarial training [4] A third
Offline RL
approach is to incorporate uncertainty estimation to evaluate the performance of the learned policy on unseen data 13
On the other hand, model-based RL tackles the problem by first learning world models and then performing planning algorithms on the learned model [22 23 35 36). In either model-based or model-free approaches, black-box approximators are used; hence, the decision is not transparent.
In offline-RL, both model-based and model-free approaches leverage black-box approximators. As a consequence, the pursuance of accountability can not be achieved through those conventional algorithms.
would like to note that, although AOC also studies control problems under the offline setting, its focus goes beyond the conservative efficient learning objective in offline-RL literature. As we have demonstrated in the Table T AOC has five distinct properties that are all crucial for accountable offline control tasks:
(P1). Conservation.
(P2). Accountability.
(P3). Low-Data Requirement.
(P4). Adaptivity.
(P5). Reward-Free.
Below, we further explain those properties and corresponding methods in turn. For each of the properties, start with introducing the definitions, followed by comparisons among AOC and MFRI,, MBRI, and BC. The discussion on MFRL and MBRL. includes the offline-RI. algorithms.
(I) Accountability: the concrete examples in the offline dataset.
making process is traceable, and the decisions can be supported by
AOC: The decision -maiing process of AOC is supported by corpus subset from the offline dataset, hence all the decisions are transparent and traceable.
MFRL: In MFRL, black-bor box value network and black-box policy network are learned with the offline dataset. There is no decision support for black-box policies.
MBRL: In MBRL, a black-box world model optimized with the offline dataset is used as proxy of the actual dynamics, and planning algorithms are then applied to such a black-box model to make decisions. Those decisions are not supported by explicit references.
19

---

BC: In BC, a black- box policy is learned through supervised learning. The output of such policy is hard be linked with specific training examples.
(2) Conservation: estimations decision outcomes are interpolated, avoiding aggressive extrapolation that is harmful in offiine control.
AOC: AOC performs conservative decision-making by using decision supports within minimal convex hull. How such decomposition in the convex hull improves conservation is justified theoretically by Proposition 3.8 (Estimation Bound) and Proposition 3.10 (Existence and Uniqueness).
MFRL: In MFRL like CQL and TD3-BC, the conservation is explicitly given as constraints distribution matching. We would note that conventional MFRL algorithms are not designed for those tasks and suffer from aggressive extrapolation.
MBRL: Similar MFRL,, external efforts should add conservation to MBRL. Because the conventional design of model-basee learning does not address such an issue.
BC: In BC, the learning objective is minimize the prediction difference. There is little can do to aid conservation.
(3) Low-Data: whether method works in the low-data regime.
AOC: The decision process of AOC only relies on few examples forming the minimal convex hull, hence the algorithm performs well under the low-data regime, making it generally applicable to many real-works data-scarce tasks.
MFRL: In MFRL,, the black-box value network and policy network can be designed to be sample-efficient.
MBRL: In MBRL, sufficient data is always required to learn an accurate world model.
BC: the performance of BC is highly dependent on the data quality. is not designed for the low-data regime.
(4) Adaptive: whether the control behavior of method can be adjusted according to additional constraints as clinical guidelines without modification or re- re-training.
AOC: by changing reference examples, i.e., the decision corpus, during test time infer- ence, AOC can seamlessly perform different types of decision-making according to user specifications.
MFRL: In MFRL, when new data is used, a new value network and policy network need to re-trained.
MBRL: In MBRL, the world model construction is independent of the data, hence the decisions can be adaptive by changing new planning algorithm top of the world model. No model re-training is needed.
BC: In BC, new model needs to be trained with a specified type of decision corpus.
(5) Reward-Frec: availability of extension to the strictly batched imitation setting where rewards are unavailable.
AOC: AOC can be extended to the strictly batched imitation setting where rewards are unavailable.
MFRL: In MFRL, the Q-values can not be calculated without the reward function.
MBRL: In MBRL, the planning algorithms do not have clear objective to optimize without reward signals.
BC: BC is not affected by the absence of reward signals, because does not need the reward to learn its policy.
B.2 Episodic Control and Nearest Neighbor Control
Episodic Memory and Episodic Control [17199 inspired by biological learning mechanism, are Follow-up works introduce various modifications and extend episodic control the continuous domains. e.g., Hu et (38) introduces Generalized Episodic Memory (GEM) which effectively organizes the state-action values of episodic memory generalizable manner and supports implicit planning on memorized trajectories. Ma et [39] leverages episodic memory in offline RI. setting with pessimistically estimated value function. Li et [40) proposing a novel state-abstractor framework for episodic control and improving the learning efficiency in continuous control benchmarks. In all those works, value function
studied as an alternative way of policy learning 37
20

---

resembling the hippocampus episodic memorization mechanism is introduced as an alternative to Q- learning 42 that updates the state-action values with temporal difference learning or Monte-Carlo
estimation 43 45
In the continuous control domains, policy gradient methods 46 47
or supervised learning- based methods [48] 49 IO 50) are then applied for the policy improvement. All of those approaches are limited to black-box value-based learning and require additional black-box policy networks in the continuous control domain, whereas our proposed method performs a transparent decision-making
process without explicit policy learning.
Explicit nearest neighbor methods that perform decision-makinn according to training-ting time similar trajectories have been studied theoretically (20] and empirically [2I]. Although those methods also enjoy transparency, they suffer from the problem of the curse of dimensionality. Moreover, defining the nearest neighbor with heuristically determined Euclidean metric also suffers the problem of aggressive extrapolation and thus is not suitable for the offline setting g 12
B.3 Explainable RL
Understanding the decisions made by RL agents is key issue in many high-stake real-world
domains such as autonomous driving [51], finance [3] and healthcare [54] (8 [55] Previous Explainable-RI RL (XRL) literature can be broadly classified with taxonomy of three classes [60]
(1) Feature importance, that includes learning policy through an explainable policy class 61 63
converting black-box models into interpretable format (64.68), and natural language 69
or saliency map based explanations 72H774): (2) Transparent Learning Process that reveals the influences of MDP ingredients during the learning process, including methods that learn predict the counterfactual outcomes for mecision-making (75777), decompose the learning objective [78]81], and identifying the crucial training datum [2] 83:: (3) Policy Level, which illustrates the long-term behavior of the agent (6). For more extensive discussions on XRL literature, we refer the readers Different from previous approaches. our work introduces the first example-bast based explanation for policy learning. Supported by training dataset examples, the execution of our control algorithm is accountable.
also to [8 88
C The Strietly Batch(Offline) Imitation Setting
The instant reward may not be contained in the offline dataset in strictly batch imitation (SBI) [3 learning settings such as clinical treatment and healthcare scenarios. In such cases, the value of actions can not be estimated through Monte Carlo, and it is generally impossible to learn belicf state based on the value prediction. In such a case, we need to adapt the Accountable Offline Controller accordingly.
In such setting, the dataset Deni = (ot, ai, of li=1....N 0411T contains only sequential observations of, i [N], [T], actions af E [M], E [T] performed by behavior policies, which is always an expert or near-expert controller, and transition histories h4 i [N], E [T] that can be composed of the former quantities.
The Accountable Offline Controller should be adapted to handle this setting. Specifically, we still define the corpus subset as
Definition C.1 (SBI Corpus Subset). A SBI Corpus Subset CSBI is defined subset of offline dataset DSBI, indexed by [C] = 21,2,.., C}- - the natural numbers between and C.
{ (08, a0,a", 11)) E DEEI c |0|
CSBI
(17)
Property C.2. (SBI Linear Restricted Belief) The policy function т 0 x x A can be decomposed as = lo b, where b: O H B &h maps the joint abservation-histon space to de dimensional belief variable b. = b(or, hz) and L: B + A is a linear function that maps the belief b, to an output l(b:)= E A.
Then any control time behavior generated by policy æ is accountable in the sense that it can be decomposed by the belief corpus, defined as
21

---

Definition C.3. (SBI Belief Corpus) A SBI Belief Corpus b(CsBi) is defined by applying the belief function b the corpus subset CSBI.
= ={0= b
b(o", bloo,,)) 10) a ,,, E CSBI B,
b(CsBI) =
(18)
on top of which define the Belief Corpus Hull in the SBI setting:
Definition C.4. (SBI Belief Corpus Hull) The SBI Belief Corpus Convex Hull spanned by corpus subset CSBI with the belief corpus b(Csmt) is the convex set
CB(CsBB) Σ b w 10, b(Cmm)), Ve [C1, Σ w = 1
(19)
followed by the concept of Minimal Hull in the SBI setting defined as
Definition C.5 (SBI Minimal Hull). Denoting the decision corpora whose belief convex hull contain belief variable 6, by
(b,)(b,) = CSBB S DSBI b, E CB(CsBI)
Among those subsets, the one that contains d, + decision corpora and has the smallest hyper-voluma forms the minimal hull, denoted by CsBi(ba):
CSBI (b.)
min Cant
dV
CB(CsBt)
Similar to the property in AOC, in the SBI setting, the control time decisions can be decomposed with examples in the offline dataset that constructs such SBI Minimal Hull,
a(arlo,,h)) = o b(o, ho) = to w b(o 19)
Σ w Lo b
Σ w
where (o a,,,,))) hO) (snr(be)-
D Further Implementation and Experiment Details
D.1
Reproduceability: Code
We elaborate implementation details in this section. Our code is available at https://
D.2 Learning the Belief Space
To efficiently encode the information in historical transition, in our work, we employ the Gated Recurrent Units (GRU) [89 to map fixed-length transition histories into embedding vector variables, followed by 3 fully connected layers as the belief function b. In principle, any other recurrent networks should also able to process such context information. Table presents the syper-parament we use in the belief learning process.
D.3 Addressing the Scalability Issue: Finding the Minimal Convex Hull Efficiently
Rigorously searching for the minimal convex hull in the belief space combinatorial optimization problem. In our implementation, we leverage heuristic search method that first reduces the search space by looking for k-nearest neighbors in the belief space, and then build the approximate minimal convex hull on top of those k-nearest neighbors of the control time belief b, With dy dy-dimensional belief space, the minimal convex hull will contain at most d, examples, hence we can set k = 2(d, + 1), and conduct the combinatorial optimization on those & examples, which is much easier than the original problem (reduce combinatorial problem Select de + out of N into Select do + 1 out of k).
22

---

Table 4: Hyperparameters in learning the belief function.
Hyper-Param
Choice
Context Model
GRU
Hidden Unit Number
128
Hidden Recurrent Layer
Batch Size
500
Epochs
4000
Optimizer
Adam
Learning Rate
0.001
Memory Length
Embedding Dimension
20
D.4 Data Generation Process: Heterogeneous Pendulum
The classical control task of Pendulum has the goal to swing up and balance pendulum using control imput. the control task, policy can apply torque to the joint order to swing the pendulum up and then maintain its upright position. The state of the system is defined by the pendulum's 's angle and angular velocity, and the action the torque applied the joint. The reward function typically provides a positive reward for keeping the pendulum upright and negative reward for large torques or deviations from the upright position.
To manifest the potential heterogeneous outcome in healthcare and generalize the study into POMDP. we consider a heterogeneous variant of the original task. There are two contradictory Pendulum systems: the normal one and the converse one. While in the first system, adding torque will lead to dynamical change according the original physical design, in the converse system, the torque inputs will be sent to the system with a negation.
We train TD3 policies in each system and merge the collected dataset together the offline dataset. In the Low-Data settings, 50000 transitions of each environment are collected, hence the dataset contains in total of 100000 transitions; in the Mid-Data regime, 150000 transitions of each environment collected, hence the dataset contains in total of 300000 transitions; in the Rich- tich-Data regime, 300000 transitions of each environment are collected, hence the dataset contains in total of 600000 transitions.
In order to achieve high performance, the agent must be able to identify the decision corpora that are collected from the same system dynamics in the control time.
D.5 Hardware and Running Time
We experiment on a machine with 2 TITAN X GPUs and 32 Intel(R) E5-2640 CPUs. In general, the computational expense of model training in AOC is cheap, as the neural networks used in AOC in general shallow and of small scale. Learning the belief space requires half the calculation of building world model. However, we acknowledge the exact calculation of the minimal convex hull large-data regime can be computationally expensive. And we have discussed our practical solution (Appendix D.3). With our proposed solution, the convex hull decomposition takes less than 10 seconds with uniform sampler that samples 100 actions randomly for every time step. Increasing the dimension of belief space size will lead to sub-linear increase in computational time with parallelization.
D.6 Baseline Implementations
Except for standardized components that we will introduce below, we use the publicly available source code when constructing the benchmark algorithms; references are in the following:
Benchmark Algorithms
kNN and INN: httpp://scckk-laamooo....eeggggp Reference: 220
BC: Implementation is stragghtfooowad using supervised learning.
MFRL: http::////ggttuubooooo/iini////n Reference: [27].
MPC: http://thhbccoo/////MMMiayyyyyup
Reference: 990
23

---

Neural Network Backbones In all baseline methods and our implementation for AOC, we use the same network structure: 3-layer MLP with a recurrent model that encodes the historical trajectory information, which is called as the Context Variable in the literature [91. Our implementation of the recurrent model is based on the open-sourced code of htttp://ggthub.comm/// httpa////m/ooosiiiieeeeeeeeeeeee q-learning In all experiments, we use the same neural network architecture and match the hyper- parameters for fair comparison, except stated otherwise (e.g., the Q-learning baseline requires larger batch size to guarantee convergence and boost stability, which will be elaborated in the following section.).
D.7 The Model-Free RL Baseline and Its Stability Issue
Our implementation of the Q-learning baseline leverages the twin-delayed techniques 227 to stabilize training. We find that Q-learning requires large batch size (i.e., 10240) and many optimization epochs to converge. In the experiment settings with more offline data, the convergence becomes even harder: the rich-data regime containing 6x more data takes 6x more training epochs to converge. And the converged performance always with high vibration, leading to worse performance.
Learning curves are shown in Figure experiments are done with 8 seeds and both averaged learning curves and individual curves are plotted.
QL. Loarning Curvelst: Low Dota
QL Leamiin Curvelst: Nid Data
OL Leaminng Curveist: Rich Data
Evoluaton Botum
-250
-29
Come
Ex 2l Rebum
Sed1
-2000
1243
4000 2000 12500
2500 2000 M00
12000
2000
1000
4100
50.00
1300 45300 69500 190i 19900
Theihing Epochs 105-10240)
Tsining Epocts (0S-102240)
"ssining Epochs (0S=10240)
Figure 6: Q-Learning learning curves. When the size of offline data increases, more training epochs are needed for the convergence, and the stability of performance at convergence is reduced, leading to larger variance and poorer performance in the rich-data regime.
E Additional Qualitative and Quantitative Results
E.1
Adaptivity: Visualizing the Decisions of AOC and Quantitative Performance
We plot the control time trajectories in experiments of Section 5.4 in Figure 7
It clearly shown that with decreasing sampling rate of n, 's trajectories (passing through the upper gate) in the offline dataset, the control time behaviors tend to choose more actions following the behaviors of n3 (passing through the lower gate).
Quantitatively, Table shows that while changing the sampling rate of trajectories from TT, the success rate does not change much while the proportions of strategies chosen by the control policy vary accordingly.
Table 5: Quantitative results in the re-sampling and sub- somplin experiments. In all settings, 100 trajectories in total generated using the proposed method. The success rate of reaching the goal and choices of solutions are presented in the table.
Re/Sub- Samp-e
x4
x3
x2
x1
x0.75
x0.5
x0.25
Success Rate
0.91
0.97
0.90
0.92
0.92
0.93
0.92
Passing Upper Gate
76
69
61
42
31
24
17
Passing Lower Gate
15
28
29
50
61
69
75
E.2 More Visualization Results and Extended Discussion on the Healthcare Dataset
We provide more qualitative results in Figure & In those figures, the colors yellow and blue indicate different treatments in the training data, separately.
24

---

Map and Sub Tasks
Decision Corpus Trajectories
Samplc Rate of my; x4
Fiod
12-
Start ft. tb
12
10
Start
10
2-
10
16
20
Sample
Sample
Sample my.
10
16
10
14
14
14-
17-
12-
12-
10-
10-
10-
Sample Rate of x0.75
Sample Rate x0.5
Sample Rate x0.25
21
2
2-
Figure Visualization of control behaviors. The first two plots show the task and collected trajectories in the offline dataset. This is the same environment we have used in Section [.3 The control time preferences can be controlled by changing the sub-sampling or re-sampling rate.
In the realm of healthcare, treatment decisions involving atypical patients - those with rare or non-standard characteristics that not closely resemble the majority of training examples - present significant challenge. These outliers often reside on the boundaries of existing treatment records of patients, rendering the optimal course of action ambiguous even for domain experts.
Our proposed method AOC provides a critical advantage in these complex scenarios. AOC constructs minimal convex hull around such patients, unearthing potential risks associated with incorrect treatment approaches. In cases where the decision corpus spans disparate treatment groups, AOC reveals the impossibility of establishing a more representative convex hull that both contains the patient and exclusively includes members of the same treatment class.
Conversely, the k-nearest neighbors (kNN) approach fails to identify these high-risk patients. The nearest neighbors for such boundary cases can all belong to the same treatment group, thus offering warning signals that the patient might be atypical and warrant special attention.
Therefore, AOC outperforms treatments, but also in its ability to flag patients whose decision basis exhibits heterogeneity at the test time. In these instances, doctors or other human experts can give particular consideration, thereby enhancing the trustworthiness of the decision system.
kNN method not merely terms of higher accuracy suggesting
25

---

Bellef Corpus: ABC
Bellef Corpus MN
Betef Corpus: ABC
Be Corpus WN
Training Data
Tra Data
Training Data
Traning Data
Test Data
Test Data
Test Data
Test Data
Ballef Corpus
Bellef Corpus
Ballel Carpus
Balief Corpus
Cimandae
Baller Cimenniie
Bodie Cir comarnii
Bellef Dimensson
Bellef Dimension I
Bel Dimersion
Be Dimension
Deliet Compus: ABC
Beite Corpast 1N8
Bett Corpus: ABC
Be ler Corpus IIN
Training Data
Taannn Data
Training Data
Teatring Data
Test Dats
Test Data
Test Data
Test Data
Bellef Corpus
Belief Corpus
Bolle' Corpus
Bellef Co
Boloe Dosartor 11
Boler Crincro
Boler Sireee
Sellef Dimenssoo
Bellef Dimension 1
Bal Dimersion 1
Be Dimension
Sullef Corpus: ABC
Bollaf Corpas: WNN
Bal of Corpus: ABC
Be Corpus: KNN
Training Data
Trai ng Data
Training Data
Training Data
two thate
lest Date
test Data Bellef Corpus
lest Deta
menzznn
Bellef Corpus
Cenasooo 2
Belee Corsus
L.Massuu O Beea
Bellef Corpus
Buise Dawe Comeenan
Sellet Oimension
Belled Dimention 1
Bet Dimersion
1
Be Dimension
Figure 8: Visualization of the control time decision supports the healthcare dataset. The two colors of the scatter plots denote different training time decisions. Test data is denoted by red cross marks, and the identified belief corpus subsets are marked with black circles.
F
Additional Experiments
F1 Trade-Off between Accountability and Performance
Experiment Setting In principle, the minimal convex hull in the d-dimensional belief space contains d + examples. However, searching for such a minimal convex hull combinatorial optimization problem and can be extremely hard in high-dimensional space with many samples. In our implementation, we leverage heuristic search method that constrains the combinatorial search inside the k-nearest neighbors of given control time belief state.
We change such a hyper-parameter the maximal number of examples in the corpus subsets can be used in building the minimal helief space convex hull. We experiment with A = 1, 10, 20, 100 separately.
Results Table shows shows the results. To make AOC work, the choice of k should at least roughly match the dimension of the latent space. Otherwise, the aggressive extrapolation will hinder the performance of AOC, including using only the nearest neighbor as approximation. While such choice naturally trades off between explainability and decision quality, our empirical study shows using a redundant set of the corpus will hinder the performance this demonstrates the necessity of using the minimal convex hull AOC.
The minimal convex hull design in AOC is crucial for performance. We recommend the choice of size constructing minimal convex hull should match the dimension of the belief space.
Take-Away:
26

---

Table 6: The cardinality of the corpus subset is a preer-pareettr that trades off between accountabil- ity and performance. While in general using a larger number of corpus examples can better support the control decisions, it also has risk of increasing the corpus residual in synthesizing the control time decision, hence hindering the performance of the control policy.
K
Performance
1
670.51 + 321.09
10
-510.39 + 311.24
20
-1.25 + 0.4
100
-2.06 + 0.55
F22 Visualizing Conservation of AOC
Experiment Setting To better illustrate the conservative behaviors introduced by the hyper- parameter E, conduct experiments Two Gates Maze environment to visualize the differences of using different choices c's.
In the Two Gates Maze task, an agent needs to navigate to goal located at (16, 8), the middle point of the right side wall, starting from (0, 8), the middle point of the left side wall. Two gates open in middle wall located at 8, hence the agent needs to pass one of those gates to reach the goal.
We generate offinn dataset from behavior policies, each of which selects of the two gates to pass the middle wall. The first two plots in Figure show the map as well as the behavior trajectories as the dataset.
We then experiment with different choices of e = [0.1, 0.3, 0.5, in AOC to perform offline control.
Results Results are shown in the last 4 plots in Figure (Purple lines denote the control time trajectories, ideal conservative policies' behaviors should be bounded by the behavior trajectories). In each setting, we roll out with 100 trajectories by AOC and visualize the behaviors, and report the averaged performance in Table 7 When a small is used, the control time behaviors show little extrapolation: trajectories are in general surrounded by the dataset behaviors. When becomes larger, more aggressive extrapolations emerge control time behaviors, leading to poorer performances.
Table 7: The in Equation
contributes to the conservative behaviors of AOC.
Performance
0.1
6.97 + 2.57
0.3
2.58 = 5.13
0.5
-3.10 = 1.00
0.7
-3.20 0.00
Take-Away: Using a small leads to more conservative behaviors in AOC. AOC performs offline control avoiding aggressive extrapolations by constructing the Minimal Hull and performing control with the decision corpora that have minimized residuals.
E3 Additional Environment: LunarLanddrrCaniinuug
Experiment Setting We additionally experiment on the LunaLLaddrComimmouus-22 environ- ment [95] for the general interests of the RL community. The LndaraLLandCCCniiuuuu-wy en- vironment is a physics-bar based simulation game in which the agent must control a lunar lander to successfully land on designated landing pad. It has 2-dim action space and 8-dim state space, both of which are continuous.
We compare AOC with the nearest-neighbor controller (1NN) 20 and its variant that using k neighbors (kNN), model-based RI. with Mode Predictive Control (MPC) (24, model-free RI. (MFRL) 27,. and behavior clone (BC) (25). We change the size of the dataset to showcase the
27

---

Map: Two Gates Maze
Trajectories
== 0.1
Start
12-
12-
10
10
10
6-
2-
2-
E 0.3
E 0.5
0.7
14
14
12-
12-
12-
10
10-
10
2-
12.5
Figure 9: Visualizing the conservative behaviors controlled by the threshold controlled by quantile number c. Using a smaller leads to more conservative behaviors hence benefiting the offline control problems where aggressive extrapolations can be dangerous.
performance difference of various methods under different settings. We experiment with different offline data availability, varying from 0.1M to 1M.
Table 8: Performance comparison on the SamarrLanddrConinuuuus environment under different settings (availability of offline data). The episodic cumulative reward each method is reported. The last row (Data) reports the performance of trajectories in the offline dataset. Higher is better.
O.IM
0.2M
0.3M
0.5M
IM
AOC
100.77 + 174.5
184.83 + 88.28
240.81 = 44.47
252.38 + 45.66
253.71 71 + 25.11
KNN
-86.07 172.57
-43.87
69.1
-3.04
116.76
59.66
143.33
-31.47
152.72
INN
-42.2 31.36
98.27 1 89.35
14.93 + 115.75
57.37 1: 59.1
-64.41 +: 69.71
BC
-130.23 + 33.23
-18..61 + 42.18
-36.91 = 57.32
18.08 35.4
54.46 + 69.16
MFRL.
92.0
82.05
165.11
54.48
221.3 19.63
219.55
38.97
254.55 +: 24.42
MPC
-31.42 + 32.09
35.52 + 43.09
-00.66 22.96
67.88 + 65.49
96.22 + 85.01
Data
149.99 133.45
207.92
38.34
171.19
100.43
169.42 96.02
184.92 69.59
Results We present the results in Table g In all settings, we find AOC is able to achieve an on-par performance of black-box decision-making algorithms, and outperforms the accountable baselines. In this environment, we find the model-based approach is not able to learn a well well-performing reward function approximator. On the other hand, different from the Pendulum-Het settings where the stability of the balanced state is essential in achieving high performance, the stability of the model-free method in this environment not an issue.
The results the properties of AOC: while being accountable, achieves similar performance as the black-bou learning algorithms and is robust under different data availability.
Take-Away:
environment again demonstrate the desired
28

---

F.4 Black-Box Policy as More Efficient Sampler
Experiment Setting In the main text, we introduce the uniform sampling approach for control time execution. While such approach is simple and effective in our accountability-sensiin sensiivvee control benchmark tasks, can be inefficient high-dimensional continuous control tasks. For the interest of the general continuous control community, we experiment with higher dimensional task in this section to stress test the capability AOC in more challenging tasks.
We experiment on the dimensional action space. Different from previous experiments where a uniform sampler applied, in this section, we use black-box box models as the more efficient sampler and leverage AOC as a post- hoe interpreter for decision accountability. Specifically, we compare AOC with a uniform sampler (AOC-Uniform) and AOC with Behavior Clone policy as a black-box sampler (AOC-BC) against the same baselines as previously. i.e., the nearest-neighbor controller (INN) (0 and variant that using k neighbors (kNN), model-based RI. with Mode Predictive Control (MPC) 24,, model-free RI (MFRL. [27]. and behavior clone (BC) 25). improve the black-box controller's performance and hence isolate the source of gain, we use the offline dataset of size 1M.
environment that has 24-dim observational space and 4-
Results Results are reported in Table 9 In this high-dimensional control task, the uniform sampler inefficient and fails to converge well-performing policy in methods, BC achieves the best performance. And AOC with BC as its sampler achieves improved performance, while the same time being accountable.
control time. Among all black-box
Table 9: Performance on the BipedalWalker-t environment. The episodic cumulative reward of each method is reported. Higher is better.
Method
Performance
kNN
1009.72 + 5.85
INN
-111.95 + 8.25
AOC-Unifome
-90.44 + 14.06
AOC-BC
276.98 = 82.78
BC
208.72 + 95.72
MFRL
18.51 + 111.53
MPC
-96.82 + 24.7
Anta-Avg-Retur
202.25 + 102.61
Take-Away: AOC can work both in isolation combined with black-box policies. AOC can be used as plug-in to add accountability to black-box controllers in post-hoc manner. In high- dimensional control tasks, uniform sampling can be inefficient and black-box samplers can alleviate such a difficulty.
F.5 Identify Control Time OOD Examples with AOC
Experiment Setting In this section, we show that AOC can be applied to OOD example identifi- cation (96) during control time. Specifically, conduct the experiment with the BipedalWalker environment. During control time, black-box controller described the last section is used, and we start to inject Gaussian noise into the control actions disturbance after the 500-th timestep to create an OOD scenario.
Results We repeat the above process for 10 times and report the averaged step-wise instant reward curve and corpus residual curve during rollouts in Figure 10 We note that in the figure in the beginning steps, the walker starts from a static state to walk, thus leading to an increase in the instant reward curve. We then label the control steps between the 100-th and the 500-th step as the stable walking phase, during which the walker receives a nearly constant instant reward (around 0.4 per step). and the corpus residual during this period is stable and close to O. The Gaussian noise is injected after the 500-th timestep, leading to clear increase in the corpus residual curve and a sudden decrease in the instant reward curve. According to the corpus residual values" sudden increase, the
29

---

0.5
Instant Reward
Corpus Residual
0.3
Noise Injection
0.2
Muyhi M1
0.1
Starting
Stable Walking Phase
OOD Phase
-0.2
Phase
0.3
100
200
360
400
500
Control Time
Figure 10: AOC can be used for OOD detection in control time. The corpus residual values can be used to detect large epistemic uncertainty OOD examples. Shaded areas in the figure denote the 3-sigma interval.
control time OOD examples can be identified according to the 3-sigma rule of thumb. (The 3-sigma intervals are marked with shaded areas in the figure).
Take-Away: Control time OOD examples can be identified by AOC according to the sudden increases the corpus residual value.
Limitations and Future Work
In this study, our exploration of accountable offline control primarily targets low-dimensional con- trol tasks, drawing inspiration from healthcare applications where treatments are typically low dimension. However, the uniform sampling approach dimensional control systems, which can be of interest in the field of robotics study. We do offer an initial exploration of using black-box samplers in AOC within our appendix, but there remains ample room for further work in enhancing the efficiency of the sampling process. Additionally, there's potential for expanding AOC into online control settings by combining it with optimism in the face of uncertainty (OFU) explorers, to pursue accountable online control. This, however, lies beyond the scope of our current paper.
propose may not perform as well in high-
Broader Impact
In this study, we examine the offline control problem, which holds significant potential for applications in costly, safety-sensitive, and critical domains such healthcare and finance. While previous works have primarily focused on efficient learning in offline settings, the accountability of offline decisions remains largely unexplored despite its importance.
In critical domains like healthcare, it's vital that decisions are based on supportive evidence. For instance, when patient is treated in certain manner, it should be based the successful outcomes of previous patients with comparable conditions who received the same treatment. The ability to trace the supportive basis of decisions enhances the process of policy reasoning and debugging, thereby improving the trustworthiness of decision-making systems.
However, we bring to light the potential risk associated with applying AOC to critical real- world decision-making systems. This risk stems from potential heterogeneous outcomes, i.e., the aleatoric uncertainty associated with decision outcomes. For example, similar patients undergoing the same treatment may experience different results. Therefore, when the variance outcomes in the corpus subset is high, users should exercise caution regarding the potential heterogencous outcomes of decisions.
30


Reviews:
[{"review_id": "uEUl800r9q", "reviewer": "Reviewer_fhj3", "summary": "I found the paper somewhat hard to read and understand, so here I’ll present a summary that’s quite different from the author’s presentation.\n\nIn offline RL, or other settings where there is a performance metric to optimize, we can consider two simple baselines:\n1. Nearest neighbors: For each action, find the most similar transition(s) in the dataset, and use those to estimate the value of the action, then take the action estimated to be best.\n2. Supervised learning: Train a model to predict the value of each action, and use that to estimate the value of actions.\n\nThe advantage of (1) is that we get a notion of explainability (visualize the nearest neighbors that were used to estimate action value), but the disadvantage is that it does not work well (because similarity in the input space may not mean that decision-making will be similar). The advantage of (2) is that it works better, but is less explainable. So the first idea is that we can get the best of both worlds by still using (2) to train a model, but then use the embeddings (i.e. the activations before the final linear layer) as inputs for a nearest neighbors approach.\n\nHowever, this can still have problems: in particular, for a new test point, the nearest neighbors may all be very tightly clustered but far away from the test point. Ideally, in such a situation, we would find nearby points in a variety of _different_ directions, and average them, so that our estimates are interpolations rather than extrapolations in the embedding space. So, instead of finding the nearest neighbors in our dataset, we find a minimal set of points from the dataset such that the current embedding falls within the convex hull of those points (or, if no such set exists, the embedding is as close as possible to the convex hull). We automatically discard any actions that are far away from the best convex hull, since they are likely OOD. This gives the author’s method: ABC.\n\n(In the paper’s presentation, the embeddings are called “beliefs”.)\n\nThe authors test their method in a variety of settings:\n\n1. Heterogeneous Pendulum: Similar to classic Pendulum, except that there is a 50% chance for the action effects to be swapped.\n2. Maze: A 2D setting with a wall separating the start and goal states, with two openings in the wall.\n3. Ward: Healthcare task, in which the task is to predict whether or not to use an oxygen therapy device.\n\nIn heterogeneous Pendulum, the authors show that ABC performs slightly better than model-free RL and model predictive control, and much better than other baselines. They also show the effect of ablating $\\epsilon$, the hyperparameter that controls which actions are considered OOD.\n\nIn Maze, the authors collect data from a variety of different behavioral policies, which have to be composed together to solve the task, and show that ABC is capable of this. There are two different ways to solve the task, corresponding to the two openings in the wall. The authors show that ABC can show both methods of solving the task, and that when visualizing the points forming the minimal convex hull for the resulting actions, they can be attributed to the behavior policies that used the same hole in the wall to solve the task. They also show that by increasing the proportion of different behavioral policies, you can control which of the two solutions ABC is more likely to use.\n\nOn Ward, the authors show that ABC performs on par with behavior cloning (BC) using a multilayer perceptron, and performs better than k-nearest-neighbors and BC using a linear model.", "questions": "**Overall view and suggestions for the authors**\n\nI quite like the idea in this paper, but currently I think the evaluation and presentation of the idea are not good enough, and so I am recommending rejection. However, I think there is the seed of a good paper here, and would likely be quite excited about a version of the paper that looked more like:\n\n1. Discussing embeddings as a useful way to get a nice structured representation, perhaps considering kNN-on-embeddings as the baseline.\n2. Identifying linear interpolation in the minimal convex hull as an alternative to the kNN decision criterion.\n3. Conducting a series of experiments that demonstrate the value of the convex hull idea, focusing particularly on questions like: (a) Are belief corpus residuals better at OOD detection than nearest-neighbor distances? (b) Does linear interpolation in a convex hull lead to better performance than kNN on embeddings? (c) Do the points in the convex hull provide a better explanation of the selected action than the k nearest neighbors in embedding space? I think it is quite plausible that convex hulls do better on all of these metrics, but the current experiments don’t show it.\n\n**Note on confidence**\n\nI’ve selected a confidence of (4) below, but I want to note that I am not very familiar with related literature, and in particular I know very little about accountability in the healthcare setting. As a result, I cannot evaluate (1) the originality of the work (maybe convex hulls have been explored before), and (2) whether the authors compared to state of the art techniques.\n\n**Questions**\n\nI’m interested in responses to any of the weaknesses I listed above, but in addition, I have some questions on specific details:\n\n1. Why do you require that the minimal corpus subset $\\tilde{C}(b_t)$ have $d_b + 1$ elements?\n2. In Section 3.5, $\\epsilon$ appears to be an absolute threshold for the belief corpus residual, but in Section 5.2, it appears to be a percentile for the belief corpus residual. Which is it?\n3. In the Maze environment, what is the performance measure for the behavioral policies that make up your dataset? Is it cumulative reward on the final task (i.e. going from (0,0) to (16,0)) or cumulative reward on each individual task (i.e. going from (0,0) to (8, 16), going from (0, 0) to (8, 8), etc), or something else entirely?", "limitations": "The weaknesses listed above are not present in the paper. Suggestions for improvements are in the previous sections.\n\nThe paper applies ABC to discrete settings, and also says that it can work in reward-free settings, but the idea for reward-free settings only works for continuous action spaces, not discrete ones. This is mostly not a big deal since the reward-free setting is not currently a major focus of the paper.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. Once I understood the idea, I found it simple and intuitive, with a clear story about why it should be helpful.\n2. The application of machine learning to healthcare is important, and accountability and conservatism are important properties to ensure in such a setting.\n3. There are a variety of experiments demonstrating the claimed properties of the method.", "weaknesses": "**Properties of ABC**\n\nThe authors list five properties that ABC satisfies. I agree with the author’s points that ABC works better with low data (at least relative to kNN) and that ABC can be used in the reward-free setting (at least for continuous actions spaces). However, I’m not convinced of the other three properties:\n\n1. Conservatism: The authors claim that ABC is conservative, I believe because they filter out actions that have a belief corpus residual that is too high. While I think the authors are probably correct, I don’t think their experiments show it: in all of the experiments that compare against baselines, black-box methods perform about the same as ABC, even though black-box methods are not normally “conservative”.\n2. Accountable: The authors claim that ABC is accountable because for any action taken by ABC, we can identify data points in the training dataset that make up the convex hull that determined that particular action, and show those to the user. However, there isn’t even a qualitative evaluation of how useful such explanations are. The closest is Figure 5, which visualizes the belief corpus as points on a 2D grid whose axes are uninterpretable (belief dimensions 1 and 2) relative to the test data point, but looking at that figure I do not feel like I have understood very much about ABC’s decision in that setting.\n3. Adaptive: To show that the ABC is adaptive, the authors perform an experiment in which they change the composition of the dataset on which ABC is trained, and show that this affects ABC’s behavior. But by this standard, essentially all algorithms are adaptive, including the baselines they compare against (e.g. behavior cloning, which they say is not adaptive in Table 1). It’s not clear why this is a unique advantage of ABC.\n\n(Incidentally, on accountability, the author’s technique is extremely similar to the technique of presenting maximally activating dataset examples to explain neuron activations, a common technique for explainability in supervised learning.)\n\n**Additional comparisons**\n\nI would like to see the authors compare ABC to the first method in my summary, i.e. training a model to predict value / actions, and then using k-nearest-neighbors on the embeddings (activations before the final linear layer). This can be thought of either as a baseline, or as an ablation (as an ablation, it corresponds to ABC without the convex hull aspects). This would be helpful in understanding the effects of the various design decisions the authors make.\n\nIf performing an experiment on accountability, then I’d like to see a comparison to the dataset examples technique applied to the kNN-on-embeddings model discussed above.\n\n**Disagreement with Section 5.2 claim**\n\nSection 5.2 notes that there are two hyperparameters: “the number of uniformly sampled actions and the threshold”. It claims that these can be unified into a single hyperparameter, the _effective action size_. However, the experiment doesn’t support this: it simply sets the number of sampled actions (which we’ll call $n_A$) to 100, and then shows the effect of varying the percentile threshold $\\epsilon$. The experiment that should be run would be to use a variety of settings of _both_ hyperparameters, and then check whether runs with similar effective action sizes $\\epsilon \\times n_A$ have similar performance: if so, then it would be justified to only think about the effective action size. However, my guess is that this will not be the case.\n\n**Minor issue with the theory**\n\n(Note: set notation doesn't seem to be working below)\n\nProposition 3.10 is false because of the requirement that the convex hull contain $d_b + 1$ examples. For example, suppose $d_b = 2$, $b_t = [1, 0]$, and $\\mathcal{D} = \\{ [7, 0], [3, 0], [-1, 0] \\}$. Note that $b_t = 0.5 * [-1, 0] + 0.5 * [3, 0]$, and so if we have $\\mathcal{C} = \\mathcal{D}$, then $b_t \\in \\mathcal{CB(C)}$, and so $r_{\\mathcal{C}}(b_t) = 0$ as required by Proposition 3.10. Definition 3.9 requires the minimal corpus subset $\\tilde{\\mathcal{C}}(b_t)$ to have 3 elements, and which means that $\\tilde{\\mathcal{C}}(b_t) = \\mathcal{D}$. However, the decomposition on the minimal hull is not unique, since we have both $b_t = 0.5 * [-1, 0] + 0.5 * [3, 0]$ as well as $b_t = 0.75 * [-1, 0] + 0.25 * [7, 0]$, contradicting Proposition 3.10.\n\nThe issue is that you require the convex hull to contain $d_b + 1$ examples. If you remove that restriction, then in the example above $\\tilde{\\mathcal{C}}(b_t) = \\{ [-1, 0], [3, 0] \\}$ and then the decomposition is unique, as desired.\n\n(I believe your current proof would also work if you remove the restriction. Currently, it doesn’t work because you remove an element from $\\tilde{\\mathcal{C}}(b_t)$ and call that a contradiction, but it is actually not a contradiction because the newly created set no longer has $d_b + 1$ elements.)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the time and effort in reviewing our paper. We will respond to each point in turn:\n\n---\n\n### 1. Properties\n\n- **Conservatism**. We demonstrate the property of Conservation through our experiments provided in Appendix F2. We explicitly visualize the behaviors of ABC under different degrees of conservation.\n\n- **Accountability**. ABC is accountable because it provides reference examples in making decisions. The high-level insight follows example-based explanations in XAI.  In our work, we put a special focus on sequential decision-making problems rather than prediction tasks in the XAI literature.\n\n- **Adaptivity**. We recognized our previous description of adaptivity could be misleading. In fact, we wish to demonstrate the property of on-the-fly adaptivity of ABC that is correlated to its property of accountability: as it links decisions to reference training data, filtering out those unwanted training data during deployment is a much easier way than training a new model like BC and other algorithms to perform a special type of decision under user specification.\n\n### 2. Ablation studies, and additional baselines\n\nTo address the reviewer's concern about empirical evaluation, we implemented two more variants of ABC and kNN as baselines, including \n- **kNN+Belief**: it uses the learned latent space for decision-making when applying kNN. This study could be regarded as the ablation study of _ABC w/o Minimal Hull_.\n- **ABC w/o Belief**: it uses the original input space for decision-making, but applies the minimal hull decomposition. This study could be regarded as the ablation study of _kNN + Minimal Hull_. \n\nResults are provided in the following extended Table 2:\n\n_The cumulative reward of each method is reported. Additional experiments are repeated with 5 seeds. **Higher is better.**_\n\n| Task              | Low-Data            | Mid-Data            | Rich-Data           |\n|-------------------|---------------------|---------------------|---------------------|\n| 1NN               | -557.07 ± 256.64    | -690.49 ± 152.59    | -512.71 ± 131.2     |\n| kNN               | -849.45 ± 91.23     | -670.51 ± 321.09    | -645.72 ± 220.33    |\n| kNN + Belief      | -659.17 ± 219.76    | -525.58 ± 436.56    | -534.02 ± 568.47    |\n| ABC w/o Belief    | -302.55 ± 426.39    | -173.84 ± 245.85    | -130.24 ± 184.12    |\n| ABC               | **-1.39 ± 1.39**    | **-1.25 ± 0.40**    | **-0.6 ± 0.08**     |\n| BC                | -422.77 ± 409.51    | -225.32 ± 340.83    | -126.74 ± 280.73    |\n| TD3               | -4.1 ± 2.76         | -11.95 ± 4.68       | -15.27 ± 6.46       |\n| CQL               | -793.89 ± 206.0     | -889.85 ± 291.99    | -805.49 ± 578.75    |\n| TD3-BC            | -844.95 ± 170.93    | -781.8 ± 337.11     | -821.02 ± 587.77    |\n| TD3-BC-Recurrent  | -82.92 ± 115.79     | -41.92 ± 59.28      | **-0.45 ± 0.57**    |\n| MPC               | **-1.5 ± 0.43**     | **-1.34 ± 0.15**    | -1.41 ± 0.26        |\n| Data-Avg-Return   | -307.81 ± 387.53    | -245.54 ± 338.65    | -208.81 ± 272.84    |\n\nWe additionally compare against CQL and TD3-BC in Section 5.1. Not surprisingly, we find the performance of the original CQL and TD3-BC on Pendulum-Het are poor. This is because they are not designed for partial observable tasks with high stochasticity. \nTherefore, we further implemented an improved version of the TD3-BC with a recurrent context encoding module [cf. Meta-Q-Learning]. In those experiments, we find the learning process of Offline-RL is not stable, leading to a large variance in the policy quality — we have observed a similar problem in our previous MFRL baseline, and reported it with details in Appendix D.7.\n\n\n### 3. Hyper-parameters\n\nWe would agree with the reviewer that using a smaller sampling size with a larger quantile number is less preferred in comparison with using a larger sampling size with a smaller quantile number, this is because the latter will lead to a more accurate estimation and more conservative behavior. \n\nWe have updated our manuscript accordingly.\n\n### 4. Theory\n\nIn the counter-example raised by the reviewer, the dimension of the belief space diminished to $1$-dim, instead of 2-dim, for the mentioned specific convex hull decomposition.\nThe key issue is indeed **the dimension of the belief space should be more clearly defined**. In this example, the minimal hull should not contain three points, because its hyper-volume (i.e., length, in this 2-D example) is not minimized. \nWe have updated the dimension of the search space from **$d_b+1$** to **maximally $d_b +1$** to enhance clarity.\n\n### 5. Threshold\n\nTo make our methodology part clear, we use $\\epsilon$ as a constant in Sec. 3.5. In practice, such a constant can be implemented through quantile thresholding (Sec.5.2). We use the same notation to emphasize that this quantile number controls the threshold. We updated the notation in Sec. 5.2 to $\\epsilon(q=0.3), \\epsilon(q=0.5)$, etc. to enhance the clarity.\n\n### 6. Terminology of _Reward-Free_\n\nIn our context, the reward-free indicates the **strictly batched imitation** settings where reward information is not accessible. Different from the normal batched control setting where an offline dataset containing $(o_t, a_t, r_t)$ is available, the reward-free setting can only leverage a dataset that is composed of $(o_t, a_t)$.\n\n---\nShould there be any additional questions or concerns, we are more than willing to provide further explanations.\n\n\n"}, {"review_id": "XGnkZONlKG", "reviewer": "Reviewer_9Kjz", "summary": "The paper presents the Accountable Batched Controller (ABC) based on the example-based explanation framework as a solution for offline control in responsibility-sensitive applications. Through experiments on simulated and real-world tasks, the method shows accountability, conservation, and adaptability.", "questions": "More in-depth analysis of Table 2 may help gain deeper insight regarding the suitability for low-data regimes.\n\nWhy is section 5.1 highlighting P1-P3?", "limitations": "As the authors also mentioned, the current accountable batched control method is limited to low-dimensional control tasks and may not perform as well in high-dimensional control systems, limiting its potential contribution.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The paper proposed a novel method for accountable batched control with decision corpus, theoretically proved the existence and uniqueness of the decomposition under mild conditions, and conducted solid experiments to verify the effectiveness and desired properties of the method.\n\nThe paper is clear-presented and well-organized.", "weaknesses": "To further improve, more experiments on real-world control tasks are needed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the time and effort in reviewing our paper. We will respond to each point in turn:\n\n---\n### 1. Additional Analysis on Table 2\n\nWe would start by providing **more empirical studies** extending the previous Table 2:\n\n_The cumulative reward of each method is reported. Additional experiments are repeated with 5 seeds. **Higher is better.**_\n\n| Task              | Low-Data            | Mid-Data            | Rich-Data           |\n|-------------------|---------------------|---------------------|---------------------|\n| 1NN               | -557.07 ± 256.64    | -690.49 ± 152.59    | -512.71 ± 131.2     |\n| kNN               | -849.45 ± 91.23     | -670.51 ± 321.09    | -645.72 ± 220.33    |\n| kNN + Belief      | -659.17 ± 219.76    | -525.58 ± 436.56    | -534.02 ± 568.47    |\n| ABC w/o Belief    | -302.55 ± 426.39    | -173.84 ± 245.85    | -130.24 ± 184.12    |\n| ABC               | **-1.39 ± 1.39**    | **-1.25 ± 0.40**    | **-0.6 ± 0.08**     |\n| BC                | -422.77 ± 409.51    | -225.32 ± 340.83    | -126.74 ± 280.73    |\n| TD3               | -4.1 ± 2.76         | -11.95 ± 4.68       | -15.27 ± 6.46       |\n| CQL               | -793.89 ± 206.0     | -889.85 ± 291.99    | -805.49 ± 578.75    |\n| TD3-BC            | -844.95 ± 170.93    | -781.8 ± 337.11     | -821.02 ± 587.77    |\n| TD3-BC-Recurrent  | -82.92 ± 115.79     | -41.92 ± 59.28      | **-0.45 ± 0.57**    |\n| MPC               | **-1.5 ± 0.43**     | **-1.34 ± 0.15**    | -1.41 ± 0.26        |\n| Data-Avg-Return   | -307.81 ± 387.53    | -245.54 ± 338.65    | -208.81 ± 272.84    |\n\nIn this updated Table, we have\n\n(1) included ablation studies (kNN+Belief, ABC w/o Belief)\n\n(2) included additional baselines in offline-RL\n\nSeveral conclusions can be drawn from the updated table:\n\n- (1) [High-Performance] Compare to all of the methods including black-box algorithms and the accountable baselines, we find** ABC is able to achieve high performance in all settings**.\n\n- (2) [Efficacy under Low-Data Regime] Compare ABC with the baselines, we observe the superiority of ABC especially under the low-data regime. In such a setting, **ABC is able to effectively solve the problem while many other methods suffer from higher instability**.\n\n- (3) [Ablation Studies] In addition to the comparison between ABC and kNN, we **additionally experiment** with kNN that works in the belief space, we find it improves the performance of kNN, demonstrating the **effectiveness of leveraging the belief space** in accountable decision making. Moreover, we demonstrate the **effectiveness of the minimal convex hull decomposition**, which is another algorithmic design, through the experiment of ABC w/o Belief (i.e., kNN with minimal convex hull decomposition). We find the results are better than the original kNN, yet significantly worse than ABC.\n\n- (4) [Instability of Offline-RL Algorithms] Conventional offline-RL algorithms focus only on the conservatism during learning from offline decision corpus in MDP tasks, hence **they fail for efficient learning in our partially observable tasks with high stochasticity**. To make those algorithms stronger, we additionally implemented a recurrent module for the TD3-BC algorithm, and compare it to ABC. We find those offline-RL algorithms **suffer from instability issues during learning and are hard to converge** to a well-performing policy. We provide an additional analysis of their learning process in Appendix D.7.\n\n- (5) [Property of Conservation] We would like to note the property of conservation is demonstrated through the _offline nature_ of those benchmark tasks. \nTo enhance the clarity of our representation, we would refer to the **added ablation study that could further help demonstrate the importance of conservation**: through the comparison between kNN and ABC w/o Belief (as they both work in the original input space) or kNN+Belief and ABC (as they both work in the belief space), the performance gain of the ABC w/o Belief and ABC over counterparts are actually based on the conservative property introduced by the minimal convex hull.\n\n### 2. More tasks\n\nTo address the reviewer's concern on more challenging tasks, we refer to additional experiments in Appendix F.3, F.4, F.5, where we demonstrate the scalability of ABC and the potential of combining ABC with black-box policies. From such a perspective, ABC can be used as a post-hoc interpretation mechanism for any given black-box algorithms\n\nTo be specific, we additionally experimented on the LunarLander-Continuous environment and the BipedalWalker environment. In those experiments, we find that the dimensionality of states is not a critical issue, but the increase in the action dimensions can be more challenging — it originates from the uniform sampling over the action space in our Algorithm. To address such a difficulty, we investigate the potential of integrating ABC with black-box samplers in F.4. \n\nWe can observe from the results that ABC can work both in isolation or combined with black-box policies. ABC can be used as a plug-in to add accountability to black-box controllers in a post-hoc manner. In high-dimensional control tasks, uniform sampling can be inefficient and black-box samplers can alleviate such a difficulty. This could potentially be a promising direction for future research.\n\n---\nShould there be any additional questions or concerns, we are more than willing to provide further explanations."}, {"review_id": "lKpLjFm2xa", "reviewer": "Reviewer_izcS", "summary": "This work proposes accountable batched control with five desirable properties. The design is motivated by the fact that the reward or feedback of trajectories are hard to obtain in high-stake responsibility-sensitive applications. The minimal hull subset of the decision corpus is constructed for the decomposition of the value function for each candidate action. Then the optimal policy is selected in terms of the weighted value function. This work demonstrates the promise of the five properties on one real-world healthcare dataset and one simulated maze environment.", "questions": "1. The term *decision corpus* is never explained. How does it differ from the trajectory in the offline data?\n2. The batched controller appears to neglect the historical decisions and fails to account for the temporal correlation among actions. How can you guarantee that the policy will receive the optimal accumulated rewards?", "limitations": "The authors have include a separate broader impact section in the Appendix.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "After the rebuttal I tentatively raised my score from 5 to 6.\n\n---\n\nThe paper is quite obscure and not easy to understand, making it challenging to grasp the complete understanding of the methodology it presented. However, I would appreciate it that the authors included a video presentation in the appendix and saved my time. In particular, the animation of the algorithm makes it easier to understand.\n\nThe introduction of the batched control is a seemingly novel contribution. The experimental setup and analysis are presented in a concrete and well-written manner, albeit deferred to the appendix due to space constraints. In addition, the authors attached an anonymous link to their code implementation.", "weaknesses": "While I may not have a comprehensive understanding of the literature, it appears that the focus of this paper leans more towards reinforcement learning rather than the chosen primary area of interpretability and explainability.\n\nI am uncertain about the reasons behind the batched controller possessing the five advantageous properties in comparison to other well-known methods, including Q-learning, model-based RL, and behavior clone. Although I skimmed through the Appendix B, the interpretation is still unclear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the time and effort in reviewing our paper. We will respond to each point in turn:\n\n---\n### 1. Definition of the Decision Corpus\nThe definition of _Decision Corpus_ is explained in line 6 in the abstract and line 61 in Sec. 3. In our context, we use _Decision Corpus_ to refer to the offline decision dataset.\nWe have improved the clarity of the definition and made it more explicit in our revision.\n\n### 2. Historical Decisions are Captured by the Belief State\n- **[No temporal correlation among actions]** We would first note our work is developed under the general Partially Observable Markov Decision Process (POMDP) setting, denoted as a tuple $(\\mathcal{X}, \\omega, \\mathcal{O}, \\mathcal{A}, \\mathcal{T}, \\mathcal{R}, \\gamma, \\rho_0)$. Under the Markovian property, **there should be no temporal correlation among actions.**\n- **[Ability to capture historical decisions]** In such a setting, it is important to capture the historical decisions and transitions, therefore, we introduced the observational transition history variable $h_t$, defined as $h_t = (o_{<t}, a_{<t}, r_{<t}) \\in \\mathcal{H} \\subseteq \\mathbb{R}^{(d_o+d_a +1)\\cdot (t-1)}$. In our work, **the belief mapping $\\mathbf{b}$ is a function of observation, action, and historical transition**, i.e., $\\mathbf{b} = \\mathbf{b}(o_t, a_t, h_t)$, to capture the information in historical observations, decisions and return.\n- **[Optimize accumulated rewards]** In order to optimize the correct learning objective of maximizing cumulative episodic return, **we use the cumulative return as $v^c$ in Eqn.(2).**\nAs a consequence, the estimated value $\\hat{v}$ approximates the true cumulative return, and ABC optimizes this cumulative reward proxy in decision-making.\n\n### 3. Extended Discussion Comparing ABC with Existing Algorithms\nBelow, we discuss each of the desired properties in turn. For each of the properties, we start with introducing the definitions of the property, followed by comparisons among ABC and MFRL (Q-learning), MBRL, and BC.\n\n(1) **Accountability**: the decision-making process is traceable, and the decisions can be supported by concrete examples in the offline dataset.\n  - ABC: the decisions of ABC are accountable, since they are generated referring to the Corpus Subset within the minimal convex hull. Those examples provide an example-based explanation of the decisions.\n  - MFRL: In MFRL, a black-box value network and black-box policy network are learned with the offline dataset. There is no decision support for the black-box policies.\n  - MBRL: In MBRL, a black-box world model optimized with the offline dataset is used as a proxy of the true dynamics, and planning algorithms are then applied to such a black-box model to make decisions. Those decisions are not supported by explicit references.\n  - BC: In BC, a black-box policy is learned through supervised learning. The output of such a policy is hard to be linked with specific training examples.\n\n(2) **Conservation**: estimations of decision outcomes are interpolated, avoiding aggressive extrapolation that is harmful in offline control.\n - ABC: ABC performs conservative decision-making by using decision supports within a minimal convex hull. How such a decomposition in the convex hull improves conservation is justified theoretically by Proposition 3.8 (Estimation Bound) and Proposition 3.10 (Existence and Uniqueness).\n - MFRL: In MFRL like CQL and TD3-BC, the conservation is explicitly given as constraints or distribution matching. We would note that conventional MFRL algorithms are not designed for those tasks and suffer from aggressive extrapolation.\n - MBRL: Similar to MFRL, the conservation should be added to MBRL through external efforts. Because the conventional design of model-based learning does not address such an issue.\n - BC: In BC, the learning objective is to minimize the prediction difference. There is little we can do to aid conservation.\n\n(3) **Low-Data**: whether a method works in the low-data regime.\n - ABC: The decision process of ABC only relies on a few examples constituting the minimal convex hull, hence the algorithm performs well under the low-data regime.\n - MFRL: In MFRL, the black-box value network and policy network can be designed to be sample-efficient.\n - MBRL: In MBRL, sufficient data is always required to learn an accurate world model.\n - BC: the performance of BC is highly dependent on the quality of data. It is not designed for the low-data regime.\n\n(4) **Adaptive**: whether the control behavior of a method can be adjusted according to additional constraints as clinical guidelines without modification or re-training.\n - ABC: by changing reference examples, i.e., the decision corpus, during test time inference, ABC can seamlessly perform different types of decision-making according to user specifications.\n - MFRL: In MFRL, when new data is used, a new value network and policy network need to be re-trained.\n - MBRL: In MBRL, the world model construction is independent of the data, hence the decisions can be adaptive by changing a new planning algorithm on top of the world model. No model re-training is needed.\n - BC: In BC, a new model needs to be trained with a specified type of decision corpus.\n\n(5) **Reward-Free**: availability of extension to the strictly batched imitation setting where rewards are unavailable.\n - ABC: We have shown the key insight of making decisions through belief space similarity can be extended to the settings without reward signals, as discussed in Appendix C.\n - MFRL: In MFRL, the Q-values can not be calculated without the reward function.\n - MBRL: In MBRL, the planning algorithms do not have a clear objective to optimize without reward signals.\n - BC: BC is not be affected by the absence of reward signals, because it does not need the reward to learn its policy.\n\n---\nShould there be any additional questions or concerns, we are more than willing to provide further explanations. "}, {"review_id": "mieOcAGhgw", "reviewer": "Reviewer_fQBY", "summary": "This paper investigates imitation learning in scenarios with limited data. The proposed approach (ABC) involves utilizing a linear combination of the belief space to generate accountable decisions.\nThe authors evaluate the performance of ABC in simulated and real-world healthcare scenarios, highlighting its ability to effectively handle batched control tasks while maintaining a high level of performance and accountability.", "questions": "1. Could you provide the results of offline reinforcement learning methods in the test environments?\n\n2. Can you provide the environment results obtained when applying the proposed approach to more challenging environments?\n", "limitations": "This work is primarily focused on simple environments and does not address more complex scenarios.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper is presented in a clear and easily understandable manner.\n2. This article investigates a highly meaningful direction, which generates reliable strategies through offline data.", "weaknesses": "1. Assuming that the value is a linear combination of belief states is a strong assumption, which may not hold true in certain tasks, particularly those that are harder and more complex.\n2. The optimization problem (Eq. 9) seems to be ill-defined. Eq. 9 could have an infinite number of solutions because the effect of $l\\circ b$ is equivalent to $cl \\circ c^{-1}b$, where $c$ represents a scalar. This would have an impact on the distance calculation when solving Eq. 10. It would be beneficial to address the limitations of the optimization procedure and conduct additional experiments to examine the influence of different solutions to the optimization problem (Eq. 9).\n3. The paper lacks a discussion and comparison with offline reinforcement learning, which is an important related work and a strong baseline.\n4. The evaluation is confined to simple environments. For more challenging tasks, please refer to [1].\n\nReference:\n[1] Fu, Justin, et al. \"D4rl: Datasets for deep data-driven reinforcement learning.\" arXiv preprint arXiv:2004.07219 (2020).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the time and effort in reviewing our paper. We will respond to each point in turn:\n\n---\n### 1. Property 3.2 is not an Assumption\nProperty 3.2 is **NOT an assumption**, and is **INDEPENDENT TO SPECIFIC TASKS**. It is about the architecture of the neural network being used (not an assumption imposed on the environment).\n\nWe wish to clarify that the belief state in our context is the **values of the last activation layer in neural networks**, hence it is independent of the task. As long as there is a neural network with a linear layer as the output layer that approximates the value function, we are able to use its last activation layer as our belief state. \n\nTherefore, we would like to note that our property 3.2 is not an assumption imposed on the environment, but a property that is satisfied by general neural network approximators, as we have noted in Remark 3.3.\n\n### 2. Eqn. (9) is Well-Defined.\n\nEqn. (9) depicts the MSE minimization using neural networks. It’s a well-defined problem.\n\nTo be specific, we consider a neural network with a linear output layer. We denote the layers before the last as $\\mathbf{b}$, and the linear output layer as $\\mathbf{l}$. We explicitly write the value function approximation minimization process in Eqn.(9). We would like to note that, different solutions of this equation correspond to different neural network parameters. And any set of parameters that minimized the loss function can be used in practice.\n\nWe do not assume the uniqueness of the optimized neural network parameters, and our optimization process following Eqn.(9) will not be affected. \nTo see this, suppose we have $\\mathbf{b}’, \\mathbf{l}’ = c^{-1}\\mathbf{b}, c\\mathbf{l}$, Eqn.(10) and Eqn.(11) will still get the identical results. This is because the calculations following Eqn.(9) only use the relative information among beliefs, rather than their absolute values.\n\n\n### 3. We added Offline RL Algorithms as Additional Baselines\n\n**The only experiment in which Offline-RL algorithms can be compared with ABC is Section 5.1.** In all other experiments, Offline-RL **does not enjoy the properties we discussed in the paper.** Nor do those Offline-RL algorithms address the issue of accountability in decision-making. \n\nTo address the reviewer's concern, we additionally compare ABC with CQL and TD3-BC in Section 5.1. Not surprisingly, we find the performance of the original CQL and TD3-BC on Pendulum-Het are poor. This is because they are not designed for partial observable tasks with high stochasticity. \nTherefore, we further implemented an improved version of the TD3-BC with a recurrent context encoding module [cf. Meta-Q-Learning]. In those experiments, we find the learning process of Offline-RL is not stable, leading to a large variance in the policy quality — we have observed a similar problem in our previous MFRL baseline, and reported it with details in Appendix D.7.\n\n_The cumulative reward of each method is reported. **Higher is better.**_\n\n| Task              | Low-Data            | Mid-Data            | Rich-Data           |\n|-------------------|---------------------|---------------------|---------------------|\n| ABC               | **-1.39 ± 1.39**    | **-1.25 ± 0.40**    | **-0.6 ± 0.08**     |\n| BC                | -422.77 ± 409.51    | -225.32 ± 340.83    | -126.74 ± 280.73    |\n| TD3               | -4.1 ± 2.76         | -11.95 ± 4.68       | -15.27 ± 6.46       |\n| CQL               | -793.89 ± 206.0     | -889.85 ± 291.99    | -805.49 ± 578.75    |\n| TD3-BC            | -844.95 ± 170.93    | -781.8 ± 337.11     | -821.02 ± 587.77    |\n| TD3-BC-Recurrent  | -82.92 ± 115.79     | -41.92 ± 59.28      | **-0.45 ± 0.57**    |\n| Data-Avg-Return   | -307.81 ± 387.53    | -245.54 ± 338.65    | -208.81 ± 272.84    |\n\n### 4. Difference between ABC and Offline-RL\n\nWe would like to note that Offline-RL is related to our work (Appdx. B.1), yet it is not our primary focus. The mentioned D4RL benchmark is famous in Offline-RL, yet it presents a weak link to the main problem of accountability studied in this work. In fact, **the only experiment that Offline-RL algorithms can be compared with ABC is Section 5.1.** In all other experiments, Offline-RL does not enjoy the properties we discussed in the paper. \n\nWe wish to clarify that **we mainly studied the accountability of offline decision-making in high-stake systems, and applied the proposed method to real-world healthcare dataset.** This is different from the normal pursuit of Offline-RL that aims at improving conservative value estimation. \n\nTo make it more explicit, we emphasized in our abstract, introduction, related work, and experiment section that what makes ABC different from existing literature including Offline-RL is its 5 properties: \n- P1: controllable conservation to avoid aggressive extrapolation;\n- P2: accountability that provides a decision basis;\n- P3: suitability for low-data regimes;\n- P4: adaptability to user specifications that allows customization;\n- P5: flexibility in strictly batched imitation settings for broader applicability.\n\nOut of those 5 properties, Offline-RL only explicitly satisfies P1.\n\nTo address the reviewer's concern about more challenging tasks, we **refer to additional experiments in Appendix F.3, F.4, F.5**, where we demonstrate the scalability of ABC and the potential of combining ABC with black-box policies. From such a perspective, ABC can be used as a post-hoc interpretation mechanism for any given black-box algorithm like CQL, TD3-BC, etc. This could be interesting and important future work but is out of the scope of this methodology paper focusing on accountability, rather than Offline-RL.\n\n---\nWe hope that these clarifications address your concerns, and we are happy to have further discussions would they remain unclear.\n"}, {"review_id": "fZqS7On8oR", "reviewer": "Reviewer_qevo", "summary": "This paper poses the problem of learning a controller from a batched dataset containing time-series observations of the world, actions, and a value estimate (e.g., for determining how to treat a patient given the results of medical tests based upon patient outcomes). This problem is considered with a POMDP formalism with execution traces. The paper imposes assumptions about the linearity of the mapping from the belief space to the value space in order to create some analytical properties about how wrong the model might be or to mitigate the model error. The algorithm, ABC, is evaluated against baselines in a set of batched-data experiments, including in a healthcare applicaiton.", "questions": "-Why is this problem different than offline reinforcement learning?\n-Why are offline RL baselines not included?\n-How do the results provide convincing evidence of accountability in a non-superficial manner?\n-For what class of problems is the linear assumption for the mapping from belief to value reasonable?\n-What is the computational complexity of the approach?\n-How scalable is the approach with the size of the state space?\n-By relying on belief space modeling, why is this approach relevant to large, real-world problems?", "limitations": "The paper does not mention \"limit\" once.", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "+The paper addresses an important problem of bringing safety to machine learning\n\n+The paper clearly states assumptions and presents logical arguments and definitions to support its thesis.\n\n+The evaluation shows positive results and does so in important domains (e.g., healthcare)", "weaknesses": "-The paper seems to be addressing the problem of offline reinforcement learning without actually addressing offline reinforcement learning. Though, this paper does cite a plethora of papers that address this topic. As such, it is difficult for the reviewer to properly contextualize this paper in this relevant prior work.\n\n-The paper appears to be missing a number of baselines for offline reinforcement learning, as shown below. Some of the baselines chosen do not have access to the same set of information available to the ABC algorithm (e.g., the BC model).\n\nChen, L., Lu, K., Rajeswaran, A., Lee, K., Grover, A., Laskin, M., Abbeel, P., Srinivas, A. and Mordatch, I., 2021. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34, pp.15084-15097.\n\nKumar, A., Zhou, A., Tucker, G. and Levine, S., 2020. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33, pp.1179-1191.\n\n-The paper develops theory to bring accountability to this problem. However, the results section provides some relatively simple computational examples and a qualitative description that is in the eye of the beholder. It would have been better to provide a clearer, more convincing test to demonstration that there are clear guarantees and fulfilled analytical properties. \n\n-One could have considered a human-subject experiment to evaluate whether this approach is really \"accountable.\" Literature on accountability could have been considered as well. For example, see:\n\nKim, B. and Doshi-Velez, F., 2021. Machine learning techniques for accountability. AI Magazine, 42(1), pp.47-52.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the time and effort in reviewing our paper. We will respond to each question in turn:\n\n---\n### Q1: Why is this problem different than offline reinforcement learning? \nA1: In this paper, we study the accountability of offline decision-making in high-stake systems, and applied the proposed method to real-world healthcare dataset. ABC performs a form of offline reinforcement learning, but unlike existing work on offline RL, it seeks to achieve P1-P5 below.\nAs we have emphasized in our abstract, introduction, related work, and experiment section that what makes ABC different from existing literature including Offline-RL is its 5 properties: \n- P1: controllable conservation to avoid aggressive extrapolation;\n- P2: accountability that provides a decision basis;\n- P3: suitability for low-data regimes;\n- P4: adaptability to user specifications that allows customization;\n- P5: flexibility in strictly batched imitation settings for broader applicability.\n\nOut of those 5 properties, **Offline-RL only satisfies P1**.\n\nWe also discussed the similarities and differences of ABC to Offline-RL in the section of Extended Related Work (Appendix B). \n\n### Q2: Why are offline RL baselines not included? \nA2: The only experiment in which Offline-RL could be compared with ABC is Sec.5.1. In all other experiments, Offline-RL does not enjoy the properties we discussed in the paper. Nor do those Offline-RL algorithms address the issue of accountability in decision-making. \n\nWe **conducted additional experiments** to better address the reviewer's concern and provide **results in the attached PDF file** due to space limitations.\n\n### Q3: How do the results provide convincing evidence of accountability in a non-superficial manner? \n\nA3 :  (1) In our main text, we use Sec.5.3 to highlight the accountability of the proposed method. Where we can have the conclusion that ABC's decision-making process exhibits strong accountability, as the **corpus subset can be tracked at every decision-making step**. This is evidenced by ABC's successful completion of a multi-stage Maze task, wherein it effectively learns from mixed trajectories generated by multiple policies at differing stages.\n\n (2) Apart from the above qualitative results that permit visualization, we also provided additional empirical evidence on the property of accountability in Section 5.5, where we study the real-world healthcare application of ABC, and highlight how **accountability helps to identify boundary examples in decision-making**. \n\n (3) Moreover, in Appendix F.5, we demonstrate how to leverage the **accountability of ABC to identify OOD examples**, this is another evidence showing the accountability of our proposed method.\n\n### Q4: For what class of problems is the linear assumption for the mapping from belief to value reasonable? \n\nA4: **Our Remark 3.3 answers this question.**\n\nThe linear relationship between the latent belief space and the value is a property rather than an assumption. As we’ve stated in Remark 3.3, this property is often the case with neural network approximators, where the belief state is the last activated layer before the final linear output layer. \n\n### Q5: What is the computational complexity of the approach?\n\nA5: **Our discussion in Appendix D.3 and D.5 can answer this question.**\n\nDue to the page limit, we postponed the discussion of computational complexity, hardware requirement, and wall-clock running time in Appendix D.3 and D.5, respectively.\nWith our proposed solution, the convex hull decomposition takes less than 10 seconds with a uniform sampler that samples 100 actions randomly for every time step. Increasing the number of sampled actions will lead to a sub-linear increase in computational time with parallelization.\n\n### Q6: How scalable is the approach with the size of the state space? \nA6: **Our experiments in Appendix F.3-F.5 are designed to answer this question.**\n\n**ABC can work both in isolation or combined with black-box policies.** ABC can be used as a plug-in to add accountability to black-box controllers in a post-hoc manner. In high-dimensional control tasks, uniform sampling can be inefficient and black-box samplers can alleviate such a difficulty.\n\n### Q7: By relying on belief space modeling, why is this approach relevant to large, real-world problems?\nA7: In this study, we examine the batch control problem, which holds **significant potential for applications in costly, risk-sensitive domains such as healthcare and finance.** While previous works have primarily focused on efficient learning in batch settings, the accountability of offline decisions remains largely unexplored despite its importance.\n\nIn healthcare, it's vital that decisions are based on a supportive basis. For instance, **when a patient is treated in a certain manner, it should be based on the successful outcomes of previous patients with comparable conditions who received the same treatment.** The ability to trace the supportive basis of decisions enhances the process of policy reasoning and debugging, thereby improving the trustworthiness of decision-making systems.\n\nWhile technically ABC is built on top of belief space modeling, the accountability generated by ABC is instance-level based on the one-to-one mapping between examples and belief states. Hence, **the use of belief space modeling in ABC does not restrict its applicability or effectiveness in real-world problems.** Therefore, the modeling of belief space enables a more nuanced understanding of individual decisions, aligning with the complexity of real-world scenarios. \n\n### Q8: The paper does not mention \"limit\" once.\nA8:  In fact, we do have a discussion section **Appendix G. Limitations and Future Work** on Page 27 of our paper.  We have updated our conclusion and refer readers to Appendix. G for limitations.\n\n---\nWe hope that these clarifications address your concerns, and we are happy to have further discussions would they remain unclear. \n"}]

Author rebuttal:
We extend our sincere gratitude to all reviewers for their insightful comments, valuable suggestions, time, and efforts in evaluating and improving our paper. 

We thank all reviewers for their affirmation of our work’s **novelty** (reviewers: 9Kjz, izcS), **presentation** (reviewers: fhj3, 9Kjz, izcS, fQBY), **evaluation** (reviewers: fhj3, 9Kjz, izcS, qevo), and **importance** (reviewers: fhj3, fQBY, qevo). 

----

To address the concerns raised by reviewers, we would respond to each of their questions respectively. Below, as a general response, we aim to outline the **key revisions and additional experimentation conducted by far**:

#### **Supplementary Experimental Evaluation**
1. **(Table 1 in attached PDF)** We conducted additional ablation studies of
  - kNN + Belief  (i.e., ABC w/o Minimal Convex Hull Decomposition)
  - ABC w/o Belief (i.e., kNN + Minimal Convex Hull Decomposition)
2. **(Table 2 in attached PDF)** We provide additional offline-RL baselines, including 
  - CQL [1]
  - TD3-BC [2]
  - TD3-BC-Recurrent that improves TD3-BC using a recurrent module in POMDP tasks [3,4].
3. **(Figure 1 in attached PDF)** We highlight the controllable conservative behavior of ABC by varying the quantile number. 
4. **(Table 3 in attached PDF)** We demonstrate the scalability of ABC by providing results on the LunarLanderContinuous environment.
5. **(Table 4 in attached PDF)** We experiment with the BipedalWalker environment to showcase how to integrate ABC with black-box controllers to add accountability.

#### **Revised Manuscript for Clarity**
1. We have revised the terminology used in Proposition 3.10 to eliminate any ambiguity regarding the dimension restriction of the minimal convex hull.
2. We have emphasized that the primary focus of ABC extends beyond the problem of offline RL. While offline RL possesses only the conservation property, ABC has five distinct properties that are all crucial for accountable batched control tasks. 
3. In Sec.3.5 and Sec.5.2, we've employed distinct symbols to represent the constant, using $ \epsilon$, and the constant as a function of a given quantile threshold, denoted as $ \epsilon(q) $, to enhance clarity.
4. We explained the 5 desired properties more explicitly in the related work section, illustrating why compared baselines may not satisfy each of those properties.


----

We hope our clarification and additional empirical studies could address the concerns raised by reviewers. Should there be any leftover questions, please let us know and we will make every effort to address them during the subsequent discussion period.


---
**_Refrences_**

[1] Kumar, Aviral, et al. "Conservative q-learning for offline reinforcement learning."

[2] Fujimoto, Scott, and Shixiang Shane Gu. "A minimalist approach to offline reinforcement learning."

[3] Ni, Tianwei, Benjamin Eysenbach, and Ruslan Salakhutdinov. "Recurrent model-free rl can be a strong baseline for many pomdps."

[4] Fakoor, Rasool, et al. "Meta-q-learning."

Meta-review: This paper is borderline, with an average score of 5.4 slightly below the acceptance threshold guidelines, and somewhat mixed reviews, with most reviewers recommending accept (7,6,5,5) and one (reviewer qevo) recommending borderline reject (4). One of reviewer qevo's main concerns is that the paper might be similar to offline RL but did not compare to offline RL baselines, a concern also identified by reviewer fQBY. The authors responded to this concern by elucidating several properties of their method not enjoyed by Offline RL, but also by adding some new results with Offline RL baselines for an experiment where it was applicable. This convinced reviewer fQBY to raise their score from a 3 to a 5, and reviewer qevo to raise their score from a 3 to a 4. Given that reviewer qevo is the only reviewer recommending reject, I attempted to engage them in further discussion during the discussion period but they did not respond. Reviewer fQBY did clarify why they raised their score, and mentioned that the additional experiments provided by the authors on multiple offline RL baselines were convincing. They also mentioned that they had initially misconstrued the optimization problem (Eq. 9), but found the further clarification provided by the authors to be helpful. In reading the detailed rebuttal provided by the authors, I found that they were able to point to additional experimental results in the appendix to address the other concerns raised by reviewer qevo. Given this, I am recommending an accept as a poster.
Decision: Accept (poster)