Budgeting Counterfactual for Offline RL
Authors: Yao Liu
Pratik Chaudhari
Rasool Fakoor
Venue: NeurIPS 2023 poster (2023)
Keywords: reinforcement learning
offline reinforcement learning
counterfactual reasoning
TL;DR: We propose a new offline RL method by limiting the number of counterfactual decisions in it and find the best allocation by dynamics programming.

Abstract: The main challenge of offline reinforcement learning, where data is limited, arises from a sequence of counterfactual reasoning dilemmas within the realm of potential actions: What if we were to choose a different course of action? These circumstances frequently give rise to extrapolation errors, which tend to accumulate exponentially with the problem horizon. Hence, it becomes crucial to acknowledge that not all decision steps are equally important to the final outcome, and to budget the number of counterfactual decisions a policy make in order to control the extrapolation. Contrary to existing approaches that use regularization on either the policy or value function, we propose an approach to explicitly bound the amount of out-of-distribution actions during training. Specifically, our method utilizes dynamic programming to decide where to extrapolate and where not to, with an upper bound on the decisions different from behavior policy. It balances between the potential for improvement from taking out-of-distribution actions and the risk of making errors due to extrapolation. Theoretically, we justify our method by the constrained optimality of the fixed point solution to our $Q$ updating rules. Empirically, we show that the overall performance of our method is better than the state-of-the-art offline RL methods on tasks in the widely-used D4RL benchmarks.

Budgeting Counterfactual for Offline RL
Yao Liu', Pratik Chaudhari 1,, Rasool Fakoor'
Amazon Web Services, "University Pennsylvania Cyaoliuai, prtic, fakoor) Camazon. fawooor)aaaazznno....ooo com
Abstract
The main challenge of offline reinforcement learning, where data is limited, arises from sequence of counterfactual reasoning dilemmas within the realm of po- tential actions: What if we were to choose different course of action? These cireumstances frequently give rise to extrapolation errors, which tend to accu- mulate exponentially with the problem horizon. Hence, it becomes crucial to acknowledge that not all decision steps are equally important to the final outcome, and to budget the number of counterfactual decisions a policy make in order to control the extrapolation. Contrary to existing approaches that use regularization either the policy or value function, we propose an approach to explicitly bound the amount out-of-distribution actions during training. Specifically, our method utilizes dynamic programming to decide where to extrapolate and where not to, with an upper bound on the decisions different from behavior policy. It balances between the potential for improvement from taking disttribiittt actions and the risk making errors due to extrapolation. Theoretically, we justify our method by the constrained optimality of the fixed point solution to our Q updating rules. Empirically, we show that the overall performance of our method is better than the state-of-tte Mtateofftthe-ar offline RL methods on tasks in the widely-used D4RL benchmarks.
Introduction
One of the primary hurdles in reinforcement learning (RL), or online RL, is its reliance on interacting with environment in order learn [47]. This can be significant barrier applying RI to real- world problems, where it may not be feasible or safe to interact with the environment directly [48]. In contrast, batch offline RL [6. 27, 30] provides more suitable framework for effectively learning policies by leveraging previously collected data to learn a policy. Notably, offline RL relies on fixed but limited dataset comprising previously collected data from an unknown policy(s) and lacks the ability to continue interacting with the environment to gather additional samples.
These limitations in offline RL give rise various challenges, with notable issue being the occurrence of extrapolation problems resulting from the scarcity training data [8, 12]. In order overcome this challenge, previous approaches to offline RL primarily focus on constraining the gap between the behavioral policy and the learned policy [4, 11, 12, 25, 40, 51], or limiting the disparity between the state-action values of logged actions in the data and extrapolated actions [8, 22, 26, 50, 53J. Alternatively, other approaches utilize learned model that avoids making predictions outside the distribution the dataset [21, 35, 55J. All these offline RL methods involve a sequence of counterfactual reasoning problems within the realm of potential actions, explicitly or implicitly. The question of "what if" we were to choose different course of action than the behavior policy often arises, leading to extrapolation errors. Hence the difficulty of the problem increases as we plan for longer horizon and involves more counterfactual decisions. For offline RL algorithms, difficult to find a balance between the potential for improvement from taking distribbution actions and the risk of making errors due to extrapolation.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

A critical aspect of offline RL which is relatively unexplored pertains to the varying importance of different actions at each step in determining the final outcome. Not all actions are created equal, and some have more influence on the outcome than others. Thus, it is not always essential to consider alternative actions or counterfactuals at every step than the one dictated by the behavior policy. Rather, the emphasis should be placed on the actions that wield the most impact, warranting careful secision-making at those steps. This limits the number of counterfactual decisions we make and assigns them to the most needed states/steps, to increase the return on the "extrapolation investment".
Unlike existing approaches that employ explicit or implicit regularization on either the policy, value functions, or both, we introduce a novel and effective algorithm, called Budgeting Counterfactual for Offfline RL (BCOL), that explicitly constrains the level of extrapolation during training. Specif- ically, our approach leverages dynamic programming to determine when and where extrapolation should occur, with a budget on the deviation from the behavior policy. This enables us to strike a balance between the potential for improvement achieved by exploring -uitt--distriuution actions and the inherent risks associated with extrapolation errors. Conceptually, such method different from regularized offline RL by allowing non-uniform constraints while keeping strict bounds on counterfactual decisions.
Our contribution. We propose a novel algorithm, BCOL, making only a few but important coun- terfactual decisions in offline RL. This idea extends the space of current offline RL methods that extensively focus on regularized methods. We demonstrate that the fixed point resulting from our Q-value updating rule corresponds to the optimal Q-value function, under the constraints that the policy deviates from the behavior policy within a budget. We conduct thorough evaluations on diverse array of tasks from the widely-utilized D4RL benchmarks [9]. In terms of overall performance, our approach exhibited favorable comparison to existing state-of-the -f-e-of-the-rrttt methods.
2 Problem Settings
We study the RL problem in the context of the infinite horizon, discounted Markov Decision Process (MDP) [41]. An MDP is defined as tuple M =< S,A,r, Po, P,y > where S is a state space, A is an action space, and both of these spaces can be infinite or continuous in nature. r ::S S x A + RR is the reward function. Po is distribution over S that the initial states are drawn from. P S x A + A(S) maps a state-action pair to distribution over state space. 7 is the discount factor over future reward. The goal in MDP is to maximize the expectation of discounted future reward Rx = E Dc >'re | ae ~ T.. An important function in MDP is the state-action value function, known as the Q function. which represents the expected future reward, given an initial state or state-action pair. Q* a) == E [E T'T, so
s, 0,00 = a].
The primary emphasis of this paper is model-free learning algorithms within the context of offline RL. The goal is to learn a target policy v that maximizes the expected future reward using a fixed dataset D. The dataset D consists of transitions (s, a, r, s', a') where can be drawn from any fixed distribution, a ~ p(((s), r = r(s, a), s' ~ P(-|s, a), a' ~ [u([(s).. Here μ denotes an unknown behavior policy that was utilized in the past collect dataset D.
In offline RL, the task of learning an optimal policy poses a greater challenge compared to online RL. This is primarily due to the fact that the target policy can be different from the behavior policy, leading to the introduction of out-of-distribution actions. This challenge is further amplified by the use of function approximation and the problem horizon the extrapolation errors from fitted Q function tend to accumulate over time-steps [8, 12]. In particular, it will results in the Q(s, a) values increasing dramatically for outto--distributio state and action pairs. Consequently, this results in the learning of a policy that is likely to perform poorly and risky at the deployment time. Many previous works on offline RL aim address these issues by some form of regularization, which might hurt the flexibility of making certain key decisions at each step (which we will return to later). However, we take a different approach in this paper where we emphasize cautiousness of making counterfactual decisions and, hence propose method that only deviates from behavior policy for a few steps.
3 Method
As mentioned earlier, prior studies in offline RL mainly address the issue of extrapolation through two types of regularization terms: policy regularization or value regularization. Policy regularization

---

typically involves utilizzig a divergence metric between current candidate policy and behavior policy while value regularization is commonly defined the positive difference between (s, a) and Q(s,a), where ~ p(-[s). In these works, the regularization terms were uniformly applied to all samples in the dataset, enforcing flat penalization on the distance between rT-|s)) and (u(((s), or flat penalization of overestimating Q(s, a) for all The utilization of a uniform regularization approach can present a problem. When strong regularization enforced, the divergence bound from behavior policy over all states are small, hence the policy may not be able to improve significantly. In such cases, the resulting policy can only be as good as the behavior policy itself. Conversely, due to the fact that trajectory distribution mismatch increase exponentially with respect to the problem horizon [8, 12, 34, 36], if the bound is not small enough, it causes significant issues such as large extrapolation errors that lead to risky and poor noor-performing policy. Therefore, applying regularization uniformly without considering the specific characteristics of each state can pose challenges and limitations in offline RI..
This work emphasizes the importance of making counterfactual deci- sions, i.e. decisions that are different from the decisions that would have been made by the behavior policy. Building upon the notion that not all decision steps carry equal importance, we introduce a novel offline RL algorithm that incorporates only a limited number of counterfactual decisions. In other words, the algorithm follows the behavior policy 1u most of the time, but it only makes counter- factual decisions (i.e. performing offline RL) in certain states. Thus intuitively it maximizes the potential improvement from counterfac- tual decisions while keeping the extrapolation error under control?. We refer to this number by the budget of counterfactual, denoted as B. The grid world MDP in Figure shows a simple example of why using a budget of counterfactuals is an effective constraint. Here, the goal is find the shortest path from the start to the goal state without passing through the fail state, given a dataset of previously recorded trajectories from different and unknown behavior policies. There are two types of trajectories in the dataset, marked as yellow and blue in Figure 1. While yellow trajectories are the most common, the shortest path requires stitching together these two types of trajectories and taking counterfactual decision (shown by dashed green line). A budget of counterfactuals ensures that the agent only takes the most beneficial counterfactual decision and prevents it from taking unnecessary counterfactual actions.
Goal
Start
-10
-10
Figure 1: Grid world example
In order to spend the budget of counterfactual strategically, a dynamic programming algorithm is utilized to plan and asses the potential policy improvement resulting from actions taken outside of the data distribution. Thus, the choice of policy on each step needs to balance between the immediate gain from taking greedy actions the current step and the benefit from future counterfactual decisions. Naturally, the policy also depends on how many counterfactual decisions it can take before it exceeds the upper bound B. We define the budget of counterfactual decisions at a time step as follows:
b = b. 1 1 (-((se,bb)) be) + p(-1s:)), bn = B,
where 1 is an indicator function. The policies studied in this paper take the current budget b, as an input, which b, is the initial budget B subtracted by the number of counterfactual steps taken before time step as shown in (1), Given that, the goal of our method is to solve the following constrained policy optimization:
max E
Σ r'rel s0 ~ Po, n s.t. b, z o, Vt 2 0, V[(s by, a:) E (M((
(TT is the support set of trajectory distribution introduced by the MDP and policy ग. Without additional statements, later we only consider statc-action- budget trajectories in ( and drop this
'There are different approaches to obtaining à [5, 8). Generally, it approximates argmax, Q(+,) :) or is sampled from the current policy.
TTechnically, extrapolation refers to of disttiibbttnionn actions. A counterfactual decision policy may still overlap with the behavior policy and thus not truly extrapolate. Whether or not actions from counterfactual policy lead to extrapolation is unknown when we only observe logged action rather than In this paper, we refer to this "possible extrapolation" extrapolation well, for the simplicity of discussion.
3

---

requirement for the ease of notation. Note that by in (2) is a function of the initial budget bu = B, policies, and states before step t. To provide more clarity, we expand the constraint (2) as follows:
2 Σ (= # ((-1s:)) - B
In order to develop our offline RL algorithm that maximizes (2), we introduce new Bellman operator that plans on backup future values well as the number of counterfactual decisions.
Definition (Counterfactual-Rddgeiee
Bellman Operator).
TenQ(s,b,a) == r(s, a) + [V (8', b)]
where Vo) max (max Q(6' l,a',, Enray Q(5 Q(d',baa)) b
En'rp ( b, a')
b b= 0
The Bellman operator TeB updates the values by taking the maximum value between terms. The first term refers to the case where a counterfactual decision is made by maximizing Q values over all possible actions the next state. This leads a decrease in the counterfactual budget. The second term involves following the behavior policy in the next state (similar to SARSA [43]) while keeping the counterfactual budget intact. By selecting the optimal backup from the two cases, the Q value for a given budget b strikes a balance between the benefits of counterfactual decisions (i.e. max,' Q(s', b- 1, d')) the next step and in further future. Since S B, there are at most B backup steps taking the maximum max,' Q(s', b a') in the backup path. This intuitively upper bounds the amount of extrapolation that the Q function can take. We can recover the standard Bellman operator on state-action Q functions by considering b = cc in (s. b, a) and the rule oc 1 = co. oo, the first max operator Vo always prefers the former term and gives us the
Since 1 = b- standard Bellman operator.
It is straightforward show the counterfactual- budgettng Bellman operator is 7-contraction. Based on that, we prove the fixed point is the optimal Q function constrained by counterfactual budget. Theorem 2. There exists a unique fixed point of TCB, and it is
oO
Q"(s,b,a) = max E
Σ
so s, ao a. b1 b,
s.t. be 2 0, 2
(5)
This theorem (proved in the appendix) indicates that the fixed point iteration of TEB will lead the optimal value function with the upper bound of counterfactual decisions. Thus it motivates to minimize a temporal difference (TD) error of the wewterffactallbudggtiiinn Bellman operator, towards the goal of only improving behavior policy over limited number of but important decision steps.
3.1 Algorithm
In this section, we derive practical offline algorithm for our approach when the counterfactual- budgeting Bellman operator (4) is estimated via function approximation. First, we need to replace the expectations in (4) by one-sample estimation to evaluate the Bellman backup with the fixed dataset D. Next, considering that the action space is continuous, we approximate the max, operator in (4) by maximizing over actions sampled from a policy network that is trained maximize the current Q function. Note that these design choices are commonly employed in previous offline RL algorithms [8. 12, 25, 26]. As a result, we obtain a sampled- based counterfactual-budee budgeeing Bellman operator denoted as TeB-
Definition (Approximate Counterfactuall-uudee
Bellman Operator).
v(s, a). Tcn = r(s,a) +7 { Qo(s',ba''
max max 1n am
Qols', 1ld',b-13 1,ū),
b> 0
0
'Here (* b, a) is defined as given b, = b rather than bo because the action is already given and the future (optimal) value should be independent which distribution drawn from.

---

Algorithm BCOL Training
Algorithm 2 BCOL Inference
Input: 80 80. T. B, D
1: Input: 0, Φ, B, M
2: for 1 to T - 1 do
2: by f- B, 80 ~ M
3: a++1 + 0c 0g VoLo B,
for t to trajectory ends do
Φ++1 6² ap V CL (0: 0: B.D)
4: T, b. t- Select Wi; sr, b, Qo)
5: end for
Of ((); Ft, 8141 ~ M
6: Return AT DTT
end for
where (s', d') sampled from D and {ax additional input to the Q functions compared with the standard Bellman operator, it does not add any extra data requirement. We can augment the (s, a, s', a') tuple from offline dataset with all where is the maximum number of counterfactual decisions that we consider.
(4.)m === sampled from Although this operator requires
S b S
To learn the Q function, we minimize the least square TD error introduced by TeB, using dataset D consisting of (s, a, s", a') sampled from behavior policy Jr. Subsequently, we update the policy by making it more likely choose actions with higher Q-values. The objective functions for the Q function and the policy π are formally defined as follows, with 0 representing the parameters of the Q function and e representing the parameters of the policy networks:
B
( (0o(s)b.a)-TTaAall -Taao((s.b.)))
Lo(0,0; (,D,) =
Σ
(6)
( (a)
b=u
C. (0,00, D,D) =
E anDD ar DaaDa (-18,6)
Q0(s,
where 0 denotes delayed target network. Algorithm describes the training process of our method! The counterfactual--udeeee budgeting Bellman operator has an interesting property that Q is monotonically increased with b, for any Q such that Q = Q.
as) 2 bs. Vs,a,b> 8'.
(8)
The proof is straightforward since Q with larger budget maximizes the value over more action sequences. This property will be followed asymptotically. However, with limited number of iterations, and function approximations, the gap is not always positive. Intuitively, enforeing the monotonic gap might reduce the search space and potentially accelerate the convergence. It is important to note that this modification does not alter the fixed point solution since Q' always has monotonic gap. Such regularizer does not enforce any behavior/pessimistic constraint to the policy. It only constrains the Q-values for the same action with different budgets, not the Q-values for different actions. This regularizer will help the dynamics programming over counterfactual actions finding a self-consistent solution.
To implement it, we introduce the following penalty term and add it to the LQ. This results in the revised form of LQ follows:
Co(0,0:0,,.DD Σ :=U
E a (15,))
(maxx(Q(ss - 11,),b+1,)))))
This penalty term minimizes the gap for the actions sampled from (- 6). As T maximizes Qo(s.b,) b, -),, which is supposed to be smaller than Qo b + 1, it can be viewed as a more efficient sampler of actions, compared to uniformly sampling, in order ensure the constraints in Equation 8.
Putting these together, we have Algorithm 1: Budgeting Counterfactual for Offline reinforcement Learning (BCOL). It uses an offline actor-critic style algorithm to learn the Q function and a policy maximizing the learned Q values. However, as the problem is to find the best policy under the constraints, greedy policy To itself is not enough for inference. The action selection method during inference is implicitly included in the definition of TeB. At test time, the policy needs to look ahead based on the current budget b, and the Q values of taking counterfactual actions v.s. taking the
NNote C, is not exact but abstract in the sense that gradient to the & as a distribution parameter cannot be calculated directly and need the so-called policy gradient estimator. We clarify our implementation in Section 3.3.
5

---

behavior policy. Based on that decision, we updated the new budget as well. We define this step as an operator below.
(u(-1s),))) if E (I(-)s,, (),[]s,)),bb 1) o.w.
-(s,b-1,00 - 1,0)
Q(+ b, ar orb a~a((s)
E
(e,ect(T,,1;,,b4 Q)
The complete inference time procedure is described in Algorithm 2. It also takes the learned policy and Q function parameters as well as an approximate behavior policy f.. In case of unknown behavior policy, f can be learned from behavior cloning or any other imitation learning algorithm. Algorithm 2 starts with an initial counterfactual budget B, takes action each time according to the condition in Select, and update the budget b, if the action is not drawn from ft.
3.2 Comparison to regularized and one-step offline RL
One of the most used methods in offline RL methods is adding policy or value regularization and constraints terms on top of a vanilla off- aff-policy RL algorithm [5, 8, 12, 22, 23, 25, 26, 50, 51], referred to as regularized methods in this paper. Our method can be viewed as an alternative to using regularized losses. Instead, we enforce a non-Markov budget constraint on the policy class, and we argue this provides several unique advantages. First, compared with the coefficients of regularization terms, the budget parameter has more clear physical explanation in most applications. Thus it is easier to tune the hyper- param-paraettrrs and explain the decisions from the policy in practice. Second, the constraints budget will never be violated in the test, it provides an additional level of safety. While regularization terms penalize the divergence during training, they cannot provide a guarantee on the divergence distance between test policy and behavior policy.
Another type of offline RL that is related to our core idea is one-step RL [3, 49]. We propose and steps. However, the "step"
leverage the concept of limiting the number of counterfactualloff-ooiiic here refers to the decision steps, and in one-step RI. it refers to the (training) iteration step. Although one-step RL only applies only one training step, the resulting policy can still be far away from the behavior policy in many different states, and give different decisions during the test. From another perspective, our method, although with limited number (or even one) of counterfactual decision steps, still applies dynamic programming to find the best allocation of counterfactual steps.
3.3 Important implementation details
Algorithm 1 provided general actor-critic framework to optimize policy value constrained by counterfactual decisions. To implement practical offline deep RL algorithm, the first design choice is how we model the policy and choose the policy gradient estimator to V VyLa. Algorithm is compatible with any off- off-policy policy gradient methods. We implement Algorithm using SAC-style [17] policy gradient for stochastic policy and TD3-style [13] policy gradient for deterministic policy, to provide generic algorithm for both stochastic and deterministic policy models. As TD3 uses deterministic policy, the value of m is 1 in TcB and the actor loss becomes Qo(s, 6, (s, 6)). In both cases, we implement target updates following SAC/TD3. We defer more details to the appendix.
Both SAC and TD3 are originally proposed for online, off off-policy RL. To better fit into the offline setting, previous offline RL methods based on these algorithms equip them with several key adapta- tions or implementation tricks. To eliminate orthogonal factors and focus on the budgeting idea in algorithm comparison, we follow these adaptations and describe them below. For SAC, prior work use twin Q functions [26], and a linear combination of the two Q values [8]. Prior work [8, 24] also sample m actions from the actor and take the maximum Q values in backup, instead of a standard actor-critic backup, which is a trick firstly introduced in [12, 15]. The entropy term in SAC is dropped in [8] as this term is mostly for online exploration. Previous TD3-based work [11] normalizes the state features and Q losses in TD3. All these adaptations are commonly used in state-of-the-ar offline RL and are mostly considered as minor implementation details rather than major algorithmic designs. We refer to SAC and TD3 with all these adaptations as offline SAC and offline TD3 in this paper and refer to the two families of algorithms SAC-style and TD3-style.
One-step RL[3] and IQL [23] for offline RL is built on top of estimating the behavior value Q" by SARSA. Unlike SAC or TD3, it is not applicable to apply the counterfactual budgeting idea on top of SARSA, since it a native on-policy method and does not consider counterfactual decisions. Thus focus on the implementation of BCOL with SAC and TD3.

---

Tisk Name
BC
10%BC
DT
KAMBO
ARMOR
IQL
Onestep
113
BCOL
COL
CDU
BCOL
+BC
(TT33)
(SAC)
halfoheetalim
42.6
42.6
77.6
47.4
55.6
62.5
hepper-m
52.9
56.9
67.6
92.8
101.4
66.3
83.3
68.4
85.8
64.6
94.9
walker2d-m m
75.3
75.0
74.0
56.9
90.7
78.3
85.6
81.5
76.7
74.5
70.7
81.1
36.6
40.6
36.6
68.9
50.5
44.2
42.5
44.4
40.9
45.
52.3
46.2
hogper
82.7
96.6
94.7
71.0
50.1
83.4
92.
87-4
walker2d-mr
26.0
62.5
66.6
85,0
85.6
73.0
71.6
80.2
49.7
83.7
87.5
86.0
haltchetthh-mm
55.2
92.9
86.5 8
93.7
93.5
86.7
93.5
91.5
86.7
67.3
66.3
86.9
hopper
52.5
107.6
$3.3
103.4
91.5
102.1
100.5
100.8
83.2
walker2å-me
107.5
109.0
1
68.3
112.2
109
110.9
110.1
108.5
103.9
mujoco total
406.7
006622
672.6
753.1
758.6
092.4
718.J
605.2
655.6
713.0
G59 .U
746.0
54.6
62.8
59.2
25.0
87.5
64.3
98.3
93.3
94.0
93.00
90.3
antmare-u-d
16.4
62.2
60.7
47.3
57.3
anmure m-p
0.0
5.4
0.0
0.0
71.2
0.3
1.7
12.4
62.4
59.5
70.0
anmmae m-d
0.0
9.5
0.0
0.0
70.0
0.0
0.3
14.0
74.3
64.6
72.3
antmare-l-p
0.0
0.0
23.2
39.6
0.0
0.0
0.0
34.2
33.0
35.6
0.0
2.4
47.5
0.0
0.0
40.7
25.3
intiaae tocal
100.2
134.2
112.2
67.0
378.0
125.3
171.3
187.7
352.9
333.5
396. 0
Total
506.9
800.4
754.8
#20.1
1070.4
841.3
874.3
1065,9
1032.5
1142.0
Table 1: Average normalized scores on D4RL tasks. Task names for MuJoCo: m-medium, mr=medium replay, me=medium expert. Task names for antmaze: u=umaze, m-medium, I=large, p=play, d=diverse. Bold numbers stand for the globally best and underlined numbers stand for the best with in the base method (SAC or TD3) group.
Looping over all budget values the loss is not efficient in practice. We implement the Q function and policy with B output heads and tensorize the loop over b. One can further reduce the computation cost by sampling budget values instead of updating all B heads over every sample.
Experiments
We evaluate our BCOL algorithm against prior offline RL methods the OpenAl gym MuJoCo tasks and AntMaze tasks in the D4RL benchmark [9]. We compare the SAC-style and TD3-style implementation of BCOL with state-of-the-art offline RI. algorithms, studying the effectiveness of budgeting counterfactual in offline RL. Our experiment results also reveal that behavior cloning with only one strategic counterfactual decision still work surprisingly well on MuJoCo tasks. Finally, study the value of our dynamic programming methods on where to spend the counterfactual budget.
Baselines. We compared BCOL with regularized methods (policy and/or value regularization) based on SAC and TD3: CQL [26], CDC [8], TD3+BC [11]. Besides regularized methods, we also compare with one-step RL method [3] and IQL [23] as other state-of-the the--rt art model-free offline RL methods, model-based offline RL, and behavior cloning, behavior cloning on the top 10% data, and decision transformer imitation learning baselines. This covers set of strong offline RL baselines. We are interested in both studying the effectiveness of counterfactual budget ideas in contrast to the closest regularized methods and state-of- the-art offline RL methods. We present a set of representative and the most performant offline RL basclines in this section, and defers the comparison against more offline RL baselines results to the Appendix due to the limit of space.
RAMBO [42] and ARMOR [52] as state-of-the-ar
Benchmark. We report the results of BCOL together with baselines on 9 OpenAl gym MuJoCo tasks and 6 AntMaze tasks in D4RL. The gym MuJoCo tasks consist of the v2 version of medium, medium-reply, and medium-expert datasets in halfcheetah, walker2d, and hopper. MuJoCo tasks generally prefer imitation-type algorithm, as it contains a large portion of near-optimal trajectories. The 6 AntMaze tasks are considered harder tasks for offline RI as they contain very few or no near-optimal trajectories, and many previous methods lack performance reports on these tasks. We exclude the random datascts in MuJoCo tasks as they are not less discriminating and none of the offline RL algorithms Iearns a meaningful policy there. We exclude expert datasets in MuJoCo tasks since they can be solved simply by behavior cloning and mismatches the consideration in most offline RL algorithm designs.
How we report the results. Due to the inconsistency in the version of D4RL in the literature, it is necessary to clarify the source of baseline results here. We retrieve the results of baseline methods from the original papers if applicable (IQL,, CQL, one-step RI, DT, RAMBO, ARMOR). We report the Rev. KI. Reg variant of one-step RI. as it shows the best performance on MuJoCo tasks [3]. For AntMaze tasks report one-step RL and decision transformer results from the IQL paper as they
7

---

BCOL SAC
BCOL TD3
BCOL SAC
BCOL TD3
400
700
M/uboC total
800
Antharre total
-00
500
200
400
100
0
300
100
100
crrgg
omagg
oenege
orega
Figure Total normalized score with different values of B and in BCOL. The left two plots show MuloCo average scores and the right two plots show AntMaze average scores.
are not reported in the original paper. We report the scores of behavior cloning and 10% behavior cloning. We report the scores of behavior cloning from the IQL paper. We report the scores of CQL from an updated version of CQL paper' for D4RL v2 environments. We report the score of TD3+BC and CDC from implementation since the original results are for D4RL v0 environments. The v2 results are generally better than the original vO scores in the original papers and are consistent with TD3+BC's v2 scores from the IQL. paper.
Table shows the results of baseline algorithms as well as two variants of BCOL (TD3-style and SAC-style) on 15 D4RL tasks. For the scores from experiment (BCOL, CDC, TD3+BC), we report the test episodic reward averaged over 300 test episodes as 10 test episodes per evaluation, the last 10 evaluations, and 3 runs with different random seeds. All algorithms run with 1M steps following the literature. We defer details like learning curves and standard deviations in the appendix.
Main results. As Table shows, BCOL-SAC outperforms prior offline RL methods for both MuloCo and AntMaze total scores. BCOL never falls behind the best methods by large margin any task. This indicates the effectiveness and robustness of the counterfactual budget in different scenarios. Especially for the total score of AntMaze, the harder tasks, BCOL-SAC outperforms most prior approaches by a large margin except IQL. We also find for both TD3-style and SAC-style algorithms, BCOL outperform the regularized method based on the same architecture, ablating orthogonal factors other than the key idea about the counterfactual budget.
Hyper-parameters: B and w. Our algorithm only adds two hyyer--prameee on top of SAC/TD3:
budget B and w. We searched the value of B in {1, 10, 50} and the value of w in {0, 1, 10. 100).
We select one set of hyper-parametent for MuJoCo (SAC-style: B = 10, w 10, stylst
B = 50, w = 10) and one set for AntMaze (SyC-style and TD3-style: B = 50, = 0) based on the overall performance. This provides fair comparison with baseline methods as all of them select their hyper- hyper-paramente parameters either as this, per task.' 2 Figure 2 shows how the total scores changes with different values of B and w. Full results with scores on each task are in the appendix.
Figure 2 also shows another surprising finding from our experiments on the MuJoCo tasks. With the budget being only one 1. BCOL (SAC) shows comparable performance to some recent offline RL work including CDC and IQL. With 10 steps out of 1000 steps, it is able to outperform most prior methods. This highlights how strategically adding a few important actions on top of behavior policy can provide strong performance in this benchmark. It also indicates that the MuloCo tasks in D4RL., even the more mixed datasets, considerably prefer the imitation-type approach.
Ablation on how we realize the counterfactual budgeting constraints. The idea of budgeting counterfactual decisions naturally requires both the planning on budgeting in Algorithm during the training and the planning on budgeting in Algorithm 2 during the testing. Ablation on the roles of budgeting itself and two planning parts is nonetheless helpful for us understand how BCOL works. In Figure 3, we report the results of three ablation methods.
The first column shows the performance without budgeting constraints. This leads to offline TD3 or SAC algorithm with all implementation adaptations used by BCOL and others [11. 12, 26]. As we expected, vanilla offline RI. without budgeting does not work well. The second column shows the performance with budgeting but without any planning on budgeting during either the training or the testing. This method realizes the budgeting by randomly assigning B decisions the policy
https:://sits.googgg. cowpellecoomieewclllffflinnee 2TD3+BC paper does not include results and hyperparameters in AntMaze. We do the pppertparameter search of their a, within the range provided in the paper, and report the highest total score with a = 3.
8

---

No Budgeting (Offline SAC)
Budgeting w/o Planning
Budgeting w/o Test Planning
Percent Difference
100
50
-50
hi--11
hop-m
W-MI
hc mr
hep-m
w-mr
he-me
hop-me
w-me
an-u
am-i-0
am-m-
airter ann an-id
hc-m
hop-m
ho-mt
hop-mr
w-mr
he-me
hop-me
w-me
am-u
am-u-d
am-m-g
am-m-
am-h-p
am-i-d
he-m
hop-m w-th
ho-mr
hop-ni
ho--e
w-me
am-i
ma mimm
No Budgeting (Offline TD3)
Budgeting w/o Planning
Budgeting w/o Test Planning
Percent Difference
100
50
-50
100
hc-m
hop-m
ho-me
hop-mm
w-mr
ho-me
hoo-Ti
w-me
am-u
an-urd
am--
am-m-- amna
the hop-m
w-m ho-mr
hop-mt
w-mr
ho-me
hoo mo- w-me
amnu
am-urd
am-m-p
am-m-d
he Mann Wan
ho-ni
hop-m!
w-nt
ho-me
hoo-me Byna
a
has
Figure Percent difference of the performance on different budgeting methods compared with the full BCOL Algorithm (hc HalfCheetah. hop Hopper, w Walker2d, am=AmtMaze). The top row
shows SAC-based experiments and the bottom row shows TD3-based experiments. TD3 plots do not include AntMaze-large tasks since the performances of BCOI. are zero. No budgeting stands for offline SAC/TD3 without the budgeting constraints (equivalent B + oo). Budgeting without planning stands for randomly selecting B steps to follow from π and the rest from jú during the test, where π is learned by offline SAC/TD3, Budgeting without test-time planning stands for randomly selecting B steps (uniformly within the max horizon) to follow from and the rest from j during the test, where स. is learned by Algorithm 1. In all settings, B is the same value as selected by BCOL.
learned by offline SAC/TD3, and the rest to the estimated behavior policy. Randomly stitching the counterfactual policy and behavior policy fails to fully realize the benefit of budgeting, since both training and inference are not budget-aware. The experiment in the third column studies if it is sufficient to be budget-aware during training. This ablation randomly selects B actions from ' trained by BCOL and the rest from the estimated behavior policy. The setting is closest to BCOL, but the lack of planning on how to spend the budget during the testing still hurts the performance. consistent. This ablation shows that
The results SAC-style and TD3-style implementations the effectiveness of BCOL relies on the collaboration of all three parts of our core idea: budgeting, planning budgeting during the training, and planning on budgeting during the testing.
5 Related Work
Most recently proposed offline reinforcement methods rely on some mechanism to force the learned counterfactual decision policy stay close the data support. One approach this end is regu- larizing the policy by its divergence with data, either by parameterization [12, 15], constraints and projection [29, 32, 44), divergence regularization [10, 11, 25], implicit regularization by weighted regression [10, 37, 39, 40, 50]. The other similar idea, which is often applied together, is regularizing the value estimate of policy in an actor-critic or learning architecture [5, 7, 8, 22, 26, 28, 33, 34, 51]. Similar regularization can also be realized by the ensemble of Q functions [1, 2, 14].
Some recent batch RL methods focus more adaptive policy, similarly to this paper, but conditioning on history [16] or confidence level [19]. A similar idea to us of making a few key decisions in one trajectory was studied [18, 46], but they focus stitching the logged trajectories in data.
In contrast to batch RL, imitation learning often does not involve dynamic programming on extrap- olated actions. It has been used as regularization in online RL [31, 38] and offline RI [11, 37]. Recently imitation learning methods using transformers and conditional training also achieves good performance on offline RL benchmarks [4. 56]. One-step RL [3] can also be viewed an extension

---

of imitation learning with one-step policy extraction [49]. Although utilizing the behavior policy as well, Our key idea different from these as train policy with awareness of behavior policy, rather than regularized by behavior cloning or imitation learning loss.
6 Discussions
The form budget considered in this work counting the different decision distributions. A natural alternative is divergences between π and This provides soft count of counterfactual decisions. However, it requires good calibration of both distributions and is more suitable to the discrete-action settings. We leave this as future work. On the theory side, natural question about the value of budgeting counterfactuals is how the benefit of less counterfactual decisions is reflected in the theoretical properties of offline RL. We leave the investigation on this for future theoretical work.
In conclusion, this paper studies offline reinforcement learning which aims to learn good counter- factual decision policy from a fixed dataset. We propose novel idea of budgeting the number of counterfactual decisions and solving the allocation problem by dynamic programming. We provide strong empirical performance over offline RL benchmarks and optimality of the fixed point solution as thcoretical justification.
10

---

References
[1] Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. In International Conference on Machine Learning, 2020.
[2] Gaon An, Seungyong Moon, Jang-Hynn Hyun Kim, and Hyun Oh Song. Uncertainty-baat -based offline reinforcement learning with diversified q-ensemble. Advances in neural information processing systems, 34:7436-7447, 2021.
[3] David Brandfonbrener, Will Whitney, Rajesh Ranganath, and Joan Bruna. Offline ri without off ofllpo evaluation. Advances in neural information processing systems, 34:4933-4944 2021.
[4] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling, 2021.
[5] Ching-An Cheng, Tengyang Xie, Nan Jiang, and Alekh Agarwal. Adversarially trained actor critic for offline reinforcement learning. In International Conference on Machine Learning. pages 1852-3878. PMLR, 2022.
[6] Damien Ernst, Pierre Geurts, and Louis Wehenkel. Tree-based batch mode reinforcement learning. Journal of Machine Learning Research, 6, 2005.
[7] Rasool Fakoor, Pratik Chaudhari, and Alexander J. Smola. P3o: Policy-on policy-off policy optimization. In Proceedings of The 35th Uncertainty in Artificial Intelligence Conference, volume 115, pages 1017-1027, 2020.
[8] Rasool Fakoor, Jonas W Mueller, Kavosh Asadi, Pratik Chaudhari, and Alexander J Smola. Continuous doubly constrained batch reinforcement learning. Advances in Neural Information Processing Systems, 34:1 1211-1107222 2021.
[9] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. 2eXi::2000.07219, 2020.
[10] Yuwei Fu, Di Wu, and Benoit Boulet. A closer look at offline rl agents. Advances in Neural Information Processing Systems, 35:8591-8604 2022.
[11] Scott Fujimoto and Shixiang Shane Gu. A minimalist approach offline reinforcement learning. Advances in neural information processing systems,
2021.
[12] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning, pages 2052-2062,
2019.
[13] Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 1587-1596. PMLR, 2018
[14] Kamyar Ghasemipour, Shixiang Shane Gu, and Ofir Nachum. Why so pessimistic? estimating uncertainties for offline through ensembles, and why their independence matters. Advances in Neural Information Processing Systems, 35:1 13:1877118881 2022.
[15] Seyed Kamyar Seyed Ghasemipour, Dale Schuurmans, and Shixiang Shane Gu. Emaq: Expected-max q-learning operator for simple yet effective offline and online arXiv:2007.1 1091, 2021.
[16] Dibya Ghosh, Anurag Ajay. Pulkit Agrawal, and Sergey Levine. Offline rl policies should be trained to be adaptive. In International Conference on Machine Learning, pages 7513-7530. PMLR, 2022.
[17] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. arXiv: 2018.
[18] Charles Hepburn and Giovanni Montana. Model-based trajectory stitching for improved offline reinforcement learning, arXiv preprint arXiv:2211 2022.

---

[19] Joey Hong, Aviral Kumar, and Sergey Levine. Confidence--onnittoone value functions for offline reinforcement learning. arXiv preprint arXiv:2212.04607, 2022.
[20] Bingyi Kang, Xiao Ma, Yirui Wang, Yang Yue, and Shuicheng Yan. Improving and benchmark- ing offline reinforcement learning algorithms. arXiv preprini SEIi::23060097722 2023.
[21] Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model-based offline reinforcement learning. arXiv:2005.05951, 2020.
[22] lly Kostrikov, Rob Fergus, Jonathan Tompson, and Ofrr Nachum. Offline reinforcement learning with fisher divergence critic regularization. In International Conference on Machine Learning, pages 5774-5783. PMLR, 2021.
[23] Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learnin, arXiv preprint
2021.
[24] Aviral Kumar, Rishabh Agarwal, Xinyang Geng, George Tucker, and Sergey Levine. Offline q- learning on diverse multi-task data both scales and generalizes. arXiv preprint arXiv:2211.15144, 2022.
[25] Aviral Kumar, Justin Fu, George Tucker, and Sergey Levine. Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction. arXiv: 190,,,99 November 2019.
[26] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative Q-Learning for Offline Reinforcement Learning. AXiv:2006.04479, June 2020.
[27] Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning. In Marco Wiering and Martijn van Otterlo, editors, Reinforcement Learning: State-of-the-Art, pages 45-73. Springer, 2012.
[28] Romain Laroche. Paul Trichelair, and Remi Tachet Des Combes. Safe policy improvement with baseline bootstrapping. In International conference on machine learning, pages 3652-3661. PMLR, 2019.
[29] Hoang Le, Cameron Voloshin, and Yisong Yue. Batch policy learning under constraints. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 3703-3712, PMLR, 2019.
[30] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 2020.
[31] Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In International Conference on Learning Representations, 2016.
[32] Yao Liu, Yannis Flet-Berliac, and Emma Brunskill. Offline policy optimization with eligible actions. In Uncertainty in Artificial Intelligence, pages 1253-1263. PMLR, 2022.
[33] Yao Liu, Adith Swaminathan, Alekh Agarwal, and Emma Brunskill. Off-policy policy gradient with stationary distribution correction. In Uncertainty Artificial Intelligence, pages 1180- 1190. PMLR, 2019.
[34] Yao Liu, Adith Swaminathan, Alekh Agarwal, and Emma Brunskill. Provably good batch off-policy reinforcement learning without great exploration. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1264-1274. Curran Associates, Inc., 2020.
[35] Tatsuya Matsushima, Hiroki Furuta, Yutaka Matsuo, Ofir Nachum, and Shixiang Gu. Deployment-efffcieent reinforcement learning via model-based offline optimization. In- ternational Conference Learning Representations, 2021.
[36] Rémi Munos and Csaba Szepesvári. Finite-time bounds for fitted value iteration. Journal of Machine Learning Research, 9(5), 2008.
12

---

[37] Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.093359 2020.
[38] Ashvin Nair, Bob McGrew, Marcin Andrychowiez. Wojciech Zaremba, and Pieter Abbeel. Over- coming exploration in reinforcement learning with demonstrations. In 2018 IEEE international conference on robotics and automation (ICRA), pages 6292-6299. IEEE, 2018.
[39] Gerhard Neumann and Jan R Peters. Fitted Q-iteration by advantage weighted regression. In Advances in Neural Information Processing Systems, pages 1177-1184, 2009.
[40] Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. CoRR, abs/1910.00177, 2019.
[41] Martin I.. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley & Sons, USA, 1st edition, 1994.
[42] Marc Rigter, Bruno Lacerda, and Nick Hawes. Rambo-rl: Robust adversarial model-based offline reinforcement learning. Advances in neural information processing systems, 35:16082- 16097, 2022.
[43] Gavin A Rummery and Mahesan Niranjan. On-line Q-learning using connectionist systems, volume 37. University of Cambridge, Department of Engineering Cambridge, UK 1994.
[44] Noveen Sachdeva, Yi Su, and Thorsten Joachims. Off-policy bandits with deficient support. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 665-975. 2020.
[45] Julian Schrittwicser, Thomas Hubert, Amol Mandhane, Mohammadamin Barekatain, Ioannis Antonoglou, and David Silver. Online and offline reinforcement learning by planning with learned model. Advances in Neural Information Processing Systems, 34:27580 27591, 2021.
[46] Scott Sussex, Omer Gottesman, Yao Liu, Susan Murphy, Emma Brunskill, and Finale Doshi- Velez. Stitched trajectories for off-policy learning. ICML Workshop, 2018.
[47] Richard Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press. 2018.
[48] Matteo Turchetta, Felix Berkenkamp, and Andreas Krause. Safe exploration in finite markov decision processes with gaussian processes. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016.
[49] Qing Wang, Jiechao Xiong, Lei Han, Han Liu, Tong Zhang, et al. Exponentially weighted imitation learning for batched historical data. Advances in Neural Information Processing Systems, 31, 2018
[50] Ziyu Wang, Alexander Novikov, Konrad Zolna, Jost Tobias Springenberg, Scott Reed, Bobak Shahriari, Noah Siegel. Josh Merel, Caglar Gulcehre, Nicolas Heess, and Nando de Freitas. Critic regularized regression. arXiv:2006.1514 2020.
[51] Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning, 1rXiv:I911.11332 2019.
[52] Tengyang Xie, Mohak Bhardwaj, Nan Jiang, and Ching-An Cheng. Armor: A model- based framework for improving arbitrary baseline policies with offline data. arXiv preprint arXiv:22/1.00 1.04538, 2022.
[53] Haoran Xu, Li Jiang, Jianxiong Li, Zhuoran Yang, Zhaoran Wang, Victor Wai Kin Chan, and Xianyuan Zhan. Offline RL with no OOD actions: In-sample learning via implicit value regularization. In The Eleventh International Conference on Learning Representations, 2023.
[54] Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offline model-based policy optimization. Advances neural 2021.
information processing systems,
13

---

[55] Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. arXiv:2005.132.9, 2020.
[56] Qinqing Zheng, Amy Zhang, and Aditya Grover. Online decision transformer. In International Conference on Machine Learning, pages 27042-27059. PMLR, 2022.
14

---

A Proof of Theorem 2
Proof. We first prove that TсB is a c-conracction with respect to || Ilac. For any function Q1, Q2 in the space, and any E b, 0:
[V. b) - (s,66.
(10)
maxx/mmx Qi 1, Banp (s), - max(max
Q2(s, 6 1, PamnQ2(56) b, a)}
(11)
$ {max (s, - 1,a Q2( b 1,))), Bash Qi(s,b,a) - Q2(8,
(12)
< <ax (1Q) 22 ||Q1 - Q211a0}
(13)
=I1Q1 Q21lao
(14)
The first line is from the definition of VQ. The second line follows from the fact that max f(z)
max g(z)l < max fx( g(x)]. The rest follows from the definition of infinity norm between Q
functions. For = 0, it is straightforward that IVo, (s, b) - (s, ()) & 1Q Q2lloc as well. So
[IVq, Vo < 1Q: Q2 22l Now we have that
[TcBQ1 Qillill> =|lr + B.V,, - BVV
(15)
Sy111 Q211
(16)
We finished the proof of TcB is y-contraction operator. Thus there exists unique fixed point of the operator TCB. Now we prove the second part of the theorem that Q = TcaQ". First, we define
8
(*(s == max E
Σ
n'r
so = b, b, s.t. by 20
(17)
In order to show Q* TCBQ2", we need to prove the following two equations:
(* a) r(s,a) + + BI(V"(s',
(18)
max/mmaxe Q* b- 1, a), Earg (s, (*(s,b)= Eazy (8, b)
b>
(19)
b
To prove Equation [18], we write bs
xc
a)) max r(s, a) +E a...b220
'ri
8,00 a. = =,
(20)
8
max r(s,a) + ns.1.::0
E
1 1 rr =s,b,, b.
(21)
=r(s,a)
+
max E
1-1 81 8 b. b.
(22)
5 8.t.bb 20
=r(s, +7 [)],
(23)
The first line holds by the linearity of expectation. The second line follows from the definition of conditional expectation. The third line follows from the fact that given s, a, does not depend on т. The fourth line follows from the definition of V and the change of index /' = + 1.
Next, we prove Equation 19. When b 0, it is obvious that Q* Q" and V VH. So V* 0) Eu (Q' O, a)). Now consider the case of b > 0.
15

---

When > O, we have that
max (s,
Σ 7'r
80
8 b,
(24)
20
max max
E
E
r'a 80
a0 a, bp b,
(25)
2o о s. (-1.8-))
E
max
E
1'n 80 0, 0, a, 00 b,
(26)
20 a-s1(-))..)
x x...b,
max
E
max
E
Σ T'r 80 ao a, bi (
(u(-ss),
xo a>.((1(o,))
x a. 20
(27)
max
E
Q' - [( = o(-1s)).o)
(28)
a~a(-1a.))
max
(s, a,, E Q" (8, a)
(29)
amp
Next, we need to prove that
max
E
r' so = bo = b, π
(30)
2.
max
max
rr
80 = 0, = a, by b, π
(31)
No &.t.b. 20 (-1,8-)
1=0
Let
π argmax E
Σ 1=0
rre 30 b, b
(32)
2.
TO argmax
max 200.m.0ammmm...mmmmmmmm
B
E
880
0, ag
bo b,
(33)
8o
1=
80
max s.1.b,
Σ r/r,
80
s. bo
b,
(34)
x
max max
E
x Are
so
0. a, bo b,
(35)
mo x s.t.b, 20 (...
then it must be the case that - (-1ss is not equal to Tr(- Define
#(-1st, bx) = { TØ (- bz)
St b. b o.w.
(36)
nT" (- (-18t, bz)
16

---

Then we have
max
E
so
b.
b,
(37)
a.t.b,
1=U
max
max
E
E
rr 80 0, a, bo b,
(38)
xo a.t. 20 awat (-1,b)
E
E
rr
80
0,00 0g
a, bo b,
(39)
am (-1a,8)
0
E
E
1r 80
0, bo b, .
(40)
amogy (-1s,b)
1=
E
s, bo 6, π
(41)
By the definition of कग and 6 0, न satisfy the constraints b, 2 0 as well. Thus it contradicts the definition of maximum. So we finished the proof
(* max/max (s. l,a), E ('s,,,,a),, a 8, b> 0 anp
(42)
This finished the proof of the fixed point of TcB is Q".
17

---

B Experimental Details
Hyperparameter
BCOL SAC
BCOL TD3
Random seeds
1,
Number of gradient steps
1e6
Hidden layer dimensions
[256, 256, 256]
[256, 256]
Activation function
ReLU
Discounting factor (?))
0.99
Target network update rate
0.005
Optimizer
Adam
Actor learning rate
3e 4
3e 4
Critic learning rate
7e 4
3e
Number of Q functions
2
Coefficient of min values (A [12])
0.75
Number of actions samples (m1)
TD3 policy noise
0.2
TD3 noise clip
0.5
TD3 policy update freq
Counterfactual budget
{1, 10, 50}
Monotonicity penalty coefficient
{0, 10, 100)
Table Hyperparameter values and model architecture details BCOL.
B.1 Hyper-parameters and infrastructures of experiments
We implement BCOL with the same architecture and hyper-parameter values as baseline methods. We list all hyperparameter values and neural architecture details in Table 2. For SAC-based imple- mentation, follow CDC's s [8] hyper-parameter values. The only exception is that we use 2 Q functions instead of 4, and we sample actions from actor (m) instead of 15. This is to save the amount of computation with small amount of performance decrease for algorithm. The baseline method (offline SAC) uses the same hyperparameter values as CDC. For TD3-based implementation, we follows TD3 [13] and TD3+BC's [11] syper-parameter values.
To generate the final results as well as learning curves, we run all algorithms with 1, 000, 000 policy gradient steps, following prior work. We evaluate algorithms every 5, 000 step before the 900, 000th gradient step, and every 1, 000 step after the 900, 000th gradient step. For each evaluation, we run 10 test episodes in the environment.
Table 3 describes the hardware infrastructure and software libraries we used to run the experiment in this paper. We will release the code to reproduce the experiment upon publication.
B.2 Important experimental setup for AntMaze
Among prior offline RL work, IQL [23] and CQL [26] outperform other algorithms with large margin in AntMaze domains. We notice that these methods also use some implementation tricks that different from other work, in AntMaze domains. In order to clarify and climinate all possible orthogonal factors, we list these implementation details below and implement our algorithm BCOL and two baselines (CDC and TD3+BC) with these implementations.
IQL uses cosine learning rate schedule for the actor learning rate and state-indepen inteiindeeendentt standard deviation in the policy network. Most prior work such as TD3+BC and CDC does not use these implementations. According to Appendix rewards for the AntMaze datasets. We apply these tricks for BCOL and two baselines (CDC and TD3+BC) that we re-implement, and report the AntMaze results in Table
the IQL paper. CQL and IQL also subtract from
Gym domains 1e for AntMaze domains. Following the suggestion from CDC authors, we found out that CDC and offline SAC perform better with a smaller learning rate in AntMaze domains. Thus we set this learning rate for our algorithm BCOL SAC and baselines (CDC, offline SAC).
We use learning rate
18

---

Machine Type GPU CPU
AWS EC2 g4dn.2xlarge
Tesla T4
Intel Xeon 2.5GHz
CUDA version
11.0
NVIDIA Driver
450.142.00
PyTorch version
1.12.1
Gym version
19.0
Python version
3.8.13
NumPy version
1.21.5
D4RI. datasets version
v2
Table Hardware infrastructure and software libraries used in this paper
B.3 Budget consumption analysis
One might wonder that if BCOL spend all the budget in the beginning trajectories. In fact, our experiment shows there are still budgets left at the end of the episode during the test, In Table 4 we present the total BCOL spent in the test (plus minus the standard deviation), averaged over seeds and last 10 policies, for AntMaze tasks. This result shows that the algorithm will not always be forced by Line 2 in Select in later steps. contrast, will plan on how to spend the budget by Line in Select.
Task
Average budget spent (B 50)
Antmaze-umaze umaze
41.00 + 0.91
Antmaze-une wee---umaazeeeeeeee
20.02 6.44
Antmaze-medium-ppee
45.94 + 1.13
A Antmaze-medium--dierss
46.04 0.92
Antmaze-large-play
43.33 + 1.81
Antmaze-large-dieers
43.75 = 1.77
Table 4: Budget Consumption
B.4 Comparison to additional baselines
Here we include the comparison against more baselines. These baselines are not discussed in the main paper either because they lack of result in the more challenging Antmaze tasks, due their relative inferior performance compared to the more recent offline RL algorithms. The numbers from the original paper if available (ATAC, RIQL, CRR+, CQL+), from other paper's report (MOPO, MoREL and COMBO is from [42] and MuZero is from [20]).
Algorithm
Mujoco Total
Antmaze Total
Total
BCOL (SAC)
746.0
396.0
1142.0
MoREL [21]
656.5
656.5
MОPO [55]
379.3
379.3
COMBO [54]
738.3
137.6
875.9
MuZero [45]
140.2
140.2
RIQL [10]
759.1
ATAC [5]
792.4
CRR+ [20]
703.9
41.9
745.8
CQL+ [20]
717.9
89.0
806.9
Table 5: Additional baselines
19

---

B.5 Complete experimental results
In this section, we include experimental details that are deferred here due to the space limitations of the main paper. First, we report the average and standard deviation (not included in Table 1) of all algorithms that we implemented in Table 6. The standard deviation is calculated across the 300 evaluation episodes for each algorithm (3 random seeds, 10 evaluation steps per random seed, and 10 episodes per evaluation.).
Task Name
TD3+BC
BCOL (TD3)
CDC
BCOL (SAC)
halfcheetah-mediun-22
48.4 4 0.3
45.0 0.4
62.5
50.1 + 0.3
hopper-medium-12 walker2d-medium--2
4.64
85.8
84.9
83.2
1.6
7..5
16.6
84.1 1.5
4.5
4.9
1.7
4
hopper-
22.1
81.2
83.2
99.8
wulker2d-medium--rqpe
8.6
45.2
8..
86.
haficherta--medduu--epptt--2
5.2
8.88
6.0
86.9 5.7
hopper--ecduuum-expper--22
100.5
7.2
99.0
walkerrad-meddiu--xxperzzz2
110.1 0.5
0.8.6
1.3.9
110.8 + 0.4
antmaze-umaze-v22
96.3
8.3
93.7+
90.3 8.4
antmarz-umaaze-diveee--y
71.7
68.0
57.3
90.0
uutmazz-mmcdium-llay--22
3.7
1233 18.7
70.0 59.5
uutnazz-mmedium-ddivees----
1.3
14.0113.1
14.3
72.3
antmaze-large-play hatmaze-largeddiverre
0.0
0.0
33.0 12.7
35.7 23.6
0.8
0.0
25.3 = 15.4
37.7 + 18.2
Table 6: Average normalized score and standard deviation on D4RL tasks.
Second, we include the learning curves these algorithms in Fig. 4 to present the details of training. Each data point in the learning curve represents evaluation step and is averaged over 30 test episodes (3 random seeds and 10 test episodes per evaluation). The shadow region represents the standard deviation over 30 test scores. For better visibility, we smooth the learning curves by moving averages with a window size of 10.
Third, we include the performance of BCOL with all searched hyper-parameter values on each task Table 7 (for BCOL SAC) and Table 8 (for BCOL TD3).
20

---

hellcheetah-meelum-v2
mopper--
welker22-meduum-2
60
60
0.00
0.25
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1,00
Gracient steps
166
Gradient steps
Gradient steps
166
hopper-meddum--pplay--2
walker226-modiim-nnaayy-
100
75
50
30
25
0.00
0.50
0.75
0.00
0.25
1.00
Gracient steps
Gradient steps
Gradient stops
Pogper-mediummeree
expperr-1222
waller22-mmddium-oxprttt
75
100
100
50
80
25
60
steps
steps
antmaze--maze-dirrree-e22
antmaze- medium-play-122
100
75
50
steps
Gradient steps
entimsae-mmdduumddivve--e-
antmaze plage-play-2
antmaze- paalnsze-laree-diveeee..
75
40
50
20
0
Gradlent
Gradient
Gradient
Figure Learning curves for CDC, BCOL (SAC), TD3+BC, and BCOL (TD3) in D4RL tasks.
21

---

w w- 100
42.5 0.4
85.8 4 10.2 02.6611221 1: 10.2
76.5 5.8 70.2 4 4.9
41.1 0.9 34.2 3.6
25.0
15.2 17.6
84.2 4.9
106. 7.2 76.2 4 15.6
4 16.5
1 18.3 49.7 = 13.0
53.3 03.3112 17.8 45.3 +133:115 15.2
0.0 + 0.0
0.0 10+0.. 0.0
0.0 + 0.0
10:00 0.0
1 0.4 93.8
0.0
10
45.0 0.4
19.2 19.2
14.5 84.2 156 3.5
0.0 0.0
0.0 0.0
0.0 +.0
0.0 0.0
=
81.8
49.6
87.8 4 26.3 108.6
+ 125 50.3
0.0
50
B 3=
45.3
28.8
30.7
1.5 42.1 41.3
8.5 62.1
72.7
31.2 = 26.8
5 19.2
0.0 + 0.0
0.0 0.0
0.0 0.0
0.0 0.0
59.9
51.0
18.8 68.0 18.0115:: 15.8 51.0
8.3 86.3
44.3
10.0
10.0
1.6
= 16.3
19
0.2
12.3 + 18.7
13.1
0.0
0.0
s w=0
9333
14.0
7.7
43.7
93.3
Table 7: Average normalize score and standard deviation of hyper parameter ablation on BCOL (TD3).
0.2
57.5
2.6 91.2 1.5 60.2
100
0.4
15.4
15.4
131
40.0
100
19.4
19.4
58.7 10.9
144 106.2 1.8
108.6
41.3 13.8
45.0
0.0
0.0
0.0
0.0
0.01
43.7
57.4
w =
78.1
79.1
10
43.5
0.6
57.7
5.0
77.5
42
39.9
1.3
51.3 12.7
0.3:1222
7.0 90.3
102.9
108.7
131
44.7
0.0 0.0
0.0
0.0
0.0
41.3
10
==
B
43.8 0.3
4 5.7
77.6 3.9
40.
13.8
10.9
20.6 6.2 85.677.0
33.0 26.9 12.31142
87.1 18.4 107.0
5
44.3 13.1171
0.0 0.0
0.0 0.0
0.0 0.00
0.0
w w=l
50.
889 50.5
15.1 55.7
5 5.7
40.1 1.0
4
15.1
48.0188.
183
0.0 0.0
0.0 0.0
0.0
0.0 0.0
w=O 0
43.7
57.3
58.3
53.7
76.8
80.6
100
4: 0.5
4 6.3
5.8
36.6
2.1
2
25.2 10.9
69.7 19.777.9 7.9 70.1 882 72.7 7 7.1 72.9 8.1
+ 21.0
339
45.3 + 12.6
+ 16.9 16.9
2.5
0.0
0.0
0.0 0.0
w =
42.8
55.5
71.6
36.6
=0 18.0 78.3
106.7 3.5 106.4 16 105.7 105.7 6.1 106.8
46.0
0.7
10
12.4044
0.4
+ 7.2
5.9
1 2.0
2 9.1
10.7 25.7 9.0
1 12.3 45.3 = 123
46.0 16.9
0.7 0
0.0 0.0
0.0 0.0
1.0
42.8
52.8
70.0 6.7 71.4
10.7 29.0
77.4 18.4 83.6 + 20.5 90.8
36.
w=l =
0.3
7.11
28.9 28.9 9.1 29.5
16.9 16.9
0.3 1.8
0.0
0.0
0.0 10 0.0
46.0
36.5 1 3.2 36.5
45.3
w=0 0
42.7 02.7 0.4
51.6 51.65.55 5.5
26.4 16.4113 11.3
44.7 + 12.6
48.3 18.31155 15.9
0.7 2.
0.0 0.0
0.0
0.0
halfcheetah- wwlcchetaalmmtiinme medium-v222
hopper-mettie meder--- v2
walker2d aalker2d-mediume meedimm-122
halfcheetah-mme sethetthh-medinee cel--mediun medium--eppaae
repl----2-2222
hopper med-um-reepeaeeeeeeee replay-22
waller22--o (oferdddmddumm medium- replay-v2
halfchectah-ni www.aah--mmmmmmaya. mu---xper--v2
hopper- hopper-mediiee medium-experttee
walker2d-meet medi-medi----- expertv22
antmaze- untmaze-umaee uumaz--22222
diver--vvs---2
antmaze cattaaze-men werr---mdiiimmpyyeeee play--22
antmaaze meee---eiummmiii diverse-v2
antmaze-Ie lurge- play-v2
diverrse--2
Task Name
antmaze-umain wee---mazeeeise
antmaaze large- leeee-lar----eeeeee
22

---

s - 100
39.7 3 1.0
5.9 50.9 + 10.4
81.4 3.1 54.9 12.6
1544
79.5 4.0 6.0
30.9 26.4 5.4 6.9
41.7 3.2
50.1 # 10.5
71.1 1: 27.6
14.7 83.3 = 14.7
49.0 29.1
0.0 0.0 0.0
0.3 4 1.8
0.0
0.0 0.0
0.3
10
46.9
17.9 58.3
41.
83.6
38.3 + 15.9 99.0 10001104
10.6
8.9 8.9 54.0 14.0111 15.8 33.3 1 20.1
0.0
0.0 0.0
0.0
0.0
0.0
9 18. 109.7
17788 64.0
0.0
0.0
50
==
B
0.3
24.2 52.2 30.5
48.1
93.6
51.9
9.6 73.8
90.3 84 91.7 91.7
70.0 + 12.6 26.7 # 19.9
72.3 + 17.5 23.7 14.3
6.0
0.0
0.0
w=
53.7
71.
7 23.6 20
==0
53.1
4.9
16.
1
9.4
65.4
4.2
2.8
31.7
61.
90.0
37.7 18.2
Table 8: Average normalize score and standard deviation of hyper parameter ablation BCOL (SAC).
42.7
21.7
35.7
= 100
1.4
1. 7.8
1 1.8
442112
1.2
88.6
46.0 5.1
83.3
92.9 13.0
110.3 110. 109.7
1 13.1
18.2
0.0 0.0
0.0 0.0
0.0 0.0
0.0 1.0
48.2
61.9
83.2
44.
99.0 11.6 92.9
196 75.7
56.3
w
10
50.1 4
83.2 7.5
84.1 1.5
46.1 1.4 46.2
99.8
7.2 86.9 #5.7
85.3
143 60.7 14.8 14.8
0.0 0.0
0.0 0.0
0.0
0.0 0.0
=
85.3
10
=
B
0.4
8.0
8.0
80.9 7.8
9.7
12.9 12.9 74.4
17.9 68.1 1
91.7
19.3 7.7 123
9.7
0.0
0.0 1.00000
w w=
52.3
71.7
80.5
8.4 55.0
w= 0
0.8
52.6
63.8
17.0
33.9
34.7
45.6
665
66.9
79.7
16.2
227 42.0 43.1
91.0799
43
14.019.
14.0
4.0
7.6
0.0
0.0 0.0
96.1
100
49.7
0.2
9.9 9.8
1.8
9
8.0
5.7
5.7
84.6 6.5 85.6
5.5
12.0
109.5-27
72. :: 1388 05.3 20.0
54.7 + 16.7 62.0 14.5
0.3 1.8
0.0
0.0
0. 0.0
44.0
w =
73.9
84.6
88.2 15.2 90.6
109.5
10
0.2
8.0
2.2
13.812
1.2
918 7.2
79.4 5.5
110.
0.0 0.0
0.3 1.8
0.0 0.0
0.0
50.4
73.5
85.1
43.8
88.2
=
B
4.4
72.2 8.4
3.0
43.1
86.0
75.2 8.6 7
79.5 5.3 81.1 5.9
194
110.6 2.3 110.7
+ 16.2
: 16.9
0.0 0.0
0.0
0.0
0.0
0.0 10 0.0
w ==l
76.3 :: 18.8 70.2
84.5 2,5 84.9
1 1.8 43.1
73.0
54.0
50.7 + 0.3
73.3 3 8.6
91.9 9.7
31.0 34.2
38.0 1:001184 18.9
0.0 0.0 0.0
0.0 0.0 0.0
0.0 0.0
0.0
w=0 0
replay-v2
hopper medimm--epppeeeeeeeeeee replay-22
waller22d-o (oferddmmddumm medium- replay-v2
halfcheetah-mmee wwwettaa-mmmim----eee menddu---pppet-22
hopper-ment wwt-mm--eeeeeeeeeeeeeeeeee expeett--v2
walker2d-mee medlumeei---mm expertv22
antmaze-umain wae---maaeeeise diver--vss---2
antmaaz meee---eiuummimi diverse-v2
diverrse--2
halfcheetah- wichheeta--meiimmee medium-v2
antmaze catmaaze-men meediu--play-a
antmaze-Ia lurge- play-v2
hopper- meder--d v2
walker2d aalker2d-mediume m-ediumm1222
antmaze- untmaze-umaee uumaz--22
Task Name
halfcheetah-mme wehhetthh-medieee wwddmm-dddeeeeeeem
antiaaze large- weree-largg-eeeeeee
23


Reviews:
[{"review_id": "JFQJcfSXZw", "reviewer": "Reviewer_4X4f", "summary": "This paper designs an new algorithm for offline RL. It also conduct experiment to validate its algorithms.", "questions": "The performance of the proposed algorithm does not demonstrate substantial superiority compared to the Contrastive Q-Learning (CQL) algorithm. It would be beneficial to provide more persuasive evidence of the proposed algorithm's efficacy.\n", "limitations": "The use of the \"budget\" concept presents limitations, as it appears applicable primarily in tabular settings. Its adaptability to continuous cases is challenging, which curtails its generalizability. The authors may want to consider elucidating the feasibility of this concept in a broader context or proposing alternative approaches for continuous settings.", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The algorithm proposed is new. The intuition behind is presented clearly.", "weaknesses": "1. The performance of the proposed algorithm does not demonstrate substantial superiority compared to the Contrastive Q-Learning (CQL) algorithm. It would be beneficial to provide more persuasive evidence of the proposed algorithm's efficacy.\n\n2. The use of the \"budget\" concept presents limitations, as it appears applicable primarily in tabular settings. Its adaptability to continuous cases is challenging, which curtails its generalizability. The authors may want to consider elucidating the feasibility of this concept in a broader context or proposing alternative approaches for continuous settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback. We hope you will consider increasing your score after reading our responses. Please let us know if there are more questions.\n\n> The performance of the proposed algorithm does not demonstrate substantial superiority compared to the Contrastive Q-Learning (CQL) algorithm.\n\nContrary to what the reviewer suggested, our results are better than the reported results with CQL. In particular, for the total score in Mujoco control tasks (746 vs 713) and, more importantly, the harder AntMaze tasks (396 vs 352.9), our algorithm BCOL surpasses the CQL performance by a large margin. (Please see the discussion about why AntMaze are more meaningful benchmark in our response to Reviewer bnLz about “comparison with CQL and CDC”.) We consider it a substantial improvement given the merits of CQL and how well it is performed in Mujoco tasks (e.g. a contemporaneous work [1] summarized the comparison between CQL and several more recent offline RL algorithms).\n\nWe would like to bring the reviewer's attention to two observations about how CQL results are mainly reported in related works. In short, we tried our best to be as fair as possible to CQL, which is different from the way CQL is reported in literature. Thus the improvement on top of CQL does not look as large as in prior work.  \n1. CQL website [2] keeps updating its performance and the performance has been significantly improved after the publication of the paper, especially after the D4RL Mujoco-v2 environments. In the meantime, many prior works [3,4,5,6] only compare to the results (in Mujoco-v0) in the original CQL paper. That explains why the gap with CQL in these papers is so large.\n2. In our submission, we reported CQL’s results on AntMaze environments from the CQL paper. However, many recent papers [1,7,8,9] compare proposed methods against their reproduced CQL results, not the original numbers in CQL. Their reproduced CQL results are often worse than the original numbers in CQL. E.g. offline results of CQL in Table 2, and Appendix C in [7] say “Our reproduced results offline are worse than the reported results, particularly on medium and large AntMaze environments.” Table 1 in [1], Table 1 in [8] and Footnote 4 on Page 8 in [9] report the results similarly. That explains why the gap with CQL in other papers is very large.\n\n\n> The use of the \"budget\" concept presents limitations, as it appears applicable primarily in tabular settings \n\nWe believe there is a misunderstanding. We would like to clarify that we derive a practical Bellman operator designed for function approximation in Section 3.1. Our method works with continuous state and continuous action spaces, as evidenced by our comprehensive experiments with gym MuJoCo and AntMaze in the D4RL benchmark. Both of these environments have continuous state and action spaces. Therefore, we are unsure of what the term \"tabular\" refers to in our context. \n\nIf the review means the budget space is discrete, we would acknowledge this in our current method. However, this is not a fundamental limit for the general “budget” method. We can define a continuous distribution distance metrics as a budget variable, and as an input to the Q function $Q(s,a,b)$. The sum over $b$ in training loss can be replaced with sampling $b$ uniformly. We would like to emphasize that our method itself shows the effectiveness of budgeting ideas, and we left the continuous budgeting implementation as a future work. We will include more discussion about this in the Discussion section in an updated version.\n\n[1] Improving and Benchmarking Offline Reinforcement Learning Algorithms. Kang et al. 2023\n\n[2] https://sites.google.com/view/cql-offline-rl\n\n[3] Chen, Lili, et al. \"Decision transformer: Reinforcement learning via sequence modeling.\" Advances in neural information processing systems 34 (2021): 15084-15097.\n\n[4] Brandfonbrener, David, et al. \"Offline rl without off-policy evaluation.\" Advances in neural information processing systems 34 (2021): 4933-4946.\n\n[5] Cheng, Ching-An, et al. \"Adversarially trained actor critic for offline reinforcement learning.\" International Conference on Machine Learning. PMLR, 2022.\n\n[6] Bhardwaj, Mohak, et al. \"Adversarial model for offline reinforcement learning.\" arXiv preprint arXiv:2302.11048 (2023).\n\n[7] Kostrikov, Ilya, Ashvin Nair, and Sergey Levine. \"Offline reinforcement learning with implicit q-learning.\" arXiv preprint arXiv:2110.06169 (2021).\n\n[8] Fakoor, Rasool, et al. \"Continuous doubly constrained batch reinforcement learning.\" Advances in Neural Information Processing Systems 34 (2021): 11260-11273.\n\n[9] Fujimoto, Scott, and Shixiang Shane Gu. \"A minimalist approach to offline reinforcement learning.\" Advances in neural information processing systems 34 (2021): 20132-20145.\n"}, {"review_id": "5cBD9JmHOZ", "reviewer": "Reviewer_wQce", "summary": "This paper gives a total new solution to offline RL, instead of introduing pessimism by behavior constraint or value regularization, budgeting the number of counterfactual decisions, which naturally reduces overestimation. This paper also gives a good formulation of the problem and provides a nice solution to solve the allocation problem by dynamic programming. This paper shows good empirical performance over offline RL benchmarks and optimality of the fixed point solution as theoretical justification.", "questions": "- Why adding the gap_penalization term (Equation 9) is necessary? Does it still do some kind of behavior regularization implicitly? Could you elabotate more on the usage of Equation 9?\n- Why SAC+BCOL can work well on antmaze tasks while TD3+BCOL can't?", "limitations": "See Weakness.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper is very well written.\n- The idea of avoid distributional shift by budgetting the number of counterfactual decision making is very novel and promising.\n- This paper gives a nice formulation of the problem and provides a solution to solve the allocation problem by dynamic programming, which is also solid and theoretical justificated.\n- This paper shows strong performance on various D4RL datasets, though not SOTA in some datasets if compared to more recent works.", "weaknesses": "- This method introduces two hyperparameters to tune: $\\omega$ and $B$.\n- It is better to have some visualization on which critical steps does the algorithm tends to make counterfactual decisions and whether it has some physical meanings. For example, on antmaze tasks, which location does the algorithm make counterfactual decisions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed and positive feedback. Please see our responses to your comments and let us know if there are more questions. \n\n> This method introduces two hyperparameters to tune \n\nThat is a fair point. Our method does have two hyperparameters, but this does not diminish its efficacy or applicability as a practical offline RL algorithm. In fact, even without the hyperparameter $\\omega$ ($\\omega=0$), our method (BCOL with SAC implementation) shows strong performance (and sometime remains state-of-the-art) in Mujoco and AntMaze tasks, as shown in Figure 1 and Table 6 of our paper. However, the issue of hyperparameters is an open problem in offline RL, and other methods such as IQL, CQL, and CDC also have various hyperparameters.\n\n> More visualization on critical steps and counterfactual decisions \n\nThat is a good suggestion, thank you. We will work on creating visualizations for the camera-ready version of our paper to show which critical steps the algorithm tends to make counterfactual decisions in AntMaze tasks. \n\n>Why adding the gap_penalization term (Equation 9) is necessary:\n\nThat is a good question. $Q(s,b+1,a) \\ge Q(s,b,a) $ is a property that must be satisfied for our desired function $Q$. However, such a property is not always held when we optimize the TD errors. Thus enforcing such constraints can be helpful to learn the desired Q function. One way to enforce it is to design neural network architectures outputting a monotonically increasing sequence of real numbers, as $Q(s, \\cdot, a)$, but it comes with additional complexity of the network structure and training. Thus we choose a simpler way of adding a regularization term to control the violence to property $Q(s,b+1,a) \\ge Q(s,b,a) $. Such a regularizer does not directly enforce a behavior constraint to the policy. Because this regularizer constraints Q gaps for the same action with different budgets, instead of gaps between different actions. But this regularizer will help the dynamics programming over counterfactual actions finding a self-consistent solution. We will include more discussion about this in the corresponding section of the updated version. \n\n>Why SAC+BCOL can work well on AntMaze tasks while TD3+BCOL can't?\n\nThat is a great question. We believe this is an issue with TD3 algorithm and offline RL based on TD3 in general. For example, the original TD3+BC paper does not report on AntMaze tasks. Our reproduced results show TD3+BC perform poorly on AntMaze. Similarly, prior work [1] observed BCQ (another TD3 based offline RL algorithm) perform poorly either on AntMaze. In the meantime, we noticed there are fewer TD3 based offline RL algorithms, in comparison with SAC based architecture. Thus our hypothesis is that TD3 based algorithms face larger challenges in offline RL especially in tasks like AntMaze where the offline algorithm needs to find a policy that stitches together multiple policies to find the optimal policy.\n\n[1] Fakoor, Rasool, et al. \"Continuous doubly constrained batch reinforcement learning.\" Advances in Neural Information Processing Systems 34 (2021): 11260-11273."}, {"review_id": "1ydyVj25yc", "reviewer": "Reviewer_FoC4", "summary": "This paper proposes a TD approach to induce counterfactual decisioning in offline RL agents. Basically, the approach suggests using a count-based budget that gives scope for making decisions that are not exhibited by the behaviour policy. The paper implements this approach in various standard benchmarks and shows that the method performs favourably to the state-of-the-art approaches. They also give a theoretical justification for why budgeting should work.", "questions": "I have listed my questions and corresponding suggestions in the above section.\n\nOverall, I like the paper's straightforward approach to increasing counterfactuals in offline RL and enjoyed reading the derivations provided. However, the paper has many unaddressed weaknesses, as pointed out above. I am inclined towards borderline rejecting the work in its current form. With my questions answered, I would love to increase my rating for this work.", "limitations": "Authors do point out the limitations around using count-based approach for budgeting and theoretical analysis of less number of counterfactuals. In addition, I urge authors to also talk about the safety considerations involved in BCOL.\n", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The strength of the paper lies in the simplicity of the proposed budgeting for counterfactual decision making. The paper identifies the gap in the existing approaches which fall short of inducing counterfactuals. The authors propose to put a hard budget on the number of counterfactual decisions taken by the agent. Building upon the constrained optimization formulation for solving the MDP, the work shows how it converges to a fixed point and argues about the optimality of the same. In addition, it gives a function approximation version for using deep learning based approaches in conjugation.", "weaknesses": "I find the following weaknesses in the approach:\n\n**Tuning of the B**: It is unclear how the budget would be tuned for different environments. Because having a high B implies applying off-the-shelf RL algorithms without considering the nuances of offline RL setting, and a low B will make the method similar to imitation learning techniques.\n\n**Lack of Imitation Learning baseline**: It would be helpful to see how the proposed BCOL compares with state-of-the-art approaches for imitation learning. This will highlight the importance of counterfactual decision-making learnt from offline dataset. I suggest authors include a competitive baseline for IL, too.\n\n**Comment on the novelty of budgeting**: Safe RL approaches have count-based constraints on agent's violations of safety. Therefore, the theory given in the present work has a very high overlap with the prior works, reducing the novelty in the optimization or the fixed point derived.\n\n**Motivation behind the counterfactual budgeting**: I am myself not aware of regularized techniques for increasing counterfactuals in offline RL, however from first principles, it seems that inducing counterfactuals might not be a good idea as offline RL might lead to safety issues when the agent is deployed. The aim of offline RL (at least to me) is finding connections in the offline data that can help improve the returns gained by taking care that the agent doesn't infer unrealistic novel behaviours from offline data that could turn harmful once deployed in the actual environment. Can authors please comment on the safety of the BCOL agents?\n\n**Why would such budgeting work**: From equations 4 and Select() given on line 185, it looks like BCOL would spend the budget in the initial steps of decision-making, and after that, it would follow the behaviour policy. Put in a different way, BCOL's budgeting seems myopic in nature; it is unclear to me how the algorithm will induce the agent to use the budget later in the decision-making when it matters the most. Can authors please comment on how the budget would be used pragmatically by the agent? It would help to see an empirical analysis of the budget expenditure against the time steps.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback. We hope you will consider increasing your score after reading our responses. Please let us know if there are more questions.\n\n> “Tuning of the B”  \n\nWe agree with the reviewer on the algorithm’s behavior with high and low B. Such a behavior (spanning over the spectrum from imitation learning to vanilla RL, with different values of hyper-parameters) is a common and desired behavior for offline RL algorithms. We would like to restate that we use a single value of B across all Mujoco tasks, and another value of B across all AntMaze tasks. This shows the algorithm is less sensitive to hyperparameters than many offline RL algorithms that tune the hyperparameters per task. It also shows that it’s feasible to tune B over one simpler validation task that shares similar properties with the target task. We also want to remark compared with the coefficient of the pessimism/uncertainty regularization in other algorithms, B has a clear physical meaning and it is more intuitive, and is easier to be set by prior knowledge about the environment.\n\n> “Lack of Imitation Learning baseline” \n\nAs the reviewer requested, we will include more imitation learning baselines in the paper. We include the results from BC on top 10% data, advantage weighted regression, and the decision transformer here, as the common imitation learning baselines in offline RL work. Please see the results in the rebuttal to all reviewers.\n\n> “the novelty of budgeting” \n\nWe thank the reviewer for pointing out the connection between our work and count-based constraints violation in safe online RL. However, there is a key difference between our setting and safe online RL: in online RL, the agent can interact with the environment to collect more data, while in offline RL, only limited data is available and exploration is not an option. This difference has a significant impact on the way that we design methods for offline RL. Additionally, while count-based constraints violation has been studied in safe RL, to the best of our knowledge, our approach of using budget for counterfactual decisions in the context of offline RL is novel (also reviewer wQce pointed that out). Specifically, our proposed algorithmic idea of using dynamic programming to solve a constrained optimization problem in an offline setting has not been proposed before. We will include a discussion of the connections with safe RL in our paper. If the reviewer can provide a more specific reference that raises concerns about the novelty of our algorithm, we would be happy to include a more detailed discussion in the paper.\n\n\n> “Motivation behind the counterfactual budgeting” \n\nWe would like to clarify that our algorithm is not “regularized techniques for increasing counterfactuals in offline RL”. In contrast, our algorithm can be viewed as *decreasing counterfactuals in offline RL*, as most offline RL methods do, but we are doing it in a simpler, more direct and explainable way. In short, RL methods are intrinsically learning over counterfactuals, and our budgeting idea is to upper bound it. \n\nMore specifically, vanilla Q learning (or any off-the-shelf online RL method based on that), without any constraints, will update the Q values from the max Q values in the next time step:\n$$ Q(s,a) \\leftarrow r(s,a) + \\gamma \\max_{a’} Q(s’,a’) $$\nThis backup will implicitly update the greedy policy with a counterfactual action $\\arg\\max_{a’} Q(s’,a’)$ unless the argmax is behavior action. Online RL algorithms such as SAC, DDPG, TD3 are all based on learning these counterfactual actions. In the offline settings, a popular approach is adding a regularization term on top of the Bellman error on Q function, such as:\n$$ - (\\max_{a’} Q(s,a’) - Q(s,a)) $$ \nWhere a is from the behavior policy. Such a regularization term prevents the Q values for counterfactual actions from being too large, thus it decreases the number of counterfactuals in the resulting Q function and its greedy policies. However, unlike our method, this type of popular regularized method cannot yield a test-time absolute upper bound of counterfactuals. In the sense of counterfactual decisions, our method is safer than other offline RL methods, except behavior cloning which follows behavior policy (factual decisions) anytime. \n\nPlease see Section 3 in our paper for more discussion.\n\n> “Why would such budgeting work” \n\nWe emphasize that our algorithm does not spend the budget in the initial steps of decision-making. The reason is that we use dynamic programming to solve the trade-off between spending the budget now to get a better action and keeping the budget for the future to get a better future value. Mathematically, it is described by the trade off between $\\max_{a’} Q(s’, b-1, a’)$ and $E_{a’ \\sim \\mu} Q(s’,b,a’)$ in Eq (4) in our paper. The maximization in the first term represents the immediate benefit of greedy counterfactual actions, but the higher budget and thus higher Q values explains the benefit of taking a factual action for now to achieve a higher value from future counterfactual actions. Always taking the greedy at the beginning is myopic and not the optimal solution to this dynamic programming problem. \n\nIn fact, our experiment shows there are still budgets left at the end of the episode during the test. We attached the total BCOL spent in the test, averaged over seeds and last 10 steps, for AntMaze tasks. \nTask | Average budget spent in test (max budget is 50) | Standard deviation of budget spent\n---|---|---\nAntmaze-umaze-v0 | 41.00 | 0.91\nAntmaze-umaze-diverse-v0  | 20.02 |  6.44\nAntmaze-medium-play-v0 | 45.94 | 1.13\nAntmaze-medium-diverse-v0 | 46.04 |  0.92\nAntmaze-large-play-v0 | 43.33 | 1.81\nAntmaze-large-diverse-v0 | 43.75 | 1.77\n\nThis result shows budget spent is less than 50, and there are budget left at the end of episodes. Thus the algorithm will not be forced to take behavior actions. In contrast, it will plan on how to spend the budget by line 1 in Select(). "}, {"review_id": "sBPGMoloFR", "reviewer": "Reviewer_bnLz", "summary": "This paper proposes a novel offline RL algorithm BCOL that builds on the idea of limiting the numbers of counterfactual decisions. Instead of enforcing policy or value regularization, BCOL follows the decisions of the behavioral policy in the majority of the states, and only makes counterfactual decisions for a limited number of times. The strategy to spend the fixed amount of opportunities for counterfactual decisions is learned through a dynamic programming algorithm. Experimental results using two different implementations of the proposed algorithm on a wide range of offline RL tasks are presented. ", "questions": "Is the algorithm still working if there is more than one behavior policy for collecting the offline data? \n\nThe sentence “Thus, the choice of policy on each step needs to balance between the Q value gain from the current step and the potential benefit from future counterfactual decisions.” is very confusing. Could the authors please elaborate more on it? \n\nWhat is the formal definition of Q(s,b,a) in Eq. (4)? It looks like the exact definition of Q(s,b,a) doesn’t matter, and the only thing that matters is the fixed point of $\\mathcal{T}_{CB}$. \n\nI understand that by definition of $\\mathcal{T}_{CB}$ that are at most B backup steps taking the max operation as the maximum value. But why does this “intuitively upper bounds the amount of extrapolation that the Q function can take”?", "limitations": "Yes. The authors have discussed the limitations of this work and pointed out future directions for continuing research. ", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "It is great that the authors present all details about their experiments and the implementation details of their proposed algorithm. \n\nThe authors present two implementations based on different state-of-the-art RL algorithms (TD3 and SAC). \n\nThe presentation about the empirical results (figures, tables, etc.) is very clear. The ablation study on the hyper-parameters is great.    \n", "weaknesses": "### Method \nIt is well-known in the offline RL community that the extrapolation needs to be careful. Limiting the level of extrapolation itself is not challenging, one can copy the behavioral policy, leading to zero extrapolation. The challenging thing is to find out where to extrapolate. The authors have argued for many times in the paper that assigning an upper bound for the number of counterfactual decisions can effectively constrain the level of extrapolation, leading to a balance between the gain of counterfactual decisions and the risk of extrapolation. However, this only explains how the level of extrapolation is limited but not the more important question: why the proposed BCOL algorithm can learn where to extrapolate. \n\nOn the intuitive level, the explanation is not enough and not clear. On the formal side, it would be great if the authors can provide some theoretical guarantees for the proposed algorithm so that the benefits of BCOL become more clear. In the current version of the paper, I fail to see enough support, either intuitive or theoretical, for the efficacy of the proposed algorithm. \n\n### Algorithm \nThe proposed algorithm induces an extra burden for learning because the Q(s,b,a), unlike the regular state-action value function, needs to approximate the value of the budget well. This at least linearly increases the difficulty of the learning problem because for every b, Q(s,b,a) needs to be approximated well so that the proposed Counterfactual-Budgeting Bellman Operator can work well. \n\n### Experiments\nIt is appreciated that the details about the experiments are presented well. \n\nHowever, the performance of BCOL doesn’t seem that impressive given the increased training cost. For example,although the BCOL(SAC) has the highest total score, it is outperformed by CQL and CDC on about half of the tasks. \n\nMoreover, the baseline methods, as diverse as they are, are dated algorithms. It may be better to include more latest baselines that also report SOTA performance in their papers (see for example, [1] and [2]) .  \n\n[1] Bhardwaj, Mohak, et al. \"Adversarial model for offline reinforcement learning.\" arXiv preprint arXiv:2302.11048 (2023).\n[2] Kang, Bingyi, et al. \"Improving and Benchmarking Offline Reinforcement Learning Algorithms.\" arXiv preprint arXiv:2306.00972 (2023). \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback. We hope you will consider increasing your score after reading our responses. Please let us know if there are more questions\n\n> “why BCOL can learn where to extrapolate\" \n\n> “not enough intuitive or theoretical support”.\n\nIndeed, it is the very central question that motivates our algorithm design and we would like to clarify it and avoid misunderstanding about it. In short, the dynamic programming formula Eq (4) in BCOL is naturally derived from the constrained objective Eq (2), to find where to extrapolate under the budget. Formally, the fixed point of Eq (4) is guaranteed to be the optimal allocation of the counterfactual or extrapolation across the time steps, by Theorem 2. BCOL is the approximate dynamics programming algorithm based on Eq (4).\n \nHere we will give a more intuitive summarization of the formal discussion in paper. To solve the problem about “where to extrapolate”, we need to balance the trade-off between spending the budget now to get better action and keeping the budget for the future to get a better future value. With BCOL, we solve the trade-off by dynamic programming. Mathematically, it is described by the selection between $\\max_{a’} Q(s’,a’,b-1)$ and $E_{a’ \\sim \\mu} Q(s’,a’,b)$ in Eq (4) in our paper. Theorem 2 says, by using such a modified Bellman backup, the fixed point is guaranteed to be the constrained optimal value function. \n\nIn summary, we respectfully disagree that we do not have enough intuitive or theoretical support in the paper. We would like to refer to the description of our algorithm from reviewer wQce, who said that our algorithm is “a solution to solve the allocation problem by dynamic programming, which is also solid and theoretical justificated.” \n\n> “Computation cost”\n\nWe acknowledge that BCOL will increase the amount of computation compared with SAC and TD3. In our paper, we discussed a stochastic approximation to reduce the linear cost on $B$ at the end of section 3.3. We remark that the computation is often not the main bottleneck in offline RL settings where the data is fixed (as discussed in all previous offline RL papers). It is more important to consider how to use the available data and leverage more information. \n\n> “comparison with CQL and CDC” \n\nTo answer your concern, we would like to explain a bit on the difference between Mujoco and AntMaze tasks, and why we think our algorithm shows a significant improvement. Mujoco datasets in D4RL have a significant fraction of near-optimal trajectories, and are considered as easier tasks by prior work e.g. IQL and CDC. AntMaze tasks require stitching parts of suboptimal trajectories that travel between different states to find a path from the start to the goal of the maze, and are more challenging and meaningful. (More discussion in section 4 of paper.) Thus we argue that the improvements on the AntMaze tasks are more significant and valuable. On AntMaze tasks, we outperform CQL and CDC by a much larger margin.\n\nWe would like to refer to our response to Reviewer 4X4f for a remark on how CQL results are reported in literature.\n \n> “latest baselines” \n\nWe thank the reviewer for the pointers and we will include more baselines in an updated version. In the response to all reviewers, we compared against the two works pointed out by the reviewer, and baseline algorithms that these two works compared with. These baselines either do not report their performance, or are significantly worse than BCOL in the harder AntMaze tasks. \n\nAdditionally, it is important to note Kang, Bingyi, et al. was only available online after the NeurIPS submission deadline, so it was not possible for us to consider it.\n\n> “Is the algorithm still working if there is more than one behavior policy?” \n\nYes. Most D4RL datasets contain data from more than one behavior policy. Please see Page 5 in the D4RL paper for further details. Our results show that our method outperforms others on this benchmark. Also BCOL does not need to know the behavior policy as well, and it will fit a behavior policy as $\\mu$. \n\n> “the confusing sentence” \n\nWe will clarify that in the updated version, however, let us explain it here. That sentence means, if we are under the constraints that we can only take at most B counterfactual decisions, we face the problem of where to allocate them. We expect the counterfactual decisions to provide improvement on top of behavior values, but taking too many of them can result in extrapolations. At some time-step, We can take counterfactual decisions immediately, and enjoy the value improvement. We can also take action from behavior policy, and this will save the budget for the future and maybe we can use it in some more influential decisions. \n\n>  “formal definition of Q in Eq 4” \n\nEquation 4 is the definition of an operator for any Q function in the function space. Thus $Q(s,b,a)$ can be any function in the function space $\\mathcal{S} \\times [B] \\times \\mathcal{A} \\to \\mathbb{R} $.\n\n> “why does this intuitively upper bounds the amount of extrapolation” \n\nGood question. The bounded extrapolation is from the comparison with standard Bellman update and RL methods based on it. In a standard Bellman backup we will update the Q function by:\n$$ Q(s,a) \\leftarrow r(s,a) + \\gamma \\max_{a’} Q(s’,a’) $$ \nPractical algorithms fitting this goal keep updating the Q values with extrapolated Q value $ \\max_{a’} Q(s’,a’) $ recursively (unless the argmax happens to be behavior policy). Such a recursive update is either ended by the episode end or discounted by $\\gamma$. Thus there are $\\frac{1}{1-\\gamma}$ or $H$ (horizon) times $\\max$ operator effectively on the optimization path, and that many extrapolated queries to the Q functions. \n\nIn contrast, our algorithm queries at most $B$ times to the extrapolated Q values in one optimization path. As $B << \\frac{1}{1-\\gamma}$ and $H$, the amount of extrapolation with our Bellman backup is much smaller than with the standard Bellman operator."}]

Author rebuttal:
We thank the reviewers for their feedback that we've used to greatly improve the paper. We have responded to the concerns of the reviewers as individual comments below. 
We are glad that the reviewers found that our method is simple and straightforward (FoC4), novel (4X4f, wQce, bnLz), and is a solid approach and theoretically justified method (wQce). They also said that the paper generally is well-written, an enjoyable reading, presents a clear message, and the intuition behind is presented clearly (4X4f, wQce, bnLz). They found our experiments to be clear and comprehensive (wQce, bnLz) with strong performance (wQce). 
We would like to clarify some key facts and summarize our main contributions here:

1) To the best of our knowledge, this is the first work to upper bound the number of counterfactual decisions in the context of offline RL. Such an objective is more straightforward, and explainable than many other offline RL methods.  

2) We propose a dynamic programming approach to optimize this objective. By planning on the benefit from different extrapolation steps, our algorithm will balance between taking immediate greedy action to Q values, and the potential benefit of taking a more advantaged action in the future at the opportunity cost. 
3) Theoretically, we prove our dynamic programming algorithm finds asymptotically the optimal way to allocate the budget of counterfactual, given an upper bound of counterfactual decisions. 
4) Through our comprehensive experimental results, we show that our algorithm outperforms most of the offline RL baselines. It verifies the effectiveness of bounding the counterfactual decisions in offline RL. 

We have included new experimental results in response to the reviewer comments from recent papers, and these results show that our method outperforms the latest SOTA by a large margin on the AntMaze task (one of the harder tasks in the D4RL benchmark) and is comparable to latest SOTA in Mujoco tasks. These latest results further strengthen our claims about the effectiveness of applicability of our method.

We attach the comparison against imitation learning baselines here. 

Task|10% BC| AWR | DT | BCOL
---|---|---|---|---
Mujoco total | 666.2 | 308.5 | 672.6 | 746.0
AntMaze total | 134.2 | 126.3 | 112.2 | 396.0

For more recent offline RL baselines, we compare BCOL against them here as well. 

Tasks | ARMOR| MoRel | MOPO | RAMBO | COMBO  | ATAC | MuZero | CRR+ | CQL+| BCOL
--- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- 
 Mujoco total | 788.6  |  656.5  |  379.3  |  753.1  |  738.3  |  792.4  |  140.2  |  703.9  |  717.9 | 746.0
AntMaze total | -  |  0  |  0   | 37.8  |  137.6 |   -  |  0  |  41.9  |  89 | 396.0

As these results show, BCOL outperforms IL baselines with large margins. BCOL is comparable to the latest SOTA in Mujoco tasks, and is substantially better in AntMaze tasks. For simplicity, we only list the total score here for all new results in rebuttal, but we will include the full results in the paper.


Meta-review: All the reviewers agree that the paper contributes a novel, interesting and effective solution approach to the well-motivated problem of offline reinforcement learning. The authors included additional experiments (comparison to imitation learning baselines, another gridworld domain, and analysis of budget consumption) that substantially strengthen the paper and should be included in the revision.
Decision: Accept (poster)