Title: Budgeting Counterfactual for Offline RL
Venue: NeurIPS 2023
Accepted: yes
Keywords: reinforcement learning

offline reinforcement learning

counterfactual reasoning
Abstract: The main challenge of offline reinforcement learning, where data is limited, arises from a sequence of counterfactual reasoning dilemmas within the realm of potential actions: What if we were to choose a different course of action? These circumstances frequently give rise to extrapolation errors, which tend to accumulate exponentially with the problem horizon. Hence, it becomes crucial to acknowledge that not all decision steps are equally important to the final outcome, and to budget the number of counterfactual decisions a policy make in order to control the extrapolation. Contrary to existing approaches that use regularization on either the policy or value function, we propose an approach to explicitly bound the amount of out-of-distribution actions during training. Specifically, our method utilizes dynamic programming to decide where to extrapolate and where not to, with an upper bound on the decisions different from behavior policy. It balances between the potential for improvement from taking out-of-distribution actions and the risk of making errors due to extrapolation. Theoretically, we justify our method by the constrained optimality of the fixed point solution to our $Q$ updating rules. Empirically, we show that the overall performance of our method is better than the state-of-the-art offline RL methods on tasks in the widely-used D4RL benchmarks.

Budgeting Counterfactual for Offline RL

Abstract

The main challenge of offline reinforcement learning, where data is limited, arises from a sequence of counterfactual reasoning dilemmas within the realm of potential actions: What if we were to choose a different course of action? These circumstances frequently give rise to extrapolation errors, which tend to accumulate exponentially with the problem horizon. Hence, it becomes crucial to acknowledge that not all decision steps are equally important to the final outcome, and to budget the number of counterfactual decisions a policy make in order to control the extrapolation. Contrary to existing approaches that use regularization on either the policy or value function, we propose an approach to explicitly bound the amount of out-of-distribution actions during training. Specifically, our method utilizes dynamic programming to decide where to extrapolate and where not to, with an upper bound on the decisions different from behavior policy. It balances between the potential for improvement from taking out-of-distribution actions and the risk of making errors due to extrapolation. Theoretically, we justify our method by the constrained optimality of the fixed point solution to our Q updating rules. Empirically, we show that the overall performance of our method is better than the state-of-the-art offline RL methods on tasks in the widely-used D4RL benchmarks.

1 Introduction

One of the primary hurdles in reinforcement learning (RL), or online RL, is its reliance on interacting with an environment in order to learn [47]. This can be a significant barrier to applying RL to realworld problems, where it may not be feasible or safe to interact with the environment directly [48]. In contrast, batch or offline RL [6, 27, 30] provides a more suitable framework for effectively learning policies by leveraging previously collected data to learn a policy. Notably, offline RL relies on a fixed but limited dataset comprising previously collected data from an unknown policy(s) and lacks the ability to continue interacting with the environment to gather additional samples.

These limitations in offline RL give rise to various challenges, with one notable issue being the occurrence of extrapolation problems resulting from the scarcity of training data [8, 12]. In order to overcome this challenge, previous approaches to offline RL primarily focus on constraining the gap between the behavioral policy and the learned policy [4, 11, 12, 25, 40, 51], or limiting the disparity between the state-action values of logged actions in the data and extrapolated actions [8, 22, 26, 50, 53]. Alternatively, other approaches utilize a learned model that avoids making predictions outside the distribution of the dataset [21, 35, 55]. All these offline RL methods involve a sequence of counterfactual reasoning problems within the realm of potential actions, explicitly or implicitly. The question of 'what if' we were to choose a different course of action than the behavior policy often arises, leading to extrapolation errors. Hence the difficulty of the problem increases as we plan for a longer horizon and involves more counterfactual decisions. For offline RL algorithms, it is difficult to find a balance between the potential for improvement from taking out-of-distribution actions and the risk of making errors due to extrapolation.

A critical aspect of offline RL which is relatively unexplored pertains to the varying importance of different actions at each step in determining the final outcome. Not all actions are created equal, and some have more influence on the outcome than others. Thus, it is not always essential to consider alternative actions or counterfactuals at every step than the one dictated by the behavior policy. Rather, the emphasis should be placed on the actions that wield the most impact, warranting careful decision-making at those steps. This limits the number of counterfactual decisions we make and assigns them to the most needed states/steps, to increase the return on the 'extrapolation investment'.

Unlike existing approaches that employ explicit or implicit regularization on either the policy, value functions, or both, we introduce a novel and effective algorithm, called Budgeting Counterfactual for Offline RL (BCOL), that explicitly constrains the level of extrapolation during training. Specifically, our approach leverages dynamic programming to determine when and where extrapolation should occur, with a budget on the deviation from the behavior policy. This enables us to strike a balance between the potential for improvement achieved by exploring out-of-distribution actions and the inherent risks associated with extrapolation errors. Conceptually, such a method is different from regularized offline RL by allowing non-uniform constraints while keeping strict bounds on counterfactual decisions.

Our contribution. We propose a novel algorithm, BCOL, making only a few but important counterfactual decisions in offline RL. This idea extends the space of current offline RL methods that extensively focus on regularized methods. We demonstrate that the fixed point resulting from our Q -value updating rule corresponds to the optimal Q -value function, under the constraints that the policy deviates from the behavior policy within a budget. We conduct thorough evaluations on a diverse array of tasks from the widely-utilized D4RL benchmarks [9]. In terms of overall performance, our approach exhibited a favorable comparison to existing state-of-the-art methods.

2 Problem Settings

We study the RL problem in the context of the infinite horizon, discounted Markov Decision Process (MDP) [41]. An MDP is defined as a tuple M = < S , A , r, P 0 , P, γ > where S is a state space, A is an action space, and both of these spaces can be infinite or continuous in nature. r : S × A → R is the reward function. P 0 is a distribution over S that the initial states are drawn from. P : S × A → ∆( S ) maps a state-action pair to a distribution over state space. γ is the discount factor over future reward. The goal in MDP is to maximize the expectation of discounted future reward v π = E [ ∑ ∞ t =0 γ t r t | a t ∼ π ] . An important function in MDP is the state-action value function, known as the Q function, which represents the expected future reward, given an initial state or state-action pair. Q π ( s, a ) := E [ ∑ ∞ t =0 γ t r t | s 0 = s, a 0 = a ] .

The primary emphasis of this paper is on model-free learning algorithms within the context of offline RL. The goal is to learn a target policy π that maximizes the expected future reward using a fixed dataset D . The dataset D consists of transitions ( s, a, r, s ′ , a ′ ) where s can be drawn from any fixed distribution, a ∼ µ ( ·| s ) , r = r ( s, a ) , s ′ ∼ P ( ·| s, a ) , a ′ ∼ µ ( ·| s ′ ) . Here µ denotes an unknown behavior policy that was utilized in the past to collect dataset D .

In offline RL, the task of learning an optimal policy poses a greater challenge compared to online RL. This is primarily due to the fact that the target policy can be different from the behavior policy, leading to the introduction of out-of-distribution actions. This challenge is further amplified by the use of function approximation and the problem horizon as the extrapolation errors from fitted Q function tend to accumulate over time-steps [8, 12]. In particular, it will results in the Q ( s, a ) values increasing dramatically for out-of-distribution state and action pairs. Consequently, this results in the learning of a policy that is likely to perform poorly and risky at the deployment time. Many previous works on offline RL aim to address these issues by some form of regularization, which might hurt the flexibility of making certain key decisions at each step (which we will return to later). However, we take a different approach in this paper where we emphasize on cautiousness of making counterfactual decisions and, hence propose a method that only deviates from behavior policy for a few steps.

3 Method

As mentioned earlier, prior studies in offline RL mainly address the issue of extrapolation through two types of regularization terms: policy regularization or value regularization. Policy regularization

typically involves utilizing a divergence metric between current candidate policy and behavior policy while value regularization is commonly defined as the positive difference between Q ( s, ˆ a ) 1 and Q ( s, a ) , where a ∼ µ ( ·| s ) . In these works, the regularization terms were uniformly applied to all samples in the dataset, enforcing a flat penalization on the distance between π ( ·| s ) and µ ( ·| s ) , or flat penalization of overestimating Q ( s, ˆ a ) for all s . The utilization of a uniform regularization approach can present a problem. When strong regularization is enforced, the divergence bound from behavior policy over all states are small, hence the policy may not be able to improve significantly. In such cases, the resulting policy can only be as good as the behavior policy itself. Conversely, due to the fact that trajectory distribution mismatch increase exponentially with respect to the problem horizon [8, 12, 34, 36], if the bound is not small enough, it causes significant issues such as large extrapolation errors that lead to a risky and poor-performing policy. Therefore, applying regularization uniformly without considering the specific characteristics of each state can pose challenges and limitations in offline RL.

This work emphasizes the importance of making counterfactual decisions , i.e., decisions that are different from the decisions that would have been made by the behavior policy. Building upon the notion that not all decision steps carry equal importance, we introduce a novel offline RL algorithm that incorporates only a limited number of counterfactual decisions. In other words, the algorithm follows the behavior policy µ most of the time, but it only makes counterfactual decisions (i.e. performing offline RL) in certain states. Thus intuitively it maximizes the potential improvement from counterfactual decisions while keeping the extrapolation error under control 2 . We refer to this number by the budget of counterfactual, denoted as B . The grid world MDP in Figure 1 shows a simple example of why using a budget of counterfactuals is an effective constraint.

Here, the goal is to find the shortest path from the start to the goal state without passing through the fail state, given a dataset of previously recorded trajectories from different and unknown behavior policies. There are two types of trajectories in the dataset, marked as yellow and blue in Figure 1. While yellow trajectories are the most common, the shortest path requires stitching together these two types of trajectories and taking a counterfactual decision (shown by dashed green line). A budget of counterfactuals ensures that the agent only takes the most beneficial counterfactual decision and prevents it from taking unnecessary counterfactual actions.

In order to spend the budget of counterfactual strategically, a dynamic programming algorithm is utilized to plan and asses the potential policy improvement resulting from actions taken outside of the data distribution. Thus, the choice of policy on each step needs to balance between the immediate gain from taking greedy actions in the current step and the benefit from future counterfactual decisions. Naturally, the policy also depends on how many counterfactual decisions it can take before it exceeds the upper bound B . We define the budget of counterfactual decisions at a time step t as follows:

̸

b t +1 = b t -✶ { π ( ·| s t , b t ) = µ ( ·| s t ) } , b 0 = B, (1) where ✶ is an indicator function. The policies studied in this paper take the current budget b t as an input, which b t is the initial budget B subtracted by the number of counterfactual steps taken before time step t as shown in (1). Given that, the goal of our method is to solve the following constrained policy optimization:

E M ( π ) is the support set of trajectory distribution introduced by the MDP and policy π . Without additional statements, later we only consider state-action-budget trajectories in E M ( π ) and drop this

requirement for the ease of notation. Note that b t in (2) is a function of the initial budget b 0 = B , policies, and states before step t . To provide more clarity, we expand the constraint in (2) as follows:

̸

In order to develop our offline RL algorithm that maximizes (2), we introduce a new Bellman operator that plans on backup future values as well as the number of counterfactual decisions.

Definition 1 (Counterfactual-Budgeting Bellman Operator).

The Bellman operator T CB updates the Q values by taking the maximum value between two terms. The first term refers to the case where a counterfactual decision is made by maximizing Q values over all possible actions in the next state. This leads to a decrease in the counterfactual budget. The second term involves following the behavior policy in the next state (similar to SARSA [43]) while keeping the counterfactual budget intact. By selecting the optimal backup from the two cases, the Q value for a given budget b strikes a balance between the benefits of counterfactual decisions (i.e. max a ′ Q ( s ′ , b -1 , a ′ ) ) in the next step and in further future. Since b ≤ B , there are at most B backup steps taking the maximum max a ′ Q ( s ′ , b -1 , a ′ ) in the backup path. This intuitively upper bounds the amount of extrapolation that the Q function can take. We can recover the standard Bellman operator on state-action Q functions by considering b ≡ ∞ in Q ( s, b, a ) and the rule ∞-1 = ∞ . Since b = b -1 = ∞ , the first max operator in V Q always prefers the former term and gives us the standard Bellman operator.

It is straightforward to show the counterfactual-budgeting Bellman operator is a γ -contraction. Based on that, we prove the fixed point is the optimal Q function constrained by a counterfactual budget.

Theorem 2. There exists a unique fixed point of T CB, and it is 1

This theorem (proved in the appendix) indicates that the fixed point iteration of T CB will lead to the optimal value function with the upper bound of counterfactual decisions. Thus it motivates us to minimize a temporal difference (TD) error of the counterfactual-budgeting Bellman operator, towards the goal of only improving behavior policy over a limited number of but important decision steps.

3.1 Algorithm

In this section, we derive a practical offline algorithm for our approach when the counterfactualbudgeting Bellman operator (4) is estimated via function approximation. First, we need to replace the expectations in (4) by one-sample estimation to evaluate the Bellman backup with the fixed dataset D . Next, considering that the action space is continuous, we approximate the max a operator in (4) by maximizing over actions sampled from a policy network that is trained to maximize the current Q function. Note that these design choices are commonly employed in previous offline RL algorithms [8, 12, 25, 26]. As a result, we obtain a sampled-based counterfactual-budgeting Bellman operator denoted as ̂ T CB.

Definition 3 (Approximate Counterfactual-Budgeting Bellman Operator).

where ( s ′ , a ′ ) sampled from D and { a k } m k =1 are sampled from π . Although this operator requires additional input b to the Q functions compared with the standard Bellman operator, it does not add any extra data requirement. We can augment the ( s, a, s ′ , a ′ ) tuple from an offline dataset with all 0 ≤ b ≤ B where B is the maximum number of counterfactual decisions that we consider.

To learn the Q function, we minimize the least square TD error introduced by ̂ T CB, using dataset D consisting of ( s, a, s ′ , a ′ ) sampled from behavior policy µ . Subsequently, we update the policy by making it more likely to choose actions with higher Q-values. The objective functions for the Q function and the policy π are formally defined as follows, with θ representing the parameters of the Q function and ϕ representing the parameters of the policy networks:

where ¯ θ denotes delayed target network. Algorithm 1 describes the training process of our method 1 . The counterfactual-budgeting Bellman operator has an interesting property that Q is monotonically increased with b , for any Q such that Q = ̂ T CB Q .

The proof is straightforward since Q with a larger budget maximizes the value over more action sequences. This property will be followed asymptotically. However, with a limited number of iterations, and function approximations, the gap is not always positive. Intuitively, enforcing the monotonic gap might reduce the search space and potentially accelerate the convergence. It is important to note that this modification does not alter the fixed point solution since Q ⋆ always has a monotonic gap. Such a regularizer does not enforce any behavior/pessimistic constraint to the policy. It only constrains the Q-values for the same action with different budgets, not the Q-values for different actions. This regularizer will help the dynamics programming over counterfactual actions finding a self-consistent solution.

To implement it, we introduce the following penalty term and add it to the L Q . This results in the revised form of L Q as follows:

This penalty term minimizes the gap for the actions sampled from π ϕ ( ·| s, b ) . As π ϕ ( ·| s, b ) maximizes Q θ ( s, b, · ) , which is supposed to be smaller than Q θ ( s, b +1 , · ) , it can be viewed as a more efficient sampler of actions, compared to uniformly sampling, in order to ensure the constraints in Equation 8.

Putting these together, we have Algorithm 1: Budgeting Counterfactual for Offline reinforcement Learning (BCOL). It uses an offline actor-critic style algorithm to learn the Q function and a policy maximizing the learned Q values. However, as the problem is to find the best policy under the constraints, greedy policy π ϕ itself is not enough for inference. The action selection method during inference is implicitly included in the definition of T CB. At test time, the policy needs to look ahead based on the current budget b t and the Q values of taking counterfactual actions v.s. taking the

behavior policy. Based on that decision, we updated the new budget as well. We define this step as an operator below.

The complete inference time procedure is described in Algorithm 2. It also takes the learned policy and Q function parameters as well as an approximate behavior policy ̂ µ . In case of unknown behavior policy, ̂ µ can be learned from behavior cloning or any other imitation learning algorithm. Algorithm 2 starts with an initial counterfactual budget B , takes action each time according to the condition in Select , and update the budget b t if the action is not drawn from ̂ µ .

3.2 Comparison to regularized and one-step offline RL

One of the most used methods in offline RL methods is adding policy or value regularization and constraints terms on top of a vanilla off-policy RL algorithm [5, 8, 12, 22, 23, 25, 26, 50, 51], referred to as regularized methods in this paper. Our method can be viewed as an alternative to using regularized losses. Instead, we enforce a non-Markov budget constraint on the policy class, and we argue this provides several unique advantages. First, compared with the coefficients of regularization terms, the budget parameter has a more clear physical explanation in most applications. Thus it is easier to tune the hyper-parameters and explain the decisions from the policy in practice. Second, the constraints on budget will never be violated in the test, it provides an additional level of safety. While regularization terms penalize the divergence during training, they cannot provide a guarantee on the divergence or distance between test policy and behavior policy.

Another type of offline RL that is related to our core idea is one-step RL [3, 49]. We propose and leverage the concept of limiting the number of counterfactual/off-policy steps. However, the 'step' here refers to the decision steps, and in one-step RL it refers to the (training) iteration step. Although one-step RL only applies only one training step, the resulting policy can still be far away from the behavior policy in many different states, and give different decisions during the test. From another perspective, our method, although with a limited number (or even one) of counterfactual decision steps, still applies dynamic programming to find the best allocation of counterfactual steps.

3.3 Important implementation details

Algorithm 1 provided a general actor-critic framework to optimize policy value constrained by counterfactual decisions. To implement a practical offline deep RL algorithm, the first design choice is how we model the policy and choose the policy gradient estimator to ∇ ϕ L π . Algorithm 1 is compatible with any off-policy policy gradient methods. We implement Algorithm 1 using SAC-style [17] policy gradient for stochastic policy and TD3-style [13] policy gradient for deterministic policy, to provide a generic algorithm for both stochastic and deterministic policy models. As TD3 uses deterministic policy, the value of m is 1 in ̂ T CB and the actor loss becomes Q θ ( s, b, π ϕ ( s, b )) . In both cases, we implement target updates following SAC/TD3. We defer more details to the appendix.

Both SAC and TD3 are originally proposed for online, off-policy RL. To better fit into the offline setting, previous offline RL methods based on these algorithms equip them with several key adaptations or implementation tricks. To eliminate orthogonal factors and focus on the budgeting idea in algorithm comparison, we follow these adaptations and describe them below. For SAC, prior work use twin Q functions [26], and a linear combination of the two Q values [8]. Prior work [8, 24] also sample m actions from the actor and take the maximum Q values in backup, instead of a standard actor-critic backup, which is a trick firstly introduced in [12, 15]. The entropy term in SAC is dropped in [8] as this term is mostly for online exploration. Previous TD3-based work [11] normalizes the state features and Q losses in TD3. All these adaptations are commonly used in state-of-the-art offline RL and are mostly considered as minor implementation details rather than major algorithmic designs. We refer to SAC and TD3 with all these adaptations as offline SAC and offline TD3 in this paper and refer to the two families of algorithms SAC-style and TD3-style.

One-step RL[3] and IQL [23] for offline RL is built on top of estimating the behavior value Q µ by SARSA. Unlike SAC or TD3, it is not applicable to apply the counterfactual budgeting idea on top of SARSA, since it is a native on-policy method and does not consider counterfactual decisions. Thus we focus on the implementation of BCOL with SAC and TD3.

Task Name

BC

10%

BC

DT

RAMBO

ARMOR

IQL

Onestep

TD3

BCOL

CQL

CDC

BCOL

+BC

(TD3)

(SAC)

halfcheetah-m

42

.

6

42

.

5

42

.

6

77.6

54

.

2

47

.

4

55

.

6

48

.

4

45

.

0

46

.

1

62

.

5

50

.

1

hopper-m

52

.

9

56

.

9

67

.

6

92

.

8

101.4

66

.

3

83

.

3

59

.

4

85

.

8

64

.

6

84

.

9

83

.

2

walker2d-m

75

.

3

75

.

0

74

.

0

86

.

9

90.7

78

.

3

85

.

6

84

.

5

76

.

7

74

.

5

70

.

7

84

.

1

halfcheetah-mr

36

.

6

40

.

6

36

.

6

68.9

50

.

5

44

.

2

42

.

5

44

.

4

40

.

9

45

.

4

52

.

3

46

.

2

hopper-mr

18

.

1

75

.

9

82

.

7

96

.

6

97

.

1

94

.

7

71

.

0

50

.

1

83

.

4

92

.

3

87

.

4

99

.

8

walker2d-mr

26

.

0

62

.

5

66

.

6

85

.

0

85.6

73

.

9

71

.

6

80

.

2

49

.

7

83

.

7

87

.

8

86

.

0

halfcheetah-me

55

.

2

92

.

9

86

.

8

93.7

93

.

5

86

.

7

93

.

5

91

.

5

88

.

7

87

.

3

66

.

3

86

.

9

hopper-me

52

.

5

110

.

9

107

.

6

83

.

3

103.4

91

.

5

102

.

1

100

.

5

106

.

8

109

.

2

83

.

2

99

.

0

walker2d-me

107

.

5

109

.

0

108

.

1

68

.

3

112.2

109

.

6

110

.

9

110

.

1

108

.

5

109

.

9

103

.

9

110

.

9

mujoco total

466

.

7

666

.

2

672

.

6

753

.

1

788.6

692

.

4

716

.

0

669

.

2

685

.

6

713

.

0

699

.

0

746

.

0

antmaze-u

54

.

6

62

.

8

59

.

2

25

.

0

-

87

.

5

64

.

3

96

.

3

93

.

3

94

.

0

93

.

6

90

.

3

antmaze-u-d

45

.

6

50

.

2

53

.

0

16

.

4

-

62

.

2

60

.

7

71

.

7

68

.

0

47

.

3

57

.

3

90

.

0

antmaze-m-p

0

.

0

5

.

4

0

.

0

0

.

0

-

71

.

2

0

.

3

1

.

7

12

.

3

62

.

4

59

.

5

70

.

0

antmaze-m-d

0

.

0

9

.

8

0

.

0

0

.

0

-

70

.

0

0

.

0

0

.

3

14

.

0

74

.

3

64

.

6

72

.

3

antmaze-l-p

0

.

0

0

.

0

0

.

0

23

.

2

-

39

.

6

0

.

0

0

.

0

0

.

0

34

.

2

33

.

0

35

.

6

antmaze-l-d

0

.

0

6

.

0

0

.

0

2

.

4

-

47

.

5

0

.

0

0

.

3

0

.

0

40

.

7

25

.

3

37

.

6

antmaze total

100

.

2

134

.

2

112

.

2

67

.

0

-

378

.

0

125

.

3

171

.

3

187

.

7

352

.

9

333

.

5

396

.

0

Total

566

.

9

800

.

4

784

.

8

820

.

1

-

1070

.

4

841

.

3

840

.

2

873

.

3

1065

.

9

1032

.

5

1142

.

0

Looping over all budget values in the loss is not efficient in practice. We implement the Q function and policy with B output heads and tensorize the loop over b . One can further reduce the computation cost by sampling budget values instead of updating all B heads over every sample.

4 Experiments

We evaluate our BCOL algorithm against prior offline RL methods on the OpenAI gym MuJoCo tasks and AntMaze tasks in the D4RL benchmark [9]. We compare the SAC-style and TD3-style implementation of BCOL with state-of-the-art offline RL algorithms, studying the effectiveness of budgeting counterfactual in offline RL. Our experiment results also reveal that behavior cloning with only one strategic counterfactual decision still work surprisingly well on MuJoCo tasks. Finally, we study the value of our dynamic programming methods on where to spend the counterfactual budget.

Baselines. We compared BCOL with regularized methods (policy and/or value regularization) based on SAC and TD3: CQL [26], CDC [8], TD3+BC [11]. Besides regularized methods, we also compare with one-step RL method [3] and IQL [23] as other state-of-the-art model-free offline RL methods, RAMBO [42] and ARMOR [52] as state-of-the-art in model-based offline RL, and behavior cloning, behavior cloning on the top 10% data, and decision transformer as imitation learning baselines. This covers a set of strong offline RL baselines. We are interested in both studying the effectiveness of counterfactual budget ideas in contrast to the closest regularized methods and state-of-the-art offline RL methods. We present a set of representative and the most performant offline RL baselines in this section, and defers the comparison against more offline RL baselines results to the Appendix due to the limit of space.

Benchmark. We report the results of BCOL together with baselines on 9 OpenAI gym MuJoCo tasks and 6 AntMaze tasks in D4RL. The gym MuJoCo tasks consist of the v2 version of medium, medium-reply, and medium-expert datasets in halfcheetah, walker2d, and hopper. MuJoCo tasks generally prefer imitation-type algorithm, as it contains a large portion of near-optimal trajectories. The 6 AntMaze tasks are considered harder tasks for offline RL as they contain very few or no near-optimal trajectories, and many previous methods lack performance reports on these tasks. We exclude the random datasets in MuJoCo tasks as they are not less discriminating and none of the offline RL algorithms learns a meaningful policy there. We exclude expert datasets in MuJoCo tasks since they can be solved simply by behavior cloning and mismatches the consideration in most offline RL algorithm designs.

How we report the results. Due to the inconsistency in the version of D4RL in the literature, it is necessary to clarify the source of baseline results here. We retrieve the results of baseline methods from the original papers if applicable (IQL, CQL, one-step RL, DT, RAMBO, ARMOR). We report the Rev. KL Reg variant of one-step RL as it shows the best performance on MuJoCo tasks [3]. For AntMaze tasks we report one-step RL and decision transformer results from the IQL paper as they

BCOLSAC

BCOLTD3

BCOLSAC

BCOLTD3

400

700

total

600

total

300

Budget

uJoCot

500

AntMazet

200

1

10

Mu

400

100

50

300

0

1

10

100

0

1

10

100

0

0

1

10

100

0

1

10

100

omega

omega

omega

omega

are not reported in the original paper. We report the scores of behavior cloning and 10% behavior cloning. We report the scores of behavior cloning from the IQL paper. We report the scores of CQL from an updated version of CQL paper 1 for D4RL v2 environments. We report the score of TD3+BC and CDC from our implementation since the original results are for D4RL v0 environments. The v2 results are generally better than the original v0 scores in the original papers and are consistent with TD3+BC's v2 scores from the IQL paper.

Table 1 shows the results of baseline algorithms as well as two variants of BCOL (TD3-style and SAC-style) on 15 D4RL tasks. For the scores from our experiment (BCOL , CDC, TD3+BC), we report the test episodic reward averaged over 300 test episodes as 10 test episodes per evaluation, the last 10 evaluations, and 3 runs with different random seeds. All algorithms run with 1M steps following the literature. We defer details like learning curves and standard deviations in the appendix.

Main results. As Table 1 shows, BCOL-SAC outperforms prior offline RL methods for both MuJoCo and AntMaze total scores. BCOL never falls behind the best methods by a large margin in any task. This indicates the effectiveness and robustness of the counterfactual budget in different scenarios. Especially for the total score of AntMaze, the harder tasks, BCOL-SAC outperforms most prior approaches by a large margin except IQL. We also find for both TD3-style and SAC-style algorithms, BCOL outperform the regularized method based on the same architecture, ablating orthogonal factors other than the key idea about the counterfactual budget.

Hyper-parameters: B and ω . Our algorithm only adds two hyper-parameters on top of SAC/TD3: budget B and ω . We searched the value of B in { 1 , 10 , 50 } and the value of ω in { 0 , 1 , 10 , 100 } . We select one set of hyper-parameters for MuJoCo (SAC-style: B = 10 , ω = 10 , TD3-style: B = 50 , ω = 10 ) and one set for AntMaze (SAC-style and TD3-style: B = 50 , ω = 0 ) based on the overall performance. This provides a fair comparison with baseline methods as all of them select their hyper-parameters either as this, or per task. 2 Figure 2 shows how the total scores changes with different values of B and ω . Full results with scores on each task are in the appendix.

Figure 2 also shows another surprising finding from our experiments on the MuJoCo tasks. With the budget being only one B = 1 , BCOL (SAC) shows comparable performance to some recent offline RL work including CDC and IQL. With 10 steps out of 1000 steps, it is able to outperform most prior methods. This highlights how strategically adding a few important actions on top of behavior policy can provide strong performance in this benchmark. It also indicates that the MuJoCo tasks in D4RL, even the more mixed datasets, considerably prefer the imitation-type approach.

Ablation on how we realize the counterfactual budgeting constraints. The idea of budgeting counterfactual decisions naturally requires both the planning on budgeting in Algorithm 1 during the training and the planning on budgeting in Algorithm 2 during the testing. Ablation on the roles of budgeting itself and two planning parts is nonetheless helpful for us to understand how BCOL works. In Figure 3, we report the results of three ablation methods.

The first column shows the performance without budgeting constraints. This leads to offline TD3 or SAC algorithm with all implementation adaptations used by BCOL and others [11, 12, 26]. As we expected, vanilla offline RL without budgeting does not work well. The second column shows the performance with budgeting but without any planning on budgeting during either the training or the testing. This method realizes the budgeting by randomly assigning B decisions to the policy

No Budgeting (Offline SAC)

Budgeting w/o Planning

Budgeting w/o Test Planning

Percent Difference

100

50

0

-50

-100

hc-m

hop-m

W-M'

hc-mr

hop-mr

w-mr

hc-me

hop-me

w-me

am-u

am-u-d

hc-m

hop-m

W-M..

hc-mr

hop-mr

W-mr

hc-me

hop-me

w-me

n-we

am-u-d

hc-m

hc-mr

hop-mr

W-mr

hc-me

hop-me

w-me

am-u

am-u-d

No Budgeting (Offline TD3)

Budgeting w/o Planning

Budgeting w/o Test Planning

Percent Difference

100

50

0

09-

-100

hc-m

hop-m

W-M

hc-mr

hop-mr

w-mr

hc-me

hop-me

w-me

am-u

am-u-d

am-m-p

am-m-d

w-ou

w-doy

W-m

hc-mr

hop-mr

w-mr

hc-me

hop-me

w-me

am-u

am-u-d

am-m-p

am-m-d

hc-m

w-doy

w-m

hc-mr

hop-mr

w-mr

hc-me

hop-me

w-me

n-we

am-u-d

am-m-p

am-m-d

learned by offline SAC/TD3, and the rest to the estimated behavior policy. Randomly stitching the counterfactual policy and behavior policy fails to fully realize the benefit of budgeting, since both training and inference are not budget-aware. The experiment in the third column studies if it is sufficient to be budget-aware during training. This ablation randomly selects B actions from π trained by BCOL and the rest from the estimated behavior policy. The setting is closest to BCOL, but the lack of planning on how to spend the budget during the testing still hurts the performance. The results on SAC-style and TD3-style implementations are consistent. This ablation shows that the effectiveness of BCOL relies on the collaboration of all three parts of our core idea: budgeting, planning on budgeting during the training, and planning on budgeting during the testing.

5 Related Work

Most recently proposed offline reinforcement methods rely on some mechanism to force the learned counterfactual decision policy to stay close to the data support. One approach to this end is regularizing the policy by its divergence with data, either by parameterization [12, 15], constraints and projection [29, 32, 44], divergence regularization [10, 11, 25], or implicit regularization by weighted regression [10, 37, 39, 40, 50]. The other similar idea, which is often applied together, is regularizing the value estimate of policy in an actor-critic or Q learning architecture [5, 7, 8, 22, 26, 28, 33, 34, 51]. Similar regularization can also be realized by the ensemble of Q functions [1, 2, 14].

Some recent batch RL methods focus on more adaptive policy, similarly to this paper, but conditioning on history [16] or confidence level [19]. A similar idea to us of making a few key decisions in one trajectory was studied [18, 46], but they focus on stitching the logged trajectories in data.

In contrast to batch RL, imitation learning often does not involve dynamic programming on extrapolated actions. It has been used as a regularization in online RL [31, 38] and offline RL [11, 37]. Recently imitation learning methods using transformers and conditional training also achieves good performance on offline RL benchmarks [4, 56]. One-step RL [3] can also be viewed as an extension

of imitation learning with one-step policy extraction [49]. Although utilizing the behavior policy as well, Our key idea is different from these as we train a policy with awareness of behavior policy, rather than regularized by behavior cloning or imitation learning loss.

6 Discussions

The form of budget considered in this work is counting the different decision distributions. A natural alternative is divergences between π and µ . This provides a soft count of counterfactual decisions. However, it requires good calibration of both distributions and is more suitable to the discrete-action settings. We leave this as future work. On the theory side, a natural question about the value of budgeting counterfactuals is how the benefit of less counterfactual decisions is reflected in the theoretical properties of offline RL. We leave the investigation on this for future theoretical work.

In conclusion, this paper studies offline reinforcement learning which aims to learn a good counterfactual decision policy from a fixed dataset. We propose a novel idea of budgeting the number of counterfactual decisions and solving the allocation problem by dynamic programming. We provide strong empirical performance over offline RL benchmarks and optimality of the fixed point solution as theoretical justification.

References

Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. In International Conference on Machine Learning , 2020.

Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty-based offline reinforcement learning with diversified q-ensemble. Advances in neural information processing systems , 34:7436-7447, 2021.

David Brandfonbrener, Will Whitney, Rajesh Ranganath, and Joan Bruna. Offline rl without off-policy evaluation. Advances in neural information processing systems , 34:4933-4946, 2021.

Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling, 2021.

Ching-An Cheng, Tengyang Xie, Nan Jiang, and Alekh Agarwal. Adversarially trained actor critic for offline reinforcement learning. In International Conference on Machine Learning , pages 3852-3878. PMLR, 2022.

Damien Ernst, Pierre Geurts, and Louis Wehenkel. Tree-based batch mode reinforcement learning. Journal of Machine Learning Research , 6, 2005.

Rasool Fakoor, Pratik Chaudhari, and Alexander J. Smola. P3o: Policy-on policy-off policy optimization. In Proceedings of The 35th Uncertainty in Artificial Intelligence Conference , volume 115, pages 1017-1027, 2020.

Rasool Fakoor, Jonas W Mueller, Kavosh Asadi, Pratik Chaudhari, and Alexander J Smola. Continuous doubly constrained batch reinforcement learning. Advances in Neural Information Processing Systems , 34:11260-11273, 2021.

Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv:2004.07219 , 2020.

Yuwei Fu, Di Wu, and Benoit Boulet. A closer look at offline rl agents. Advances in Neural Information Processing Systems , 35:8591-8604, 2022.

Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems , 34:20132-20145, 2021.

Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning , pages 2052-2062, 2019.

Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In Proceedings of the 35th International Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research , pages 1587-1596. PMLR, 2018.

Kamyar Ghasemipour, Shixiang Shane Gu, and Ofir Nachum. Why so pessimistic? estimating uncertainties for offline rl through ensembles, and why their independence matters. Advances in Neural Information Processing Systems , 35:18267-18281, 2022.

Seyed Kamyar Seyed Ghasemipour, Dale Schuurmans, and Shixiang Shane Gu. Emaq: Expected-max q-learning operator for simple yet effective offline and online rl. arXiv:2007.11091 , 2021.

Dibya Ghosh, Anurag Ajay, Pulkit Agrawal, and Sergey Levine. Offline rl policies should be trained to be adaptive. In International Conference on Machine Learning , pages 7513-7530. PMLR, 2022.

Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. arXiv:1801.01290 , 2018.

Charles A Hepburn and Giovanni Montana. Model-based trajectory stitching for improved offline reinforcement learning. arXiv preprint arXiv:2211.11603 , 2022.

Joey Hong, Aviral Kumar, and Sergey Levine. Confidence-conditioned value functions for offline reinforcement learning. arXiv preprint arXiv:2212.04607 , 2022.

Bingyi Kang, Xiao Ma, Yirui Wang, Yang Yue, and Shuicheng Yan. Improving and benchmarking offline reinforcement learning algorithms. arXiv preprint arXiv:2306.00972 , 2023.

Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel : Model-based offline reinforcement learning. arXiv:2005.05951 , 2020.

Ilya Kostrikov, Rob Fergus, Jonathan Tompson, and Ofir Nachum. Offline reinforcement learning with fisher divergence critic regularization. In International Conference on Machine Learning , pages 5774-5783. PMLR, 2021.

Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:2110.06169 , 2021.

Aviral Kumar, Rishabh Agarwal, Xinyang Geng, George Tucker, and Sergey Levine. Offline qlearning on diverse multi-task data both scales and generalizes. arXiv preprint arXiv:2211.15144 , 2022.

Aviral Kumar, Justin Fu, George Tucker, and Sergey Levine. Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction. arXiv:1906.00949 , November 2019.

Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative Q-Learning for Offline Reinforcement Learning. arXiv:2006.04779 , June 2020.

Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning. In Marco Wiering and Martijn van Otterlo, editors, Reinforcement Learning: State-of-the-Art , pages 45-73. Springer, 2012.

Romain Laroche, Paul Trichelair, and Remi Tachet Des Combes. Safe policy improvement with baseline bootstrapping. In International conference on machine learning , pages 3652-3661. PMLR, 2019.

Hoang Le, Cameron Voloshin, and Yisong Yue. Batch policy learning under constraints. In Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pages 3703-3712. PMLR, 2019.

Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 , 2020.

Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In International Conference on Learning Representations , 2016.

Yao Liu, Yannis Flet-Berliac, and Emma Brunskill. Offline policy optimization with eligible actions. In Uncertainty in Artificial Intelligence , pages 1253-1263. PMLR, 2022.

Yao Liu, Adith Swaminathan, Alekh Agarwal, and Emma Brunskill. Off-policy policy gradient with stationary distribution correction. In Uncertainty in Artificial Intelligence , pages 11801190. PMLR, 2019.

Yao Liu, Adith Swaminathan, Alekh Agarwal, and Emma Brunskill. Provably good batch off-policy reinforcement learning without great exploration. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 1264-1274. Curran Associates, Inc., 2020.

Tatsuya Matsushima, Hiroki Furuta, Yutaka Matsuo, Ofir Nachum, and Shixiang Gu. Deployment-efficient reinforcement learning via model-based offline optimization. In International Conference on Learning Representations , 2021.

Rémi Munos and Csaba Szepesvári. Finite-time bounds for fitted value iteration. Journal of Machine Learning Research , 9(5), 2008.

Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359 , 2020.

Ashvin Nair, Bob McGrew, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Overcoming exploration in reinforcement learning with demonstrations. In 2018 IEEE international conference on robotics and automation (ICRA) , pages 6292-6299. IEEE, 2018.

Gerhard Neumann and Jan R Peters. Fitted Q-iteration by advantage weighted regression. In Advances in Neural Information Processing Systems , pages 1177-1184, 2009.

Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. CoRR , abs/1910.00177, 2019.

Martin L. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming . John Wiley & Sons, USA, 1st edition, 1994.

Marc Rigter, Bruno Lacerda, and Nick Hawes. Rambo-rl: Robust adversarial model-based offline reinforcement learning. Advances in neural information processing systems , 35:1608216097, 2022.

Gavin A Rummery and Mahesan Niranjan. On-line Q-learning using connectionist systems , volume 37. University of Cambridge, Department of Engineering Cambridge, UK, 1994.

Noveen Sachdeva, Yi Su, and Thorsten Joachims. Off-policy bandits with deficient support. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , pages 965-975, 2020.

Julian Schrittwieser, Thomas Hubert, Amol Mandhane, Mohammadamin Barekatain, Ioannis Antonoglou, and David Silver. Online and offline reinforcement learning by planning with a learned model. Advances in Neural Information Processing Systems , 34:27580-27591, 2021.

Scott Sussex, Omer Gottesman, Yao Liu, Susan Murphy, Emma Brunskill, and Finale DoshiVelez. Stitched trajectories for off-policy learning. ICML Workshop , 2018.

Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction . MIT press, 2018.

Matteo Turchetta, Felix Berkenkamp, and Andreas Krause. Safe exploration in finite markov decision processes with gaussian processes. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 29. Curran Associates, Inc., 2016.

Qing Wang, Jiechao Xiong, Lei Han, Han Liu, Tong Zhang, et al. Exponentially weighted imitation learning for batched historical data. Advances in Neural Information Processing Systems , 31, 2018.

Ziyu Wang, Alexander Novikov, Konrad Zolna, Jost Tobias Springenberg, Scott Reed, Bobak Shahriari, Noah Siegel, Josh Merel, Caglar Gulcehre, Nicolas Heess, and Nando de Freitas. Critic regularized regression. arXiv:2006.15134 , 2020.

Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv:1911.11361 , 2019.

Tengyang Xie, Mohak Bhardwaj, Nan Jiang, and Ching-An Cheng. Armor: A modelbased framework for improving arbitrary baseline policies with offline data. arXiv preprint arXiv:2211.04538 , 2022.

Haoran Xu, Li Jiang, Jianxiong Li, Zhuoran Yang, Zhaoran Wang, Victor Wai Kin Chan, and Xianyuan Zhan. Offline RL with no OOD actions: In-sample learning via implicit value regularization. In The Eleventh International Conference on Learning Representations , 2023.

Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offline model-based policy optimization. Advances in neural information processing systems , 34:28954-28967, 2021.

Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. arXiv:2005.13239 , 2020.

Qinqing Zheng, Amy Zhang, and Aditya Grover. Online decision transformer. In International Conference on Machine Learning , pages 27042-27059. PMLR, 2022.

A Proof of Theorem 2

Proof. We first prove that T CB is a γ -contraction with respect to ∥ · ∥ ∞ . For any function Q 1 , Q 2 in the space, and any s ∈ S , b > 0 :

The first line is from the definition of V Q . The second line follows from the fact that | max f ( x ) -max g ( x ) | ≤ max | f ( x ) -g ( x ) | . The rest follows from the definition of infinity norm between Q functions. For b = 0 , it is straightforward that | V Q 1 ( s, b ) -V Q 2 ( s, b ) | ≤ ∥ Q 1 -Q 2 ∥ ∞ as well. So ∥ V Q 1 -V Q 2 ∥ ∞ ≤ ∥ Q 1 -Q 2 ∥ ∞ Now we have that

We finished the proof of T CB is a γ -contraction operator. Thus there exists a unique fixed point of the operator T CB. Now we prove the second part of the theorem that Q ⋆ = T CB Q ⋆ .

First, we define

In order to show Q ⋆ = T CB Q ⋆ , we need to prove the following two equations:

To prove Equation [18], we write Q ⋆ ( s, b, a ) as:

The first line holds by the linearity of expectation. The second line follows from the definition of conditional expectation. The third line follows from the fact that given s, a , s ′ does not depend on π . The fourth line follows from the definition of V ⋆ and the change of index t ′ = t +1 .

Next, we prove Equation 19. When b = 0 , it is obvious that Q ⋆ = Q µ and V ⋆ = V µ . So V ⋆ ( s, 0) = E a ∼ µ [ Q ⋆ ( s, 0 , a )] . Now we consider the case of b > 0 .

When b > 0 , we have that

Next, we need to prove that

Let

If

then it must be the case that π ⋆ ( ·| s, b ) is not equal to π ⋆ 0 ( ·| s, b ) . Define

Then we have

By the definition of π ⋆ 0 and b > 0 , ¯ π satisfy the constraints b t ≥ 0 as well. Thus it contradicts the definition of maximum. So we finished the proof of

This finished the proof of the fixed point of .

T CB is Q ⋆

B Experimental Details

B.1 Hyper-parameters and infrastructures of experiments

We implement BCOL with the same architecture and hyper-parameter values as baseline methods. We list all hyperparameter values and neural architecture details in Table 2. For SAC-based implementation, we follow CDC's [8] hyper-parameter values. The only exception is that we use 2 Q functions instead of 4, and we sample 5 actions from actor ( m ) instead of 15. This is to save the amount of computation with a small amount of performance decrease for our algorithm. The baseline method (offline SAC) uses the same hyperparameter values as CDC. For TD3-based implementation, we follows TD3 [13] and TD3+BC's [11] hyper-parameter values.

To generate the final results as well as learning curves, we run all algorithms with 1 , 000 , 000 policy gradient steps, following prior work. We evaluate algorithms every 5 , 000 step before the 900 , 000 th gradient step, and every 1 , 000 step after the 900 , 000 th gradient step. For each evaluation, we run 10 test episodes in the environment.

Table 3 describes the hardware infrastructure and software libraries we used to run the experiment in this paper. We will release the code to reproduce the experiment upon publication.

B.2 Important experimental setup for AntMaze

Among prior offline RL work, IQL [23] and CQL [26] outperform other algorithms with a large margin in AntMaze domains. We notice that these methods also use some implementation tricks that are different from other work, in AntMaze domains. In order to clarify and eliminate all possible orthogonal factors, we list these implementation details below and implement our algorithm BCOL and two baselines (CDC and TD3+BC) with these implementations.

IQL uses a cosine learning rate schedule for the actor learning rate and a state-independent standard deviation in the policy network. Most prior work such as TD3+BC and CDC does not use these implementations. According to Appendix B in the IQL paper, CQL and IQL also subtract 1 from rewards for the AntMaze datasets. We apply these tricks for BCOL and two baselines (CDC and TD3+BC) that we re-implement, and report the AntMaze results in Table 1.

B.3 Budget consumption analysis

One might wonder that if BCOL spend all the budget in the beginning of trajectories. In fact, our experiment shows there are still budgets left at the end of the episode during the test. In Table 4 we present the total BCOL spent in the test (plus minus the standard deviation), averaged over seeds and last 10 policies, for AntMaze tasks. This result shows that the algorithm will not always be forced by Line 2 in Select in later steps. In contrast, it will plan on how to spend the budget by Line 1 in Select .

B.4 Comparison to additional baselines

Here we include the comparison against more baselines. These baselines are not discussed in the main paper either because they lack of result in the more challenging Antmaze tasks, or due to their relative inferior performance compared to the more recent offline RL algorithms. The numbers are from the original paper if available (ATAC, RIQL, CRR+, CQL+), or from other paper's report (MOPO, MoREL and COMBO is from [42] and MuZero is from [20]).

B.5 Complete experimental results

In this section, we include experimental details that are deferred here due to the space limitations of the main paper. First, we report the average and standard deviation (not included in Table 1) of all algorithms that we implemented in Table 6. The standard deviation is calculated across the 300 evaluation episodes for each algorithm (3 random seeds, 10 evaluation steps per random seed, and 10 episodes per evaluation.).

Second, we include the learning curves of these 4 algorithms in Fig. 4 to present the details of training. Each data point in the learning curve represents one evaluation step and is averaged over 30 test episodes (3 random seeds and 10 test episodes per evaluation). The shadow region represents the standard deviation over 30 test scores. For a better visibility, we smooth the learning curves by moving averages with a window size of 10 .

Third, we include the performance of BCOL with all searched hyper-parameter values on each task in Table 7 (for BCOL SAC) and Table 8 (for BCOL TD3).

hopper-medium-v2

walker2d-medium-v2

60

80

75

40

60

50

20

40

25

0

20

00.00

0.00

0.25

0.50

0.75

1.00

0.00

0.25

0.50

0.75

1.00

0.25

0.50

0.75

1.00

Gradient steps

1e6

Gradient steps

1e6

Gradientsteps

1e6

hopper-medium-replay-v2

walker2d-medium-replay-v2

100

50

75

75

40

50

50

30

25

25

0.00

0.25

0.50

0.75

1.00

0.00

0.25

0.500.75

1.00

0.00

0.25

0.50

0.75

1.00

Gradient steps

1e6

Gradient steps

1e6

Gradient steps

1e6

hopper-medium-expert-v2

walker2d-medium-expert-v2

75

100

100

80

50

50

25

60

0.00

0.25

0.50

0.75

1.00

0.00

0.25

0.50

0.75

1.00

0.00

0.25

0.50

0.75

1.00

Gradientsteps

1e6

Gradient steps

1e6

Gradient steps

1e6

antmaze-umaze-v2

antmaze-umaze-diverse-v2

100

75

75

50

50

50

25

25

0

0

0

0.00

0.25

0.50

0.75

1.00

0.00

0.25

0.50

0.75

1.00

0.00

0.25

0.50

0.75

1.00

Gradient steps

1e6

Gradient steps

1e6

Gradient steps

1e6

antmaze-large-play-v2

antmaze-large-diverse-v2

75

40

40

50

25

20

20

0

0

0

0.00

0.25

0.50

0.75

1.00

0.00

0.25

0.50

0.75

1.00

0.00

0.25

0.50

0.75

1.00

Gradient steps

1e6

Gradient steps

1e6

Gradient steps

1e6

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel offline RL algorithm BCOL that builds on the idea of limiting the numbers of counterfactual decisions. Instead of enforcing policy or value regularization, BCOL follows the decisions of the behavioral policy in the majority of the states, and only makes counterfactual decisions for a limited number of times. The strategy to spend the fixed amount of opportunities for counterfactual decisions is learned through a dynamic programming algorithm. Experimental results using two different implementations of the proposed algorithm on a wide range of offline RL tasks are presented.\n\nSTRENGTHS:\nIt is great that the authors present all details about their experiments and the implementation details of their proposed algorithm. \n\nThe authors present two implementations based on different state-of-the-art RL algorithms (TD3 and SAC). \n\nThe presentation about the empirical results (figures, tables, etc.) is very clear. The ablation study on the hyper-parameters is great.\n\nWEAKNESSES:\n### Method \nIt is well-known in the offline RL community that the extrapolation needs to be careful. Limiting the level of extrapolation itself is not challenging, one can copy the behavioral policy, leading to zero extrapolation. The challenging thing is to find out where to extrapolate. The authors have argued for many times in the paper that assigning an upper bound for the number of counterfactual decisions can effectively constrain the level of extrapolation, leading to a balance between the gain of counterfactual decisions and the risk of extrapolation. However, this only explains how the level of extrapolation is limited but not the more important question: why the proposed BCOL algorithm can learn where to extrapolate. \n\nOn the intuitive level, the explanation is not enough and not clear. On the formal side, it would be great if the authors can provide some theoretical guarantees for the proposed algorithm so that the benefits of BCOL become more clear. In the current version of the paper, I fail to see enough support, either intuitive or theoretical, for the efficacy of the proposed algorithm. \n\n### Algorithm \nThe proposed algorithm induces an extra burden for learning because the Q(s,b,a), unlike the regular state-action value function, needs to approximate the value of the budget well. This at least linearly increases the difficulty of the learning problem because for every b, Q(s,b,a) needs to be approximated well so that the proposed Counterfactual-Budgeting Bellman Operator can work well. \n\n### Experiments\nIt is appreciated that the details about the experiments are presented well. \n\nHowever, the performance of BCOL doesn’t seem that impressive given the increased training cost. For example,although the BCOL(SAC) has the highest total score, it is outperformed by CQL and CDC on about half of the tasks. \n\nMoreover, the baseline methods, as diverse as they are, are dated algorithms. It may be better to include more latest baselines that also report SOTA performance in their papers (see for example, [1] and [2]) .  \n\n[1] Bhardwaj, Mohak, et al. \"Adversarial model for offline reinforcement learning.\" arXiv preprint arXiv:2302.11048 (2023).\n[2] Kang, Bingyi, et al. \"Improving and Benchmarking Offline Reinforcement Learning Algorithms.\" arXiv preprint arXiv:2306.00972 (2023).\n\nQUESTIONS:\nIs the algorithm still working if there is more than one behavior policy for collecting the offline data? \n\nThe sentence “Thus, the choice of policy on each step needs to balance between the Q value gain from the current step and the potential benefit from future counterfactual decisions.” is very confusing. Could the authors please elaborate more on it? \n\nWhat is the formal definition of Q(s,b,a) in Eq. (4)? It looks like the exact definition of Q(s,b,a) doesn’t matter, and the only thing that matters is the fixed point of $\\mathcal{T}_{CB}$. \n\nI understand that by definition of $\\mathcal{T}_{CB}$ that are at most B backup steps taking the max operation as the maximum value. But why does this “intuitively upper bounds the amount of extrapolation that the Q function can take”?\n\nLIMITATIONS:\nYes. The authors have discussed the limitations of this work and pointed out future directions for continuing research.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a TD approach to induce counterfactual decisioning in offline RL agents. Basically, the approach suggests using a count-based budget that gives scope for making decisions that are not exhibited by the behaviour policy. The paper implements this approach in various standard benchmarks and shows that the method performs favourably to the state-of-the-art approaches. They also give a theoretical justification for why budgeting should work.\n\nSTRENGTHS:\nThe strength of the paper lies in the simplicity of the proposed budgeting for counterfactual decision making. The paper identifies the gap in the existing approaches which fall short of inducing counterfactuals. The authors propose to put a hard budget on the number of counterfactual decisions taken by the agent. Building upon the constrained optimization formulation for solving the MDP, the work shows how it converges to a fixed point and argues about the optimality of the same. In addition, it gives a function approximation version for using deep learning based approaches in conjugation.\n\nWEAKNESSES:\nI find the following weaknesses in the approach:\n\n**Tuning of the B**: It is unclear how the budget would be tuned for different environments. Because having a high B implies applying off-the-shelf RL algorithms without considering the nuances of offline RL setting, and a low B will make the method similar to imitation learning techniques.\n\n**Lack of Imitation Learning baseline**: It would be helpful to see how the proposed BCOL compares with state-of-the-art approaches for imitation learning. This will highlight the importance of counterfactual decision-making learnt from offline dataset. I suggest authors include a competitive baseline for IL, too.\n\n**Comment on the novelty of budgeting**: Safe RL approaches have count-based constraints on agent's violations of safety. Therefore, the theory given in the present work has a very high overlap with the prior works, reducing the novelty in the optimization or the fixed point derived.\n\n**Motivation behind the counterfactual budgeting**: I am myself not aware of regularized techniques for increasing counterfactuals in offline RL, however from first principles, it seems that inducing counterfactuals might not be a good idea as offline RL might lead to safety issues when the agent is deployed. The aim of offline RL (at least to me) is finding connections in the offline data that can help improve the returns gained by taking care that the agent doesn't infer unrealistic novel behaviours from offline data that could turn harmful once deployed in the actual environment. Can authors please comment on the safety of the BCOL agents?\n\n**Why would such budgeting work**: From equations 4 and Select() given on line 185, it looks like BCOL would spend the budget in the initial steps of decision-making, and after that, it would follow the behaviour policy. Put in a different way, BCOL's budgeting seems myopic in nature; it is unclear to me how the algorithm will induce the agent to use the budget later in the decision-making when it matters the most. Can authors please comment on how the budget would be used pragmatically by the agent? It would help to see an empirical analysis of the budget expenditure against the time steps.\n\nQUESTIONS:\nI have listed my questions and corresponding suggestions in the above section.\n\nOverall, I like the paper's straightforward approach to increasing counterfactuals in offline RL and enjoyed reading the derivations provided. However, the paper has many unaddressed weaknesses, as pointed out above. I am inclined towards borderline rejecting the work in its current form. With my questions answered, I would love to increase my rating for this work.\n\nLIMITATIONS:\nAuthors do point out the limitations around using count-based approach for budgeting and theoretical analysis of less number of counterfactuals. In addition, I urge authors to also talk about the safety considerations involved in BCOL.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

comments: SUMMARY:
This paper gives a total new solution to offline RL, instead of introduing pessimism by behavior constraint or value regularization, budgeting the number of counterfactual decisions, which naturally reduces overestimation. This paper also gives a good formulation of the problem and provides a nice solution to solve the allocation problem by dynamic programming. This paper shows good empirical performance over offline RL benchmarks and optimality of the fixed point solution as theoretical justification.

STRENGTHS:
- The paper is very well written.
- The idea of avoid distributional shift by budgetting the number of counterfactual decision making is very novel and promising.
- This paper gives a nice formulation of the problem and provides a solution to solve the allocation problem by dynamic programming, which is also solid and theoretical justificated.
- This paper shows strong performance on various D4RL datasets, though not SOTA in some datasets if compared to more recent works.

WEAKNESSES:
- This method introduces two hyperparameters to tune: $\omega$ and $B$.
- It is better to have some visualization on which critical steps does the algorithm tends to make counterfactual decisions and whether it has some physical meanings. For example, on antmaze tasks, which location does the algorithm make counterfactual decisions.

QUESTIONS:
- Why adding the gap_penalization term (Equation 9) is necessary? Does it still do some kind of behavior regularization implicitly? Could you elabotate more on the usage of Equation 9?
- Why SAC+BCOL can work well on antmaze tasks while TD3+BCOL can't?

LIMITATIONS:
See Weakness.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

comments: SUMMARY:
This paper designs an new algorithm for offline RL. It also conduct experiment to validate its algorithms.

STRENGTHS:
The algorithm proposed is new. The intuition behind is presented clearly.

WEAKNESSES:
1. The performance of the proposed algorithm does not demonstrate substantial superiority compared to the Contrastive Q-Learning (CQL) algorithm. It would be beneficial to provide more persuasive evidence of the proposed algorithm's efficacy.

2. The use of the "budget" concept presents limitations, as it appears applicable primarily in tabular settings. Its adaptability to continuous cases is challenging, which curtails its generalizability. The authors may want to consider elucidating the feasibility of this concept in a broader context or proposing alternative approaches for continuous settings.

QUESTIONS:
The performance of the proposed algorithm does not demonstrate substantial superiority compared to the Contrastive Q-Learning (CQL) algorithm. It would be beneficial to provide more persuasive evidence of the proposed algorithm's efficacy.

LIMITATIONS:
The use of the "budget" concept presents limitations, as it appears applicable primarily in tabular settings. Its adaptability to continuous cases is challenging, which curtails its generalizability. The authors may want to consider elucidating the feasibility of this concept in a broader context or proposing alternative approaches for continuous settings.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-09