SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\pi_ {0} ^ {\star} = \operatorname * {a r g m a x} _ {\pi_ {0}} \max  _ {\pi s. t. b _ {t} \geq 0} \mathbb {E} _ {a \sim \pi (\cdot | s, b)} \left[ \mathbb {E} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} r _ {t} \mid s _ {0} = 0, a _ {0} = a, b _ {0} = b, \pi \right] \right] \tag {33}
$$
$$
\begin{array}{l} \max  _ {\pi s. t. b _ {t} \geq 0} \mathbb {E} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} r _ {t} \mid s _ {0} = s, b _ {0} = b, \pi \right] (34) \\ <   \max  _ {\pi_ {0}} \max  _ {\pi s. t. b _ {t} \geq 0} \mathbb {E} _ {a \sim \pi (\cdot | s, b)} \left[ \mathbb {E} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} r _ {t} \mid s _ {0} = 0, a _ {0} = a, b _ {0} = b, \pi \right] \right], (35) \\ \end{array}
$$
then it must be the case that  $\pi^{\star}(\cdot |s,b)$  is not equal to  $\pi_0^\star (\cdot |s,b)$ . Define
$$
\bar {\pi} (\cdot | s _ {t}, b _ {t}) = \left\{ \begin{array}{l l} \pi_ {0} ^ {\star} (\cdot | s _ {t}, b _ {t}) & s _ {t} = s, b _ {t} = b \\ \pi^ {\star} (\cdot | s _ {t}, b _ {t}) & \mathrm {o . w .} \end{array} \right. \tag {36}
$$
$$
\begin{array}{l} \max  _ {\pi s. t. b _ {t} \geq 0} \mathbb {E} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} r _ {t} \mid s _ {0} = s, b _ {0} = b, \pi \right] (37) \\ <   \max  _ {\pi_ {0}} \max  _ {\pi s. t. b _ {t} \geq 0} \mathbb {E} _ {a \sim \pi (\cdot | s, b)} \left[ \mathbb {E} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} r _ {t} \mid s _ {0} = 0, a _ {0} = a, b _ {0} = b, \pi \right] \right] (38) \\ <   \underset {a \sim \pi_ {0} ^ {\star} (\cdot | s, b)} {\mathbb {E}} \left[ \mathbb {E} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} r _ {t} \mid s _ {0} = 0, a _ {0} = a, b _ {0} = b, \pi^ {\star} \right] \right] (39) \\ <   \underset {a \sim \pi_ {0} ^ {\star} (\cdot | s, b)} {\mathbb {E}} \left[ \mathbb {E} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} r _ {t} \mid s _ {0} = 0, a _ {0} = a, b _ {0} = b, \bar {\pi} \right] \right] (40) \\ = \mathbb {E} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} r _ {t} \mid s _ {0} = s, b _ {0} = b, \bar {\pi} \right] (41) \\ \end{array}
$$
By the definition of  $\pi_0^\star$  and  $b > 0$ ,  $\bar{\pi}$  satisfy the constraints  $b_{t} \geq 0$  as well. Thus it contradicts the definition of maximum. So we finished the proof of
$$
V ^ {\star} (s, b) = \max  \left\{\max  _ {a} Q ^ {\star} (s, b - 1, a), \mathbb {E} _ {a \sim \mu} Q ^ {\star} (s, b, a) \right\}, \forall s \in \mathcal {S}, b > 0 \tag {42}
$$
This finished the proof of the fixed point of  $\mathcal{T}_{\mathrm{CB}}$  is  $Q^{\star}$
Hyperparameter	BCOL SAC	BCOL TD3
Random seeds	{0,1,2}
Number of gradient steps	1e6
Hidden layer dimensions	[256,256,256]	[256,256]
Activation function	ReLU
Discounting factor (γ)	0.99
Target network update rate	0.005
Optimizer	Adam
Actor learning rate	3e-41	3e-4
Critic learning rate	7e-4	3e-4
Number of Q functions	2
Coefficient of min Q values (λ in [12])	0.75	1
Number of actions samples (m)	5	-
TD3 policy noise	-	0.2
TD3 noise clip	-	0.5
TD3 policy update freq	-	2
Counterfactual budget B	{1,10,50}
Monotonicity penalty coefficient ω	{0,1,10,100}
Table 2: Hyperparameter values and model architecture details in BCOL.
B.1 Hyper-parameters and infrastructures of experiments
We implement BCOL with the same architecture and hyper-parameter values as baseline methods. We list all hyperparameter values and neural architecture details in Table 2. For SAC-based implementation, we follow CDC's [8] hyper-parameter values. The only exception is that we use  $2Q$  functions instead of 4, and we sample 5 actions from actor  $(m)$  instead of 15. This is to save the amount of computation with a small amount of performance decrease for our algorithm. The baseline method (offline SAC) uses the same hyperparameter values as CDC. For TD3-based implementation, we follows TD3 [13] and TD3+BC's [11] hyper-parameter values.
To generate the final results as well as learning curves, we run all algorithms with 1,000,000 policy gradient steps, following prior work. We evaluate algorithms every 5,000 step before the 900,000th gradient step, and every 1,000 step after the 900,000th gradient step. For each evaluation, we run 10 test episodes in the environment.
Table 3 describes the hardware infrastructure and software libraries we used to run the experiment in this paper. We will release the code to reproduce the experiment upon publication.
B.2 Important experimental setup for AntMaze
Among prior offline RL work, IQL [23] and CQL [26] outperform other algorithms with a large margin in AntMaze domains. We notice that these methods also use some implementation tricks that are different from other work, in AntMaze domains. In order to clarify and eliminate all possible orthogonal factors, we list these implementation details below and implement our algorithm BCOL and two baselines (CDC and TD3+BC) with these implementations.
IQL uses a cosine learning rate schedule for the actor learning rate and a state-independent standard deviation in the policy network. Most prior work such as TD3+BC and CDC does not use these implementations. According to Appendix B in the IQL paper, CQL and IQL also subtract 1 from rewards for the AntMaze datasets. We apply these tricks for BCOL and two baselines (CDC and TD3+BC) that we re-implement, and report the AntMaze results in Table 1.
<sup>1</sup>We use learning rate  $3e - 4$  for Gym domains and  $1e - 4$  for AntMaze domains. Following the suggestion from CDC authors, we found out that CDC and offline SAC perform better with a smaller learning rate in AntMaze domains. Thus we set this learning rate for our algorithm BCOL SAC and baselines (CDC, offline SAC).
Machine Type	AWS EC2 g4dn.2xlarge
GPU	Tesla T4
CPU	Intel Xeon 2.5GHz
CUDA version	11.0
NVIDIA Driver	450.142.00
PyTorch version	1.12.1
Gym version	19.0
Python version	3.8.13
NumPy version	1.21.5
D4RL datasets version	v2
B.3 Budget consumption analysis
One might wonder that if BCOL spend all the budget in the beginning of trajectories. In fact, our experiment shows there are still budgets left at the end of the episode during the test. In Table 4 we present the total BCOL spent in the test (plus minus the standard deviation), averaged over seeds and last 10 policies, for AntMaze tasks. This result shows that the algorithm will not always be forced by Line 2 in Select in later steps. In contrast, it will plan on how to spend the budget by Line 1 in Select.
Table 3: Hardware infrastructure and software libraries used in this paper
Task	Average budget spent (B = 50)
Antmaze-umaze	41.00 ± 0.91
Antmaze-umaze-diverse	20.02 ± 6.44
Antmaze-medium-play	45.94 ± 1.13
Antmaze-medium-diverse	46.04 ± 0.92
Antmaze-large-play	43.33 ± 1.81
Antmaze-large-diverse	43.75 ± 1.77
B.4 Comparison to additional baselines
Here we include the comparison against more baselines. These baselines are not discussed in the main paper either because they lack of result in the more challenging Antmaze tasks, or due to their relative inferior performance compared to the more recent offline RL algorithms. The numbers are from the original paper if available (ATAC, RIQL, CRR+, CQL+), or from other paper's report (MOPO, MoREL and COMBO is from [42] and MuZero is from [20]).
Table 4: Budget Consumption
Algorithm	Mujoco Total	Antmaze Total	Total
BCOL (SAC)	746.0	396.0	1142.0
MoREL [21]	656.5	0	656.5
MOPO [55]	379.3	0	379.3
COMBO [54]	738.3	137.6	875.9
MuZero [45]	140.2	0	140.2
RIQL [10]	759.1	-	-
ATAC [5]	792.4	-	-
CRR+ [20]	703.9	41.9	745.8
CQL+ [20]	717.9	89.0	806.9
Table 5: Additional baselines
B.5 Complete experimental results
In this section, we include experimental details that are deferred here due to the space limitations of the main paper. First, we report the average and standard deviation (not included in Table 1) of all algorithms that we implemented in Table 6. The standard deviation is calculated across the 300 evaluation episodes for each algorithm (3 random seeds, 10 evaluation steps per random seed, and 10 episodes per evaluation.).
Task Name	TD3+BC	BCOL (TD3)	CDC	BCOL (SAC)
halfcheetah-medium-v2	48.4 ± 0.3	45.0 ± 0.4	62.5 ± 1.2	50.1 ± 0.3
hopper-medium-v2	59.4 ± 4.6	85.8 ± 10.2	84.9 ± 8.4	83.2 ± 7.5
walker2d-medium-v2	84.6 ± 1.6	76.5 ± 5.8	70.7 ± 16.6	84.1 ± 1.5
halfcheetah-medium-replay-v2	44.5 ± 0.5	41.1 ± 0.9	52.3 ± 1.7	46.2 ± 1.0
hopper-medium-replay-v2	50.1 ± 22.2	81.8 ± 19.2	87.4 ± 13.2	99.8 ± 1.4
walker2d-medium-replay-v2	80.2 ± 7.6	49.6 ± 15.2	87.8 ± 5.4	86.0 ± 3.3
halfcheetah-medium-expert-v2	91.5 ± 5.2	89.0 ± 3.8	66.3 ± 7.0	86.9 ± 5.7
hopper-medium-expert-v2	100.5 ± 11.8	106.9 ± 7.2	83.2 ± 16.3	99.0 ± 11.6
walker2d-medium-expert-v2	110.1 ± 0.5	108.6 ± 0.4	103.9 ± 7.5	110.8 ± 0.4
antmaze-umaze-v2	96.3 ± 5.5	93.3 ± 8.3	93.7 ± 8.7	90.3 ± 8.4
antmaze-umaze-diverse-v2	71.7 ± 15.3	68.0 ± 15.8	57.3 ± 26.2	90.0 ± 8.9
antmaze-medium-play-v2	1.7 ± 3.7	12.3 ± 18.7	59.5 ± 15.6	70.0 ± 12.6
antmaze-medium-diverse-v2	1.3 ± 4.3	14.0 ± 13.1	64.7 ± 14.3	72.3 ± 17.5
antmaze-large-play-v2	0.0 ± 0.0	0.0 ± 0.0	33.0 ± 12.7	35.7 ± 23.6
antmaze-large-diverse-v2	0.3 ± 1.8	0.0 ± 0.0	25.3 ± 15.4	37.7 ± 18.2
Table 6: Average normalized score and standard deviation on D4RL tasks.
Second, we include the learning curves of these 4 algorithms in Fig. 4 to present the details of training. Each data point in the learning curve represents one evaluation step and is averaged over 30 test episodes (3 random seeds and 10 test episodes per evaluation). The shadow region represents the standard deviation over 30 test scores. For a better visibility, we smooth the learning curves by moving averages with a window size of 10.
Third, we include the performance of BCOL with all searched hyper-parameter values on each task in Table 7 (for BCOL SAC) and Table 8 (for BCOL TD3).
Figure 4: Learning curves for CDC, BCOL (SAC), TD3+BC, and BCOL (TD3) in D4RL tasks.
Task Name	B=1	B=10	B=50
ω=0	ω=1	ω=10	ω=100	ω=0	ω=1	ω=10	ω=100	ω=0	ω=1	ω=10	ω=100
halfcheetah-medium-v2	42.7 ± 0.4	42.7 ± 0.3	42.8 ± 0.4	42.8 ± 0.5	43.7 ± 0.4	43.8 ± 0.3	43.5 ± 0.6	43.7 ± 0.4	44.3 ± 1.2	45.3 ± 0.3	45.0 ± 0.4	42.5 ± 0.4
hopper-medium-v2	51.6 ± 5.5	55.2 ± 7.1	52.8 ± 7.2	55.5 ± 6.3	53.7 ± 5.7	59.2 ± 5.7	57.7 ± 5.0	78.1 ± 15.4	7.7 ± 10.0	59.9 ± 28.8	85.8 ± 10.2	61.1 ± 10.2
walker2d-medium-v2	72.1 ± 4.9	70.0 ± 6.7	71.4 ± 5.9	71.6 ± 5.8	76.8 ± 4.0	77.6 ± 3.9	77.5 ± 4.2	79.1 ± 3.1	0.2 ± 1.6	51.0 ± 30.7	76.5 ± 5.8	70.2 ± 4.9
halfcheetah-medium-replay-v2	36.5 ± 3.2	36.5 ± 2.8	36.5 ± 2.0	36.6 ± 2.1	40.1 ± 1.0	40.4 ± 0.7	39.9 ± 1.3	40.0 ± 1.0	41.3 ± 1.5	42.1 ± 0.8	41.1 ± 0.9	34.2 ± 3.6
hopper-medium-replay-v2	28.9 ± 9.1	29.5 ± 10.7	29.0 ± 9.1	28.3 ± 8.9	40.8 ± 14.1	46.6 ± 13.8	51.3 ± 12.7	57.4 ± 19.4	43.7 ± 8.5	62.1 ± 19.0	81.8 ± 19.2	25.0 ± 8.1
walker2d-medium-replay-v2	26.4 ± 11.3	25.3 ± 10.7	25.7 ± 9.0	25.2 ± 10.9	57.3 ± 8.9	56.5 ± 10.9	56.3 ± 12.2	58.7 ± 10.9	57.5 ± 16.3	72.7 ± 7.4	49.6 ± 15.2	17.6 ± 8.1
halfcheetah-medium-expert-v2	69.7 ± 7.9	70.1 ± 8.2	72.7 ± 7.1	72.9 ± 8.1	80.6 ± 6.2	85.6 ± 7.0	90.3 ± 2.6	91.2 ± 1.5	60.2 ± 14.5	84.2 ± 5.6	89.0 ± 3.8	84.2 ± 4.9
hopper-medium-expert-v2	77.4 ± 18.4	83.6 ± 20.5	90.8 ± 18.0	78.3 ± 21.0	33.0 ± 26.9	82.3 ± 14.2	102.9 ± 14.4	106.2 ± 5.8	1.9 ± 1.2	31.2 ± 26.8	106.9 ± 7.2	76.2 ± 45.6
walker2d-medium-expert-v2	106.7 ± 3.5	106.4 ± 4.6	105.7 ± 6.1	106.8 ± 3.9	87.4 ± 18.4	107.0 ± 5.5	108.7 ± 0.3	108.6 ± 0.4	-0.2 ± 0.0	87.8 ± 26.3	108.6 ± 0.4	93.8 ± 16.5
antmaze-umaze-v2	44.7 ± 12.6	45.3 ± 12.3	45.3 ± 12.3	45.3 ± 12.6	58.3 ± 15.1	55.7 ± 15.4	41.3 ± 13.1	41.3 ± 13.8	93.3 ± 8.3	86.3 ± 12.5	50.3 ± 13.3	49.7 ± 13.0
antmaze-umaze-diverse-v2	48.3 ± 15.9	46.0 ± 16.9	46.0 ± 16.9	46.0 ± 16.9	48.0 ± 18.3	44.3 ± 17.1	44.7 ± 18.0	45.0 ± 18.8	68.0 ± 15.8	51.0 ± 19.2	53.3 ± 17.8	45.3 ± 15.2
antmaze-medium-play-v2	0.7 ± 2.5	0.3 ± 1.8	0.7 ± 2.5	0.7 ± 2.5	0.0 ± 0.0	0.0 ± 0.0	0.0 ± 0.0	0.0 ± 0.0	12.3 ± 18.7	0.0 ± 0.0	0.0 ± 0.0	0.0 ± 0.0
antmaze-medium-diverse-v2	0.0 ± 0.0	0.0 ± 0.0	0.0 ± 0.0	0.0 ± 0.0	0.0 ± 0.0	0.0 ± 0.0	0.0 ± 0.0	0.0 ± 0.0	14.0 ± 13.1	0.0 ± 0.0	0.0 ± 0.0	0.0 ± 0.0
antmaze-large-play-v2	0.0 ± 0.0	0.0 ± 0.0	0.0 ± 0.0	0.0 ± 0.0	0.0 ± 0.0	0.0 ± 0.0	0.0 ± 0.0	0.0 ± 0.0	0.0 ± 0.0	0.0 ± 0.0	0.0 ± 0.0	0.0 ± 0.0
antmaze-large-diverse-v2	0.0 ± 0.0	0.0 ± 0.0	0.0 ± 0.0	0.0 ± 0.0	0.0 ± 0.0	0.0 ± 0.0	0.0 ± 0.0	0.0 ± 0.0	0.0 ± 0.0	0.0 ± 0.0	0.0 ± 0,0	0.0 ± 0.0
Table 7: Average normalize score and standard deviation of hyper-parameter ablation on BCOL (TD3).
Task Name	B=1	B=10	B=50
ω=0	ω=1	ω=10	ω=100	ω=0	ω=1	ω=10	ω=100	ω=0	ω=1	ω=10	ω=100
halfcheetah-medium-v2	50.7 ± 0.3	50.7 ± 0.4	50.4 ± 0.2	49.7 ± 0.2	52.6 ± 0.8	52.3 ± 0.4	50.1 ± 0.3	48.2 ± 0.4	53.1 ± 1.5	53.7 ± 0.3	46.9 ± 0.4	39.7 ± 1.0
hopper-medium-v2	73.3 ± 8.6	72.2 ± 8.4	73.5 ± 8.0	73.9 ± 9.8	63.8 ± 17.0	71.7 ± 8.0	83.2 ± 7.5	61.9 ± 7.8	4.9 ± 6.2	71.0 ± 17.9	58.3 ± 5.9	50.9 ± 10.4
walker2d-medium-v2	84.5 ± 2.5	84.9 ± 3.0	85.1 ± 2.2	84.6 ± 0.4	33.9 ± 34.7	80.9 ± 7.8	84.1 ± 1.5	83.2 ± 1.8	42.7 ± 24.2	52.2 ± 30.5	81.4 ± 3.1	54.9 ± 12.6
halfcheetah-medium-replay-v2	43.7 ± 1.8	43.1 ± 2.5	43.8 ± 1.2	44.0 ± 1.8	45.6 ± 1.7	46.1 ± 1.4	46.2 ± 1.0	44.2 ± 1.2	21.7 ± 16.6	48.1 ± 1.4	41.0 ± 1.6	15.4 ± 6.7
hopper-medium-replay-v2	91.9 ± 9.7	86.0 ± 11.0	91.8 ± 7.2	93.2 ± 8.0	96.1 ± 6.5	92.9 ± 8.7	99.8 ± 1.4	88.6 ± 12.6	11.6 ± 13.1	93.6 ± 9.9	79.5 ± 16.9	4.0 ± 6.0
walker2d-medium-replay-v2	75.2 ± 8.6	76.5 ± 9.8	79.4 ± 5.5	81.1 ± 5.7	66.9 ± 12.9	74.4 ± 11.3	86.0 ± 3.3	82.1 ± 5.1	9.4 ± 9.5	51.9 ± 14.0	30.9 ± 26.4	5.4 ± 6.9
halfcheetah-medium-expert-v2	79.5 ± 5.3	81.1 ± 5.9	84.6 ± 6.5	85.6 ± 5.5	79.7 ± 7.6	80.5 ± 7.2	86.9 ± 5.7	83.3 ± 5.7	65.4 ± 9.6	73.8 ± 9.0	83.6 ± 5.9	41.7 ± 3.2
hopper-medium-expert-v2	76.3 ± 18.8	70.2 ± 19.4	88.2 ± 15.2	90.6 ± 12.0	16.2 ± 17.9	68.1 ± 17.0	99.0 ± 11.6	92.9 ± 13.0	4.2 ± 2.8	38.3 ± 15.9	99.0 ± 10.6	50.1 ± 10.5
walker2d-medium-expert-v2	110.6 ± 2.3	110.7 ± 0.7	110.8 ± 0.4	109.5 ± 2.7	42.0 ± 43.4	110.3 ± 1.2	110.8 ± 0.4	109.7 ± 0.4	64.7 ± 31.7	93.0 ± 18.3	109.7 ± 0.7	71.1 ± 27.6
antmaze-umaze-v2	31.0 ± 34.2	73.0 ± 16.2	72.0 ± 13.8	65.3 ± 20.0	91.0 ± 7.9	91.7 ± 8.2	85.3 ± 9.6	75.7 ± 13.1	90.3 ± 8.4	91.7 ± 8.6	64.0 ± 14.7	83.3 ± 14.7
antmaze-umaze-diverse-v2	38.0 ± 18.9	54.0 ± 16.9	54.7 ± 16.7	62.0 ± 14.5	4.3 ± 8.4	55.0 ± 14.3	60.7 ± 14.8	56.3 ± 18.2	90.0 ± 8.9	54.0 ± 15.8	33.3 ± 20.1	49.0 ± 29.1
antmaze-medium-play-v2	0.0 ± 0.0	0.0 ± 0.0	0.0 ± 0.0	0.3 ± 1.8	14.0 ± 19.3	7.7 ± 12.3	0.0 ± 0.0	0.0 ± 0.0	70.0 ± 12.6	26.7 ± 19.9	0.0 ± 0.0	0.0 ± 0.0
antmaze-medium-diverse-v2	0.0 ± 0.0	0.0 ± 0.0	0.3 ± 1.8	0.0 ± 0.0	4.0 ± 7.6	9.7 ± 8.4	0.0 ± 0.0	0.0 ± 0.0	72.3 ± 17.5	23.7 ± 14.3	0.0 ± 0.0	0.3 ± 1.8
antmaze-large-play-v2	0.0 ± 0.0	0.0 ± 0.0	0.0 ± 0.0	0.0 ± 0.0	0.0 ± 0.0	0.0 ± 0.0	0.0 ± 0.0	0.0 ± 0.0	35.7 ± 23.6	2.0 ± 6.0	0.0 ± 0.0	0.0 ± 0.0
antmaze-large-diverse-v2	0.0 ± 0.0	0.0 ± 0.0	0.0 ± 0.0	0.0 ± 0.0	0.0 ± 0.0	0.0 ± 0.0	0.0 ± 0.0	0.0 ± 0.0	37.7 ± 18.2	0.0 ± 0.0	0.0 ± 0.0	0.0 ± 0.0
Table 8: Average normalize score and standard deviation of hyper-parameter ablation on BCOL (SAC).