SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Comparison of ReBRAC with the ensemble-based methods is presented in Table 16, Table 17, and Table 18. We add the following ensemble-based methods: RORL for each domain (Yang et al., 2022), SAC-N/EDAC (An et al., 2021) for the Gym-MuJoCo and Adroit tasks<sup>3</sup> and MSG (Ghasemipour et al., 2022) for AntMaze tasks<sup>4</sup>. The mean-wise best results among algorithms are highlighted with bold, and the second-best performance is underlined. Our approach, ReBRAC, shows competitive results on the Gym-MuJoCo datasets. On AntMaze tasks, ReBRAC achieves state-of-the-art results among ensemble-free algorithms and a good score compared to ensemble-based algorithms. And on Adroit tasks, our approach outperforms both families of algorithms.
Table 16: ReBRAC evaluation on the Gym domain. We report the final normalized score averaged over 10 unseen training seeds on v2 datasets. CQL, SAC-N and EDAC scores are taken from An et al. (2021). RORL scores are taken from Yang et al. (2022).
Ensemble-free	Ensemble-based
Task Name	TD3+BC	IQL	CQL	SAC-RND	SAC-N	EDAC	RORL	ReBRAC, our
halfcheetah-random	30.9 ± 0.4	19.5 ± 0.8	31.1 ± 3.5	27.6 ± 2.1	28.0 ± 0.9	28.4 ± 1.0	28.5 ± 0.8	29.5 ± 1.5
halfcheetah-medium	54.7 ± 0.9	50.0 ± 0.2	46.9 ± 0.4	66.4 ± 1.4	67.5 ± 1.2	65.9 ± 0.6	66.8 ± 0.7	65.6 ± 1.0
halfcheetah-expert	93.4 ± 0.4	95.5 ± 2.1	97.3 ± 1.1	102.6 ± 4.2	105.2 ± 2.6	106.8 ± 3.4	105.2 ± 0.7	105.9 ± 1.7
halfcheetah-medium-expert	89.1 ± 5.6	92.7 ± 2.8	95.0 ± 1.4	108.1 ± 1.5	107.1 ± 2.0	106.3 ± 1.9	107.8 ± 1.1	101.1 ± 5.2
halfcheetah-medium-replay	45.0 ± 1.1	42.1 ± 3.6	45.3 ± 0.3	51.2 ± 3.2	63.9 ± 0.8	61.3 ± 1.9	61.9 ± 1.5	51.0 ± 0.8
halfcheetah-full-replay	75.0 ± 2.5	75.0 ± 0.7	76.9 ± 0.9	81.2 ± 1.3	84.5 ± 1.2	84.6 ± 0.9	-	82.1 ± 1.1
hopper-random	8.5 ± 0.6	10.1 ± 5.9	5.3 ± 0.6	19.6 ± 12.4	31.3 ± 0.0	25.3 ± 10.4	31.4 ± 0.1	8.1 ± 2.4
hopper-medium	60.9 ± 7.6	65.2 ± 4.2	61.9 ± 6.4	91.1 ± 10.1	100.3 ± 0.3	101.6 ± 0.6	104.8 ± 0.1	102.0 ± 1.0
hopper-expert	109.6 ± 3.7	108.8 ± 3.1	106.5 ± 9.1	109.8 ± 0.5	110.3 ± 0.3	110.1 ± 0.1	112.8 ± 0.2	100.1 ± 8.3
hopper-medium-expert	87.8 ± 10.5	85.5 ± 29.7	96.9 ± 15.1	109.8 ± 0.6	110.1 ± 0.3	110.7 ± 0.1	112.7 ± 0.2	107.0 ± 6.4
hopper-medium-replay	55.1 ± 31.7	89.6 ± 13.2	86.3 ± 7.3	97.2 ± 9.0	101.8 ± 0.5	101.0 ± 0.5	102.8 ± 0.5	98.1 ± 5.3
hopper-full-replay	97.9 ± 17.5	104.4 ± 10.8	101.9 ± 0.6	107.4 ± 0.8	102.9 ± 0.3	105.4 ± 0.7	-	107.1 ± 0.4
walker2d-random	2.0 ± 3.6	11.3 ± 7.0	5.1 ± 1.7	18.7 ± 6.9	21.7 ± 0.0	16.6 ± 7.0	21.4 ± 0.2	18.1 ± 4.5
walker2d-medium	77.7 ± 2.9	80.7 ± 3.4	79.5 ± 3.2	92.7 ± 1.2	87.9 ± 0.2	92.5 ± 0.8	102.4 ± 1.4	82.5 ± 3.6
walker2d-expert	110.0 ± 0.6	96.9 ± 32.3	109.3 ± 0.1	104.5 ± 22.8	107.4 ± 2.4	115.1 ± 1.9	115.4 ± 0.5	112.3 ± 0.2
walker2d-medium-expert	110.4 ± 0.6	112.1 ± 0.5	109.1 ± 0.2	104.6 ± 11.2	116.7 ± 0.4	114.7 ± 0.9	121.2 ± 1.5	111.6 ± 0.3
walker2d-medium-replay	68.0 ± 19.2	75.4 ± 9.3	76.8 ± 10.0	89.4 ± 3.8	78.7 ± 0.7	87.1 ± 2.4	90.4 ± 0.5	77.3 ± 7.9
walker2d-full-replay	90.3 ± 5.4	97.5 ± 1.4	94.2 ± 1.9	105.3 ± 3.2	94.6 ± 0.5	99.8 ± 0.7	-	102.2 ± 1.7
Average w/o full-replay	66.8	70.1	70.1	79.5	82.4	82.9	85.7	78.0
Average	70.3	72.9	73.6	82.6	84.4	85.2	-	81.2
B.4 SAC-RND
${}^{3}$  SAC-N and EDAC score 0 on medium and large AntMaze tasks (Tarasov et al., 2022).
<sup>4</sup>MSG numerical results are not available for Gym-MuJoCo tasks and Adroit tasks were not benchmarked.
Table 17: ReBRAC evaluation on AntMaze domain. We report the final normalized score averaged over 10 unseen training seeds on v2 datasets. CQL scores are taken from Ghasemipour et al. (2022). RORL scores are taken from Yang et al. (2022).
Ensemble-free	Ensemble-based
Task Name	TD3+BC	IQL	CQL	SAC-RND	RORL	MSG	ReBRAC, our
antmaze-umaze	66.3 ± 6.2	83.3 ± 4.5	74.0	97.0 ± 1.5	97.7 ± 1.9	97.9 ± 1.3	97.8 ± 1.0
antmaze-umaze-diverse	53.8 ± 8.5	70.6 ± 3.7	84.0	66.0 ± 25.0	90.7 ± 2.9	79.3 ± 3.0	88.3 ± 13.0
antmaze-medium-play	26.5 ± 18.4	64.6 ± 4.9	61.2	38.5 ± 29.4	76.3 ± 2.5	85.9 ± 3.9	84.0 ± 4.2
antmaze-medium-diverse	25.9 ± 15.3	61.7 ± 6.1	53.7	74.7 ± 10.7	69.3 ± 3.3	84.6 ± 5.2	76.3 ± 13.5
antmaze-large-play	0.0 ± 0.0	42.5 ± 6.5	15.8	43.9 ± 29.2	16.3 ± 11.1	64.3 ± 12.7	60.4 ± 26.1
antmaze-large-diverse	0.0 ± 0.0	27.6 ± 7.8	14.9	45.7 ± 28.5	41.0 ± 10.7	71.3 ± 5.3	54.4 ± 25.1
Average	28.7	58.3	50.6	60.9	65.2	80.5	76.8
Table 18: ReBRAC evaluation on Adroit domain. We report the final normalized score averaged over 10 unseen training seeds on v1 datasets. BC, CQL, EDAC and RORL scores are taken from Yang et al. (2022).
Ensemble-free	Ensemble-based
Task Name	BC	TD3+BC	IQL	CQL	SAC-RND	RORL	EDAC	ReBRAC, our
pen-human	34.4	81.8 ± 14.9	81.5 ± 17.5	37.5	5.6 ± 5.8	33.7 ± 7.6	51.2 ± 8.6	103.5 ± 14.1
pen-cloned	56.9	61.4 ± 19.3	77.2 ± 17.7	39.2	2.5 ± 6.1	35.7 ± 35.7	68.2 ± 7.3	91.8 ± 21.7
pen-expert	85.1	146.0 ± 7.3	133.6 ± 16.0	107.0	45.4 ± 22.9	130.3 ± 4.2	122.8 ± 14.1	154.1 ± 5.4
door-human	0.5	-0.1 ± 0.0	3.1 ± 2.0	9.9	0.0 ± 0.0	3.7 ± 0.7	10.7 ± 6.8	0.0 ± 0.1
door-cloned	-0.1	0.1 ± 0.6	0.8 ± 1.0	0.4	0.2 ± 0.8	-0.1 ± 0.1	9.6 ± 8.3	1.1 ± 2.6
door-expert	34.9	84.6 ± 44.5	105.3 ± 2.8	101.5	73.6 ± 26.7	104.9 ± 0.9	-0.3 ± 0.1	104.6 ± 2.4
hammer-human	1.5	0.4 ± 0.4	2.5 ± 1.9	4.4	-0.1 ± 0.1	2.3 ± 2.3	0.8 ± 0.4	0.2 ± 0.2
hammer-cloned	0.8	0.8 ± 0.7	1.1 ± 0.5	2.1	0.1 ± 0.4	1.7 ± 1.7	0.3 ± 0.0	6.7 ± 3.7
hammer-expert	125.6	117.0 ± 30.9	129.6 ± 0.5	86.7	24.8 ± 39.4	132.2 ± 0.7	0.2 ± 0.0	133.8 ± 0.7
relocate-human	0.0	-0.2 ± 0.0	0.1 ± 0.1	0.2	0.0 ± 0.0	0.0 ± 0.0	0.1 ± 0.1	0.0 ± 0.0
relocate-cloned	-0.1	-0.1 ± 0.1	0.2 ± 0.4	-0.1	0.0 ± 0.0	0.0 ± 0.0	0.0 ± 0.0	0.9 ± 1.6
relocate-expert	101.3	107.3 ± 1.6	106.5 ± 2.5	95.0	3.4 ± 4.5	47.8 ± 13.5	-0.3 ± 0.0	106.6 ± 3.2
Average w/o expert	11.7	18.0	20.8	11.7	1.0	9.6	17.4	25.5
Average	36.7	49.9	53.4	40.3	12.9	41.0	21.9	58.6
Table 19: Average normalized score over the final evaluation and ten unseen training seeds on D4RL tasks for 2 types of normalization: LayerNorm (LN) and Feature Norm (FN). The symbol  $\pm$  represents the standard deviation across the seeds. For both variatns we tune hyperparameters using the same grid.
Task Name	ReBRAC + LN	ReBRAC + FN
halfcheetah-random	29.5 ± 1.5	31.4 ± 2.7
halfcheetah-medium	65.6 ± 1.0	66.1 ± 1.2
halfcheetah-expert	105.9 ± 1.7	104.1 ± 3.7
halfcheetah-medium-expert	101.1 ± 5.2	100.9 ± 4.7
halfcheetah-medium-replay	51.0 ± 0.8	54.7 ± 1.0
halfcheetah-full-replay	82.1 ± 1.1	81.5 ± 1.6
hopper-random	8.1 ± 2.4	8.2 ± 2.2
hopper-medium	102.0 ± 1.0	102.4 ± 0.2
hopper-expert	100.1 ± 8.3	99.7 ± 11.7
hopper-medium-expert	107.0 ± 6.4	107.7 ± 6.4
hopper-medium-replay	98.1 ± 5.3	91.0 ± 15.3
hopper-full-replay	107.1 ± 0.4	106.7 ± 0.5
walker2d-random	18.4 ± 4.5	0.0 ± 0.7
walker2d-medium	82.5 ± 3.6	81.8 ± 4.7
walker2d-expert	112.3 ± 0.2	110.7 ± 3.2
walker2d-medium-expert	111.6 ± 0.3	100.6 ± 34.3
walker2d-medium-replay	77.3 ± 7.9	80.2 ± 8.1
walker2d-full-replay	102.2 ± 1.7	101.2 ± 2.7
Gym-MuJoCo average	81.2	79.3
antmaze-umaze	97.8 ± 1.0	96.8 ± 1.6
antmaze-umaze-diverse	88.3 ± 13.0	88.5 ± 7.5
antmaze-medium-play	84.0 ± 4.2	84.1 ± 10.1
antmaze-medium-diverse	76.3 ± 13.5	75.6 ± 13.7
antmaze-large-play	60.4 ± 26.1	55.0 ± 30.0
antmaze-large-diverse	54.4 ± 25.1	66.4 ± 7.4
AntMaze average	76.8	77.7
pen-human	103.5 ± 14.1	107.0 ± 13.8
pen-cloned	91.8 ± 21.7	84.9 ± 20.1
pen-expert	154.1 ± 5.4	151.6 ± 4.7
door-human	0.0 ± 0.1	0.0 ± 0.0
door-cloned	1.1 ± 2.6	0.1 ± 0.1
door-expert	104.6 ± 2.4	105.1 ± 1.4
hammer-human	0.2 ± 0.2	0.2 ± 0.1
hammer-cloned	6.7 ± 3.7	10.1 ± 9.5
hammer-expert	133.8 ± 0.7	133.4 ± 1.5
relocate-human	0.0 ± 0.0	0.0 ± 0.0
relocate-cloned	0.9 ± 1.6	1.2 ± 1.9
relocate-expert	106.6 ± 3.2	108.7 ± 3.0
Adroit average w/o expert	25.5	25.4
Adroit average	58.6	58.5
D Feature Normalization
E Computational costs
Table 20: Computational costs for algorithms in Table 2.
Algorithm	Number of runs	Approximate hours per run
TD3+BC, tuning	360	0.3
IQL, tuning	1440	1.8
ReBRAC, tuning	1440	0.4
TD3+BC, eval	180	0.2
IQL, eval	180	1.8
SAC-RND, eval	180	1.8
ReBRAC, eval	180	0.4
Sum	3960	4032.0
Table 21: Computational costs for algorithms in Table 3 and Table 17.
Algorithm	Number of runs	Approximate hours per run
TD3+BC, tuning	96	0.5
IQL, tuning	480	2.1
ReBRAC, tuning	384	0.6
TD3+BC, eval	60	0.5
IQL, eval	60	2.0
SAC-RND, eval	60	2.9
MSG, eval	60	5.1
ReBRAC, eval	60	0.4
Sum	1260	1940.4
Table 22: Computational costs for algorithms in Table 4.
Algorithm	Number of runs	Approximate hours per run
TD3+BC, tuning	240	0.3
IQL, tuning	960	1.8
SAC-RND, tuning	1200	1.1
ReBRAC, tuning	960	0.3
TD3+BC, eval	120	0.2
IQL, eval	120	1.9
SAC-RND, eval	120	1.1
ReBRAC, eval	120	0.3
Sum	3840	3828.0
Table 23: Computational costs for algorithms in Table 5.
Algorithm	Number of runs	Approximate hours per run
ReBRAC, tuning	600	10.6
ReBRAC, eval	75	10.5
Sum	675	7147.5
Table 24: Computational costs for algorithms in Table 8 and Figure 2.
Algorithm	Number of runs	Approximate hours per run
ReBRAC, ablations eval	1104	1.4
Sum	1104	1545.6
F Expected Online Performance
(c) halfcheetah-expert EOP.
(a) halfcheetah-random EOP.
(d) halfcheetah-medium-expert EOP.
(b) halfcheetah-medium EOP.
(e) halfcheetah-medium-replay EOP.
(f) halfcheetah-full-replay EOP.
Figure 3: TD3+BC, IQL and ReBRAC visualised Expected Online Performance under uniform policy selection on HalfCheetah tasks.
(a) hopper-random EOP.
(d) hopper-medium-expert EOP.
Figure 4: TD3+BC, IQL and ReBRAC visualised Expected Online Performance under uniform policy selection on Hopper tasks.
(b) hopper-medium EOP.
(e) hopper-medium-replay EOP.
(c) hopper-expert EOP.
(f) hopper-full-replay EOP.
(a) walker2d-random EOP.
(d) walker2d-medium-expert EOP.
Figure 5: TD3+BC, IQL and ReBRAC visualised Expected Online Performance under uniform policy selection on Walker2d tasks.
(b) walker2d-medium EOP.
(e) walker2d-medium-replay EOP.
(c) walker2d-expert EOP.
(f) walker2d-full-replay EOP.
Table 28: TD3+BC, IQL and ReBRAC Expected Online Performance under uniform policy selection on AntMaze tasks.
umame	medium-play	large-play	umame-diverse	medium-diverse	large-diverse
Policies	T3D+BC	IQL	ReBRAC	T3D+BC	IQL	ReBRAC	T3D+BC	IQL	ReBRAC	T3D+BC	IQL	ReBRAC	T3D+BC	IQL	ReBRAC	T3D+BC	IQL	ReBRAC
1	12.4 ± 24.8	64.3 ± 12.0	87.5 ± 10.9	7.5 ± 11.7	22.7 ± 29.5	75.0 ± 14.8	0.0 ± 0.0	10.9 ± 16.2	52.7 ± 21.4	9.6 ± 19.2	52.8 ± 11.1	71.9 ± 13.8	21.3 ± 26.8	65.3 ± 26.3	0.2 ± 0.2	6.7 ± 10.3	56.8 ± 17.0
2	22.3 ± 29.8	71.1 ± 9.2	93.2 ± 6.2	13.0 ± 13.3	37.5 ± 30.9	82.7 ± 8.9	0.0 ± 0.0	18.6 ± 17.8	64.1 ± 13.0	17.3 ± 23.0	58.7 ± 11.3	79.3 ± 11.2	19.2 ± 14.0	35.0 ± 27.7	79.2 ± 16.0	0.3 ± 0.2	11.6 ± 11.6
3	30.3 ± 31.0	74.3 ± 7.5	95.1 ± 3.7	17.0 ± 13.2	47.4 ± 28.5	85.4 ± 3.7	0.0 ± 0.0	24.0 ± 16.9	68.5 ± 9.3	23.4 ± 24.0	62.5 ± 10.4	83.1 ± 8.4	23.8 ± 12.7	44.0 ± 25.1	84.2 ± 9.8	0.4 ± 0.2	15.2 ± 11.4
4	36.9 ± 30.8	76.2 ± 7.5	96.1 ± 4.8	20.0 ± 15.0	54.7 ± 22.7	86.7 ± 10.5	0.0 ± 0.0	27.9 ± 18.8	78.3 ± 23.3	28.3 ± 23.3	65.1 ± 11.2	86.5 ± 11.3	26.9 ± 11.1	44.8 ± 17.8	0.4 ± 0.0	17.9 ± 13.7	7.5 ± 10.2
5	41.7 ± 29.1	77.5 ± 5.9	96.4 ± 1.8	22.2 ± 11.7	58.7 ± 21.4	87.5 ± 3.5	0.0 ± 0.0	30.8 ± 13.6	72.2 ± 6.0	32.3 ± 22.5	67.0 ± 8.2	36.5 ± 9.9	29.0 ± 14.4	87.6 ± 4.9	0.5 ± 0.1	19.9 ± 9.7	73.5 ± 6.2
6	78.4 ± 14.4	86.7 ± 1.4	96.7 ± 1.4	-	61.5 ± 13.8	88.1 ± 3.2	-	32.8 ± 15.1	73.1 ± 4.9	68.4 ± 8.7	88.6 ± 5.7	-	-	88.6 ± 3.5	-	21.5 ± 14.1	74.5 ± 5.9
7	-	79.2 ± 4.9	96.9 ± 1.1	-	64.1 ± 15.3	88.3 ± 2.9	-	34.3 ± 10.3	73.7 ± 4.1	-	69.4 ± 6.4	88.4 ± 5.3	-	58.8 ± 12.9	88.9 ± 3.2	-	22.7 ± 8.0
8	-	79.8 ± 4.6	97.0 ± 0.9	-	65.7 ± 12.8	88.9 ± 2.7	-	35.5 ± 8.9	74.2 ± 3.5	-	70.2 ± 5.7	89.0 ± 4.9	-	60.1 ± 10.7	89.3 ± 2.8	-	23.7 ± 7.2
9	-	80.3 ± 1.1	97.1 ± 0.7	-	66.8 ± 10.4	89.2 ± 2.5	-	36.5 ± 7.9	74.5 ± 2.9	-	70.8 ± 10.1	89.5 ± 4.5	-	61.1 ± 11.0	89.6 ± 2.4	-	24.7 ± 7.2
10	-	80.7 ± 4.0	97.1 ± 0.6	-	67.6 ± 9.0	89.4 ± 2.4	-	37.0 ± 6.7	74.8 ± 2.5	-	71.3 ± 4.6	89.9 ± 4.2	-	61.8 ± 7.5	89.8 ± 2.1	-	25.1 ± 5.9
11	-	81.1 ± 3.8	97.2 ± 0.6	-	68.2 ± 7.5	89.6 ± 2.2	-	37.5 ± 5.9	75.0 ± 2.1	-	71.7 ± 4.2	90.3 ± 3.9	-	62.4 ± 6.3	90.8 ± 1.9	-	25.6 ± 5.4
12	-	81.4 ± 3.8	97.1 ± 0.5	-	69.8 ± 8.1	89.8 ± 2.1	-	37.0 ± 4.9	75.0 ± 2.8	-	72.0 ± 5.8	90.3 ± 3.6	-	63.0 ± 6.3	90.8 ± 1.6	-	26.6 ± 7.2
13	-	81.6 ± 3.3	97.3 ± 0.4	-	69.0 ± 5.2	90.8 ± 2.0	-	38.2 ± 4.6	75.2 ± 1.6	-	72.3 ± 3.5	90.9 ± 3.3	-	63.0 ± 9.3	90.3 ± 1.5	-	26.4 ± 4.5
14	-	81.9 ± 3.1	97.3 ± 0.4	-	69.2 ± 4.4	90.1 ± 1.9	-	38.5 ± 4.2	75.3 ± 1.4	-	72.6 ± 3.2	91.1 ± 3.1	-	63.3 ± 3.8	90.4 ± 1.3	-	26.7 ± 4.2
15	-	82.1 ± 3.1	97.3 ± 0.4	-	69.2 ± 3.9	90.2 ± 1.8	-	38.0 ± 3.9	75.4 ± 1.2	-	72.8 ± 3.0	91.3 ± 2.9	-	63.5 ± 3.7	90.4 ± 1.2	-	27.8 ± 5.5
16	-	82.2 ± 2.8	97.3 ± 0.3	-	69.5 ± 3.1	90.3 ± 1.7	-	39.0 ± 3.5	75.4 ± 1.1	-	72.9 ± 2.8	91.4 ± 2.7	-	63.6 ± 2.8	90.5 ± 1.1	-	27.2 ± 3.6
17	-	82.4 ± 2.7	-	-	69.6 ± 2.6	-	-	39.1 ± 3.3	-	-	73.1 ± 2.7	-	-	63.7 ± 2.4	-	27.4 ± 3.4
18	-	82.5 ± 2.5	-	-	69.8 ± 2.6	-	-	39.3 ± 3.1	-	-	73.2 ± 2.5	-	-	63.8 ± 2.1	-	27.4 ± 3.2
19	-	82.7 ± 2.4	-	-	69.8 ± 1.9	-	-	39.4 ± 2.9	-	-	73.4 ± 2.4	-	-	63.9 ± 1.8	-	27.7 ± 3.0
20	-	82.8 ± 2.3	-	-	69.8 ± 1.6	-	-	39.6 ± 2.8	-	-	73.5 ± 2.3	-	-	64.0 ± 1.6	-	27.9 ± 2.9
(b) antmaze-medium-play EOP.
(a) antimaze-umaze EOP.
(d) antimaze-umaze-diverse EOP.
Figure 6: TD3+BC, IQL and ReBRAC visualised Expected Online Performance under uniform policy selection on AntMaze tasks.
(e) antmaze-medium-diverse EOP.
(c) antimaze-large-play EOP.
(f) antmaze-large-diverse EOP.
Table 29: TD3+BC, IQL and ReBRAC Expected Online Performance under uniform policy selection on Pen tasks.
human	cloned	expert
Policies	TD3+BC	IQL	ReBRAC	TD3+BC	IQL	ReBRAC	TD3+BC	IQL	ReBRAC
1	42.9 ± 32.0	87.1 ± 4.1	69.9 ± 28.2	33.6 ± 23.2	73.8 ± 5.8	65.8 ± 32.7	73.9 ± 65.2	130.1 ± 2.8	136.9 ± 25.4
2	60.4 ± 25.4	89.4 ± 2.9	85.8 ± 20.7	44.9 ± 25.4	76.7 ± 4.8	83.6 ± 21.8	108.9 ± 52.9	131.7 ± 2.1	149.0 ± 13.6
3	68.7 ± 18.9	90.3 ± 2.5	92.8 ± 14.8	52.6 ± 25.2	78.0 ± 4.9	90.8 ± 15.4	125.6 ± 38.7	132.4 ± 1.8	152.4 ± 7.0
4	73.0 ± 14.2	90.9 ± 2.2	96.3 ± 10.9	58.1 ± 24.1	79.0 ± 5.2	94.6 ± 12.2	134.1 ± 27.8	132.8 ± 1.6	153.5 ± 3.8
5	75.6 ± 11.2	91.4 ± 2.0	98.4 ± 8.3	62.3 ± 22.6	79.7 ± 5.4	97.0 ± 10.4	138.5 ± 20.0	133.1 ± 1.5	154.1 ± 2.2
6	-	91.7 ± 1.8	99.7 ± 6.5	-	80.3 ± 5.5	98.7 ± 9.3	-	133.4 ± 1.4	154.3 ± 1.5
7	-	92.0 ± 1.7	100.5 ± 5.2	-	80.9 ± 5.6	100.0 ± 8.4	-	133.6 ± 1.3	154.5 ± 1.1
8	-	92.2 ± 1.6	101.1 ± 4.4	-	81.4 ± 5.7	101.1 ± 7.6	-	133.7 ± 1.2	154.7 ± 0.9
9	-	92.3 ± 1.4	101.6 ± 3.7	-	81.9 ± 5.8	101.9 ± 7.0	-	133.9 ± 1.1	154.8 ± 0.8
10	-	92.5 ± 1.3	101.9 ± 3.3	-	82.3 ± 5.9	102.6 ± 6.4	-	134.0 ± 1.0	154.8 ± 0.7
11	-	92.6 ± 1.2	102.2 ± 2.9	-	82.7 ± 5.9	103.1 ± 5.9	-	134.1 ± 0.9	154.9 ± 0.6
12	-	92.7 ± 1.2	102.4 ± 2.6	-	83.0 ± 5.9	103.6 ± 5.4	-	134.1 ± 0.9	154.9 ± 0.6
13	-	92.8 ± 1.1	102.6 ± 2.4	-	83.4 ± 5.9	104.0 ± 5.0	-	134.2 ± 0.8	155.0 ± 0.5
14	-	92.9 ± 1.0	102.8 ± 2.2	-	83.7 ± 5.9	104.4 ± 4.6	-	134.2 ± 0.7	155.0 ± 0.5
15	-	92.9 ± 1.0	102.9 ± 2.0	-	84.0 ± 5.8	104.7 ± 4.3	-	134.3 ± 0.7	155.0 ± 0.4
16	-	93.0 ± 0.9	103.0 ± 1.8	-	84.3 ± 5.8	104.9 ± 4.0	-	134.3 ± 0.6	155.1 ± 0.4
17	-	93.0 ± 0.9	103.1 ± 1.7	-	84.6 ± 5.7	105.1 ± 3.8	-	134.4 ± 0.6	155.1 ± 0.4
18	-	93.1 ± 0.8	103.2 ± 1.6	-	84.8 ± 5.7	105.3 ± 3.5	-	134.4 ± 0.6	155.1 ± 0.4
19	-	93.1 ± 0.8	103.3 ± 1.4	-	85.0 ± 5.6	105.5 ± 3.3	-	134.4 ± 0.5	155.1 ± 0.4
20	-	93.2 ± 0.8	103.3 ± 1.3	-	85.3 ± 5.5	105.7 ± 3.1	-	134.5 ± 0.5	155.1 ± 0.4
(a) pen-human EOP.
(b) pen-cloned EOP.
Figure 7: TD3+BC, IQL and ReBRAC visualised Expected Online Performance under uniform policy selection on Pen tasks.
(c) pen-expert EOP.
Table 30: TD3+BC, IQL and ReBRAC Expected Online Performance under uniform policy selection on Door tasks.
human	cloned	expert
Policies	TD3+BC	IQL	ReBRAC	TD3+BC	IQL	ReBRAC	TD3+BC	IQL	ReBRAC
1	-0.2 ± 0.1	4.4 ± 1.2	-0.1 ± 0.1	-0.1 ± 0.3	1.6 ± 0.8	0.3 ± 0.9	50.7 ± 46.3	102.1 ± 5.7	75.4 ± 43.0
2	-0.1 ± 0.1	5.0 ± 1.0	-0.0 ± 0.1	0.0 ± 0.3	2.0 ± 0.6	0.6 ± 1.2	75.6 ± 39.5	104.7 ± 2.5	96.1 ± 25.0
3	-0.1 ± 0.1	5.4 ± 1.0	-0.0 ± 0.0	0.1 ± 0.3	2.2 ± 0.5	0.8 ± 1.4	88.2 ± 30.4	105.4 ± 1.3	102.3 ± 13.5
4	-0.1 ± 0.1	5.6 ± 0.9	-0.0 ± 0.0	0.2 ± 0.3	2.4 ± 0.4	1.0 ± 1.6	94.9 ± 22.9	105.7 ± 0.8	104.4 ± 7.3
5	-0.1 ± 0.1	5.8 ± 0.9	-0.0 ± 0.0	0.2 ± 0.3	2.5 ± 0.4	1.2 ± 1.7	98.6 ± 17.2	105.8 ± 0.6	105.2 ± 4.1
6	-	5.9 ± 0.9	0.0 ± 0.0	-	2.5 ± 0.3	1.3 ± 1.8	-	105.9 ± 0.5	105.6 ± 2.4
7	-	6.0 ± 0.9	0.0 ± 0.0	-	2.6 ± 0.3	1.5 ± 1.8	-	106.0 ± 0.5	105.8 ± 1.5
8	-	6.1 ± 0.8	0.0 ± 0.0	-	2.6 ± 0.3	1.6 ± 1.9	-	106.1 ± 0.4	105.9 ± 1.0
9	-	6.2 ± 0.8	0.0 ± 0.0	-	2.6 ± 0.2	1.8 ± 1.9	-	106.1 ± 0.4	105.9 ± 0.7
10	-	6.3 ± 0.8	0.0 ± 0.0	-	2.7 ± 0.2	1.9 ± 1.9	-	106.1 ± 0.4	106.0 ± 0.5
11	-	6.4 ± 0.8	0.0 ± 0.0	-	2.7 ± 0.2	2.0 ± 2.0	-	106.2 ± 0.4	106.0 ± 0.4
12	-	6.4 ± 0.7	0.0 ± 0.0	-	2.7 ± 0.2	2.2 ± 2.0	-	106.2 ± 0.3	106.0 ± 0.3
13	-	6.5 ± 0.7	0.0 ± 0.0	-	2.7 ± 0.2	2.3 ± 2.0	-	106.2 ± 0.3	106.0 ± 0.2
14	-	6.5 ± 0.7	0.0 ± 0.0	-	2.7 ± 0.2	2.4 ± 2.0	-	106.2 ± 0.3	106.0 ± 0.2
15	-	6.6 ± 0.7	0.0 ± 0.0	-	2.7 ± 0.2	2.5 ± 2.0	-	106.3 ± 0.3	106.0 ± 0.2
16	-	6.6 ± 0.6	0.0 ± 0.0	-	2.7 ± 0.2	2.6 ± 1.9	-	106.3 ± 0.3	106.1 ± 0.2
17	-	6.6 ± 0.6	0.0 ± 0.0	-	2.8 ± 0.2	2.7 ± 1.9	-	106.3 ± 0.3	106.1 ± 0.2
18	-	6.7 ± 0.6	0.0 ± 0.0	-	2.8 ± 0.2	2.7 ± 1.9	-	106.3 ± 0.3	106.1 ± 0.1
19	-	6.7 ± 0.6	0.0 ± 0.0	-	2.8 ± 0.2	2.8 ± 1.9	-	106.3 ± 0.3	106.1 ± 0.1
20	-	6.7 ± 0.5	0.0 ± 0.0	-	2.8 ± 0.2	2.9 ± 1.9	-	106.3 ± 0.2	106.1 ± 0.1
(a) door-human EOP.
Figure 8: TD3+BC, IQL and ReBRAC visualised Expected Online Performance under uniform policy selection on Door tasks.
(b) door-cloned EOP.
(c) door-expert EOP.
Table 31: TD3+BC, IQL and ReBRAC Expected Online Performance under uniform policy selection on Hammer tasks.
human	cloned	expert
Policies	TD3+BC	IQL	ReBRAC	TD3+BC	IQL	ReBRAC	TD3+BC	IQL	ReBRAC
1	0.2 ± 0.0	1.6 ± 0.4	0.3 ± 0.2	1.1 ± 0.8	1.9 ± 1.1	4.5 ± 5.5	60.2 ± 55.4	128.7 ± 1.1	101.9 ± 49.9
2	0.3 ± 0.0	1.8 ± 0.4	0.3 ± 0.2	1.6 ± 0.8	2.5 ± 1.1	7.3 ± 6.1	90.0 ± 48.4	129.3 ± 0.9	124.3 ± 26.8
3	0.3 ± 0.0	1.9 ± 0.4	0.4 ± 0.2	1.8 ± 0.7	2.8 ± 1.0	9.2 ± 6.2	105.7 ± 38.1	129.6 ± 0.7	130.1 ± 13.6
4	0.3 ± 0.0	2.0 ± 0.4	0.4 ± 0.2	2.0 ± 0.6	3.1 ± 0.9	10.6 ± 6.2	114.3 ± 29.3	129.7 ± 0.5	131.9 ± 7.0
5	0.3 ± 0.0	2.1 ± 0.4	0.5 ± 0.2	2.1 ± 0.6	3.3 ± 0.9	11.7 ± 6.2	119.2 ± 22.4	129.8 ± 0.4	132.6 ± 3.8
6	-	2.1 ± 0.4	0.5 ± 0.2	-	3.4 ± 0.8	12.6 ± 6.1	-	129.9 ± 0.3	133.0 ± 2.4
7	-	2.2 ± 0.4	0.5 ± 0.2	-	3.5 ± 0.7	13.3 ± 6.0	-	129.9 ± 0.2	133.2 ± 1.7
8	-	2.2 ± 0.4	0.5 ± 0.2	-	3.6 ± 0.7	14.0 ± 5.9	-	129.9 ± 0.2	133.4 ± 1.4
9	-	2.3 ± 0.3	0.6 ± 0.2	-	3.7 ± 0.6	14.6 ± 5.8	-	130.0 ± 0.1	133.5 ± 1.2
10	-	2.3 ± 0.3	0.6 ± 0.2	-	3.7 ± 0.6	15.1 ± 5.7	-	130.0 ± 0.1	133.6 ± 1.1
11	-	2.3 ± 0.3	0.6 ± 0.1	-	3.8 ± 0.5	15.5 ± 5.6	-	130.0 ± 0.1	133.7 ± 1.0
12	-	2.3 ± 0.3	0.6 ± 0.1	-	3.8 ± 0.5	16.0 ± 5.4	-	130.0 ± 0.1	133.8 ± 0.9
13	-	2.4 ± 0.3	0.6 ± 0.1	-	3.9 ± 0.5	16.3 ± 5.3	-	130.0 ± 0.1	133.9 ± 0.8
14	-	2.4 ± 0.3	0.6 ± 0.1	-	3.9 ± 0.4	16.7 ± 5.2	-	130.0 ± 0.1	133.9 ± 0.7
15	-	2.4 ± 0.3	0.6 ± 0.1	-	3.9 ± 0.4	17.0 ± 5.1	-	130.0 ± 0.1	134.0 ± 0.6
16	-	2.4 ± 0.3	0.6 ± 0.1	-	4.0 ± 0.4	17.3 ± 4.9	-	130.0 ± 0.1	134.0 ± 0.6
17	-	2.4 ± 0.2	0.6 ± 0.1	-	4.0 ± 0.4	17.5 ± 4.8	-	130.0 ± 0.0	134.0 ± 0.5
18	-	2.4 ± 0.2	0.6 ± 0.1	-	4.0 ± 0.3	17.8 ± 4.7	-	130.0 ± 0.0	134.0 ± 0.5
19	-	2.5 ± 0.2	0.6 ± 0.1	-	4.0 ± 0.3	18.0 ± 4.6	-	130.0 ± 0.0	134.1 ± 0.4
20	-	2.5 ± 0.2	0.7 ± 0.1	-	4.0 ± 0.3	18.2 ± 4.5	-	130.0 ± 0.0	134.1 ± 0.4
(a) hammer-human EOP.
(b) hammer-cloned EOP.
Figure 9: TD3+BC, IQL and ReBRAC visualised Expected Online Performance under uniform policy selection on Hammer tasks.
(c) hammer-expert EOP.
Table 32: TD3+BC, IQL and ReBRAC Expected Online Performance under uniform policy selection on tasks.
human	cloned	expert
Policies	TD3+BC	IQL	ReBRAC	TD3+BC	IQL	ReBRAC	TD3+BC	IQL	ReBRAC
1	-0.2 ± 0.1	0.2 ± 0.2	-0.1 ± 0.1	-0.2 ± 0.1	-0.0 ± 0.1	0.5 ± 0.8	21.4 ± 43.2	106.0 ± 1.4	73.5 ± 44.3
2	-0.2 ± 0.1	0.2 ± 0.2	-0.1 ± 0.1	-0.2 ± 0.1	0.0 ± 0.1	0.9 ± 0.9	38.8 ± 51.9	106.8 ± 1.0	96.0 ± 27.4
3	-0.1 ± 0.1	0.3 ± 0.2	-0.0 ± 0.0	-0.2 ± 0.1	0.1 ± 0.1	1.2 ± 0.9	52.6 ± 54.0	107.2 ± 0.8	103.7 ± 15.9
4	-0.1 ± 0.1	0.3 ± 0.2	-0.0 ± 0.0	-0.1 ± 0.1	0.1 ± 0.1	1.4 ± 0.9	63.7 ± 53.1	107.4 ± 0.7	106.7 ± 9.4
5	-0.1 ± 0.1	0.4 ± 0.2	-0.0 ± 0.0	-0.1 ± 0.0	0.1 ± 0.1	1.6 ± 0.9	72.5 ± 50.7	107.5 ± 0.6	107.9 ± 5.8
6	-	0.4 ± 0.2	-0.0 ± 0.0	-	0.1 ± 0.1	1.7 ± 0.8	-	107.6 ± 0.5	108.6 ± 3.8
7	-	0.4 ± 0.2	-0.0 ± 0.0	-	0.1 ± 0.1	1.8 ± 0.8	-	107.7 ± 0.5	108.9 ± 2.6
8	-	0.4 ± 0.2	0.0 ± 0.0	-	0.1 ± 0.1	1.9 ± 0.8	-	107.7 ± 0.5	109.2 ± 1.9
9	-	0.5 ± 0.2	0.0 ± 0.0	-	0.1 ± 0.1	2.0 ± 0.7	-	107.8 ± 0.4	109.3 ± 1.5
10	-	0.5 ± 0.2	0.0 ± 0.0	-	0.1 ± 0.1	2.1 ± 0.7	-	107.8 ± 0.4	109.4 ± 1.3
11	-	0.5 ± 0.2	0.0 ± 0.0	-	0.2 ± 0.1	2.1 ± 0.7	-	107.8 ± 0.4	109.5 ± 1.1
12	-	0.5 ± 0.2	0.0 ± 0.0	-	0.2 ± 0.1	2.2 ± 0.6	-	107.9 ± 0.4	109.6 ± 1.0
13	-	0.5 ± 0.2	0.0 ± 0.0	-	0.2 ± 0.1	2.2 ± 0.6	-	107.9 ± 0.4	109.7 ± 0.9
14	-	0.5 ± 0.2	0.0 ± 0.0	-	0.2 ± 0.1	2.3 ± 0.6	-	107.9 ± 0.4	109.8 ± 0.9
15	-	0.5 ± 0.2	0.0 ± 0.0	-	0.2 ± 0.1	2.3 ± 0.6	-	107.9 ± 0.4	109.8 ± 0.8
16	-	0.5 ± 0.2	0.0 ± 0.0	-	0.2 ± 0.1	2.3 ± 0.5	-	108.0 ± 0.4	109.9 ± 0.8
17	-	0.6 ± 0.2	0.0 ± 0.0	-	0.2 ± 0.1	2.4 ± 0.5	-	108.0 ± 0.4	109.9 ± 0.8
18	-	0.6 ± 0.2	0.0 ± 0.0	-	0.2 ± 0.1	2.4 ± 0.5	-	108.0 ± 0.3	109.9 ± 0.8
19	-	0.6 ± 0.2	0.0 ± 0.0	-	0.2 ± 0.1	2.4 ± 0.5	-	108.0 ± 0.3	110.0 ± 0.7
20	-	0.6 ± 0.1	0.0 ± 0.0	-	0.2 ± 0.1	2.5 ± 0.4	-	108.0 ± 0.3	110.0 ± 0.7
(a) relocate-human EOP.
(b) relocate-cloned EOP.
Figure 10: TD3+BC, IQL and ReBRAC visualised Expected Online Performance under uniform policy selection on Relocate tasks.
(c) relocate-expert EOP.
G D4RL tasks ablation
Table 33: ReBRAC ablations for halfcheetah tasks. We report final normalized score averaged over 4 unseen training seeds.
Ablation	random	medium	expert	medium-expert	medium-replay	full-replay	Average
TD3+BC, paper	11.0 ± 1.1	48.3 ± 0.3	96.7 ± 1.1	90.7 ± 4.3	44.6 ± 0.5	-	-
TD3+BC, our	2.2 ± 0.0	44.6 ± 0.4	93.8 ± 0.1	91.9 ± 2.3	40.5 ± 1.6	69.3 ± 0.7	57.0
TD3+BC, tuned	30.1 ± 1.4 (+0.7%)	55.4 ± 0.9 (-15.2%)	95.5 ± 0.5 (-9.6%)	91.9 ± 2.3 (-11.4%)	45.1 ± 1.7 (-11.0%)	74.1 ± 2.9 (-9.5%)	65.3 (-10.3%)
ReBRAC w/o LN	32.0 ± 1.5 (+7.0%)	64.3 ± 4.1 (-1.5%)	61.7 ± 20.4 (-41.5%)	86.7 ± 0.9 (-16.3%)	52.8 ± 2.4 (+4.1%)	82.1 ± 2.1 (+0.2%)	63.2 (-13.1%)
ReBRAC w/o layer	27.8 ± 3.4 (-7.0%)	65.0 ± 1.6 (-0.4%)	74.4 ± 26.7 (-29.5%)	86.7 ± 8.8 (-16.3%)	50.4 ± 0.7 (-0.5%)	80.9 ± 1.1 (-1.2%)	64.1 (-11.9%)
ReBRAC w/o actor penalty	31.8 ± 4.1 (+6.3%)	64.5 ± 0.7 (-1.2%)	4.3 ± 4.3 (-95.9%)	71.6 ± 12.3 (-30.9%)	38.0 ± 27.2 (-25.0%)	59.3 ± 41.3 (-27.5%)	44.9 (-38.3%)
ReBRAC w/o critic penalty	28.1 ± 1.6 (-6.0%)	65.7 ± 1.4 (+0.6%)	104.2 ± 5.9 (-1.3%)	100.5 ± 3.1 (-3.0%)	50.7 ± 0.1 (0.0%)	81.7 ± 1.2 (-0.2%)	71.8 (-1.3%)
ReBRAC w/o large batch	21.0 ± 15.7 (-29.7%)	65.8 ± 0.7 (+0.7%)	62.6 ± 24.3 (-40.7%)	85.2 ± 7.3 (-17.8%)	50.7 ± 1.1 (0.0%)	81.9 ± 1.4 (0.0%)	61.2 (-15.9%)
ReBRAC	29.9 ± 1.2	65.3 ± 1.1	105.6 ± 1.5	103.7 ± 3.9	50.7 ± 0.6	81.9 ± 1.4	72.8
Table 34: ReBRAC ablations for hopper tasks. We report final normalized score averaged over 4 unseen training seeds.
Ablation	random	medium	expert	medium-expert	medium-replay	full-replay	Average
TD3+BC, paper	8.5 ± 0.6	59.3 ± 4.2	107.8 ± 7.0	98.0 ± 9.4	60.9 ± 18.8	-	-
TD3+BC, our	10.3 ± 1.8	53.2 ± 2.2	108.7 ± 5.3	75.8 ± 8.9	64.5 ± 24.9	49.9 ± 9.6	60.4
TD3+BC, tuned	10.3 ± 1.8 (+51.5%)	57.6 ± 6.8 (-43.2%)	110.7 ± 2.1 (+21.1%)	106.2 ± 2.5 (-3.4%)	64.5 ± 24.9 (-30.8%)	106.2 ± 2.2 (-0.6%)	75.9 (-10.6%)
ReBRAC w/o LN	12.2 ± 13.3 (+79.4%)	1.0 ± 0.6 (-99.0%)	111.1 ± 1.0 (+21.6%)	112.4 ± 0.7 (+2.3%)	57.4 ± 25.0 (-38.5%)	107.2 ± 2.0 (+0.4%)	66.8 (-21.3%)
ReBRAC w/o layer	8.8 ± 0.6 (+29.4%)	101.8 ± 0.2 (+0.4%)	103.7 ± 5.1 (+13.5%)	104.1 ± 7.7 (-5.3%)	97.5 ± 3.5 (+4.5%)	106.5 ± 0.3 (-0.3%)	87.0 (+2.5%)
ReBRAC w/o actor penalty	7.5 ± 4.6 (+10.3%)	1.7 ± 1.2 (-98.3%)	1.1 ± 0.5 (-98.8%)	1.6 ± 1.6 (-98.5%)	24.4 ± 8.7 (-73.8%)	27.7 ± 23.4 (-74.1%)	10.6 (-87.5%)
ReBRAC w/o critic penalty	7.4 ± 1.1 (+8.8%)	102.3 ± 0.5 (+0.9%)	103.4 ± 8.6 (+13.1%)	111.2 ± 0.7 (+1.2%)	83.1 ± 30.9 (-10.9%)	107.5 ± 0.1 (+0.7%)	85.8 (+1.1%)
ReBRAC w/o large batch	8.6 ± 0.5 (+26.5%)	98.9 ± 5.2 (-2.5%)	98.8 ± 13.4 (+8.1%)	107.8 ± 2.9 (-1.9%)	96.2 ± 7.6 (+3.1%)	106.6 ± 0.2 (-0.2%)	86.1 (+1.4%)
ReBRAC	6.8 ± 3.4	101.4 ± 1.5	91.4 ± 4.7	109.9 ± 3.0	93.3 ± 7.5	106.8±0.6	84.9