SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Algorithm 4 DROP: Online fine-tuning (embedding-level)
Require: Env, last  $T$  checkpoint models:  $\beta_{t}(\mathbf{a}|\mathbf{s},\mathbf{z})$  and  $f_{t}(\mathbf{s},\mathbf{a},\mathbf{z})(t = 1,\dots ,T)$    
1:  $R_{\mathrm{MAX}} = -\infty$    
2:  $\beta_{\mathrm{best}}\gets$  None.   
3:  $f_{\mathrm{best}}\gets$  None.   
4:  $k_{\mathrm{best}}\gets 0$    
5: while  $t = 1,\dots ,T$  do   
6: while  $k = 1,\dots ,K_{\mathrm{max}}$  do   
7:  $\mathbf{s}_0 = \mathrm{Env}\cdot \mathrm{Reset}()$  # Conduct embedding inference with DROP-Grad or DROP-Grad-Ada   
8: Return  $\leftarrow$  Evaluate  $\beta_{t}$  and  $f_{t}$  on Env, setting  $\pi^{*}(\mathbf{a}|\mathbf{s}) = \beta (\mathbf{a}|\mathbf{s},\mathbf{z}^{*}(\mathbf{s}_{0}))$  or  $\beta (\mathbf{a}|\mathbf{s},\mathbf{z}^{*}(\mathbf{s}))$  , where we conduct  $k$  gradient ascent steps to obtain  $\mathbf{z}^{*}(\mathbf{s}_{0})$  or  $\mathbf{z}^{*}(\mathbf{s})$    
9: if  $R_{\mathrm{MAX}} <   \mathrm{Return}$  then   
10: Update the best checkpoint models:  $\beta_{\mathrm{best}}\gets \beta_t,f_{\mathrm{best}}\gets f_t$    
11: Update the best gradient update step:  $k_{\mathrm{best}}\gets k$    
12: Update the optimal return:  $R_{\mathrm{MAX}}\gets$  Return.   
13: end if   
14: end while   
15: end while   
Return:  $\beta_{\mathrm{best}},f_{\mathrm{best}}$  and  $k_{\mathrm{best}}$
Online fine-tuning (embedding-level): The embedding-level fine-tuning aims to find a suitable gradient ascent step that is used to conduct the embedding inference in DROP-Grad or DROP-Grad-Ada. Thus, we enumerate a list of gradient update steps and pick the best update step (according to the episode returns).
Codebase. Our code is based on d3rlpy: https://github.com/takuseno/d3rlpy. We provide our source code in the supplementary material.
Computational resources. The experiments were run on a computational cluster with 22x GeForce RTX 2080 Ti, and 4x NVIDIA Tesla V100 32GB for 20 days.
Comparison with iterative offline RL baselines. Here, we compare the performance of DROP (Grad, Best-Ada, and Grad-Ada) to iterative offline RL baselines (BEAR [33], BCQ [20], CQL [35], BRAC-p [67], and TD3+BC [19]) that perform iterative bi-level offline RL paradigm with (explicit or implicit) value/policy regularization in inner-level. In Table 10, we present the results for AntMaze, Gym-MuJoCo, and Adroit suites in standard D4RL benchmark (*-v0), where we can find that DROP-Grad-Ada performs comparably or surpasses prior iterative bi-level works on most tasks: outperforming (or comparing) these policy regularized methods (BRAC-p and TD3+BC) on 25 out of 33 tasks and outperforming (or comparing) these value regularized algorithms (BEAR, BCQ, and CQL) on 19 out of 33 tasks.
Table 9: The number  $(N)$  of sub-tasks, the number  $(M)$  of trajectories in each sub-task, and the dimension  $(\dim(\mathbf{z}))$  of the embedding for each sub-task.
Domain	Task Name	Parameters (*-v0)	Parameters (*-v2)
N	M	dim(z)	N	M	dim(z)
Maze 2D	umaze	500	5	5
medium	150	50	5
large	100	15	5
Antmaze	umaze	500	50	5	150	50	5
umaze-diverse	500	50	5	150	50	5
Medium-play	500	50	5	150	50	5
Medium-diverse	500	50	5	150	50	5
Large-play	500	50	5	150	50	5
Large-diverse	500	50	5	150	50	5
halfcheetah	random	1000	1	5	1000	1	5
medium	100	2	5	100	2	5
medium-expert	1000	1	5	1000	1	5
medium-replay	50	10	5	50	10	5
hopper	random	100	2	5	100	2	5
medium	100	5	5	100	5	5
medium-expert	100	2	5	100	2	5
medium-replay	50	5	5	10	30	5
walker2d	random	500	2	5	500	2	5
medium	50	5	5	50	5	5
medium-expert	50	5	5	50	5	5
medium-replay	1000	5	5	10	50	5
door	cloned	1000	2	5
expert	500	5	5
human	50	3	5
hammer	cloned	1000	1	5
expert	500	5	5
human	20	3	5
pen	cloned	500	5	5
expert	500	5	5
human	50	5	5
relocate	cloned	500	5	5
expert	500	5	5
human	50	4	5
Ablation studies. In Figure 11, we provide more results for the ablation of the conservative regularization term in Equation 8 in the main paper. We can find that for the halfcheetah-medium and hopper-medium tasks, the performance of DROP-Grad-Ada w/o Reg depends on the choice of the gradient update steps, showing that too small or too large number of gradient update step deteriorates the performance. Such a result is also consistent with COMs [62], which also observes the sensitivity of naive gradient update (i.e., w/o Reg) to the number of update steps used for design input inference. By comparison, the conservative score model learned with DROP-Grad-Ada exhibits more stable and robust performance to the gradient update steps.
Further, we also find that in walker2d-medium and walker2d-medium-expert tasks, the naive gradient update  $(w / o Reg)$  does not affect performance significantly across a wide range of gradient update steps. The main reason is that although the excessive gradient updates lead to faraway embeddings, conditioned on the inferred embeddings, the learned contextual behavior policy can safeguard against the embeddings distribution shift. Compared to prior model-based optimization that conducts direct gradient optimization (inference) over the design input itself, such "self-safeguard" is a special merit in the offline RL domain as long as we reframe the offline RL problem as one of model-based optimization and conduct inference over the embedding space. Thus, we encourage the research community to pursue further into this model-based optimization view for the offline RL problem.
Figure 11: The performance comparison of DROP-Grad-Ada and DROP-Grad-Ada w/o Reg, where we ablate the conservative regularization for the  $w / o$  Reg implementation. The y-axis denotes the normalized return, the x-axis denotes the number of gradient-ascent steps used for embedding inference at deployment.
Table 10: Comparison of our method to prior offline methods that perform iterative (regularized) RL paradigm on D4RL. We take the baseline results of BEAR, BCQ, CQL, and BRAC-p from Fu et al. [18], and the results of TD3-BC from Fujimoto and Gu [19]. For all results of our method (DROP), we average the normalized returns across 5 seeds; for each seed, we run 10 evaluation episodes. For proper comparison, we use  $\triangle$  and  $\triangle$  to denote DROP (*-Ada) achieves comparable or better performance compared with value and policy regularized offline RL methods respectively.
Task Name	Value Reg.	Policy Reg.	DROP-
BEAR	BCQ	CQL	BRAC-p	TD3+BC	Grad	Best-Ada	Grad-Ada
antmaze	umaze	73.0	78.9	74.0	50.0	-	72.0	78.0▲▲	80.0▲▲
umaze-diverse	61.0	55.0	84.0	40.0	-	48.0	62.0▲	66.0▲
medium-play	0.0	0.0	61.2	0.0	-	24.0	34.0▲	30.0▲
medium-diverse	8.0	0.0	53.7	0.0	-	20.0	24.0▲	30.0▲
large-play	0.0	6.7	15.8	0.0	-	24.0	36.0▲▲	42.0▲▲
large-diverse	0.0	2.2	14.9	0.0	-	14.0	20.0▲▲	26.0▲▲
halfcheetah	random	25.1	2.2	35.4	24.1	10.2	2.3	2.3	2.3
medium	41.7	40.7	44.4	43.8	42.8	42.4	42.9▲▲	43.1▲▲
medium-expert	53.4	64.7	62.4	44.2	97.9	86.6	88.5▲	88.9▲
medium-replay	38.6	38.2	46.2	45.4	43.3	39.5	40.4	40.3
hopper	random	11.4	10.6	10.8	11.0	11.0	5.1	5.4	5.5
medium	52.1	54.5	58.0	32.7	99.5	57.5	60.3▲	59.5▲
medium-expert	96.3	110.9	98.7	1.9	112.2	103.5	102.5	105.9▲
medium-replay	33.7	33.1	48.6	0.6	31.4	48.0	83.4▲▲	87.4▲▲
walker2d	random	7.3	4.9	7.0	-0.2	1.4	2.8	3.0▲	3.0▲
medium	59.1	53.1	79.2	77.5	79.7	76.5	75.8▲▲	79.1▲▲
medium-expert	40.1	57.5	111.0	76.9	101.1	107.5	106.8▲▲	106.9▲▲
medium-replay	19.2	15.0	26.7	-0.3	25.2	37.4	60.9▲▲	61.9▲▲
door	cloned	-0.1	0.0	0.4	-0.1	-	0.5	2.5	2.7▲▲
expert	103.4	99.0	101.5	-0.3	-	98.6	102.2▲▲	102.6▲▲
human	-0.3	0.0	9.9	-0.3	-	3.3	1.9▲	3.0▲
hammer	cloned	0.3	0.4	2.1	0.3	-	0.3	0.3▲	0.3▲
expert	127.3	107.2	86.7	0.3	-	65.7	73.3▲	77.7▲
human	0.3	0.5	4.4	0.3	-	1.1	0.3	2.1▲
pen	cloned	26.5	44.0	39.2	1.6	-	76.7	77.1▲▲	82.4▲▲
expert	105.9	114.9	107.0	-3.5	-	113.1	118.6▲▲	116.7▲▲
human	-1.0	68.9	37.5	8.1	-	71.1	85.2▲▲	81.5▲▲
relocate	cloned	-0.3	-0.3	-0.1	-0.3	-	0.1	0.5▲▲	0.2▲▲
expert	98.6	41.6	95.0	-0.3	-	2.5	6.2▲	5.4▲
human	-0.3	-0.1	0.2	-0.3	-	0.0	0.0	0.0