SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In test-time, we thus can dynamically adapt the outer-level optimization, setting the inferred optimal policy  $\pi^{*}(\mathbf{a}|\mathbf{s}) = \beta (\mathbf{a}|\mathbf{s},\mathbf{z}^{*}(\mathbf{s}))$ , where  $z^{*}(\mathbf{s}) = \arg \max_{z}f(\mathbf{s},\beta (\mathbf{a}|\mathbf{s},\mathbf{z}),\mathbf{z})$ . Specifically, at any state  $\mathbf{s}$  in the test-time, we can perform gradient ascent to find the optimal behavior embedding:  $z^{*}(\mathbf{s}) = \mathbf{z}_{K}^{\circ}(\mathbf{s}) := \mathrm{GradAscent}(\mathbf{s},\mathbf{z}_{0}^{\circ},K)$ , where  $\mathbf{z}_0^\circ$  is the starting point and for  $k = 0,1\dots ,K - 1$
$$
\mathbf {z} _ {k + 1} ^ {\circ} (\mathbf {s}) \leftarrow \mathbf {z} _ {k} ^ {\circ} (\mathbf {s}) + \eta \nabla_ {\mathbf {z}} f (\mathbf {s}, \beta (\mathbf {a} | \mathbf {s}, \mathbf {z}), \mathbf {z})) | _ {\mathbf {z} = \mathbf {z} _ {k} ^ {\circ}}. \tag {9}
$$
3.4 Connection to Prior Non-iterative Offline Methods
In Table 1, we summarize the comparison with prior representative non-iterative offline RL methods. Intuitively, our DROP (using returns to decompose  $\mathcal{D}$ ) is similar in spirit to F-BC $^6$  and RvS-R [14],
$^{6}$ F-BC performs BC over filtered trajectories with high returns.
Table 1: Comparison of non-iterative bi-level offline RL methods, where we use  $\mathsf{R}(\tau)$  to represent the return of a sampling trajectory  $\tau$  and use  $\mathsf{R}(\mathbf{s},\mathbf{a})$  to represent the expected return of trajectories starting from  $(\mathbf{s},\mathbf{a})$ . The checkmark in A2 indicates whether the exploitation to "?" is regularized for outer-level optimization and that in A3 indicates whether test-time adaptation is supported.
Inner-level	Outer-level	"[?" in A1	A2	A3
F-BC	BC over filtered τ with high R(τ)	—	π(a|s)	×	×
RvS-R [14]	minπ - E[log π(a|s, R(τ))]	handcraft Rtarget	π(a|s, ·)	×	×
Onestep [6]	minQ L(Q(s, a), R(s, a))	arg maxa Qβ(s, β(a|s))	Qβ(s, a)	×	✓
COMs [62]	minf L(f(βτ), R(τ))	arg maxβ f(β)	f(β)	✓	×
DROP (ours)	minf L(f(s, a, z), R(s, a, z))	arg maxz f(s, β(a|z), z)	f(s, a, z)	✓	✓
both of which use return  $\mathsf{R}(\tau)$  to guide the inner-level optimization. However, both F-BC and RvS-R leave q2 unanswered. In outer-level, F-BC can not enable policy extrapolation, which heavily relies on the data quality in offline tasks. RvS-R needs to handcraft a target return (as the contextual variable for  $\pi (\mathbf{a}|\mathbf{s},\cdot))$  , which also probably triggers the potential distribution mismatch between the hand-crafted contextual variable (i.e., the desired return) and the actual rollout return induced by the learned policy when conditioning on the given contextual variable.
Diving deeper into the bi-level optimization, we can also find DROP combines the advantages of Onestep [6] and COMs [62], where Onestep answers Q3 and performs outer-level optimization in action space (arg max $_{\mathbf{a}}$ ), COMs answers Q2 and performs that in parameter space (arg max $_{\beta}$ ), while our DROP answers both Q2 and Q3 and performs that in embedding space (arg max $_{\mathbf{z}}$ ). Specifically, DROP thus allows us to (A2) conduct safe extrapolation in outer-level and (A3) perform test-time adaptation in testing rather than simply conducting outer-level optimization at initial states as in COMs.
3.5 Practical Implementation
Algorithm 1 DROP (Training)	Algorithm 2 DROP (Testing/Deployment)
Require: Dataset of trajectories: D = {τ}.	Require: Env, β(a|s, z), and f(s, a, z).
1: Decompose D into N sub-sets D[N].	1: s0 = Env Reset().
2: Initialize φ(z|n), β(a|s, z), and f(s, a, z).	2: while not done do
3: while not converged do	3: Test-time adaptation:
z*(st) = arg maxz f(st, β(at|st, z), z).
4: Sample a sub-task: Dn ~ D[N].
5: Learn φ, β, and f with Equations 6 and 8.	4: Sample action: at ~ β(at|st, z*(st)).
6: end while	5: Step Env: st+1 ~ P(st+1|st, at).
Return: β(a|s, z) and f(s, a, z).	6: end while
We now summarize the DROP algorithm (see Algorithm 1 for the training phase and Algorithm 2 for the testing phase). During training (inner-level optimization), we alternate between updating  $\phi(\mathbf{z}|n)$ ,  $\beta(\mathbf{a}|\mathbf{s},\mathbf{z})$ , and  $f(\mathbf{s},\mathbf{a},\mathbf{z})$ , wherein we update  $\phi$  with both maximum likelihood loss and TD-error loss in Equations 6 and 8. During testing (outer-level optimization), we use the gradient ascent as specified in Equation 9 to infer the optimal embedding  $z^{*}$ . Instead of simply sampling a single starting point  $\mathbf{z}_0^{\circ}$ , we choose  $N$  starting points corresponding to all the embeddings  $\{\mathbf{z}_n\}$  of sub-tasks, and then choose the optimal  $\mathbf{z}^{*}(\mathbf{s})$  from those updated embeddings for which the learned  $f$  outputs the highest score, i.e.,  $\mathbf{z}^{*}(\mathbf{s}) = \arg \max_{\mathbf{z}} f(\mathbf{s},\beta(\mathbf{a}|\mathbf{s},\mathbf{z}),\mathbf{z})$  s.t.  $\mathbf{z} \in \{\mathrm{GradAscent}(\mathbf{s},\mathbf{z}_n,K)|n = 1,\dots,N\}$ . Then, we sample action from  $\pi^{*}(\mathbf{a}|\mathbf{s}) \coloneqq \beta(\mathbf{a}|\mathbf{s},\mathbf{z}^{*}(\mathbf{s}))$ .
Offline RL. In offline RL, learning with offline data is prone to exploiting OOD state-actions and producing over-estimated values, which makes vanilla iterative policy/value optimization challenging. To eliminate the problem, a number of methods have been explored, in essence, by either introducing a policy/value regularization in the iterative loop or trying to eliminate the iterative loop itself.
Iterative methods: Sticking with the normal iterative updates in RL, offline policy regularization methods aim to keep the learning policy to be close to the behavior policy under a probabilistic
For more illustrative examples w.r.t. this mismatch, we refer the reader to Figure 6 of the RvS-R paper [14].
distance [8, 19, 30, 33, 45, 46, 52, 54, 58, 67, 73]. Some works also conduct implicit policy regularization with variants of importance sampling [39, 47, 51]. Besides regularizing policy, it is also feasible to constrain the substitute value function in the iterative loop. Methods constraining the value function aim at mitigating the over-estimation, which typically introduces pessimism to the prediction of the Q-values [9, 27, 35, 41, 48, 49] or penalizes the value with an uncertainty quantification [3, 5, 56, 68], making the value for OOD state-action more conservative. Similarly, another branch of model-based methods [29, 71, 72, 57] also perform iterative bi-level updates, alternating between regularized evaluation and improvement. Different from this iterative paradigm, DROP only evaluates values of behavior policies in the inner-level optimization, avoiding the potential overestimation for values of learning policies and eliminating error propagation between two levels.
Non-iterative methods: Another complementary line of work studies how to eliminate the iterative updates, which simply casts RL as a weighted or conditional imitation learning problem (Q1). Derived from the behavior-regularization RL [22, 64], the former conducts weighted behavior cloning: first learning a value function for the behavior policy, then weighing the state-action pairs with the learned values or advantages [1, 12, 66, 54]. Besides, some works propose an implicit behavior policy regularization that similarly does not estimate values of new learning policies: initializing the learning policy with a behavior policy [50] or performing only a "one-step" update (policy improvement) over the behavior policy [20, 24, 75]. For the latter, this branch method typically builds upon the hindsight information matching [4, 15, 28, 36, 44, 43, 55, 65], assuming that the future trajectory information can be useful to infer the middle decision that leads to the future and thus relabeling the trajectory with the reached states or returns. Due to the simplicity and stability, RvS-based methods advocate for learning a goal-conditioned or reward-conditioned policy with supervised regression [10, 13, 14, 21, 25, 42, 60, 69]. However, these works do not fully exploit the non-iterative framework and fail to answer the raised questions, which either does not regularize the inner-level optimization when exploiting "?" in outer-level (Q2), or does not support test-time adaptation (Q3).
Offline MBO. Similar to offline RL, the main challenge of MBO is to reason about uncertainty and OOD values [7, 16], since a direct gradient-ascent against the learned score model can easily produce invalid inputs that are falsely and highly scored. To counteract the effect of model exploitation, prior works introduce various techniques, including normalized maximum likelihood estimation [17], model inversion networks [32], local smoothness prior [70], and conservative objective models (COMs) [62]. Compared to COMs, DROP shares similarities with the conservative model, but instantiates on the embedding space instead of the parameter space. Such difference is nontrivial, not only because DROP avoids the adversarial optimization employed in COMs, but also because DROP allows test-time adaptation, enabling dynamical inference across states at testing.
In this section, we present our empirical results. We first give examples to illustrate the test-time adaptation. Then we evaluate DROP against prior offline RL algorithms on the D4RL benchmark. Finally, we provide the computation cost regarding the test-time adaptation protocol. For more offline-to-online fine-tuning results, ablation studies w.r.t. the decomposition rules and the conservative regularization, and training details on the hyper-parameters, we refer the readers to the appendix.
Illustration of test-time adaptation. To better understand the test-time adaptation of DROP, we include four comparisons that exhibit different embedding inference rules at testing/deployment:
(1) DROP-Best: At initial state  $\mathbf{s}_0$ , we choose the best embedding from those embeddings of behavior policies,  $\mathbf{z}_0^*(\mathbf{s}_0) = \arg \max_{z} f(\mathbf{s}_0, \beta(\mathbf{a}_0 | \mathbf{s}_0, \mathbf{z}), \mathbf{z})$  s.t.  $\mathbf{z} \in \mathbf{z}_{[N]} := \{\mathbf{z}_1, \dots, \mathbf{z}_N\}$ , and keep this embedding fixed for the entire episode, i.e., setting  $\pi^*(\mathbf{a}_t | \mathbf{s}_t) = \beta(\mathbf{a}_t | \mathbf{s}_t, \mathbf{z}_0^*(\mathbf{s}_0))$ .
(2) DROP-Grad: At initial state  $\mathbf{s}_0$ , we conduct inference (gradient ascent on starting point  $\mathbf{z}_0^*(\mathbf{s}_0)$ ) with  $\mathbf{z}^*(\mathbf{s}_0) = \arg \max_z f(\mathbf{s}_0, \beta(\mathbf{a}_0|\mathbf{s}_0, \mathbf{z}), \mathbf{z})$ , and keep  $\mathbf{z}^*(\mathbf{s}_0)$  fixed throughout the test rollout.
(3) DROP-Best-Ada: We adapt the policy by setting  $\pi^{*}(\mathbf{a}_{t}|\mathbf{s}_{t}) = \beta (\mathbf{a}_{t}|\mathbf{s}_{t},\mathbf{z}_{0}^{*}(\mathbf{s}_{t}))$  , where we choose the best embedding  $\mathbf{z}_0^* (\mathbf{s}_t)$  directly from those embeddings of behavior policies for which the score model outputs the highest score, i.e.,  $\mathbf{z}_0^* (\mathbf{s}_t) = \arg \max_{\mathbf{z}}f(\mathbf{s}_t,\beta (\mathbf{a}_t|\mathbf{s}_t,\mathbf{z}),\mathbf{z})$  s.t.  $\mathbf{z}\in \mathbf{z}_{[N]}$
(4) DROP-Grad-Ada (as described in Section 3.5): We set  $\pi^{*}(\mathbf{a}_{t}|\mathbf{s}_{t}) = \beta (\mathbf{a}_{t}|\mathbf{s}_{t},\mathbf{z}^{*}(\mathbf{s}_{t}))$  and choose the best embedding  $\mathbf{z}^{*}(\mathbf{s}_{t})$  from those updated embeddings of behavior policies, i.e.,  $\mathbf{z}^{*}(\mathbf{s}_{t}) = \arg \max_{\mathbf{z}}f(\mathbf{s}_{t},\beta (\mathbf{a}_{t}|\mathbf{s}_{t},\mathbf{z}),\mathbf{z})$  s.t.  $\mathbf{z}\in \{\mathrm{GradAscent}(\mathbf{s}_t,\mathbf{z}_n,K)|n = 1,\dots ,N\}$ .
Figure 3: Visualization of the embedding inference (a, b, c1, c2) and performance comparison (d).  $\mathbf{z}_{[N]}$  denotes embeddings of all behavior policies.  $\mathbf{z}_0^*(\mathbf{s}_0)$ ,  $\mathbf{z}^*(\mathbf{s}_0)$ ,  $\mathbf{z}_0^*(\mathbf{s}_t)$ , and  $\mathbf{z}^*(\mathbf{s}_t)$  denote the selected embeddings in DROP- Best, Grad, Best-Ada, and Grad-Ada implementations respectively.
Figure 4: (Left) Aggregate metrics [2] on 12 D4RL tasks. Higher median, IQM, and mean scores are better. (Right) Each row shows the probability of improvement that the algorithm X on the left outperforms algorithm Y on the right. The CIs are estimated using the percentile bootstrap (95%) with stratified sampling. For all results of our method, we average the normalized returns across 5 seeds; for each seed, we run 10 evaluation episodes. (GA: Grad-Ada. BA: Best-Ada.)
In Figure 3, we visualize the four different inference rules and report the corresponding performance in the halfcheetah-medium-expert task [18]. In Figure 3 (a), we set the starting point as the best embedding  $\mathbf{z}_0^*(\mathbf{s}_0)$  in  $\mathbf{z}_{[N]}$ , and perform gradient ascent to find the optimal  $\mathbf{z}_0^*(\mathbf{s}_0)$  for DROP-Grad. In Figure 3 (b), we can find that at different time steps, DROP-Best-Ada chooses different embeddings for  $\beta(\mathbf{a}_t|\mathbf{s}_t,\cdot)$ . Intuitively, performing such a dynamical embedding inference enables us to combine different embeddings, thus stitching behavior policies at different states. Further, in Figure 3 (c1, c2), we find that performing additional inference (with gradient ascent) in DROP-Grad-Ada allows to extrapolate beyond the embeddings of behavior policies, and thus results in sequential composition of new embeddings (policies) across different states. For practical impacts of these different inference rules, we provide the performance comparison in Figure 3 (d), where we can find that performing gradient-based optimization (\*-Grad-*') outperforms the natural selection among those embeddings of behavior policies/sub-tasks (\*-Best-*'), and rollout with adaptive embedding inference (DROP-*-Ada) outperforms that with fixed embeddings (DROP-Best and DROP-Grad). In subsequent experiments, if not explicitly stated, DROP takes the DROP-Grad-Ada implementation by default.
Empirical performance on benchmark tasks. We evaluate DROP on a number of tasks from the D4RL dataset and make comparisons with prior non-iterative offline RL counterparts<sup>8</sup>.
As suggested by Agarwal et al. [2], we provide the aggregate comparison results to account for the statistical uncertainty. In Figure 4, we report the median, IQM [2], and mean scores of DROP and prior non-iterative offline baselines in 12 D4RL tasks (see individual comparison results in Appendix C). We can find our DROP (Best-Ada and Grad-Ada) consistently outperforms prior non-iterative baselines in both median, IQM, and mean metrics. Compared to the most related baseline Onestep and COMs (see Table 1), we provide the probability of performance improvement in Figure 4 right. We can find that DROP-Grad-Ada shows a robust performance improvement over Onestep and COMs, with an average improvement probability of more than  $80\%$ .
Comparison with latent policy methods. Note that one additional merit of DROP is that it naturally accounts for hybrid modes in  $\mathcal{D}$  by conducting task decomposition in inner-level, we thus compare DROP to latent policy methods (PLAS [74] and LAPO [11]) that use conditional variational autoencoder (CVAE) to model offline data and also account for multi-modes in offline data. Essentially, both our DROP and baselines (PLAS and LAPO) learn a latent policy in the inner-level optimization, except that we adopt the non-iterative bi-level learning while baselines are instantiated
Due to the page limit, we provide the comparison with prior iterative baselines in Appendix C.
Table 2: Comparison in AntMaze and Gym-MuJoCo domains (v2). Results are averaged over 5 seeds; for each seed, we run 10 evaluation episodes. u: umaze (antmaze). um: umaze-medium. ul: umaze-large. p: play. d: diverse. wa: walker2d. ho: hopper. ha: halfcheetah. r: random. m: medium.
u	u-d	um-p	um-d	ul-p	ul-d	wa-r	ho-r	ha-r	wa-m	ho-m	ha-m	mean
CQL	74.0	84.0	61.2	53.7	15.8	14.9	-0.2	8.3	22.2	82.1	71.6	49.8	44.8
IQL	87.5	62.2	71.2	70.0	39.6	47.5	5.4	7.9	13.1	77.9	65.8	47.8	49.7
PLAS	70.7	45.3	16.0	0.7	0.7	0.3	9.2	6.7	26.5	75.5	51.0	44.5	28.9
LAPO	86.5	80.6	68.5	79.2	48.8	64.8	1.3	23.5	30.6	80.8	51.6	46.0	55.2
DROP	90.5	92.2	74.1	82.9	57.2	63.3	5.2	20.8	32.0	82.1	74.9	52.4	60.6
DROP std	±2.4	±1.7	±3.9	±3.5	±5.5	±2.4	±1.6	±0.3	±2.5	±5.2	±2.8	±2.2
under the iterative paradigm. By answering Q3, DROP permits test-time adaptation, enabling us to dynamically stitch "skills"(latent behaviors/embeddings as shown in Figure 3) and thus encouraging high-level abstract exploitation in testing. However, the aim of introducing the latent policy in PLAS and LAPO is to regularize the inner-level optimization, which fairly answers Q2 in the iterative offline counterpart but can not provide the potential benefit (test-time adaptation) by answering Q3.
Note in our naive DROP implementation, we heuristically use the return to conduct task decomposition (motivated by RvS-R), while PLAS and LAPO take each trajectory as a sub-task and use CVAE to learn the latent policy. For a fair comparison, here we also adopt CVAE to model the offline data and afterward take the learned latent embedding in CVAE as the embedding of behaviors, instead of conducting return-guided task decomposition. We provide implementation details (DROP+CVAE) in Appendix A.4 and the comparison results in Table 2 (taking DROP-Grad-Ada implementation). We can observe that our DROP consistently achieves better performance than PLAS in all tasks, and performs better or comparably to LAPO in 10 out of 12 tasks. Additionally, we also provide the results of CQL and IQL [31]. We can find that DROP also leads to significant improvement in performance, consistently demonstrating the competitive performance of DROP against state-of-the-art offline iterative/non-iterative baselines.
Computation cost. One limitation of DROP is that conducting test-time adaptation at each state is usually expensive for inference time. To reduce the computation cost, we find that we can resample (infer) the best embedding  $\mathbf{z}^*$  after a certain time interval, i.e.,  $\mathbf{z}^*$  does not necessarily need to be updated at each testing state. In Figure 5, we plot the average inference time versus different models and DROP implementations with varying inference time intervals. We see that compared to Diffuser (one offline RL method that also supports test-time adaptation, see quantitative performance comparison in Table 4) [26], DROP can achieve significantly lower computation costs. In Figure 5 top-right, we also illustrate the trade-off between performance and runtime budget as we vary the inference time intervals. We find that by choosing a
Figure 5: Computation cost across different models and DROP implementations with varying inference time intervals. The number in parenthesis denotes the inference time intervals. In top-right, we also provide the average performance of DROP on AntMaze tasks when using different inference time intervals. (Diffuser* denotes the warm-start implementation of Diffuser.)
suitable time interval, we can reduce the inference cost with only a modest drop in performance:  $\text{DROP}(50)$  brings less computational burden while providing stable performance.
Comparison to a "distilled" DROP implementation. Our answer to Q3 is that we conduct outer-level optimization in the testing phase. Similarly, we can design a "distilled" DROP implementation: we keep the same treatment as DROP for Q1 and Q2, but we perform outer-level optimization on the existing offline data instead of on the testing states, i.e., conducting optimization in  $\mathbf{z}$ -space for
Table 3: Comparison to "distilled" DROP (D- DROP), using Grad-Ada and CVAE implementation.
wa-r	ho-r	ha-r	wa-m	ho-m	ha-m
CQL	-0.2	8.3	22.2	82.1	71.6	49.8
IQL	5.4	7.9	13.1	77.9	65.8	47.8
D-DROP	4.5	18.9	27.8	84.2	67.1	50.9
DROP	5.2	20.8	32.0	82.1	74.9	52.4
all states in the dataset. Then, we can "distill" the resulting contextual policy and the inferred
$\mathbf{z}^*$  into a fixed rollout policy. As shown in Table 3, we can see that such a "distilled" DROP implementation achieves competitive performance, approaching DROP's overall score and slightly outperforming CQL and IQL, which further supports the superiority afforded by the non-iterative offline RL paradigm versus the iterative one.
Comparison to adaptive baselines. One merit of DROP resides in its test-time adaptation ability across rollout states. Thus, here we compare DROP with two offline RL methods that also support test-time adaptation: 1) APEV [23] learns an uncertainty-adaptive policy and dynamically updates a con
Table 4: Comparison to adaptive baselines (APE-V and Diffuser). DROP uses Grad-Ada and CVAE implementation.
wa-mr	ho-mr	ha-mr	wa-me	ho-me	ha-me
APE-V	82.9	98.5	64.6	110.0	105.7	101.4
Diffuser	70.6	93.6	37.7	106.9	103.3	88.9
DROP	83.5	96.3	50.9	109.3	107.2	102.2
textual belief vector at test states, and 2) Diffuser [26] employs a score/return function to guide the diffusion denoising process, i.e., score-guided action sampling. As shown in Table 4, we can see that in medium-replay (mr) and medium-expert (me) domains (v2), DROP can achieve comparable results to APE-V, with a clear improvement over Diffuser.
In this work, we introduce non-iterative bi-level offline RL, and based on this paradigm, we raise three questions (Q1, Q2, and Q3). To answer that, we reframe the offline RL problem as one of MBO and learn a score model (A1), introduce embedding learning and conservative regularization (A2), and propose test-time adaptation in testing (A3). We evaluate DROP on various tasks, showing that DROP gains comparable or better performance compared to prior methods.
Limitations. DROP also has several limitations. First, the offline data decomposition dominates the following bi-level optimization, and thus choosing a suitable decomposition rule is a crucial requirement for policy inference (see experimental analysis in Appendix A.2). An exciting direction for future work is to study generalized task decomposition rules (e.g., our  $\text{DROP} + \text{CVAE}$  implementation). Second, we find that when the number of sub-tasks is too large, the inference is unstable, where adjacent checkpoint models exhibit larger variance in performance (such instability also exists in prior offline RL methods, discovered by Fujimoto and Gu [19]). One natural approach to this instability is conducting online fine-tuning (see Appendix A.3 for our empirical studies).
Going forward, we believe our work suggests a feasible alternative for generalizable offline robotic learning: by decomposing a single robotic dataset into multiple subsets, offline policy inference can benefit from performing MBO and the test-time adaptation protocol.
[1] Abbas Abdolmaleki, Jost Tobias Springenberg, Yuval Tassa, Remi Munos, Nicolas Heess, and Martin Riedmiller. Maximum a posteriori policy optimisation. arXiv preprint arXiv:1806.06920, 2018.
[2] Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Marc Bellemare. Deep reinforcement learning at the edge of the statistical precipice. Advances in neural information processing systems, 34:29304-29320, 2021.
[3] Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty-based offline reinforcement learning with diversified q-ensemble. Advances in Neural Information Processing Systems, 34, 2021.
[74] Wenxuan Zhou, Sujay Bajracharya, and David Held. Plas: Latent action space for offline reinforcement learning. arXiv preprint arXiv:2011.07213, 2020.
[75] Zifeng Zhuang, Kun Lei, Jinxin Liu, Donglin Wang, and Yilang Guo. Behavior proximal policy optimization. arXiv preprint arXiv:2302.11312, 2023.
A More Experiments
A.1 Single Behavior Policy v.s. Multiple Behavior Policies
A.2 Decomposition Rules
In DROP algorithm, we explicitly decompose an offline task into multiple sub-tasks, over which we then reframe the offline policy learning problem as one of offline model-based optimization. In this section, we discuss three different designs for the task decomposition rule.