A Policy-Guided Imitation Approach for Offline Reinforcement Learning
Authors: Haoran Xu
Li Jiang
Jianxiong Li
Xianyuan Zhan
Venue: NeurIPS 2022 Accept (2022)
Keywords: Offline RL

Abstract: Offline reinforcement learning (RL) methods can generally be categorized into two types: RL-based and Imitation-based. RL-based methods could in principle enjoy out-of-distribution generalization but suffer from erroneous off-policy evaluation. Imitation-based methods avoid off-policy evaluation but are too conservative to surpass the dataset. In this study, we propose an alternative approach, inheriting the training stability of imitation-style methods while still allowing logical out-of-distribution generalization. We decompose the conventional reward-maximizing policy in offline RL into a guide-policy and an execute-policy. During training, the guide-poicy and execute-policy are learned using only data from the dataset, in a supervised and decoupled manner. During evaluation, the guide-policy guides the execute-policy by telling where it should go so that the reward can be maximized, serving as the \textit{Prophet}. By doing so, our algorithm allows \textit{state-compositionality} from the dataset, rather than \textit{action-compositionality} conducted in prior imitation-style methods. We dumb this new approach Policy-guided Offline RL (\texttt{POR}). \texttt{POR} demonstrates the state-of-the-art performance on D4RL, a standard benchmark for offline RL. We also highlight the benefits of \texttt{POR} in terms of improving with supplementary suboptimal data and easily adapting to new tasks by only changing the guide-poicy.

A Policy-Guided Imitation Approach for Offline Reinforcement Learning
Haoran Xu
Jiang Jianxiong Li
Xianyuan Zhan +
JD Technology. Beijing, China
Tsinghua University, Beijing, China
Shanghai AI Laboratory, Shanghai, China
fryanxhr,
Abstract
Offline reinforcement learning (RL) methods can generally be categorized into two types: Rl.-hased and imitation-hased. RI.-based methods could in principle enjoy out-of-distribution generalization but suffer from erroneous off-policy evaluation. Imitation- methods avoid off-policg evaluation but are too conservative to surpass the dataset. In this study, we propose an alternative approach, inheriting the training stability of imitation-style methods while still allowing logical out-of- distribution generalization. We decompose the conventional reward-ma seardd-maximzzing policy in offline RL into guide-policy and execute-policy. During training, the guide-policy and execute-policy are learned using only data from the dataset, in a supervised and decoupled manner. During evaluation, the guide-policy guides the ssecute-policy by telling where it should go so that the reward can be maximized. By doing so, algorithm allows catosscompsiiionnlitty from the dataset, rather than action-compositioonll. this approach Policy-guided Offline RL (POR). POR demonstrates the state- of-the-ar the-art performance on D4RL, standard benchmark for offline RL. We also highlight the benefits of POR in terms of improving with supplementary suboptimal data and easily adapting to new tasks by only changing the guide-policy. Code is available https: //g/ thub. com/ ryanxhr/PRR.
conducted in prior imitation-style methods. We dumb
Introduction
Offline RL, also known batch RL, allows learning policies from previously collected data, without online interactions [29, 31]. It is promising area for bringing RL into real-world domains, such as roboties [21], healthcare [47] and industrial control [62]. In such scenarios, arbitrary exploration with untrained policies is costly or dangerous, but sufficient prior data is available. While most off-policy RL algorithms are applicable in the offline setting by filling the replay buffer with offline data, improving the policy beyond the level of the behavior policy often entails querying the value function (i.e.. Q function) about values of actions that were not in the dataset. Those out-of-distribution actions can be deemed as adversarial examples of the Q function [28], which cause extrapolation error of the Q-function. The error will be accumulated by the deadly triad issue [50], propagate across the state-cction space through the iterative dynamic programming procedure [4], and can not be climinated without requiring growing batch of online samples [31].
To alleviate this issue, prior model-free offline RL methods typically add a behavior regularization term to the policy improvement step, to limit how far it deviates from the behavior policy. This can be achieved explicitly by calculating some divergence metrics [26, 54, 38, 13], or implicitly
'Equal contribution. Correspondence to Haoran Xu, Xianyuan Zhan.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

by regularizing the learned value functions to assign low values to out-of- of-distribution actions [28, 24, 2]. Nevertheless, this imposes a trade-off between accurate value estimation (more behavior regularization) and maximum policy improvement (less behavior regularization). To avoid this problem, recently a branch of methods bypass querying the values of unseen actions by performing some kind of imitation learning on the dataset'. This can be achieved by filtering trajectories based on their return [40, 8], or reweighting transitions based on how advantageous they could be under the behavior policy [4, 25]. or just be conditioned
some variables without any dataset reweighting [7, 10]. Although imitation-style methods enjoy a stable training process and are able to effectively perform multi-step dynamic programming by assigning the proper reweighting weight conditioned variable [25, 10], they only allow action- compositioooi from the dataset, lose the ability to surpass the dataset by out-of-distributin generalization, which only appears in RL-based methods.
In this work, we propose an alternative approach. We aim at inheriting the training stability of imitation-style methods while still allowing logical out-of-distributiin generalization. To do so, we propose Policy-guided Offline RL (POR), an algorithm that employs two policies to solve tasks: a guide-policy and an execute-policy. The job of the guide-policy is to learn the optimal next state given the current state, and the job of the execute-policy is to learn how different actions can produce different next states, given the current state. By this manner, we decompose the original task of RL (i.e., reward- reward-maximizing maximizing) into two distinct yet complementary tasks, and this decomposition makes each task much casier to be solved. During evaluation, the guide-policy serves guide for the execute- policy by telling it where to go so that the reward can be maximized. By doing so, our algorithm allows conpostooooii from the dataset, rather than action compostiooom conducted in prior work, which owns logical autoof-distribution generalization.
Our method is easy to implement by only adding the learning of a quide-policy, the training stage of the guide-policy and execute-policy are decoupled, using in-sample learning from the dataset. We test POR in widely-used D4RL offline RL benchmarks and demonstrates the state-of-the-art performance, especially on low-quality datasets that require out-of-distribution generalization to achieve a high score. We also show that by decoupling the learning of guide-policy and execute-policy we can enhance the guide-policy with supplementary suboptimal data, or re-learn the guide-policy to adapt to new tasks with different reward functions, without changing the execute-policy.
2 Related Work
RL-based offline approach A large portion of offline RL methods is RL-based. RL-based methods typically augment existing off-policy methods (e.g., Q-learning or actor-critic) with a behavior regularization term. The primary ingredient of this class of methods is to propose various policy regularizers to ensure that the learned policy does not stray too far from the behavior policy. These regularizers can appear explicitly as divergence penalties [54, 26, 13], implicitly through weighted behavior cloning [52, 40, 38), or more directly through careful parameterization of the policy [14, 65]. Another way to apply behavior regularizers is via modification of the critic learning objective to incorporate some form of regularization to encourage staying near the behavioral distribution and being pessimistic about unknown state-action pairs [37, 28, 24, 58]. There also several works incorporating behavior regularization through the use of uncertainty. The uncertainty quantification can be done via Monte Carlo dropout [55] or explicit ensemble models [2. 61, 22, 63]. Note that the behavior regularization weight is crucial in RL-based methods. With a small weight, RL-based methods could in principle enjoy -istr--distriuution generalization since they perform true dynamic programming. However, a small weight will make erroneous off-policy evaluation due to distribution shift and the policy is extremely vulnerable to the value function misestimation.
Imitation-based offfine approach Another line of methods, on the contrary, performs some kind of imitation learning on the dataset, without the need do off off-policy evaluation. When the dataset is trajectories, we can simply clone the actions observed in the dataset [41], perform some kind of filtering conditioning to extract useful transitions. For instance, recent work filters trajectories based on their return [8, 40], or directly filters individual transitions based on how advantageous these could be under the behavior policy and then clones them [4, 25, 5, 16, 56]. Conditioned BC methods are based on the idea that every transition or trajectory
good enough or contains high-performin
'The core difference between RL-based and imitation- based methods is that RL-based methods leam a value function of policy æ while imitation based methods don't.
2

---

is optimal when conditioned on the right variable [15, 10]. In this way, after conditioning, the data becomes optimal given the value of the conditioned variable. The conditioned variable could be the cumulative return [7, 19, 27], or the goal information if provided [34, 15, 10, 59]. A recent study [10] shows that goal-conditioning can be super useful D4RL. AntMaze datasets. However, this introduces the assumption that we have some prior information about the structure of the task, which is beyond the standard offline RL setup.
Our method can be viewed as a combination of RL-based and imitation-based methods. We adopt an imitation-style manner to train the goal-policy and execute-policy. During evaluation, however, following the
the execute-pon seelucy-pllcyy could produce out-of-distribution actions by leveraging the generalization capacity of the function approximator.
3 Preliminaries
We consider the standard fully observed Markov Decision Process (MDP) setting [46]. An MDP can be represented as M (S, T, rr,,,; p:7) where S is the state space, A the action space, T(-ss, a) is the transition probability distribution function, r(s, a) is the reward function, p is the initial state distribution and 7 is the discount factor, we assume 7 E (0, 1) in this work. The goal of RL is to find policy (alss) S x A + [0, 1] that maximizes the expected cumulative discounted reward (or called return) along a trajectory as
Offline RL
20
max E
Σ
r'r ( ur) 80 = 8, ag = a, 8, ~ ((1ss-1, ar ~ a (+18)) for & 2
1 (I)
In this work, we focus on the offline setting. The goal is learn policy from a fixed dataset D ,... (s rf),
{r' = ((sb. ag. sffrr))) v (si, are collected by different policies, where H the time horizon. The dataset can be heterogenous and suboptimal, we denote the underlying behavior policy of 'D as μl, which represents the conditional distribution p(a(s) observed in the dataset.
of, si)
aip, sij rir) } [= consisting of trajectories that
RL via Supervised Learning Conventional RL methods generally either compute the derivative of (1) with respect to the policy directly via policy gradient methods [43, 20], or estimate value function or Q-function by means of TD learning [53], or both [44, 17]. In the RL via Supervised Learning framework, we avoid the complex and potentially high-variance policy gradient estimators, as well as the complexity of temporal difference learning. Instead, we perform conditioned behavior cloning on some extra information, such as goal state, cumulative return, or language description [34, 15]. When applying the framework to the offline RL setting, we can take the offline dataset 'D as input and find the salcome-conditiones policy æ that optimizes
EE- Ex..D,teUnif((,,) () w- [log x (arlse,
(2)
where an outcome conditioned on the remaining trajectory, i.e., w ~ 9 (-ITEH), and we use Tay = (s,, .,....,8)) 85) to denote fragment of the trajectory.
4 Policy-guided Offline Reinforcement Learning
We now continue to describe our proposed approach, POR. To have a clear understanding of the out-of cut-of-diitribution generalization of POR, we begin with motivating example which demonstrates that by allowing state-compositionaaltyy on the dataset, the agent is able to generalize to a much better policy than doing action- compositionality. Then describe how we accomplish this idea in the more general case and introduce the algorithm details of POR. Finally, we give a thcoretical analysis of POR, we are interested to see how close the policy induced by POR and the optimal policy could be.
4.1 A Motivating Example
We first use a simple, didactic toy example to illustrate the importance of state-stitching for offline RL. Let's consider the navigation task shown in Figure 1, where the goal is to find the shortest path from the starting location (bottom-left corner) to the goal location (top-right corner) the grid world. The state and action space in this task are discrete, the state consists of the (x, y) coordinate
3

---

and the agent could choose to move to any of its nearby 8 grids (i.e., the action space is 8). The agent receives reward of 0 for entering the goal state and for all other transitions. The offline dataset (green-arrowes línes) consists of two suboptimal trajectories from the start location to the goal location, plus transitions of taking random actions at random states.
We use the grid color (more saturated to purple means higher value) to show the value function (ss) com- puted from the offline dataset. We compare two meth- ods that use the value function differently. At state St D, the first method chooses the action og in the dataset that leads si with the highest V(s)): the second method chooses the action that leads to s' in the dataset with the highest V(&).. note that a not necessarily to be within the dataset. The first method doing action "stitching" in the dataset while the second method is doing state "stitching". Although both methods are able to "stitch" suboptimal trajecto- ries together from the dataset and successfully travel from the start location to the goal location, the action- stitching method takes more steps (11 steps) com- pared with the state-stitching method (7 or 8 steps), as shown in Figure 1.
-2
-8
-10
toy example the grid world that requires navigation from the start location (bottom- left corner) to the goal location (top-right corner). The offline dataset is colored with green, trajec- tories obtained from action-stitching method and statc-stitching method are colored with blue and red, respectively. Note that both action-stitching and state-stitching methods have two equal-valued action choices the start location (1 and ->), re- sulting in two different trajectories in both blue and red lines.
Figure 1:
The cetionn-sitchinn method introduced above is ac- tually an abstract methodology of imitation- based methods, which imitate the transitons the dataset unequally with different weight choices. However, these methods are too conservative to find the optimal trajectory, especially when there does not exist one in the offline dataset. To generate better-than than-dataset trajectories (e.g., red-arrowed lines), the agent needs to take logical -iitt-ddistriuttion actions.
Note that in the toy example, we give the agent the freedom to choose any action that leads to the highest-vale value state the dataset. This allows the most ofitt--diitributtin generalization and is realizable in some simple tasks (e.g., discrete state and action space) with high action coverage, for instance, in the toy example, given historical (s, a, 8') results at (5, 4). the agent can successfully recognize that taking upper-right will arrive at (6, 5). However, in the more general case, especially the continuous state and action space setting, this may make the agent do erroneously-generaie actions the same state hardly be observed twice. To perform logical out-of-distributton generalization, we need a Prophet, i.c., the guide-policy, to help the agent by telling it which state should (high reward) and can (logical generalization) go to. It turns out that the involvement of the guide-policy also brings some theoretical meaning, we will discuss that in Section 4.4.
4.2 Learning the Guide-Policy
Recall that the goal of the guide-polies is to guide the execute-policy about which state it should go to. To accomplish that, we train a state value function V(s) : + IR. The training of uses only (s. s') samples in the offline dataset, it doesn't ' suffer from overestimation because there's 're no dut-of-distribution actions involved. To approximate the optimal value function in the dataset, we adopt tricks from recent work [25, 35] by giving the fz loss with different weight using expectile regression, yielding the following asymmetric lg loss
min E (err, L (r + 7 V (8') V=()) where Lz(u) = |7 1(u < O)
(3)
can be seen that when r 1/2, this operator is reduced Bellman expectation operator, while when F + 1, this operator approaches Bellman optimality operator. Note that our learning objective bears similarity with IQL [25], however, here we aim to learn a state value function while IQL aims to learn state-action value function so as to extract the policy. To do this, IQL. needs to learn both Q and V (V is used to isolate the effect of state and approximate the expectile of Q only with respect to the action distribution), the training of V and Q are coupled and may affect each other.

---

Simply maximizing the guide-policy with respect to V(s) will result in state where the execute- policy may make erroneous generalization. To alleviate this issue, we add behavior cloning term to the learning objective of the guide-policy. Denote the guide-policy as (s) : - S, the learning objective of g is given by
max (=D (s)( + log gh (s'1s),
where the weight a serves as the trade-off between guiding to space with high-reward and space that the execute-policy ought to have a correct generalization.
We also give an alternative learning objective that eliminates the need of a by using the residual (r + VV (s') - V&(s)) as the behavior cloning weight, which we find work well in practice:
max E (', ((rtt log (s
(S)
4.3 Learning the Pxecute-Policy: Training and Evaluation
After learning the guide-poliey, now we turn the execute-policy. Since the job of the execute-policy to have a strong generalization ability, we adopt the RL via Supervised Learning framework by condi- tioning the execute-polle peecccy on that encountered in the dataset. To be more specific, denote the execute- policy as (8, s') : S x S + [0. 1]. During training. performs supervised learning by maximizing the likelihood of the actions given the states and next
Algorithm 1 Policy Guided Offline RL
Require: D, T. a (optional).
// Training
2: Initialize Va, Vol, 9w. T0
for 1 1,2,..., do
4: Sample transitions (s,r, 8') ~ DD Update Va by E3.(3)
6: Update Iu by (4) or (5)
states, yielding the following objective
7: Update Var by o' + AΦ + (1 A)
max (s.D,s))) [log To
(6)
8: end for
9: for t = 1,2. M do
in some scenarios with low data quality, we also add the exponential residual weight to (6) to mitigate the effect of those bad actions (same as (5)).
10: Sample transitions (s, a, sa) D
11: Update Te by (q.
12: end for
13: // Evaluation
During evaluation, given a state the final action is determined by both the guide-policy and the exccute- policy, by
14: Get initial state s, set d as False
15: while not d do
16: Get action a form (7)
a = arg max (a(a gs)
(7)
17: Roll out a and get (6',1,4
a
18: Set 8 s
Note that our learning objective (6) differs from pre- vious RL via Supervised Learning methods in that we only use the next state s' the conditioned variable, there's 's no need to estimate the cumulative return [7], which may be highly suboptimal in certain cases [11, 3]. Our method also works in settings where we don't 't know the goal information [10]. Also, during evaluation, previous methods show that the choice of conditioned variable is crucial important as little changes will cause significant performance difference [7]. In our method, owing the existence of the guide-policy, the optimal conditioned variable can be automatically generated.
19: end while
Our final algorithm, POR, consists of three supervised stages: learning V. learning g, and learning We summarize the training and evaluation procedure of POR in Algorithm 1. Note that the training of g and fully decoupled, this brings POR some nice properties improve with supplementary data or transfer to new tasks, we will further investigate it in the experiments.
4.4 Analysis
In this section, we give theoretical analysis of POR. Concretely, we aim to 1) give the lower bound of the performance difference between POR and the optimal policy n², and 2) analyze how the guide-policy g will influence this bound.
We begin by introducing the notation and assumptions used in our analysis. We denote P(s, s') x + A as the inverse transition operator. We denote ag and a as the ground truth of T (s, gs)))
and 7 (s. s'), respectively. We also denote sup [| s') all as the upper bound of the approximation error in the dataset during training. We then introduce the following two assumptions.
5

---

Assumption 1. (Non-lazy MDP) The inverse transition operator P(s, s') is L1 Lipisschhtzz continuous, i.e, [|P(s), si) P(2,, s/yوl S L1 [ss s1) ($2, 2/)1l
Assumption 2. (Lipschitz continuous function approximators) The execute-policy we trained is Lz-Lisschitz continuous, i.e., [|T( $1) π (82, s S L2l1(51, s1) - (82, s,)ll-
Assumption holds when actions do have effects state transitions (i.e. if a # O. then ||s' - s|| 6, > 0). We refer the MDP that satisfies Assumption as non-lazy MDP and others as lazy MDP. should be mentioned that most real-world tasks belong to the non-lazy MDP case, it is meaningless to study under the lazy MDP case because in lazy MDP, action changes will have little effect on state transitions, making the policy learning in vain. Assumption 2 is mild assumption that frequently utilized in plenty of works [51, 36, 32).
Then in Theorem I, show that under these two assumptions, the gap between the optimal action and the action output by POR can be bounded.
Theorem 1. (Single step gap to optimal action). The single-step gap between optimal action and the action induced by our method can be bounded as
(T((s -a (L1 L2)llg(s) s'1l
|10g a")
(8)
12 lg
Theorem states that single step optimal gap is related to three parts: 1, 12, 11. 1 is the generalization performance factor. It can be found that a small value of [[g(s) - s'Il enables a small 4, which indicates the necessity to constrain the guide-policy to stay close to the dataset. l2 denotes the suboptimality constant. Intuitively, a loosened constraint on the guide-policy may induce a small suboptimality constant but on the other hand, may suffer from the risk of high value of 4, The guide-policy serves as a trade-off between 4 and /2, we could balance the constraint strength and suboptimality by adjusting in (4) to obtain the lowest bound. 03 is related to the approximation eror on the training data, which can be reduced via improving the representation ability of function approximators but may suffer from high L2 due to the overfitting caused by over-pavann werr-praaeterizzttonn 1 also depends on the Lipschitz constant L1 and L2. Note that a smooth inverse transition operator will result in small value of L, and L2 (because (s,, s') is trained to approximate P(s, s')). this means our method may achieve better performance under a smoother inverse transition.
Based on Theorem 1, we can also give the performance bound of our method by replacing the in Theorem 3.2 in [15] with the RHS of (8): JTT") (T) I sup,,a, (4 + 12 + l2)T, where T is the horion of each episode, h is the time step.
5 Experiments
We present empirical evaluations of POR in this section. We first evaluate POR against other baseline algorithmss D4RL [12] benchmark datasets. We then explore deeper the guide-policy about the benefits the decoupled training process. We finally establish ablation studies on the execute-policy.
5.1 Comparative Experiments on D4RL Benchmark Datasets
We first evaluate our approach on D4RL MuloCo and AntMaze datasets [12). Notice that most of MuJoCo datasets (except random and medium datasets) consist a large fraction of near-optimal trajectories, while AntMaze datasets contain few or no near-optimal trajectories. Hence those low-quality datasets can serve as a good testbed for quantifying the compositionalityy ability.
We compare POR with both RL-based and imitation-based baselines. For RL-based baselines, we select the best one from state-of-the-at art algorithms, including BCQ [14], BEAR [26], BRAC (54), CQL [28] and TD3+BC [13]. For imitation- baselines, we further split them into Weighted BC methods and Conditioned BC methods. Weighted BC methods include 10%BC [7], One-step RL [4] and IQL [24]. 10%BC is filtered version of BC that runs behavior cloning only the top 10% high-return trajectories in the dataset. One-step RL and IQL can be viewed as using different weights to do behavior cloning (see Appendix A for discussion). Conditioned BC methods include DT [7] and RvS [10]. DT built with Transformer and conditioned on the cumulative return. RvS well studied the effect of different conditional information different tasks. The score of RvS is

---

Table 1: Averaged normalized scores of POR against other baselines. The scores are taken the final 10 evaluations with seeds. In the Bext RL Baseline column, the algorithm with the best performance among RL-based algorithms (BCQ [14], BEAR [26], BRAC [54], CQL [28], TD3+BC [13]) is presented, where the previous four are took from their original papers and the last one is from our implementation. POR achieved the highest scores in 11 out of 18 tasks.
Weighted BC
Conditioned BC
Best RL
D4RL Dataset
Baseline
10%BC
One-step
IQL
DT
RvS
POR (ours)
antmaze-u
62.8
64.3
87.5=2.6
59.2
65.4
90.6 #7.1
89.0"00
antmaze-u-d
50.2
60.7
63.2
53.0
60.9
71.3 112.
61.0
antmaze-m-p
5.4
0.3
71.2:17.3
0
58.1
84.6 1:56
68.0
antmazc-m-d
0
70.0+
0
67.3
79.2 +:311
68.0
antmaze-l-p
0
39.6:15.8
0
32.4
58.0
12.4
18,
antmaze-I-d
0
47.5
0
36.9
73.4 48.5
45.6
antmaze mean
21.4
20.8
63.6 +.8.3
18.7
53.5
76.2
+-8.1
58.4
halfcheetah-r
5.4
3.7-10.2
11.2:22.9
29 +007
20.0
hopper-r
4.2
5.2:10.2
7.9+0.4
12::2.1
14.2 BEAR
walker2d-r
6.7
5.6:10.6
5,9:0.5
6.3
8.3 COL.
halfcheetah-m
42.5
48.4
47.410.22
42.6110.1
41.6
48.8 -:0.5
48, 18.3
hopper-m
56.9
59.6+22.5
66.2+5.7
67.6+1.0
60.2
98.2 1.6
59.3 TD94C
walker2d-m
75.0
81.8:1..2
78.31187
74.01:144
71.7
81.1
83.7 TDS-BC
halfcheetah-m--r
40.6
3..1
44.2-11.2
36.610.88
38.0
43.5
45.5 ogi.
hopper-m-r
75.9
97.5++0.7
94.7#1.66
82.7+7.0
73.5
98.9 +:211
95.0
walker2d-m-r
62.5
12.5
73.8
66.611.00
60.6
7..6116.9
81.8 TDS4BC
halfcheetah-m-<
92.9
93.411..6
86.715..3
86.8111.3
92.2
94.7 -:222
91.6
hopper-m-e
110.9
103.3:1..9
91.5:114.3
107.6 1.8
101.7
90.0==12
105. 10L
walker2d-m-c
109.0
113.0 +004
109.6:11.0
108.1
106.0
109.14:0.7
110.1 110.1 T05BC
locomotion mean
55.5
58.2 +2.0
57.6 4:6.6
56 1.4
53.7
65.6 +:2.7
63.5
chosen by the higher score between RvS-R and Rvs-G, which use the reward-to-go and oracle goal information as the conditioned variable, respectively. Note that we also categorize our method POR into conditioned BC methods because the execute-policy is learned in similar manner other conditioned BC methods. Note that the results of Filter BC are from our own implementation, and the results of IQL on MuJoCo random tasks are re-runed using the author-provides implementation. Other results are taken directly from their corresponding papers. To enable fairness and consistency in the evaluation process, we train our algorithm for million time steps and evaluate every 5000 time steps that consist of 10 episodes. Full experimental details are included in Appendix D.1 and the learning curve can found Appendix E.
The results are shown in Table 1, in MuJoCo locomotion tasks, POR performs competitively to the best performance of prior methods in high-quality datasets, while achieving much better performance than other methods in low-quality datasets (c.g., random and medium um datasets). On the more challenging AntMaze tasks, POR outperforms all other baselines by a large margin. It even surpassed RvS that uses the oracle goal information to learn the policy. We conjecture that the success in all those datasets should be credited to the cat-of-distribttio generalization ability of POR.
5.2 Validation Experiments on
Generalization
In this section, we try validate our hypothesis about the out-of-distribution generalization of POR. We are especially interested to see how the guide-policy is learned in those high-dimensional tasks in D4RL. To do so, we compare the distribution of g(s) generated by the guide-policy cammaze-large-dia
with the distribution of s' from the offline dataset. We choose popper-medium-regy
and two datasets for evaluation. For each dataset, we randomly collect 100, 000 (s. s') transition tuples. To visualize the results clearly, we plot the distribution of g(s) and s' with t-Distributed Stochastie Neighbor Embedding (1-SNE) [18], we also visualize the true values of all states using the optimal value function V* (s)..
7

---

Guide policy
Dataset
Guide-poliy
Dataset
(a) antmaze- large- diverse
hyper-mediuum-eplag
Figure 2: Visualizations the distribution of states generated by the guide-policy and states in the dataset, on antmaze- marzeelaage-. warrarrggiiee diverse and nopper-mediure -replaay datasets. The left panel of cach subfigure shows the 100, 000 data points of g(s) (blue) and (green). The right panel of each subfigure shows the corresponding value of each state. The distribution of g()) and s' bear some similarity but g(s) is more prone to generate high- value states, especially antmaze-large -irgeedeeeesee dataset. Better
clear comparison.
Hopper
Walker2d
Main
99
More
Norualized Score
80
60-
.3
48
50-
49
40-
40-
20
TD3+BC
CQL
POR (ours)
TD3+BC
CQL
POR (ours)
Figure 3: Normalize scores of different algorithms with different training schemes. While the "main" and "more" training scheme can be applied all methods, the "mix" training scheme is only available POR hecause decouple the training process of and æ. With "mix" training scheme, POR outperforms other algorithms by a large margin, in both hopper and walker2d environments.
The results are shown Figure it can be found that in both tasks, the distribution of g(s) bears some similarity to the distribution of s'. However, when looking at the non-overlapped area of g(s) and s', we found that g(s) has a much higher value than s', especially in the antmaze-large large-diverre dataset. Those outtt--distributiin and high-value areas are exactly where POR generalized to, note that the high-value benefit the single-step generalization could also be accumulated and propagated through time steps, resulting in much better policy.
5.3 Investigations on the Guide-Policy
In this section, we investigate what the potential benefit decoupling the training process of the guide-policy and execute-policy. We found that the execute- pooicceppocyyy is not task-specific and only cares about the effect of actions on different states, one can reuse its powerful generalization ability when encountered with additional suboptimal data or adapting the guide-police as is task-specific.
task. We only need to re-train
Improve guide-policy by additional suboptimal data We first study the setting where already have a policy learned from small yet high-quality dataset De, and now we get supplementary large dataset D., D. may be sampled from or multiple behavior policies, it provides higher data coverage but could be suboptimal. We are interested to see whether we can use D, to obtain a better policy. This setting is realistic as low-quality datasets appear more often in real-world tasks [60, 57].
Naïvely, we can combine D. and D, as a new offline dataset D, then run any off-the-shelf offline RL algorithm. We label this training scheme as "more". We also label the original training scheme that only uses D. as "main". We test these two training schemes on two tate-of-the-ar offline RI. algorithms, TD3+BC and CQL. The results shown in Figure 3 indicate that both "main" and "more"

---

(a) Four-room
(b) Four-room-tne
(c) Four-room-key
Figure 4: Rollout trajectories of POR with the same pre-trained execute- execute-policy but different guide policies in different four-room tasks, with color becoming more saturated as time progresses. The svecute-police is pre trained in task A and remains unchanged in task and task C. All three tasks require the agent find path from the start to the goal Besides, task B requires the agent not to fall into the river and task C requires the agent to get the key of as well as avoiding the river we before arriving at goal location.
are insufficient to learn a good policy, The reason why "main" fails is that the resulting policy may not be able to generalize due to the limited size. While "more" can generalize better than "main" due to access to much larger dataset, the policy will be negatively impacted by the low-quality data in Do as both TD3+BC and CQL contain the behavior regularization term.
Due to the availability of decoupled training, POR is able to use different training scheme, we can keep the execute-policy learned from D. unchanged but use the combination of DD and D. to re-train the guide-policy Note that in this scheme, we allow the use of action-free dataset D., which often appears real-world scenarios (i.e., video data and third-person demonstrations) [49]. The value function will be more accurate and better generalized when combining De with D. to have larger data coverage (see exploration in online RL as an example). We label this unique training procedure as "mix". It can be shown in Figure 3 that, by doing so, POR could have super large improvement over "main" and "more". The version "mix" brings POR up to 99, close to the upper limit of the performance in the hopper environment, with only limited data. In the walker2a environment, POR with "mix" " also outperforms all other baselines.
Change guide-policy in new tasks Then, we study whether we can use the guide-policy to do lightweight adaptation new tasks, without changing the execute-police We conduct experiments on the continuous variant of the classic four-room environment [19]. The training data consists of trajectories collected by a goal-reaching controller with the start and end locations sampled randomly non-wall locations, mixed with data collected by random policy. In this environment, we introduce three different tasks: Task A (Four-room) needs the agent to travel from the start location to the goal location, besides that, Task B (Four-room rrom-riveee rrieer) and task C (Four- (Four-room key) require the agent to bypass the river or get a key before arriving at the goal location.
We first train the guide-policy and the paccute-policyy π to solve task A. In task B and task C, we re-train the guide-policy using the corresponding reward function in that task and reuse the execute- policy π from Task A. In Figure 4, we can see that by doing so, POR is able to successfully solve task and task C. In detail, in task B, POR is aware of the river and correctly bypasses In task C, POR only chooses to pass the bottom door, this is owing to the awareness of the importance of obtaining the key before reaching the goal. This result shows strong adaptation ablity of the guide-policy for different tasks, as well as the generalization ability of the execute-policy across different tasks.
6 Conclusions and Limitations
In this work, we propose a novel offline RL approach, POR, which leverages the training stability of imitation-style methods while still encouraging logical out-of-distribution generalization. POR allows state-composttionality rather than action- cetion-compssttionllly from the dataset. Through theoretical analysis and extensive experiments, show that POR outperforms prior methods variety of

---

datasets, especially those with low quality. We also empirically demonstrate the additional benefits of POR in terms of improving with supplementary suboptimal data and easily adapting to new tasks. One limitation of POR is that the prediction emorr of the guide-policy may be large when the state space is high-ddmennional. However, this can be alleviated with the help of representation learning [30, 64).
Acknowledgmentts This work is supported by fundings from AsiaInfo. We thank Michael Janner for providing the code of the continuous four-room environment. We thank Junwu Xiong, James Zhang for their feedback on earlier versions of the manuscript.
References
[1] Gaon An, Seungyong Moon, Jang-,ang-Hie Jang-Hyun Kim, and Hyun Oh Song. Uncertainty-bant based offline reinforcement learning with diversified q-ensemble. Proc. of NeurlPS, 2021.
[2] Chenjia Bai, Lingxiao Wang, Zhuoran Yang, Zhi-Hong Deng, Animesh Garg, Peng Liu, and Zhaoran Wang. Pessimistic bootstrapping for uncertainty-driven offline reinforcement learning. In Proc. of ICLR, 2021.
[3] David Brandfonbrener, Alberto Bietti, Jacob Buckman, Romain Laroche, and Joan Bruna. When does conditiond supervised learning work for offline reinforcement learning? In Proc. of NeurlPS, 2022.
[4] David Brandfonbrener, William Whitney, Rajesh Ranganath, and Joan Bruna. Offline without off-policy evaluation. Proc. of NeurlPS, 2021.
[5] David Brandffobrenee, William F Whitney, Rajesh Ranganath, and Joan Bruna. Quantile filtered imitation learning. ArXiv preprint, 2021.
[6] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. ArXiv preprint, 2016.
[7] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Proc. of NeurlPS, 2021.
[8] Xinyue Chen, Zijian Zhou, Zheng Wang, Che Wang, Yanqiu Wu, and Keith W. Ross. BAIL: best-action imitation learning for batch deep reinforcement learning. In Proc. of NeurlPS, 2020.
[9] Paul Christiano, Zain Shah, Igor Mordatch, Jonas Schneider, Trevor Blackwell, Joshua Tobin, Pieter Abbeel, and Wojeiech Zaremba. Transfer from simulation real world through learning deep inverse dynamics model. ArXiv preprint, 2016.
[10] Scott Emmons, Benjamin Eysenbach, Ilya Kostrikov, and Sergey Levine. Rvs: What is essential for offline rl via supervised learning? ArXiv preprint, 2021.
[11] Benjamin Eysenbach, Soumith Udatha, Sergey Levine, and Ruslan Salakhutdinov. Imitating past successes can be very suboptimal. In Proc. of NeurIPS, 2022.
[12] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. ArXiv preprint, 2020.
[13] Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. ArXiv preprint, 2021.
[14] Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In Proc. of ICML, pages 1582-1591, 2018
[15] Dibya Ghosh, Abhishek Gupta. Ashwin Reddy, Justin Fu, Coline Manon Devin, Benjamin Eysenbach, and Sergey Levine. Learning to reach goals via iterated supervised learning. In Proc. of ICLR, 2021.
[16] Caglar Gulcehre, Sergio Gómez Colmenarejo, Ziyu Wang, Jakub Sygnowski, Thomas Paine, Konrad Zoina, Yutian Chen, Matthew Hoffman, Razvan Pascanu, and Nando de Freitas. Regu- larized behavior value estimation. ArXiv preprint, 2021.
10

---

[17] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proc. of ICML, pages I 1856-1865, 2018.
[18] Geoffrey E. Hinton and Sam T. Roweis. Stochastic neighbor embedding. In Proc. of NeurIPS, pages 833-840, 2002.
[19] Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. Proc. of NewrlPS, 2021.
[20]
Sham M. Kakade. A natural policy gradient. In Proc. of NeurlPS, pages 1531-1535 2001.
[21] Dmitry Kalashnikov, Jacob Varley, Yevgen Chebotar, Benjamin Swanson, Rico Jonschkowski, Chelsea Finn, Sergey Levine, and Karol Hausman. Mt-opt: Continuous multi-task robotic reinforcement learning at scale. ArXiv preprint, 2021.
[22] Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model-based offline reinforcement learning. In Proc. of NeurIPS, 2020.
[23] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proc. of ICLR, 2015.
[24] Ilya Kostrikov, Rob Fergus, Jonathan Tompson, and Ofir Nachum. Offline reinforcement learning with fisher divergence critic regularization. Proc. of ICML, pages 5774-5783 2021.
[25] Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-leaming, ArXiv preprint, 2021.
[26] Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off- policy q-learning via bootstrapping error reduction. In Proc. of NeurlPS, pages 11761-11771, 2019.
[27] Aviral Kumar, Xue Bin Peng, and Sergey Levine. Reward-conditioned policies. ArXiv preprint, 2019.
[28] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. In Proc. of NeurIPS, 2020.
[29] Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning. In Reinforcement learning. 2012.
[30] Michael Laskin, Aravind Srinivas, and Pieter Abbeel. CURL: contrastive unsupervised repre- sentations for reinforcement learning. Proc. of ICMI., pages 5639-5650, 2020.
[31] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. ArXiv preprint, 2020.
[32] Hsueh-Ti Derek Liu, Francis Williams, Alec Jacobson, Sanja Fidler, and Or Litany. Learning smooth neural functions via lipschitz regularization. ArXiv preprint, 2022.
[33] Minghuan Liu, Zhengbang Zhu, Yuzheng Zhuang, Weinan Zhang, Jianye Hao, Yong Yu, and Jun Wang. Plan your target and learn your skills: Transferable state-only imitation learning via decoupled policy optimization. ArXiv preprint, 2022.
[34] Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, and Pierre Sermanet. Learning latent plans from play. In Conference on robot learning. 2020.
[35] Xiaoteng Ma, Yigin Yang, Hao Hu, Qihan Liu, Jun Yang, Chongjic Zhang, Qianchuan Zhao, and Bin Liang. Offline reinforcement learning with value-based episodic memory. In Proc. of ICLR, 2022.
[36] Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In Proc. of ICLR, 2018
11

---

[37] Ofir Nachum, Bo Dai, Ilya Kostrikov, Yinlam Chow, Lihong Li, and Dale Schuurmans, Al- gaedice: Policy gradient from arbitrary experience. ArXiv preprint, 2019.
[38] Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine. Accelerating online rein- forcement learning with offline datasets. ArXiv preprint, 2020.
[39] Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell. Curiosity-driven explo- ration by self-supervised prediction. In Proc. of ICML, pages 2778-2787, 2778-2777,21111 2017.
[40] Xue Bin Peng, Aviral Kumar, Grace 7.hang, and Sergey Levine. advantage-weighhee regression: Simple and scalable off-policy reinforcement learning. ArXiv preprint, 2019.
[41] Dean Pomerleau. Alvinn: An autonomous land vehicle in a neural network. In Proc. of NeurlPS, 1989.
[42] Antonin Rafin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Emestus, and Noah Dormann. babll-basselness: Reliable reinforeement learning implementations. Journal of Machine Learning Research, pages 1-8, 2021.
[43] John Schulman, Sergey Levine, Pieter Abbeel, Michael I. Jordan, and Philipp Moritz. Trust region policy optimization. In Proc. of ICML, pages 1889-1897, 2015.
[44] David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin A. Ried- miller. Deterministic policy gradient algorithms. In Proc. of ICML,, pages 387-395, 2014.
[45] Rupesh Kumar Srivastava, Pranav Shyam, Filipe Mutz, Wojeiech Jaškowski, and Jürgen Schmid- huber. Training agents using upside-down reinforcement learning. ArXiv preprint, 2019.
[46] Richard Sutton, Andrew G Barto, et al. Introduction to reinforcement learning. MIT press Cambridge, 1998.
[47] Shengpu Tang and Jenna Wiens. Model selection for offline reinforcement learning: Practical considerations for healthcare settings. In Proc. of ML4H, 2021.
[48] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In Proc. of IROS, 2012.
[49] Faraz Torabi, Garrett Warnell, and Peter Stone. Behavioral cloning from observation. In Proc. of IJCAL, pages 4950-4957, 2018.
[50] Hado Van Hasselt, Yotam Doron, Florian Strub, Matteo Hessel, Nicolas Sonnerat, and Joseph Modayil. Deep reinforcement learning and the deadly triad. ArXiv preprint, 2018.
[51] Aladin Virmaux and Kevin Scaman. Lipschitz regularity of deep neural networks: analysis and efficient estimation. Proc. of NeurlPS, pages 1839-3848, 2018.
[52] Ziyu Wang, Alexander Novikov, Konrad Zolna, Josh Merel, Jost Tobias Springenberg. Scott Reed, Bobak Shahriari, Noah Y. Siegel, Çaglar Gülçehre, Nicolas Heess, and Nando de Freitas. Critic regularized regression. In Proc. of NeurlPS, 2020.
[53] Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning, 1992.
[54] Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning, ArXiv preprint, 2019.
[55] Yue Wu, Shuangfei Zhai, Nitish Srivastava, Joshua M. Susskind, Jian Zhang, Ruslan Salakhut- dinov, and Hanlin Goh. Uncertainty weighted actor-critic for offline reinforcement learning. In Proc. of ICML, pages 1319-11322. 2021.
[56] Haoran Xu. Xianyuan Zhan, Jianxiong Li, and Honglei Yin. Offline reinforcement learning with soft behavior regularization. ArXiv preprint, 2021.
[57] Haoran Xu, Xianyuan Zhan, Honglei Yin, and Huiling Qin. Discriminator-weighin offline imitation learning from suboptimal demonstrations. In Proc. of ICMI., pages 24725-24742, 2022.
12

---

[58] Haoran Xu, Xianyuan Zhan, and Xiangyu Zhu. Constraints pemalized q-learning for safe offline reinforcement learning. In Proc. of AAAL, pages 8753-8760, 2022.
[59] Rui Yang, Yiming Lu, Wenzhe Li, Hao Sun, Meng Fang, Yali Du, Xiu Li, Lei Han, and Chongjie Zhang. Rethinking goal-conditioned supervised learning and its connection to offline rl. ArXiv preprint, 2022.
[60] Tianhe Yu, Aviral Kumar, Yevgen Chebotar, Karol Hausman, Chelsea Finn, and Sergey Levine. How to leverage unlabeled data in offline reinforcement learning. ArXiv preprint, 2022.
[61] Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. MOPO: model-based offline policy optimization. In Proc. of NeurIPS, 2020.
[62] Xianyuan Zhan, Haoran Xu, Yue Zhang, Xiangyu Zhu, Honglei Yin, and Yu Zheng. Deepther- mal: Combustion optimization for thermal power generating units using offline reinforcement learning. In Proc. of AAAI, pages 4680-4688, 2022.
[63] Xianyuan Zhan, Xiangyu Zhu, and Haoran Xu. Model-based offline planning with trajectory pruning. In Proc. of IJCAI, 2022.
[64] Amy Zhang, Rowan Thomas McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. Learning invariant representations for reinforcement learning without reconstruction. In Proc. of ICLR, 2021.
[65] Wenxuan Zhou, Sujay Bajracharya, and David Held. Latent action space for offline reinforce- ment learning. In Conference on Robot Learning, 2020.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Section 6.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Section
4.4.
(b) Did you include complete proofs of all theoretical results? [Yes] See Appendix C.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed reproduce the main ex- perimental results (either the supplemental material or as a URL)? [Yes] See the supplement material.
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [Yes] See Appendix D.
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [Yes] In all related figures.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] See Apendix D.
4. If you using existing assets (e.g., code, data, models) or curating/releasing
assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]] Data is from [12].
(b) Did you mention the license of the assets? [Yes] The license is Apache 2.0.
(c) Did you include any assets either in the supplemental material
URL? [Yes]
See supplement for code and four room dataset.
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A] Data is simulated.
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A] Data is simulated.
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "j_HThPtGF1h", "reviewer": "Reviewer_iPbB", "summary": "This paper proposes an approach to offline RL that first learns a policy $\\pi(s' \\mid s)$ for predicting the next *state*, and then learns another policy for extracting the corresponding action. The next-state policy (\"guide policy\") is trained to maximize a learned value function while staying close to the next states in the dataset. The action policy (\"execution policy\") is trained using a standard inverse action loss. Empirically, the proposed method outperforms prior methods on many benchmark tasks, especially ones involving sub-optimal data. Additional experiments show that the method affords efficient transfer to new tasks by finetuning just one of the two policies.\n\n--------------------\n**After author response**: The authors did a great job addressing my questions and concerns. I continue to think that this is a strong paper and should be accepted.", "questions": "1. Since the guide policy is predicting the next state, it's output could be very high dimensional, right? Does this pose a problem?\n2. How does network capacity affect the performance of the *guide* policy?\n3. Eq. 4 -- How carefully does $\\alpha$ have to be tuned?\n4. For the experiments in Fig 3, I suspect that most of the benefit from the \"mixed\" approach is that learning the execute policy on the expert data effective amounts to behavioral cloning (e.g., see analysis in [3]). If so, this benefit can be injected into the TD3+BC and CQL methods, too. For example, for TD3+BC, we could learn the Q-function on the combined dataset and train the actor on just the expert data. *How would these \"mixed\" versions of TD3+BC and CQL perform?*\n4. Can the method be used to perform transfer across agents with different action spaces? E.g., learn a guide policy for one robot and an execute policy for another robot, and then mix and match them.\n\n\n[3] https://arxiv.org/pdf/2206.03378v1.pdf, Section 3.2", "limitations": "Yes, the limitations are well addressed.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n* The idea is novel and makes intuitive sense.\n* The experimental results are strong, with a lot of care put into making them reproducible.\n* The additional experiments were good for building intuition into the method.\n* Fig 1 is great for building intuition (though the description could be clarified).\n* It's great that the paper provides theoretical results relating the policy learning losses to the performance of the resulting policy.\n\nWeaknesses\n* It's not entirely clear *why* the proposed method should work, especially since the guide policy's job of predicting the next state seems  onerous in high-dimensional tasks (e.g., AntMaze).\n* The writing is legible, but grammatical and spelling errors are frequent.\n\n\nMinor writing comments:\n* Abstract -- I didn't understand the method after reading the abstract (but the last two paragraphs of the intro do provide a good succinct description)\n* L12 \"Prophet\" -- I found this pretty confusing, as it was unclear how this relates to the execute and guide policies. I might recommend renaming everything to refer to its inputs and outputs (e.g., inverse action policy, next state policy)\n* L16 \"we highlight\" -- Great contributions!\n* L25 \"requires\" --> \"often entails\" (this isn't a strict requirement)\n* L29 \"can not ...\" -- Cite.\n* Introduction -- At some point, it would be useful to more formally characterize the difference between the IL and RL methods being discussed (e.g., \"We use imitation learning to refer to methods that do not learn a value function.\")\n* L49 \"optimal next state ... \" -- Great explanation.\n* L55 \"which owns logical\" -- I didn't understand this.\n* L60 \"We also show ...\" -- Great description of the contributions.\n* L64 \",\" --> \".\"\n* L140 \"second method\" -- Does this second method implicitly assume access to an oracle execute policy? If the action has never been taken before, then how would the agent know which action to take?\n* L141 \"acction\" --> \"action\"\n* L164 \"orronumously\" --> \"erroneously\"\n* L170 -- L179 -- It would be good to clarify that none of this material is novel, but instead is taken from IQL. This will help the reader distinguish the contributions of the paper.\n* Sec. 4.3 -- Cite many other papers that use inverse action models [1, 2], which are more relevant than RvS.\n* L206 \"conditioning variable can be automatically generated\" -- Great point.\n* Assumption 1 -- Is $(s_1, s_1')$ treated as a big, concatenated vector when taking the norm? Which norm is this?\n* Assumption 2 -- It seems like this can be made to hold by simply restricting the policy class. That is, this is not an assumption on the problem, and instead could be treated as part of the algorithm.\n* L224 -- L229, \"meaningless\", \"in vain\", \"plenty of worls\" -- Too colloquial for technical writing.\n* Eq 7 -- Precisely, how is $a^*$ defined?\n* Theorem 1 -- Does this suggest that $\\pi$ should be trained with the $\\ell-\\infty$ norm used for $\\epsilon$, or that $\\alpha = L_1 + L_2$?\n* Theorem 1 -- It would be good to acknowledge the limitation that computing the $\\ell-\\infty$ norm for $\\epsilon$ is quite hard, and its value will be quite large, potentially making the bound loose.\n* L253 \"benfits\" --> \"benefits\"\n* L255 \"on D4RL\" --> \"on the D4RL\"\n* L256 \"are consisted of\" --> \"consist of\"\n* (many places) -- \"Imitation\" --> \"imitation\"\n* L267 -- L275 -- Great job making the results reproducible.\n* Fig 2 -- I was unsure how to interpret the right panel of these plots. It seems like the higher values are in the bottom/left, but I'm unsure how that relates to $g(s)$ and $s'$.\n* L308 \"this two approach as\" --> \"these two approaches as\"\n* L318 \"lable\" --> \"label\"\n* L320 \"up tp\" --> \"up to\"\n* L361 \"may be biased\" -- I didn't understand this.\n\n[1] https://arxiv.org/pdf/1912.12773.pdf\n\n[2] http://proceedings.mlr.press/v70/pathak17a/pathak17a.pdf", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ELLD1V0c9mL", "reviewer": "Reviewer_ED7f", "summary": "This paper proposes a totally new paradigm to achieve Offline RL, compared to the alternatives that stitch actions for constraining the target policy to stay close to the behavior policy. Instead of composing actions, the proposed $POR$ allows state compositionality through learning a guided policy $g(s)$ producing **target state to reach** and optimizing the execute policy $\\pi(a|s, s')$ conditioned on the goal state in an imitation learning manner. After that, the agent makes decisions according to $\\underset{a}{\\mathrm{argmax}}\\ \\pi(a|s, g(s))$ in the inference stage. \n\nThe decoupled two-stage learning process benefits from the Imitation-based method's stability and the RL-based method's out-of-distribution generalizability. Theoretical analysis shows the lower bound of the proposed method and how this bound will be influenced by the guided policy $g(s)$. The comprehensive experiments show that $POR$ achieves SOTA on *Mujoco* and *AntMaze* dataset and new applications are enabled by this new learning paradigm. ", "questions": "Q1: As stated in the paper, $POR$ can pick logical out-of-distribution actions for reaching the state-to-go produced by $g(s)$. However, in my view,  although the $g(s)$ can generate an out-of-distribution goal state $s'$ for $\\pi$, the lack of transition $(s, a, s')$ may prevent $\\pi$ from arriving $s'$. Let's consider the toy example. When the agent is at $s=(2, 0)$ and the $\\hat{s}=g(s)$ indicates that $\\hat{s}=(3, 1)$ has the highest value, the policy is supposed to select the upper right action according to $\\hat{a}=\\underset{a}{\\mathrm{argmax}}\\ \\pi(a|s, \\hat{s})$. However, there is no such a transition $(s, \\hat{a}, \\hat{s})$ contained in the dataset, and hence the training policy $\\pi$ can not know how to arrive $(3, 1)$. Would the policy $\\pi$ fails to reach the target state $\\hat{s}$ in this situation? ", "limitations": "I am wondering whether this method can still work well on discrete control tasks.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n1) It is an interesting idea to design a goal-conditioned policy for Offline RL tasks. The learned policy only encodes state reachability via supervised learning and thus can learn stably. In addition, the out-of-distribution generalization is completed through stitching states via a variant of *Bellman Operator* and yields $s'=g(s)$ for providing a high-value target goal $s'$. \n2) The paper is well.\n3) Authors do comprehensive experiments to evaluate the new method and discuss its potential applications in the real world.\n\nWeakness and typos:\n1) Action selection should follow **Eq.(6)** in Algorithm 1, line 16 :).\n2) The axes can be added to Figure. 1 showing the toy example, so readers can know if the start index is 0 or 1. It indeed takes some time for me to find (5,4) and (6,5)\n3) The results of t-NSE are not clear enough, especially in Fig. 2, left panel, ant-maze-large. It only shows that the guided policy $g(s)$ can select some out-of-distribution goal states. However, it is hard to observe that these out-of-distribution states are in higher value than the states contained in the dataset. Consider using circles in the figure to highlight which area shows the $g(s)$ manages to generate high-value OOD states to help understand this figure.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "6h8nVNvyfnG", "reviewer": "Reviewer_PSxU", "summary": "The paper studies offline reinforcement learning (RL) and proposes to decouple the reward-maximizing objective in RL into learning a guide-policy and an execute-policy. For learning the guide-policy, the authors adopt the expectile regression and an additional BC loss. For learning the execute-policy, the author performs supervised learning by maximizing a likelihood. During the test, the guide-policy tells the execute-policy where it should go, and then the execute-policy conditioned on the current state and goal-state outputs the action. The authors also show the effectiveness of the method on a range of tasks (including the standard D4RL tasks and some transfer tasks). ", "questions": "1. What is the direct benefit of using expectile regression to learn g_\\omega rather than \\pi(a|s). The authors state that \"we only need to learn a value function while IQL needs an additional Q-function to extract the policy\". However, the authors also introduce an additional guide-policy, as well as the execute-policy. \n2. Intuitively, the expectile regression can sometimes eliminate the OOD issue in offline RL, but the authors additional introduce a behavior cloning term in Equation 4. My question is, if we replace the expectile regression in Equation 3 with the standard regression loss and then adopt the behavior cloning loss in Equation 4, what will be the difference in performance. More ablation experiments are required. \n3. The authors use both expectile regression and BC loss for learning the guide-policy.  Doesn't this make learning too conservative?\n4. In essence, the proposed method requires learning addition execute-policy. I do not see why there would be an added advantage over IQL on the D4RL tasks. \n5. In Section 4.1, the authors state that \"to generate better-than-dataset trajectories, the agent needs to take logical out-of-distribution actions\". However, the toy example assumes that offline data includes transitions of taking random actions at random states. So, is there any OOD action in this motivating example? Further, does the learned execute-policy enable the output of the so-called \"logical\" out-of-distribution actions? \n6. In Equation 4, are sampled states s and s' at two adjacent time steps? If so, I think the execute-policy will not output the (logical) out-of-distribution actions, given the current state s and goal-state s' are two adjacent states.", "limitations": "yes", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n+ The presented method is simple and well-motivated.\n+ The paper is overall well-executed. \n+ Experiments cover various scenarios, showing the effectiveness on D4RL tasks and the generalization ability of the execute-policy.\n\nWeaknesses:\n+ The proposed method is somewhat incremental, which combines the expectile regression (and a BC loss in Equation 4) for eliminating the OOD issue in offline setting and supervised goal-reaching execute-policy learning. \n+ The authors adopt the expectile regression from IQL, but do not explain the direct benefits of using expectile regression to learn g_\\omega rather than \\pi(a|s). \n+ Some references are misquoted, eg. seeing Line 264.\n+ The description of some references is inaccurate. For example, the authors state that One-step RL [3] and IQL use different weights to do behavior cloning. Is that true? \n+ The representation of some formulas is not rigorous enough. For example, a_0 in Equations 1 should be a_0 \\sim \\pi(a_0|s_0). a_t and s_t in Equation 2 should be given a clear description, eg. the relation to \\tau. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "-hNZGQ4AAQh", "reviewer": "Reviewer_3s43", "summary": "This paper proposes a new algorithm for the offline RL problem (learning a high-performing policy from a fixed dataset instead of directly interacting with an environment). The main idea is to decompose the agent into a *guide-policy* —— which suggests the highest value next state to transition to given the current state —— and an *execute-policy*, which produces the action that is likely to transition the agent to a given state in one step. By using this decomposition, the paper claims to avoid the pitfalls of methods that use an action-value function, while retaining the stability of imitation learning methods.\n", "questions": "Please address the weaknesses listed above (and fix spelling).", "limitations": "$\\alpha$ is a new \"conservative-ness\" hyperparameter similar to other offline RL algorithms, and it is not clear how to set it for a new use problem. If the authors have any algorithmic or practical suggestions for this, it would be great to have them included in the paper.\n", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\nStrengths:\n- The proposed decomposition of the agent and training process into guide and execute policies is a new and interesting contribution. \n- The main contributions of the paper are communicated clearly, with motivating examples and discussions to make the ideas understandable.\n- The experiments are reasonably well designed and explore interesting properties of the proposed POR algorithm beyond benchmark scores. I particularly like Sec. 5.3 for giving readers an interesting insight into how POR can be useful in certain situations.\n\nWeaknesses:\n- The proposed execute-policy is presented as an instance of RL via supervised learning popular more recently (citation to UDRL [1] is missing here), and this framework is discussed multiple times, but what is actually learned is in fact known as an \"inverse dynamics model\", so it would be more relevant to cite references to prior work on those. [2] is a nice paper from 2016 that learned one using neural networks, but the concept is older and more general.\n- The experiment in Sec. 5.3 is quite interesting, but it is unclear why the execute policy is not trained on $\\mathcal{D}_o$. If I understand correctly, the execute policy is not a typical policy but an inverse dynamics model as mentioned above, and it can be trained on data from any policy.\n- The paper contains a large number of spelling mistakes, which feels unprofessional. A spelling checker would easily find these.\n\n[1] Srivastava, R.K., Shyam, P., Mutz, F., Jaśkowski, W. and Schmidhuber, J., 2019. Training agents using upside-down reinforcement learning. arXiv preprint arXiv:1912.02877.\n\n[2] Christiano, P., Shah, Z., Mordatch, I., Schneider, J., Blackwell, T., Tobin, J., Abbeel, P. and Zaremba, W., 2016. Transfer from simulation to real world through learning deep inverse dynamics model. arXiv preprint arXiv:1610.03518.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept