GOPlan: Goal-conditioned Offline Reinforcement Learning by Planning with Learned Models
Authors: Mianchu Wang
Rui Yang
Xi Chen
Hao Sun
Meng Fang
Giovanni Montana
Venue: Accepted by TMLR ()

Abstract: Offline Goal-Conditioned RL (GCRL) offers a feasible paradigm for learning general-purpose policies from diverse and multi-task offline datasets. Despite notable recent progress, the predominant offline GCRL methods, mainly model-free, face constraints in handling limited data and generalizing to unseen goals. In this work, we propose Goal-conditioned Offline Planning (GOPlan), a novel model-based framework that contains two key phases: (1) pretraining a prior policy capable of capturing multi-modal action distribution within the multi-goal dataset; (2) employing the reanalysis method with planning to generate imagined trajectories for funetuning policies. Specifically, we base the prior policy on an advantage-weighted conditioned generative adversarial network, which facilitates distinct mode separation, mitigating the pitfalls of out-of-distribution (OOD) actions. For further policy optimization, the reanalysis method generates high-quality imaginary data by planning with learned models for both intra-trajectory and inter-trajectory goals. With thorough experimental evaluations, we demonstrate that GOPlan achieves state-of-the-art performance on various offline multi-goal navigation and manipulation tasks. Moreover, our results highlight the superior ability of GOPlan to handle small data budgets and generalize to OOD goals.

Published in Transactions on Machine Learning Research (05/2024)
GOPlan: Goal-conditioned Offline Reinforcement Learning by Planning with Learned Models
Mianchu Wang'
Mianchu. Wong@warriick.ce. uk
University of Warwick
Rui Yang*
ryanggam@@romnnec.uuu..kkk
Hong Kong Universiby of Science and Technology
Xi Chen Tainghua University
pochenzi@ttsigguua.eeu.
Hao Sun
hs hs7889cca.acc.k
University of Cambridge
Meng Fang University of Liverpool
Meng. Cenng,Pangalivverpooocc..u
Giovanni Montanal
g. ..mottnna@varuicc.ccom uk
University of Warwick:
Reviewed on OpenReview: https: /ttps::/oppeeeee opwnreview.ee.e. net/ forum? 20KAm
Abstract
Offline Conditoond RL (GCRL) offers a feasible paradigm for learning general-purpose policies from diverse and multi-task offline datasets. Despite notable recent progress, the predominant offline GCRL methods, mainly model-free, face constraints in handling limited data and generalizing to unseen goals. In this work, propose coal-conditioned Offline Planning (GOPlan), novel based framework that contains two key phases: (1) pretraining a prior policy capable of capturing multi-modal action distribution within the multi-goal dataset; (2) employing the reanalysis method with planning to generale imagined trajectories for funetuning policies. Specifically, we base the prior policy on an advantage- weighted conditioned generative adversarial network, which facilitates distinct mode sepa- ration, mitigating the pitfalls of out- aattof-diisrrbbutio (OOD) actions. For further policy op- timization, the reanalysis method generates high-quality imaginary data by planning with learned models for both intra-trajectory and inter-trajectory goals. With thorough experi- mental evaluations, we demonstrate that GOPlan achieves stato-of-the-an performance on varions offline multi-goal navigation and manipulation tasks. Moreover, our results highlight the superior ability of GOPlan to handle small data budgets and generalize to OOD goals.
1 Introduction
Offline reinforcement learning (RT) (Fujimoto al.) 2019 Kumar al.) 2020 Levine et al.) 2020 Janner
et al. 2021 enables learning policies from offline dataset without online interactions with the environment, offering an efficient and safe approach for real-world applications, e-g., robolics, autonomous driving, and healthcare [Levine et al. 2020). Developing a general-purpose policy capable of multiple skills is partic-
ularly appealing for RL. Offline goal-conditioned RI. (GCRL) (Chebotar et al.] 2021 Yang et al.) (022b) offers such a way to learn multiple skills from diverse and multi-goal datasets. far, prior works in this
EEuul contribution. Cooresponding authors.

---

Published in Transactions on Machine Learning Research (05/2024)
Learning Priors
Learning by Planning
Intra & trajectory reanalysis
Dynamics models
Model-based
planning
Offline dataset
Prior policy
Policy finetuning
Figure 1: The two-stage framework of GOPlan: pretraining prior policy and group of dynamics models, and finetuning policy with imagined trajectories generated by the reanalysis method.
direction (Chebotar al.) 2021 Yang et al., 2022b Ma et al. 2022 Mezghani et al. 2022 mainly focus
avoiding out-of-distribution (OOD) actions and learning to reach all in-dataset goals. Despite recent progress, efficiently mastering diverse skills with limited data and achieving OOD goal generalization remain significant challenging.
Model-based RL (Schrittwieser et al. 2020 Hafner et al. 2020 Moerland et al. 2023 is a natural choice to
overcome these two challenges for offline GCRL, as enables efficient learning from limited data (Deisenroth) Schritlwieser el. al. 2021 and enjoys better gener-
& Rasmussen 2011 Argenson & Dulac-Arnold
2021
alization (van Steenkiste et al. 2019 Yu et al. 2020 Lee et al.) 2020p. The advantages of model-based
RL in addressing the aforementioned challenges emanate from its ability to utilize more direct supervision information rather than relying solely on scalar rewards. Additionally, world models used in model-based RL are trained using supervised learning, which is more stable and reliable than bootstrapping (Yu et al. 2020). recent notable moodel-based method, reanalyse (Schrittwieser et al.) 2020 2021), employs model- based value estimation and planning to generate improved value and policy target for given states, showing advantageous performance for both online and offline RL and holding the potential for handling the limited data and OOD generalization challenges. However, it should be noted that reanalysis is primarily designed for single-task RL and cannot be directly applied for offline GCRL. This is in part due to the challenges presented by multi-goal datasets, which contain trajectories collected for heterogeneous goals. For effective long-term planning in offline GCRL, accurately modeling action modes in these datasets while avoiding OOD actions is crucial. In addition, the qucstion of what goal to plan for, must be addressed in reanalysis for oflline GCRL, which is not concern for single-task RI.
To this end, we introduce Gonl-conditioned Offline Planning (GOPlan), novel model-based algorithm designed for offline GCRL. A diagram of GOPlan is shown in Figure | GOPlan consists of two main stages, pretraining stage and reanalysis stage. During the pretraining stage, GOPlan trains policy using advantage- Conditioned Generative Adversarial Network (CGAN) to capture the multi-modal action distribution in the heterogeneous multi-goals dataset. The pretrained policy exhibits notable mode separation to avoid OOD actions and is improved towards high-reward actions, making it suitable for offline planning. A group of dynamics models also learned during this stage and will be used for planning and uncertainty quantification. During the reanalysis stage, GOPlan finetunes the policy with imagined trajectories for further policy optimization. Specifically, GOPlan generates reanalysis buffer by planning using the policy and learned models for both iutra-trajector and inter-trajectory goals. Given state in a trajectory, intra-trajectory goals are along the same trajectory as the state, and inter-trajectory goals lic in different trajectories. We quantify the uncertainty of the planned trajectories based on the disagreement of the models (Pathak et al.) 20199 Yu et al., 2020 to avoid going excessively outside the dynamics model's support. The imagined data with small uncertainty can be high-quality demonstrations that can enhance the agent's 's ability to achieve both in-dataset and out-of-dataset goals. By iteratively planning generate better data and finetune the policy with ad weighhee CGAN, the reanalysis stage significantly improves the
2

---

Published in Transactions on Machine Learning Research (05/2024)
policy's performance while also reducing the requirement for large offiine dataset. Furthermore, GOPlan can leverage the generalization of dynamics models to plan for OOD goals, which is a particularly promising feature for offline GCRL.
In the experiments, we evaluate GOPlan across multiple multi-goal navigation and manipulation tasks, and demonstrate that GOPlan outperforms prior state-of-the-art (SOTA) offline GCRL algorithms. Moreover, we extend the evaluation to the small data budget and the OOD goal settings, in which GOPlan exhibits superior performance compared to other algorithms by substantial margin. To summarize, our main contributions are three folds:
1. We propose GOPlan, novel model-based offline GCRL algorithm that can address challenging settings with limited data and OOD goals.
GOPlan consists of two stages, pretraining a prior policy via -aantage-weightee CGAN and fine- tuning the policy with high-quality imagined trajectories generated by planning.
3. Experiments validate GOPlan's efficacy in benchmarks as well as two challenging settings.
2 Related Work
Offline RL.
Addressing OOD actions for offline RL is essential to ensure safe policies during deployment.
Several works propose to avoid OOD actions by enforcing constraints on the policy (Wang et al.] 2018 Wu
ltt al.
2019 Nair et al., 2021] or penalizing the Q values of OOD state-action pairs (Kumar et al. 2020 An
et al. uni-modal Gaussian policy, which cannot capture the
2021 Bai et al. 2022 Yang et al. 2022a Sun al.) 2022). However, these methods generally use a action distribution of the heterogencous
offline dalasel. To address the multi-modality issue, PLAS (Zhou el. al.]
2020) decodes variables sampled
2022] leverages advantage- weighted latent space to further optimize the policy. A recent work (Yang et al.
from a variational auto-encoder (VAE) latent space, while LAPO (Chen et al.
2022c) demonstrates GAN's
ability to capture multiple action modes. Based on GAN, DASCO (Vuong et al., 2022
proposes a dual-
generator algorithm to maximize the expected return. Distinctively, we propose to capture the multi-modal distribution in multi-goal dataset via ddvaataage-wiihhte CGAN.
Offline GCRL. In offline GCRL, agents learn goal-conditioned policies from static dataset. One di- rection for offline GCRL is through goal-conditioned supervised learning (GCSL), which directly performs imitation learning on the relabelled data (Ghosh et al., 2021 Emmons et al. 2022.. When the data is suboptimal or noisy, weighted GCST, (Yang et al. 2022b; Wang et al. 2024 Yang et al.
2023) with multiple
weighting criteria is a more powerful solution. Other directions include optimizing the state occupancy on the targeted goal distribution (Ma et al. 2022) and contrastive RL (Eysenbach et al. 2022). Additionally,
Mezghani et al. (2022) design a self-supervised dense reward learning method to solve offline GCRL with long-term planning. Different from prior works, our work alleviates the multi-modality problem raised from the multi-goal dataset via addantage- CGAN, and efficiently utilizes dynamics models for reanalysis and policy improvement.
Model-based RL. The ability to reach any in-dataset goals requires an agent to grasp the invariance underlying different tasks. The invariance can be represented by the dynamics of the environment that can facilitate RL (Schrittwieser et al.] 2020 Hafner et al.) 2020 Nagabandi et al. 2020 Hansen et al. 2022:
Rigter et al. 2022) and GCRL (Charlesworth & Montana 2020 Yang et al. 2021). D. In the offline setting, an
ensemble of dynamics models is used to construct a pessimistic MDP (Kidambi et al. 2020 Yu et al. 2020
Reanalysis (Schrittwieser et al. 2021) uses Monte-Carlo tree search (Coullo] 2007 with learned model to generate new training targets for a given state. These dynamics models could be deterministic models (Yang et al. 2021), Gaussian models (Nagabandi et al., 2020), or recurrent models with latent space (Hafner eet al. 2019 2020). GOPlan chooses the deterministic models due to its simplicity, and with the recurrent models it can be extended to high-dimensional state space. Imagined trajectories during offline planning may contain high uncertainty, resulting in inferior performance. MOPP (Zhan et al.) (202) measures the uncertainty by the disagreement of the dynamics models and prunes uncertain trajectories. However, prior
3

---

Published in Transactions on Machine Learning Research (05/2024)
model-based offiine RL works are not designed for the multi-goal setting. Instead, perform model-based planning for both intra-trajectory and inter-trajectory goals, iteratively finetuning the GAN-based policy with high-quality and low-uncertainty imaginary data.
3 Preliminaries
Goal-conditioned RL. Goal-conditioned RL is generally formulated as Goal-Conditioned Markov Deci- sion Processes (GCMDP), denoted by a tuple < S,A,g, P, P, > where S, A, and g are the state, action and goal spaces, respectively. P(s++ st. az) is the transition dynamics, r is the reward function, and y is the discount factor. An agent learns goal-conditoned policy S + maximize the expected discounted cumulative return:
T Σ
2'm( ar 90
(T) = Py
(1)
(-
sc-1mP((18,,00))
where Pg and Po(so) are the distribution of the goals and initial states, and T corresponds to the length of an episode. The expected value of a state-goal pair is defined as: (* = E, Et so = A sparse reward is non-zero only when the goal is reached, i.e., a(s, 1 ([])()) g S s), where S 9 is the state-to-gen mapping (Andrychowicz et al. 2017) and e is threshold. This sparse reward function is accessible to the agent during the learning process. For offline GCRL, the agent can only learn from an oflline dataset B without interacting with the enviromment.
Rcanalysis. MuZero Reanalyse (Schrittwicser et al. 2020 2021 | introduces reanalysis to perform itcrative
model-based value and policy improvement for states in the dataset, resulting in an ongoing cycle of refine- ment through updated predictions and improved generated training data. Specifically, MuZero Reanalyse updates its parameters 0 for K-step model rollouts via supervised learning:
(e(0) Σ Is (TT++ss P()
Σ (z vt)
Σ 1' (uc+ss))))
(2)
where pt, of,rt rf are action, value, and reward predictions generated by the model 0. Their training largets Tith: 24t are generated via search tree, and Ut*k is the true reward. In contimuou control settings, the loss functions ((,,,, (1,,,, (())) are mean square error (MSE) between the inputs.
4 Methodology
This section introduces the two-stage GOPlan algorithm, specially designed for offline GCRL. During the pretraining stage, we implement an trrantage-weighter CGAN establish a proficient prior policy for capturing multi-modal action distribution in offline datasets, which is suitable for subsequent model-based planning. In the rcanalysis stage, enhance the performance of the agent by cnabling planning with learned models and multiple goals, resulting in a significant policy improvement. The overall framework is depicted in Figure
4.1 Learning Priors from Offline Data
Learning Prior Policy. The initial step involves learning policy that can generate in-distribution and high-reward actions from multi-goal offline data. Due to the nature of collecting data for multiple heteroge- neous goals, these datasets can be highly multi-modal (Lynch et al. 2020 Yang et al. 2022b), meaning that a state can have multiple valid action labels. The potential conflict between these actions poses a significant lcarning challenge. Unlike prior works using Gaussian (Yang et al. 2022b Ma et al., 2022), conditional Variational Auto-encoder (CVAE) (Zhou al. 2020 Chen al.] (2022) and Conditioned Generative Ad- versarial Network (CGAN) YYang et. al. 2022c), we employ Weighted CGAN the prior policy. Tn Figure
4

---

Published in Transactions on Machine Learning Research (05/2024)
using policy
using
15
15
1.0
10
0.5
0.
-0.5
-10
-1.
-15
-10
-05
05
25
10
15
200
-20
-15
-10
-55
60
10
15
2.
-2.
-15
-10
-05
0.0
05
3.0
15
20
(a-1)
(b-1)
(o-1)
(d-1)
Action reward the dataset
Action distr iuttoontioonooo using weigttel-Cuusiinnngee
Action distribution using weighted-CVA
Action distrizttion using weightsdscCAA
1.5
10
05
00
0.
-05
-0.3
055
-03
-100
1.0
-10
-1.5
-15
-1.5
-15
-15
(a-2)
(b-2)
(o-2)
(d-2)
Figure 2: An example about modeling the multi-modal behavior policy while maximizing average rewards. The z-axis represents the state, and the y-axis represents the multi-modal action. (a-1) shows the action distribution of the offline dataset. (a-2) shows the corresponding reward distribution. (b-1) and (b-2) illustrate the action distributions generated by Gaussian and Weighted Gaussian. (c-1) and (c-2) illustrate action distributions from CVAE and Weighted CVAE. (d-1) and (d-2) illustrate action distributions from CGAN and Weighted CGAN.
2 we compare six different models : Gaussian, Weighted Gaussian, CVAE, Weighted CVAE, CGAN, and Weighted CGAN on multi-modal dataset with imbalanced rewards, where high rewards have less frequency, illustrated in Figure (a-2).. The weighting scheme for Weight CVAE and Weighted CGAN is based on advantage ro-weighting YYnn et al. 2022b) and directly use rewards as weights in this example.
In Figure Weighted CGAN outperforms other models by exhibiting more distinct mode separation. As a result, the policy generates fewer OOD actions by reducing the number of interpolations between modes. In contrast, other models all suffer from interpolating between modes. Even though VAE models perform better than Gaussian models, they are still prone to interpolation due to the regularization of the Euclidean norm on the Jacobian of the VAE decoder (Salmona al. 2022). Furthermore, without employing advantage- weighting, both the CVAE and CGAN models mainly capture the denser regions of the action distribution, but fail to consider their importance relative to the rewards associated with each mode.
Based on the empirical results, the utilization of the advantage-weighted CGAN model for modeling the prior policy from multi-modal oflline data demonstrates notable advantages for oflline GCRL. In this framework, the discriminator is responsible for distinguishing high-quality actions in the offline dataset from those gener- ated by the policy, while the generative policy is designed to generate actions that outsmart the discriminator in an adversarial process. This mechanism encourages the policy to produce actions that closely resemble high-quality actions from the offline dataset. In Section 4.3 we will elaborate on the specific defmitiio of the advantage weighted weighte CGAN model.
Learning Dynamics Models. We also train a group of N dynamic models (Mp, H from the offline data to predict the residual between current states and next states. Each model Ms. minimizes the following loss:
[(vr)) (( [1]MMs (St, az) (S141-sp)
Then the predicted next state is Bi+1 = s, + My, (St, at). As the dynamics models are trained through supervised lcarning, making them more stable and better equipped with generalization abilitics compared to bootstrapping (Yu et al. 2020). In the subsequent section, we will illustrate how the learned models can be utilized to enhance the training efficiency and generalization ability of offline GCRL.
5

---

Published in Transactions on Machine Learning Research (05/2024)
Intra-trajectory reanalysis
Start state
Goal state
Last state imagined trajectory
Obstacle
Dataset trajectory Imagined trajectory
(a-1)
(a-2)
(a-3)
Inter-trajectory reanalysis
I Uncertainty range
dataset trajectory
Save imagined trajectory
Do nothing
(b-2)
(B-3)
Figure 3: Musttatiion of intra-trajectory and inter-trajectory reanalysis. There are six scenarios: (a-1) the imagined trajectory is valid and better than the original trajectory; (a-2) the imagined trajectory fails reach the goal within the same number of steps as the original trajectory; (b-1) valid imagined trajectory connects the state to inter-trajectory goal; (b-2) valid imagined trajectory that does not achieve the desired goal; (a-3) (b-3) invalid imagined trajectories with large uncertainty.
4.2 Learning by Planning with Learned Models
Based on the learned prior policy and dynamics models, we reanalyse and finetune the policy in this learning procedure. As the dynamics information encapsulated in the offline dataset remains invariant across different goals, utilize this property equip our policy with the capacity to achieve a broad spectrum of goals by reanalysing the current policy for both intra-trajectory and inter-trajectory goals. The reanalysis procedure is shown in Figure where apply model-based planning method to generate trajectories for selected goals and save potential trajectories that can help improve the policy and have small uncertainty. By integrating this imagined data to update the policy, we repeat the aforementioned process to iteratively improve the policy.
Model-based Planning. We first introduce our planning method that serves intra-trajectory and inter- trajectory reanalysis. Our planner is similar to prior model-predictive approaches (Nagabandi al., 2020:
Argenson & Dulac-Arnold 2021 Charlesworth & Montana 2020 Schrittwieser et al. 2021] where the planner employs a model to simulate multiple imaginary trajectories, assigns scores to the initial actions, and generates actions based on these scores. Following prior work (Charlesworth & Montana 2020 given the current state 8, and the selected goal g. our planner firstly samples C' initial actions {4f) c-1 from the prior generative policy π and predicts C next states { based on a randomly selected dynamics model Mp.- To score each initial action, the planner duplicates every next state H times and generates imagined trajectories of K steps with the policy and a randomly chosen dynamics model at each step. For each initial action af, the planner averages and normalizes all cumulative returns commencing from that initial action as R(as), where the reward for every generated step is computed by the sparse reward function.
Σe cnR(of)a
Finally, the planner's output action for st, is 0r De om
where k is a hyper uyper-parameter tuning the
exponentially weights for actions. It worth noting that advantag--weighted CGAN effectively provides diverse in-dataset actions with higher rewards, enabling efficient and safe action planning for offline GCRL. The planning algorithm outlined in Algorithm 2 provided in Appendix A

---

Published in Transactions on Machine Learning Research (05/2024)
Intra-trajectory and Inter-traeectory Reanalysis. Offine goal-conditioned planning involves the chal- lenges of choosing planning goals and selecting appropriate imaginary data for policy training. In this paper, we present a novel approach that provides solutions to these challenges by incorporating intra-trajectory and inter-trajectory reanalysis, as illustrated in Figure 3 Specifically, our approach includes designed mech- anisms for assigning goals and selecting generated trajectory to be stored in a reanalysis buffer Bre for subsequent policy improvement.
Since the primary objective of offine GCRL is reach all in-dataset goals, utilizing all the states in the dataset goals appears to be natural choice. These goals can be classified into two categories: intra- trajectory goals, which are along the same trajectory as the starting state, and inter-trajectory goals, which lie in different trajectories. The division criterion is determined by the existence of a path between the state and the goal. Intra-trajectory goals refer to those that already have a path that links the state to them and hence, they can be reached with high probability because the successful demonstration has been provided. In contrast, inter-trajoctory goals may not have a valid path to be reached from the given state. Therefore, different criterion is needed for the two situations.
The main idea of the two types of reanalysis is to store improved trajectories than the dataset and remove invalid trajectories with large uncertainty. The criterion of removing unrealistic imagined trajectories with large uncertainty is shared by both intra-trajectory and inter-trajectory reanalysis, because imitating these trajectories can lead the policy excessively outside the support of the dynamics model. To achieve this, we measure the the uncertainty of a trajectory by accessing the disagreement of a group of learned dynamics models. Specifically, given trajectory T {so, 0o, $1, a1, sH}, the uncertainty is defined as the maximal disagreement on the transition within the trajectory:
N Σ (sk, 56+1l12.
U(T) = max OSKSH N
(4)
where Ma. is the i-th model and 8k+1 is the mean prediction over N models: 8k+1 = 1/2 Eim Mo. (sks an).
Intra-trajectory reanalysis. randomly samples trajectory and two states st, SL+k T, where k > 0. After that, the model-based planning approach mentioned earlier is then employed to generate an action at each step from St with the aim of reaching ((5+4), transiting according to the mean prediction of 2 E NJ. There are mainly three cases: (a-1) if the planner reaches (($++*) with fewer than k steps and the uncertainty of the imagined trajectory is less than a threshold value u, the generated virtual trajectory is stored into a rcanalysis buffer Bre. In cases where the (a-2) policy fails to reach Φ($++)) within k steps or (a-3) the generated virtual trajectory has larger uncertainly than u, the original trajectory Tea+ is included in Bre because
enhance learning the original successful demonstrations during fine-tuning,
Inter-trajectory reanalysis. involves randomly selecting a state in trajectory Ti and another state 8g in a different trajectory 72, followed by model-based planning from 8, to reach Φ(8g). There are also three cases: (b-1) if the planner successfully reaches d(sg) with an uncertainty less than u, we include the generated virtual trajectory 7 into the reanalysis buffer Bre. (b-2) Moreover, situations where the policy fails to attain the intended goal state o(sg) but the associated trajectory exhibits low levels of uncertainty, the virtually achieved goals along the generate trajectory are labeled the intended goal for the tra trajocyory.
This approach enables the utilization of failed virtual trajectories to enhance the diversity of the reanalysis buller B (b-3) Additionally, if the imagined trajectory has high uncertainly, we do not save such invalid trajectory.
4.3 Policy Optimization
In GOPlan, the policy is updated during both the pretraining stage and the finetuning stage. We train the policy x via aevintage-weighted CGAN, which can be formulated as the following objective:
max minE, minn (o..r g)~B [u(se, ar, 9) log D(st, ar g))
D
(5)
+
[log(1 - D(st, D(st,d' d'gg)
7

---

Published in Transactions on Machine Learning Research (05/2024)
Here, D is the discriminator weighted by w. a(s,a.g) = exp(A(ss a. a. .)) is the exponential advantage weight, where ((8,g) serves as a normalizing factor to ensure that ΣaeA w(s, a, g)) 8. = 1. is the behavior policy underlying the relabeled offline dataset B. The advantage function can be estimated by a learned value function Vo.: A(st, ar, 9) = r(s,, az 9) + +oVo, (8++1.9) Vo, (S).g). The value function Vo, (5t.
is updated to minimize the TD loss:
[(0,)) = B( [VVo, (st. g) - yr)
(6)
where the target Ve rr + V. (5141,9) and the target network Vo, is slowly updated to improve training stability.
To optimize the minmax objective in Eq.equation 5 we train a discriminator D and a generative policy æ separately. The discriminator D, with parameter Od. learns to minimize the following loss function:
[(0a) g)) [w(st, az 9) log Døg (s,, (.,)) - =.s..))B.:= [log(11 - Dea a', g))l, a'-mTe (8..9.2)
(7)
where P(z) constitutes random noise that follows diagonal Gaussian distribution. The discriminator's output is passed through sigmoid function, thereby constraining the output to lie within (0, 1). To deceive the discriminator, the policy network with parameter 0,, is trained to minimize the following loss function:
[(0x)
(ss. -B) [log(1 Da d' .g)]l.
a a'a (569.2)
Through this training process, the policy network is capable of producing actions that closely resemble high- quality actions in the dataset and can enjoy policy improvement similar to prior advantage-weigee works WWag et al. 2018
Peng et al. 2019
4.4 Overall Algorithm
In the pretraining stage, we train the dynamics models M E [1, the discriminator Des the value function Vo, and the policy Te, until convergence. In the subsequent reanalysis stage, we generate new trajectories through intra-trajectory and inter-trajectory reanalysis, save them into the reanalysis buffer, and then employ the reanalysis buffer to finetune the value function, the discriminator, and the policy, The process of rcanalysis and fine-tuning is repeated over iterations to improve the policy performance. Algorithm D outlines the details of the two main stages of the overall algorithm.
5 Experiments
We conduct a comprehensive evaluation of GOPlan across a collection of continuous control tasks (Plappert) et al. 2018 Yang et al.) (002bb) with multiple goals and sparse rewards. In addition, we also assess the efficacy of GOPlan in settings with limited data budgets and OOD goal generalization.
Environments and Datasets. To conduct benchmark experiments, we utilize offline datasets from Yang et al. 2022b). The dataset contains 1 x 10 transitions for low-dimensional tasks and 2 x 10 for four high- dimensional tasks (FetchPush, FetchPick, FetchSlide and HandReach). Furthermore, to demonstrate the ability to handle small data budgets, we also integrate an additional group of small and extra small datasets, referred to with a suffix "-s" and "-es", containing only 10% and 1% of the number of transitions. To assess GOPlan's ability to generalize to OOD goals, we leverage four task groups from (Yang al., 2023): : FetchPush Left- Left-Right, FetchPush Near-Far, FetchPick Light, and FetchPick Low- High, each consisting of a dataset and multiple tasks. For instance, the dataset of FetchPush Left-Right contains trajectories where both the initial object and achieved goals are on the right side of the lable. As such, the independent identically distributed (IID) task assesses agents handling object and goals on the right side (i.e., Right2Right), while the other tasks in the group assess OOD goals or starting positions, such as Right2Left, Left2Right, and Left For further information regarding the environments and datasets used in our evaluation, we refer readers to Appendix B
8

---

Published in Transactions on Machine Learning Research (05/2024)
Algorithm Goal-conditioned Offline Planning (GOPlan).
Initialise: N dynamics models {1x discrim- inator 0d. policy 0,,, goal-conditioned value function 0,; offline dataset B and reanalysis
def' Intra_trajl,
1: ($1, 81+1, 91+K:9) B, sc 3t
2: for k = 0,... K do
buffer Bre; the state-to-goal mapping 6.
3: a++a = Plan($s 1 ((s++K))
1: # Pre-train
4: SE+R+11 = Ma (Betks (z4k)
2: while not converges do Update (v))== using B. Update 0, using B.
5: if US ((+)+):0(+k) > u then
6: return {81, s, SttK Stt.gg)
Eq.
end if
Update 84 using B.
Eq.
if $1+R+1 achieves ((5++) then
Update 0x using B.
Eq.
9: return {st, Set1. B (at44i,,0(ss)) (($(+k))
7: end while
10: end if
8:
11: end for
9: # Finetune
12: return S11,
10: for !.... I do
def Inter_traj
11: for j = ..... intra do
13: so ~ B, Sy ~ B, 80 so
12:
Intra traj()
14: for t t= ..... do
13:
Bye Bree Ut
15: a (1a ((s)))
14:
end for
16: $111 = M Ma N (6),
15:
for j = 1,... Tinter do
17: if ( 4x) > then
16:
7 = Inter traj()
18: return (0}
17:
Br = Bre UT
19: end if
18:
end for
20: if $141 achieves o(sy) then
19:
Finetune 0, using Bre-
Eq.
21: return (so, $1, S411, ((sg))
20: Finetune Og using Bre-
Eq.
22: end if
21:
Finetune Oz using Brc.
Eq.
23: end for
22:
end for
24: return
{so, $1, S7, 6(3))}
We compare GOPlan against SOTA offline GCRL algorithms, WGCSL Yang et al. 2022b contrastive RL (CRL) [Esenbbach et al.] 2022), actionable models (AM) (Chebotar et al. 2021]. GCSL Ghosh et al. 2021), and modified offine RL methods, such as TD3-BC (g-TD3-BC))) (Fujimoto
Experimental Setup.
exponential advantage weighting (GEAW) (Wang et al. 2018 Trajectory Transformer (TT) 2 Decision Transformer (DT) (Chen et al. 2021 We denote variant approach of GOPlan as GOPlan2, which employs besting-time model-based planning with candidate actions from the The testing- gesting-time planning method aligns with the model-based planning approach described 4.2 For all experiments, we report the average returns with standard deviation across different
& Gu 2021).
(Janner et al.
GOPlan policy.
in Section
random seeds.
Implementation details are provided Appendix C
5.1 Benchmarks and Small Dataset Results
Table [ demonstrates the performance of GOPlan and baselines in the benchmark tasks. As shown in the table, GOPlan shows improved performance over current SOTA model-free algorithms, achieving the highest average return on 8 out 10 tasks. Notably, unlike prior online model-based GCRL approaches (Charlesworth & Montana, 2020 Yang al. 2021) that fail to handle high dimensional tasks like HandReach, GOPlan works well for these tasks. This perhaps can be attributed to the fact that GOPlan incorporates both the strengths of model-free advantage- weighted
weighttte policy optimization and model-based reanalysis techniques.
Apart from the benchmark results, we conduct experiments under a small data setting that only has th of the dataset used for benchmark experiments, and we observe that GOPlan surpasses other baselines by a significant margin. Specifically, GOPlan delivers an average improvement of 23% compared to the best- performing bascline CRL, and 38% compared to WGCSL. It is noteworthy that GOPlan alone achieves an average return over 10 on the challenging HandReach-s task, while other baselines fail on the HandReach-s task with small data budget and high-dimensional states.
9

---

Published in Transactions on Machine Learning Research (05/2024)
Table 1: Average return with standard deviation on the offline goal-conditioned benchmark and the small
dataset setting.
Task
GOPlan
BC
GCSL
WGCSL
GEAW
AM
CRL
g-TDB-BC
g-TT
g-DT
PointReach
46.09
39.36
39.27
44. 2011
42.96
43.5620.6
44.33
43.69
45.39
45.
PointRoous
43.
33.17
a3.
36. 25.
36.02 22..0
33.45+19
40. 40.1221.
42.32=a.
42.4620.8
43 13.20
Reacher
40.67
35.72+0.3
36.42=o.3
40.57=0.2
38. 80.
37.4844.1
37. 37.7
37.39+0.
38. 28.79
65
SawyerReach
10.43.0.3
32.91103
33.65
40. 10.12
37.4210.3
40.9110.
36.73103
30.9611.7
40.251oa
41.10 10.3
SawyerDnor
40.4
35.0310.2
35.67
42.81
40.03
42.4910.5
38.38.0.4
35.90
44.23 44.3210.4
FetchReach
40.3
42.03
41.72=a.a
46.33
45.01
46.5010.1
46.10 10.11
45.31
47.15.0.1
47 47.03
FetchPush
39.15.0.6
31.56+n.6
28.56_a.9
39. 39.11
37 47.42
30.49+21
36.32=n.6
30.83
38 38.90=o.s
38 63
PetchPick
37.00 11.1
31,7511.2
25.22_oa
34.37 20.5
34.5640.5
34.07
35.77.0.2
36.51
36.65 36.6520.8
35. 2..2
FetchSlide
10.08
0.84
3.05_a.
10.73
4.5511.7
6.9211.2
9.91
5.88
9.61a110
8.20
HandReach
28.28 15.3
0.06
0.57
26.73 1.2
0.81 11.5
0.0210.0
6.46
5.21
24.1211.4
22.80
Average
37.70
28.24
27.71
36.13
31.76
31.58
33.25
31.42
36.76
36.49
PetchPushes
37.3110.3
25.5411.
26.30 20.7
32.35
33.65 33.68
32.93
31.72.1.5
30.9210.
37.02 37.02.1.1
37 37.10.0.1
FetchPick-s
32.8510.3
23.0511.0
23.71
29.12
30.9210.
25.5613.
32.27.0.8
29.0614.0
32.5311.5
31.23
FetchSlide-s
1.04
0..1
0.98
0.22go.1
0.30
1.97
4.74
0.1610.2
2.39
1.60
HandReach-s
10.11
0.164n.1
0.13_a.1
0.12
0.03
0.0840.1
0.45.0.3
1.6
4.22.0.6
5.69
RetchPush--rs
18.91 21:2
10.37
5.96
13.25
10.35
11.28,
16. 13.1.8
6.2720.9
15.76.1.2
16.56.1.4
FetchPick--s
14.19
4.16120
1.67g0s
6.49
3.17
4.16
13.12.0.8
5.96
12.28
11.82
Average
19.73
10.63
9.79
13.59
13.07
12.66
16.40
12.33
17.37
17.33
g-TD3-80
40
35
30
25
20
15
5
0-
RotchPush Logh-
FetchPush Mear-Rar
RotchRick Loft-Right
PatchPick Low-High
Figure 4: Average performance OOD generalization tasks over 5 random seeds. The error bars depict the upper and lower bounds of the returns within each task group.
5.2 Generalization to OOD Goals
We also conducted experiments on the four OOD generalization task groups in Figure 4 The results demonstrate that GOPlan outperforms other baselines. Specifically, by planning actions for OOD goals, GOPlan2 showcases significant improved performance, outperforming WGCSL by nearly 20% on average and enjoying less variance across TID and OOD tasks than recent SOTA method GOAT (Yang et al. 2023) for OOD goal generalization. The full results of this experiment can be found in Appendix[D.. [.I] The results obtained from our experiments provide empirical evidence in support of the effectiveness of using dynamics models for enhancing generalization towards achieving OOD goals.
5.3 Robustness to Noise
In this experimental setup, we investigated the robustness of various algorithms modified FetchReach environments, which are subjected different levels of Gaussian action noise applied the policy action outputs. The datasets for these environments were obtained from Ma et all (2022). As shown in Figure GOPlan exhibits robust performance in this setup even under noise level of 1.5.
10

---

policy approach and its potential for Published in Transactions on Machine Learning Research (05/2024)
GOPan GCSL
Discounted Retur
64
Noise Level
Figure 5: Stochastic environment evaluation.
5.4 Finetuning with Different Models
In offline GCRL, the reward function is always accessible for the agent, so we do not need to learn reward model. However, it is feasible to learn a world model, including the transitions and the rewards. We design two variants: GOPlan with reward models, which we do not use the environment-define reward function, and instead train reward model using the offline dataset with a binary cross entropy loss to predict the sparse reward; GOPlan with RSSM, in which we replace the environment-defmed reward function and our learned transition models with the RSSM world model used in Dreamer (Hafner et. al. 2019 2020). Since the inference of the RSSM model consists of stochastic latent code, we directly indicate the uncertainty by its standard deviation. In Figure 2 we evaluate the two variants with GOPlan in FetchPush, FetchPick and SawyerDoor. We found that GOPlan with reward models shows similar performance as the vanilla GOPlan. However, GOPlan with RSSM shows slight performance decline.
FetchPush
FetchPick
SawyerDoor
GOPlan GOPlan with reward models
39.15
37.01+1.1 37.10+1.0
44.42+0.3
30.5
44.32.0.3
GOPlan with RSSM
38.20
36.40
43.20
Table 2: Comparsions between GOPlan, GOPlan with learned reward models, and GOPlan with RSSM.
5.5 Ablations
This section investigates the impact of different components the performance of GOPlan.
Different prior policy for planning. Initially, compare different models the prior policy for planning. We denote planning with model "X" as "X-plan", and advanlage- CGAN as "ACGAN". Based on the results in Table 3 GOPlan2 and ACGAN-plan exhibit superior performance over Gaussian and VAE models, owing to the performant OOD action avoidance capabilitics, which are advantagoous for long-term planning. Furthermore, policies that incorporate advantage weighting, including ACGAN, Weighted CVAE, and Weighted Gaussian, have shown superior performance compared to their unweighted counterparts. This underscores the efficacy of the application in planning scenarios.
Ablation on the two stages of GOPlan. Next, we compare the performance of three different variants of GOPlan to assess the significance of the pretraining and finetuning stages in GOPlan. The variants
include: (1) ACGAN (iee., GOPlan without finetuning), (2) CGAN with w(s, a, g) = 1 in Equequation 7 (3) ACGAN-plan with model-basod planning with action candidates from ACGAN. Results presented Figures (a)(b) demonstrate that ACGAN consistently outperforms CGAN in terms of average return. Furthermore, ACGAN-pla is stable and efficient, but imposes excessive computation for online interaction
11

---

Published in Transactions on Machine Learning Research (05/2024)
Regum
Maa mio00rr
130000 Z00N0 2033 Gradiont 40000
30000
508000
200880 10:00
40000
20000
231 vas Pan 1988 17403 V98S 17123 WED
240 100
PER TOt Lnve 118e 1IVE 2000
(a) FetchPick pretrain
(b) SawyerDoor pretrain
FetchPick finetune
(d) SawyerDoor finetune
Figure 6: Ablation studies of GOPlan. The static lines in (a) and (b) are the convergent performance of
COPlan, i.e, ACGAN after finetuning.
Table 3: Comparison of different prior policies for planning.
Algorithm
FetchPick
FetchPush
SawyerDoor
GOPlan2
38.20
39.75
45.13
ACGAN-plan
37.01
38.42
44.42.0.a
CGAN-plan
33.95+2.3
36.51+2.6
42.04+1.0
weighted CVAE-plan
35.02.0.9
39.11
42.88
CVAF-plan
30.32+1.2
34.36+1.1
39. 73.777
weighted Gaussian-plan
36.41
38.62
43.71
Gaussian-plan
29.95+1.0
30.97
36.61
(30 Hz GTX 3090). After finetuning ACGAN with our reanalysis methods, GOPlan achieves comparable performance as ACGAN-plan, and offers faster interaction (500 Hz).
Inter-trajectory and intra-trajectory reanalysis. During finetuning, both inter-trajectory and intra- trajectory reanalysis techniques are used in GOPlan. To assess the effectiveness of each approach, we compare the individual impacts and examine their joint influence on performance, as illustrated in Figures 0 (c). Specifically, GOPlan generates 50% trajectories for each reanalysis, while inter-trajectoor"(( "intra- trajectory") performs 100% inter-trajectory (or (traatrrajecoor)) reanalysis. Our findings are threefold: (1) inter-trajcctory reanalysis can introduce distribution shift that initially decreases performance, while intra- trajectory reanalysis has such effect. (2) Both intra and inter-trajectory reanalysis lead considerable improvement over the pretrained policy after sufficient finetuning steps. (3) The two reanalysis methods can work synergistically to yicld the best performance.
mocats
models
Return
Fnvironments
Figure Ablation studies of GOPlan on different ensemble sizes.
Ensemble sizes and uncertainty truncation. We investigates the influence of different ensemble sizes
of the dynamics models the performance of GOPlan. We use 2, 5, 10 dynamics models in the finetuning
12

---

Published in Transactions on Machine Learning Research (05/2024)
stage to generate data for reanalysis. The results, presented in Figure 0 demonstrate the robustness of COPlan to varying ensemble sizes, with all ensemble sizes exceeding the performance of a single dynamics model. This observation underscores the significance of uncertainty qualification through ensembles for GOPlan's performance. Notably, when only one model employed, the inability to estimate uncertainty for truncaling trajectories with high uncertainly leads lo a degradation performance. Furthermore, in the FetchPush task, where the variance in performance relatively small, the average performance exhibits a stable improvement as the number of models increases.
6 Conclusions and Discussions
This paper proposes GOPlan, a novel model- offline GCRL algorithm designed to effectively learn gen- eral goal-reaching policies from diverse and multi-goal offiine datasets. GOPlan comprises two components: pretraining prior policy using an advantage-weighe weighted CGAN and finetuning the policy with reanalysis. The mrantag-wweightee CGAN exhibits distinct mode separation and enhances the action distribution based on advantage values, thereby mitigating the OOD action issue and promoting long-term planning for offline GCRL, In addition, the reanalysis method generates high-performing and low-uncertainty imaginary samples by planning with learned models towards both intra-trajectory and inter-trajectory goals. Exper- imental results demonstrate that GOPlan achieves SOTA performance on offline GCRL benchmark tasks. Importantly, GOPlan yields greater improvements for challenging settings with limited data and OOD goal generalization, highlighting its potential advantages for practical scenarios.
Despite its promising features, GOPlan could be enhanced to efficiently learn from human demonstrations. which exhibit more diverse patterns than our lesting benchmarks (Shafiullah et al. 2022 Lynch et al. 2019). Future research directions also include extending GOPlan to nigh-dimensional setting. which may require planning with advanced world models as proposed (Janner et al., 2021 Wang et al. 2023 Janner et al. 2022), ; as well planning latent spaces as studied in (Hafner et al.] 2019 Nguyen et al. 2021p. In addition, researchers should consider its potential ethical issues when applying the technique to real-world applications, such as the misuse of technology and the safety in scenarios where mistakes could be hazardous.
Acknowledgements
We would like to thank Yue Jin, the editors and the reviewers for their comments to improve the paper. We also express our gratitudc to Tristan Tomilin for his contribution through his presentation on this paper at the Goal-conditioned RL workshop at NeurIPS 2023. GM acknowledges support from UKRI Turing AI Acceleration Fellowship (EPSRC EP /V0024488//)
References
Gaon An, Seungyong Moon, Jang- Hyun Kim, and Hyun Oh Song. Uncertainty-based offline reinforcement learning with diversified q-ensemble. In Advances in Neural Information Processing Systems, 2021.
Marcin Andrychowiez, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojeiech Zaremba. Hindsight experience replay. In Advances in Neural Information. Processing Systems, 2017.
Arthur Argenson and Gabriel Dulac Arnol. Model-based offline planning. In International Conference on Learning Representations, 2021.
Chenjia Bai, Lingxiao Wang, Zhuoran Yang, Zhi-Hong Deng, Animesh Garg, Peng Liu, and Zhaoran Wang. Pessimistic bootstrapping for uncertainty-driven offline reinforcement learning. In International Confer- ence on Learning Representations, 2022.
Henry Charlesworth and Giovanni Montana. PlanGAN: Model-based planning with sparse rewards and multiple goals. In Advances in Neural Information Processing Systems, 2020.
13

---

Published in Transactions on Machine Learning Research (05/2024)
Yevgen Chebotar, Karol Hausman, Yao Lu, Ted Xiao, Dmitry Kalashnikov, Jacob Varley, Alex Irpan, Benjamin Eysenbach, Ryan C Julian, Chelsea Finn, and Sergey Levine. Actionable models: Unsupervised offline reinforcement learning of robotic skills. In International Conference on Machine Learning, 2021.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Picter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforeement learning via soquence modeling. In Aduances in Newral Information Processing Systems, 2021.
Xi Chen, Ali Ghadirzadeh, Tianhe Yu, Jianhao Wang, Alex Yuan Gao, Wenzhe Li, Liang Bin, Chelsea Finn, and Chongjie Zhang. I.APO: Latent-variable srantage-weighted policy optimization for offline reinforcement learning. In Advances in Neural Information Processing Systems, 2022.
Rémi Coulom. Efficient selectivity and backup operators in monte-carlo tree search. In H. Jaap van den Herik, Paolo Ciancarini, and H. H. L. M. (Jeroen) Donkers (eds.), Computers and Games, pp. 72-83, Berlin, Heidelberg, 2007. Springer Berlin Heidelberg. ISBN 978-3--44--55388-6
Marc Deisenroth and Carl E. Rasmussen. PILCO: A model-based and data-efficient approach to policy search. In International Conference on Machine Learning, 2011.
Scott Emmons, Benjamin Eysenbach, Ilya Kostrikov, and Sergey Levine. RvS:
What is essential for offline RL via supervised learning? In International Conference Learning Representations, 2022.
Benjamin Eysenbach, Tianjun Zhang, Sergey Levine, and Ruslan Salakhutdinov. Contrastive learning as goad-condit reinforcement learning. In Advances in Newral Information Processing Systems, 2022.
Scott Fujimoto and Shixiang Gu. A minimalist. approach to offline reinforcement learning. In Advances in Newral Information Processing Systems, 2021.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. International Conference on Machine Learning, 2019.
Dibya Ghosh, Abhishek Gupta, Ashwin Reddy, Justin Fu, Coline Manon Devin, Benjamin Eysenbach, and Sergey Levine. Learning to reach goals via. iterated supervised learning. In International Conference Learning Representations, 2021.
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International Conference on Machine Learning, 2019.
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream control: Learning behaviors by latent imagination. In International Conference Learning Representations, 2020.
Nicklas A. Hansen, Hao Su, and Xiaolong Wang. Temporal difference learning for model predictive control. International Conference on Machine Learning, 2022.
Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. In Advances in Neural Information Processing Systems, 2021.
Michael Janner, Yilun Du, Joshua B. Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis, 2022.
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. MOReL: Model-basse offline reinforcement learning. In Advances in Neural Information Processing Systems, 2020.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Tevine. Conservative Q-learning for offline rein- forcement learning. In Advances in Newral Information Processing Systems, 2020.
Kimin Lec, Younggyo Sco, Scunghyun Leo, Honglak Lee, and Jinwoo Shin. Context-aware dynamics model for generalization in model-based reinforcement learning. In International Conference on Machine Learning, 2020.
14

---

Published in Transactions on Machine Learning Research (05/2024)
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offine reinforcement learning: Tutorial, review, and perspectives on open problems, 2020.
Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, and Pierre Sermanet. Learning latent plans from play. In Conference on Robot Learning, 2019.
Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, and Pierre Sermanet. Learning latent plans from play, In Anmual Conference on Robot Learning, 2020.
Yecheng Jason Ma, Jason Yan, Dinesh Jayaraman, and Osbert Bastani. Offlinc goal-conditioned reinforce- ment learning via f-advantage regression. In Advances in Neurol Information Processing Systems, 2022.
Lina Mezghani, Sainbayar Sukhbaatar, Piotr Bojanowski, Alessandro Lazaric, and Karteek Alahari. Learning goal-conditioned policies offline with self-supervised reward shaping. In Annual Conference on Robot Learning, 2022.
Thomas M Mocrland, Joost Brockens, Aske Plaat, Catholijn M Jonker, et al. Model-based reinforcement learning: A survey. Foundations and Trends0 in Machine Learning, 16(1):1-118, 2023.
Anusha Nagabandi, Kurt Konolige, Sergey Levine, and Vikash Kumar. Deep dynamics models for learning dexterous manipulation. In Annual Conference on Robot Learning, 2020.
Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. AWAC: Accelerating online reinforcement learning with offline datasets, 2021.
Tung D Nguyen, Rui Shu, Tuan Pham, Hung Bui, and Stefano Ermon. Temporal predictive coding for model-based planning in latent space. In International Conference on Machine Learning, 2021.
Deepak Pathak, Dhiraj Gandhi, and Abhinav Gupla. Self-supervised exploration via disagreement. In International Conference on Machine Learning, 2019.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Enrantage-weighter regression: Simple and scalable off-policy reinforcement learning, 2019.
Matthias Plappert, Marcin Andrychowicz, Alex Ray, Bob McGrew, Bowen Baker, Glemn Powell, Jonas Schneider, Josh Tobin, Macick Chociej, Peter Welinder, Vikash Kumar, and Wojeiech Zaremba. Multi- goal reinforcement learning: Challenging robotics environments and request for research, 2018
Mare Rigler, Bruno Lacerda, and Nick Hawes. RAMBO-RT: Robust adversarial model-based offline rein- forcement learning. In Advances in Newral Information Processing Systems, 2022.
Antoine Salmona, Valentin de Bortoli, Julie Delon, and Agnès Desolneux. Can push-forward generative
models fit multimodal distributions?, 2022.
Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, Timothy Lillicrap, and David Silver. Mastering atari, go, chess and shogi by planning with learned model. Nature, 588(7839):604 -69, December 2020.
Julian Schrittwieser, Thomas Hubert, Amol Mandhane, Mohammadamin Barekatain, Ioannis Antonoglou, and David Silver. Online and offline reinforcement learning by planning with a learned model. In Advances in Neural Information Processing Systems, 2021.
Nur Muhammad Mahi Shafullahh, Zichen Jeff Cui, Ariuntuya Altanzaya, and Lerrel Pinto. Behavior trans- formers: Cloning k modes with one stone. In Advances in Neural Information Processing Systems, 2022.
Hao Sun, Lei Han, Rui Yang, Xiaoteng Ma, Jian Guo, and Bolci Zhou. Exploit reward shifting value-basod deep-rl: Optimistic curiosity- based exploration and conservative exploitation via linear reward shaping. Advances in Neural Information Processing Systems, 35:37719-3777 37734, 2022.
bassd
15

---

Published in Transactions on Machine Learning Research (05/2024)
Sjoerd van Steenkiste, Klaus Greff, and Jürgen Schmidhuber. A perspective on objects and systematic generalization in model-based RT, 2019.
Quan Vuong, Aviral Kumar, Sergey Levine, and Yevgen Chebotar. Dual generator offline reinforcement lcarning, 2022.
Mianchu Wang, Yue Jin, and Giovanni Montana. Goul-conditioned oflline reinforcement learning through state space partitioning, Machine Learning, Feb 2024.
Pengqin Wang, Meixin Zhu, and Shaojie Shen. Entropy: Environment transformer and offline policy opti- mization, 2023.
Qing Wang, Jiechao Xiong, Lei Han, peng sun, Han Liu, and Tong Zhang. Exponentially weighted imitation learning for batched historical data. In Advances in Neural Information Processing Systems, 2018.
Yifan Wu, George Tucker, and Ofrr Nachum. Behavior regularized offline reinforcement learning, 2019.
Rui Yang, Meng Fang, Lei Han, Yali Du, Feng Luo, and Xiu Li. MHER: Model-based hindsight experience replay. In Deep RI. Workshop NeurIPS 2021, 2021.
Rui Yang, Chenjia Bai, Xiaoteng Ma, Zhaoran Wang, Chongjie Zhang, and Lei Han. RORL: Robust offline reinforcement learning via conservative smoothing. Advances in Neural Information Processing Systems, 35:23851 22128851-23666, 2022a.
Rui Yang, Yiming Lu, Wenzhe Li, Hao Sun, Meng Fang, Yali Du, Xiu Li, Lei Han, and Chongjie Zhang, Re- thinking goal-conditioned supervised learning and its connection to offline RL. In International Conference on Learning Representations, 2022b.
Rui Yang, Yong Lin, Xiaoteng Ma, Hao Hu, Chongjie Zhang, and Tong Zhang. What is essential for unseen goal generalization of offline goal-conditioned RL? In International Conference on Machine Learning, 2023.
Shentao Yang. Zhendong Wang, Huangjie Zheng, Yihao Feng, and Mingyuan Zhou, A behavior regularized implicit policy for offline reinforcement learning, 2022c.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. MOPO: bosel-based offline policy optimization. In Advances in Neural Information Pro- cessing Systems, 2020.
Xianyuan Zhan, Xiangyu Zhu, and Haoran Xu. Model-based offline planning with trajectory pruning, In International Joint Conference Arlificial Intelligence, 2022.
Wenxuan Zhou, Sujay Bajracharya, and David Held. PLAS: Latent action space for offline reinforcement learning, 2020.
A Model-based Planning
In GOPlan's ttraa-rraecctoy and inter-trajectory reanalysis, GOPlan invokes function named Plan, which is model-based planning method, that employs model to simulate multiple imaginary trajectories, assigns scores to the initial actions, and generates the final action based on these scores. A variety of the model- based planning methods could be used here, such as model-predietive control (MPC) (Nagabandi el al., 2020) and Monte-Carlo tree search (MCTS) (Schrittwieser et al., 2021). In our implementation, we use a similar planning algorithm as PlanGAN (Charlesworth & Montana (2020) as demonstrated in Algorithm Our planning method can be refined by including value function that estimates the return horizon T (Argenson & Dulac-Arnold 2021), which leave for future work.
16

---

Published in Transactions on Machine Learning Research (05/2024)
Algorithm 2 Model-based Plamning.
Initialise: N dynamics models {Mi policy n, current state so: goal g, reward function
for I,... C' do
10)
3: 4 T(50, 2)
> Sample C' initial actions (aile=
4: i Uniform N)
5: 89 = (00 d))
D Predict next states {$i]e-1-
6: for h = 1,... H do
% = %
Duplicate every next state H times.
for k 1,,., K do 1()11
10:
hh = = ( kt 9:2) 2)
11:
i~ Uniform(1,
M)
12:
11 - Mi (g end for
b Generate H trajectories of K steps.
13:
14:
Re, = -E r(8 (h
15: end for
16: R. Σ# h=1 Ro,h
Average all cumulative returns.
17: R. E..1" fle
b Normalize all cumulative returns.
18: end for
ex Re
19: a"
Exponentially weight the actions.
20: return a
High
Low
Left
Right
(b)
Figure 8: Goal-conditioned tasks for OOD generalization. (a) Push Left-Right (b) Pick Low-High.
B Environments and Datasets
We use a set manipulation and navigation environments to evaluate the algorithm, including ReintReach, PointRooms, Reacher, SawyerReach, SawyerDoor, ReachReach, FetchPush, FetchPick, FetchSlide and Han- dReach. The environments are illustrated in Figure g All of the environments have continuous state space, action space and goal space. The details of the environments can be found in Appendix F in (Yang al., 2022b). The offline datasets (Andrychowicz et al. 2017), where the actions from the policy are perturbed by adding Gaussian noise with zero mean and 0.2 standard deviation to increase the diversity and multi-modality of the dataset Yang
collected by pre-trained policy using DDPG and hindsight relabelling
et al. 2022b
We also introduce four groups tasks for evaluating OOD generalization ability YYngg et al. 2023 There are four task groups: FetchPush Left-Right, FetchPush Near-Fat Far, FetchPick Left-Right, FetchPick Low-High We illustrate FetchPush Right and FetchPick Low-High in Figure 8 In the data collection, we only
17

---

Published in Transactions on Machine Learning Research (05/2024)
(b)
(c)
(d)
(e)
(I)
(g)
(h)
(i)
(i)
Figure Goal-conditioned tasks. (a) PointReach, (b) PointRooms, (c) Reacher, (d) SawyerReach, (e)
SawyerDoor, (f) FetchReach, (g) FetchPush, (h) FetchPick, (i) FetchSlide, and (j) HandReach.
store the trajectories whose achieved goals are all in the IID (independent identically distributed) region. The IID region is defined by each task group, shown in Table 5 The dataset division standard refers to the location requirements of initial states and desired goals for IID tasks (e.g., Right2Right). For OOD tasks, at least one of the initial state or the desired goal does not satisfy the IID requirement (c.g., Right2Left, 2ight Left2 We collect relatively smaller datasets of 5000 trajectories. Once we train a policy on the offline dataset, we evaluate the policy both on the IID tasks and the OOD tasks.
C Implementation Details
To implement GOPlan, we train a fully implicit policy, denoted as T(a | Sts 9, with = representing a 64-dimensional diagonal Gaussian noise. All associated models, including the value function and dynamics models, utilize 3-layer multi-layer perceptrons with 512 units in each layer. Furthermore, the policy network incorporates batch normalization, while the discriminator network employs leaky ReLU activation function. We use Adam optimizer with a learning rate of x 10-4. During the fine-tuning phase, we gather Jintra 2000 and Jinter = 2000 trajectories for antra-trajectory reanalysis and inter-trajectory reanalysis, respectively. After each collection, the prior policy is finetuned with 500 gradient steps. This process performed a total of 10 times. In order to show the difference in the ablation study, we use Jintra = 200 and Jinter = 200 there. Table 4 gives list and description of them, as well as their default values.
Additional Experiments
In this section, we provide more experiments study our proposed algorithm.
D.1 OOD Generalization Tasks
In the main paper, present the statistical results of experiments conducted four OOD generalization task groups (see Figure FetchPush Left-Right, FetchPush Near-Far, FetchPick Left-Right, and FetchPick Low-High. The complete set of results for this experiment is shown in Table G Among these tasks, GOPlan with online planning (GOPlan2) demonstrates the highest performance in both IID and OOD tasks, most of the task groups, GOPlan is comparable with the recent SOTA algorithm GOAT (Yang et al. 2023). while it outperforms previous approaches. These results validate the effectiveness of incorporating dynamics models to enhance generalization for OOD goals.
18

---

Published in Transactions on Machine Learning Research (05/2024)
Table 4: papeer-praamttrs..
Symbol
Description
Default Value
7
discount factor
0.98
B
coefficient in the exponential advantage weight
60
N
Number of dynamics models
3
ACGAN noise dimension
64
Range of the exponential weight
[0, 10]
Batch size
512
1
Finetuning episodes
10
Lintra
# collected intra-trajcctoriis each episode
1000
Linter
# collected Inter-traj sree-trajecctories each episode Uncertainty threshold
1000
u
0.1
Finetuning updates every episode
1000
Pretraining updates
5 x 10
Size of reanalysis buffer
2 x 106
K
Exponential weight
2
Planning horizon
20
Number of initial actions
20
Copies of initial actions
10
Table 5: Information about Task Groups and Datasets.
Datasets (TTss Group)
IID region
IDD task
OOD task
Dataset Division Standard
Right2Left,
Push Left-Right
Right
Right
Ligh2Right, Left2Left.
the object's coordinate value the initial position
Near2Far,
Push Near-Faz
Near
Near2Near
Far2Near, Far2Far.
the distance between the object and the initial position 0.15
Right2Left, Righ, Left
Pick Left-Right
Right
Right2Right
the object's y coordinate value the initial position
Pick Low Low-High
Low
Low2Low
Low2High.
object's coordinate value < 0.6
19

---

Published in Transactions on Machine Learning Research (05/2024)
Table 6: Average returns with standard deviations on the OOD benchmark. The results of GOAT are taken from the original paper Yang et al. 2023
Task Group
Task
GOPlan
GOPlan2
GOAT
GCSL
WGCSL
AM
g g-TD3-BC
Right2Left
27.17
33.65
24.9_24
15.7811.
25.44.1.2
11.93
17.73
Right
26.64
34.26413
21.3_2.7
15.98+1.
25.13
10.4844.8
18.3541.2
FetchPush Left-Right
Left2Left
26.8413.2
3451
23.0_28
16.4810.8
24.91
12.4444.7
17.31
Right2Right
26.87
34.21+-2.1
38.9 38.9_1.0
16.42
26. 26.14
12.35+4.0
18.12
Average
26.88
34.16
27.78
16.16
25.41
11.80
17.88
Far2Far
33.3221.8
38.47
16.2_o.
25. 25.7241.3
34.7220.8
18.3844.7
29.30
Near2Near
33.59
38.7411.0
34.9_1.3
25.9641.1
34.73
18.86
19.93
FetchPush Near-Far
Near2Far
34.26 38.5340.7
25.3_2.1
26.58
34.69
17-5444.2
29.86
Far2Near
33.79
37.82
13.0
26.01
34.65
18.13
30.06
Average
33.74
38.39
24.85
26.07
34.70
18.23
29.79
Legt2Right
27.99
32.9242.1
33.4_o.
13.03
26.69
25.9241.7
16.7841.4
Left2Left
28.29
33.74.16
32.3
13.7211.5
26.26
25.89
18.33
FetchPick Left-Right
Reght2Teeft
27.59+1.D
33.72+1.
32.3_o.7
13.55+1.
26.55+1.4
25.57+1.4
18.5442.
Right2Right
28.07
34.39
36.7_0.6
14.1312.1
26.17
25.9711.1
17.5342.3
Average
27.99
33.69
33.68
13.61
26.42
25.84
17.79
Low2Low
31.43=1.6
34.37
40.2_n.2
16.5541.3
30.8440.8
8.60
21.9642.4
FetchPick Low-High
Low2High
31.6621.8
34.72
26.222.4
16.75
31.50
8.7012.7
21.7342.1
Average
31.55
34.54
33.20
16.65
31.17
8.65
21.84
20


Reviews:
[{"review_id": "ZGM6crtzah", "reviewer": "Reviewer_5fHZ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n- The main claims are supported by fairly comprehensive experimental results\n- The method's motivation is straightforward.\n- Fig. 2 is helpful in understanding the nature of the policy distributions.\n- Fig. 3 is, overall, very helpful to help the reader understand most of the intuition behind the reanalysis method.\n- The technical description is overall, fairly clear. The algorithmic pseudocode blocks are helpful.\n\n# Weaknesses\n- It appears that in 13/14 tasks, the difference between the method's average performance and the next-best method's average performance is smaller than the sum of their performance standard deviations, i.e., there is no environment in which the method's performance improvement over the next-best method is inarguably substantial. One explanation could be that, in 13/14 tasks, there isn't substantial room for improvement over the next-best method's performance, i.e., that the next-best method is near-optimal. This suggests that harder tasks may be required in order to illustrate a clearer performance difference between the methods in more than 1 task. It would therefore significantly improve the paper if a performance comparison was performed on a task on which all methods are known to be far from optimal (e.g. a more difficult task, or a task for which the optimal performance is calculable and reported), in order to rule out the case that the existing evaluation is limited to \"solved\" tasks.\n- It seems fairly odd that the final action selection strategy used during planning is an exponentially weighted average of the initial actions (Alg. 2, line 19). I am aware that prior work has considered this strategy (e.g. Charlesworth & Montana 2020), however, the MuZero MCTS used an argmax over an upper-confidence bound (Eq. 2 from https://arxiv.org/pdf/1911.08265.pdf). The paper would be improved with an experiment that varies the action selection strategy between the current one, the originally-proposed one from MuZero, and perhaps also a sampling-based one, drawing $a$ from the categorical distribution defined with unnormalized mass per bin $c$ as $\\kappa \\cdot R_c$.  Furthermore, it would improve the paper to run the method on a non-continuous control task, because I suspect that tasks with discrete actions would pose more of a challenge to the action-blending strategy used here.\n- The employed dynamics models and environments are deterministic (aside the experiment in 5.3, in which the environment is deterministic), which limits our understanding of how well the approach would work in more complex environments with significant stochasticity.\n- Eq. 2 is not self-contained: $l^p$, $l^v$, and $l^r$ are not defined in this paper. The paper should be self-contained.\n- Missing citations to MCTS on p3: [A]\n- p3-p4: The paper is not clear which rewards are used to compute the cumulative return $R(a_t^c)$. It seems that it could be either rewards computed using the original sparse reward function, or rewards computed using the reward model. I found the answer (the former) by looking at the supplement. The main text should be updated to clarify this uncertainty.\n- Fig. 3 says there are \"six scenarios\" for reanalysis, but it's not clear that the six scenarios presented in Fig. 3 actually cover the space of possible outcomes of reanalysis. Specifically, the scenario in which the trajectory plans into an obstacle and the uncertainty ranges are small is not mentioned (i.e. the uncertainty estimate is inaccurate) -- instead, only the scenario in which the trajectory plans into an obstacle and the uncertainty range is large is presented (i.e. the uncertainty estimate is accurate). Either the caption should be fixed or those additional scenarios (one for intra-trajectory and 1 for inter-trajectory) should be added to the figure. Perhaps one way to add it would be to add other trajectories to a-2 and b-2 that intersect the obstacle, do not achieve the goal, and have low uncertainty.\n\n## Citations\n- [A] Coulom, Rémi. \"Efficient selectivity and backup operators in Monte-Carlo tree search.\" International conference on computers and games. Berlin, Heidelberg: Springer Berlin Heidelberg, 2006.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a new method for goal-conditioned offline reinforcement learning that combines a multi-modal policy distribution with an MCTS-based approach for fine-tuning. Experimental results measure the approach's performance on a variety of continuous control environments and illustrate that the approach is competitive with the next-best approach on every task. Ablations measure the relative performance of a variety of the approach's components (most of the main components) and confirm the efficacy of each of the main components investigated.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The first two weaknesses are my main concerns. While I would recommend acceptance in the current state, they would significantly strengthen the work in my view.", "broader_impact_concerns": "I have no broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "BM0NGczpfg", "reviewer": "Reviewer_bUdD", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n- overall the paper is written clearly and presented well. \n- extensive experiment results and ablations are provided to support the empirical claims. \n- combining a number of technical components and design for the goal-conditioned setting can be novel\n- results look strong\n\n\nWeaknesses:\n- some of the components of the proposed method are from previous works and not new\n- the proposed method consists of a number of components and can be more complex to use and can have a larger computation overhead compared to alternatives\n- would be good to see a discussion on how much additional computation cost does the proposed method bring in. Is it significantly slower than some of the baselines you compared to? \n\n\nSome questions: \n- Figure 2 is interesting, however, without quantification of the results, it is a little unclear to me how much advantage do we get from using CGAN compared to CVAE, and also do we still have such a behavior in the high-dimension state-action space setting? And when you compare them do these models have the same capacity, is the comparison fair? \n- Maybe I missed sth, I am a little not convinced how important is the multi-modal action distribution problem in goal-conditioned RL, can you remind me what is the main evidence this is critical? If I have a simpler network but just higher capacity will it achieve the same results?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes GOPlan, a new model-based method for goal-conditioned offline RL setting. This method has two training stages, and use a number of different techniques to reach the final strong performance, including planning, training with imgined trajectories, using ensemble-based uncertainty estimation to avoid OOD data. The proposed method achieves a new SOTA performance on a number of benchmark tasks and settings.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- The anonymous code link in appendix is expired, can you fix it? \n- Can you provide a discussion on computation overhead comparing the proposed method and other baselines?", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "5JrMaFSokQ", "reviewer": "Reviewer_UJw9", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\n- Generally, the paper is well-written with its ideas clearly conveyed. The illustrations well convey the high-level framework of GOPlan (Fig. 1) and possible outcomes of reanalysis (Fig. 3).  The pseudocode is also quite helpful, making the procedure easy to follow. The hyperparameters and experimental settings are clearly listed in the supplementary materials. Some details, though, could be improved; see “weakness” section, point 1 for advice.\n\n- The proposed idea, specifically, the reanalysis part, is quite interesting. The inter- and intra-trajectory selection of rollout effectively and actively builds up a topological graph between states that appear in the dataset, which is an elegant design.\n\n- The results are convincing with extensive ablation study; the ablations in Sec. 5.2 to 5.4 clearly show the robustness of the proposed method against noise, out-of-distribution goals, and small dataset. The effectiveness of the dynamic model ensemble is also examined.\n\n**Weaknesses:**\n\t\n- Some details of the presentation of this paper could be improved. For example, the terms $l^p, l^v, l^r$ in Eq. 2 are not properly introduced.\n\n- There is neither limitation nor broader impact statement included in the paper.\n\n- Though this paper briefly mentions world models in the beginning, it does not discuss in detail the feasibility of actually learning a world model instead of only dynamics model (that is, learning everything including transitions and reward), and neither cite or compare with representative world model papers such as Dreamerv3 [1].\n\n- (This is not a limitation; it is more of a question:) Why is case (a-2) of the intra-trajectory reanalysis leading to discarding the new trajectory? Is it possible to include the trajectory but with a relabeled goal as what the method does in case (b-2)?\n\n**References:**\n\n[1] Hafner, D., Pasukonis, J., Ba, J., & Lillicrap, T. Mastering diverse domains through world models. arXiv, 2023.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes GOPlan, a novel model-based offline goal-based RL method that is capable of capturing the multi-modal action distribution and dealing with out-of-distribution actions. The method has two main components: \n\n1) Learning prior policy with an advantage-weighted CGAN and an ensemble of dynamics models that predict next state from offline data; \n\n2) Finetune the policy with imaginary rollouts in the model. The rollouts are conducted by planning with the learned prior policy, where GOPlan first samples actions and generates many next steps, then takes the next step as weighted average of the next steps (weights are the future return over many imaginary rollouts). Specially, the imaginary rollouts have two sources: intra-trajectory or inter-trajectory, where the former tries to reach from one state in an existing trajectory in the dataset to another, and the latter tries to reach from one state in an existing trajectory to another state in another existing trajectory. The generated trajectory could be accepted, accepted with goal relabeled (for intra-trajectory), or discarded.\n\nThe method is tested on various testbeds which proves its superiority to existing offline goal-based RL methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Below is the change that is critical to securing my recommendation for acceptance:**\n\nMore carefully discuss in related work and preferably compare with world model methods such as Dreamer in the experiments.\n\n**Below are the changes that strengthen the work:**\n\n- Include decision diffuser [1] (and possibly decision transformer [2]) as baseline. Decision diffuser is a newer solution superior to trajectory transformer, is generative-model-based, and can work with goal-based RL, which corresponds to “satisfying constraints” in the decision diffuser paper.\n- Discuss the possibility of using other generative models, such as normalizing flow and diffusion models as the agent to address the multimodal issue;\n\n- Check the math symbols appear in the paper and make sure they are properly introduced (specifically, see weakness point 1);\n\n- Include a discussion about the limitation of the proposed method, and preferably also an impact statement. For example, while this work mostly deals with simulated environments and thus does not have much direct impact on real-world applications, it is generally an automation of decision-making that could lead to loss of job, abused use of technology and human deskilling. Besides, hallucination of the generated trajectory caused by inaccurate modeling could also be harmful in some scenarios where making mistakes is dangerous.\n\n**References:**\n\n[1] Ajay, Anurag, et al. \"Is conditional generative modeling all you need for decision-making?.\" In ICLR, 2023.\n\n[2] Chen, Lili, et al. \"Decision transformer: Reinforcement learning via sequence modeling.\" In NeurIPS, 2021.", "broader_impact_concerns": "This paper does not include a broader impact statement, and since the testbed of the paper is all on simulation tasks, I do not feel there are any major broader impact concerns. In spite of this, it would be great if the author could include a broader impact statement in the revised version (see requested changes).", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper offers a new methodology for learning and refining offline goal-conditioned RL policies -- leveraging planning techniques over a learned dynamics model and initial policy to generate additional synthetic trajectories. Results suggest the method is sound and empirically comparable to state-of-the-art baselines. The AE recommend a minor revision to ensure the additional experiments provided in the author response are fully incorporated into the manuscript prior to acceptance.
Decision: Accept (poster)