Goal-Conditioned Predictive Coding for Offline Reinforcement Learning
Authors: Zilai Zeng
Ce Zhang
Shijie Wang
Chen Sun
Venue: NeurIPS 2023 poster (2023)
Keywords: reinforcement learning
offline RL
self-supervised learning
TL;DR: Sequence modeling helps offline RL via Goal-Conditioned Predictive Coding

Abstract: Recent work has demonstrated the effectiveness of formulating decision making as supervised learning on offline-collected trajectories. Powerful sequence models, such as GPT or BERT, are often employed to encode the trajectories. However, the benefits of performing sequence modeling on trajectory data remain unclear. In this work, we investigate whether sequence modeling has the ability to condense trajectories into useful representations that enhance policy learning. We adopt a two-stage framework that first leverages sequence models to encode trajectory-level representations, and then learns a goal-conditioned policy employing the encoded representations as its input. This formulation allows us to consider many existing supervised offline RL methods as specific instances of our framework. Within this framework, we introduce Goal-Conditioned Predictive Coding (GCPC), a sequence modeling objective that yields powerful trajectory representations and leads to performant policies. Through extensive empirical evaluations on AntMaze, FrankaKitchen and Locomotion environments, we observe that sequence modeling can have a significant impact on challenging decision making tasks. Furthermore, we demonstrate that GCPC learns a goal-conditioned latent representation encoding the future trajectory, which enables competitive performance on all three benchmarks.

Goal-Conditioned Predictive Coding for Offline
Reinforcement Learning
Zilai Zeng Brown University
Ce Zhang Brown University
Shijie Wang Brown University
Chen Sun Brown University
Abstract
Recent work has demonstrated the effectiveness of formulating decision making supervised learning on offline-collected trajectories. Powerful sequence models, such as GPT or BERT, are often employed to encode the trajectories. However, the benefits performing sequence modeling on trajectory data remain unclear. this work, investigate whether sequence modeling has the ability condense trajectories into useful representations that enhance policy learning. We adopt a two-stage framework that first leverages sequence models encode trajectory-lever representations, and then learns a goal-conditioned policy employing the encoded representations its input. This formulation allows consider many existing supervised offline RI methods as specific instances of our framework. Within this framework, we introduce Goal-Conditioned Predictive Coding (GCPC), a sequence modeling objective that yields powerful trajectory representations and leads performant policies. Through extensive empirical evaluations on AntMaze, FithhaKitchen and Locomotion environments, we observe that sequence modeling can have a significant impact on challenging decision making tasks. Furthermore, we demonstrate that GCPC learns goal- sall-coditioned latent representation encoding the future trajectory, which enables competitive performance on three bench- marks. Our code is available at https: httee:///roommoaee palm. g github. ioriio//ooo /G/CC//
1 Introduction
Goal-conditioned imitation learning [13, 15, 20] has recently emerged promising approach to solve offline reinforcement learning problems. Instead of relying on value-based methods, they directly learn policy that maps states and goals (e.g. expected returns, or target states) to the actions. This is achieved by supervised learning on offline collected trajectories (i.e. sequences of state, action, and reward triplets). This learning paradigm enables the resulting framework to avoid bootstrapping for reward propagation, a member of the "Deadly Triad" [45] of RL which is known to lead to unstable optimization when combining with function approximation and off- learning, and also allows the model to leverage large amounts of collected trajectories and demonstrations. The emergence of RL supervised learning trajectory data coincides with the recent success of the Transformer architecture [46, 20] and its applications to sequence modeling, such as GPT [8] for natural language and VPT [3] for videos. Indeed, several recent works have demonstrated the effectiveness of sequence modeling for offline RL [10, 27, 34, 44, 49, 9, 42]. Most of these approaches apply the Transformer architecture to jointly learn the trajectory representation and the policy. When the sequence models are unrolled into the future, they can also serve as "world models" [21, 22] for many RL applications. By leveraging advanced sequence modeling objectives from language and visual domains [11, 4, 24], such methods have shown competitive performance in various challenging tasks [16, 5].
Despite the enthusiasm and progress, the necessity of sequence modeling for offline reinforcement learning has been questioned by recent work [15]: With a simple but properly tuned neural architecture (e.g. multilayer perceptron), the trained agents can achieve competitive performance several
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

challenging tasks while taking only the current state and the overall goal as model inputs. In some tasks, these agents even significantly outperform their counterparts based on sequence modeling. These observations naturally motivate the question: Is explicit sequence modeling of trajectory data necessary for offline RL? And if so, how should it be performed and utilized?
To properly study the impact of sequence modeling for decision making, we propose to decouple trajectory representation learning and policy learning. We adopt two-stage framework, where sequence modeling can be applied to learn the trajectory representation, the policy, or both. The connection between the two stages can be established by leveraging encoded trajectory representations for policy learning, or by transferring the model architecture and weights (i.e. the policy can be initialized with the same pre-trained model used for trajectory representation learning). These stages can be trained separately, or jointly and end-to-end. This design not only facilitates analyzing the impacts of sequence modeling, but is also general such that prior methods can be considered as specific instances, including those that perform joint policy learning and trajectory representation learning [10, 34, 49], and those that learn a policy directly without sequence modeling [15].
Concretely, we aim at investigating the following questions: (1) Are offline trajectories helpful due to sequence modeling, or simply by providing more data for supervised policy learning?
(2) What would be the most effective trajectory representation learning objectives to support policy learning? Should the sequence models learn to encode history experiences [10], future dynamics [27], or both [19]?
(3) As the same sequence modeling framework may be employed for both trajectory representation learning and policy learning [34, 9, 49], should they share the same training objectives or not?
We design and conduct experiments on the AntMaze, FrankaKitchen, and Locomotion environments to answer these questions. We observe that sequence modeling, if properly designed, can effectively aid decision-making when its resulting trajectory representation is used as an input for policy learning. We also find that there is discrepancy between the optimal self-supervised objective for trajectory representation learning, and that for policy learning. These observations motivate us to propose specific design of the two-stage framework: It compresses trajectory information into compact bottlenecks via sequence modeling pre-training The condensed representation is then used for policy learning with a simple MLP-based policy network. We observe that goal-conditioned predictive coding (GCPC) is the most effective trajectory representation learning objective. It enables competitive performance across all benchmarks, particularly for long-horizon tasks. We attribute the strong empirical performance of GCPC to the acquisition of goal-conditioned latent representations about the future, which provide crucial guidance for decision making.
To summarize, our main contributions include:
We propose to decouple sequence modeling for decision making into a two-stage framework, namely trajectory representation learning and policy learning. It provides a unified view for several recent reinforcement learning via supervised learning methods.
We conduct a principled empirical exploration with our framework to understand if and when sequence modeling of offline trajectory data benefits policy learning.
We discover that conliconditioned predictive coding (GCPC) serves as the most effective sequence modeling objective to support policy learning. Our overall framework achieves competitive performance on AntMaze, FrankaKitchen and Gym Locomotion benchmarks.
2
Related Work
What is essential for Offline RL. Offline reinforcement learning aims to obtain effective policies by leveraging previously collected datasets. Prior work usually adopts dynamic programming [31, 18] or supervised behavioral cloning (BC) methods [43, 3J. Recent approaches [10, 27] demonstrate the effectiveness of solving decision making tasks with sequence modeling, whereas RvS [15] establishes a strong MLP baseline for conditional behavioral cloning. To further understand what are the essential components for policy learning. researchers have investigated the assumptions required to guarantee the optimality of return- conditionn supervised learning [7], and examined offline RL agents from three fundamental aspects: representations, value functions, and policies [17]. Another concurrent work [6] compares the preferred conditions (e.g. data, task, and environments) to perform Q-learning and imitation learning. In our work, we seek to understand how sequence modeling may benefit.
offline RL and study the impacts of different sequence modeling objectives.

---

Transformer for sequential decision-making Sequential decision making has been the subject of extensive research over the years. The tremendous success of the Transformer model for natural language processing [11, 40] and computer vision [24, 35] has inspired numerous works that seek to apply such architectures for decision making, and similarly motivates our work. Prior work has shown how to model sequential decision making as autoregressive sequence generation problem to produce a desired trajectory [10], while others have explored the applications of Transformer for model-based RL [27, 47] and multi-task learning [41, 9J. Our work aims to utilize Transformer- models to learn good trajectory representations that can benefit policy learning.
Masked Autoencoding. Recent work in NLP and CV has demonstrated masked autoencoding (MAE) as an effective task for self-suuer representation learning [11, 8, 4, 24]. Inspired by this, Uni[MASK] [9], MaskDP [34], and MTM [49] propose to train unified masked autoencoders on trajectory data by randomly masking the states and actions to be reconstructed. These models can be directly used to solve range of decision making tasks (e.g. return-conditioned BC, forward dynamics, inverse dynamics, etc.) by varying the masking patterns at inference time, without relying on task-specific fine-tuning. In contrast, we decouple trajectory representation learning and policy learning into two stages. Unlike approaches which perform value-based offline RL (e.g. TD3, Actor-Critic) after pretraining [34, 49], our policy learning stage adopts imitation learning setup where the policy can be learned without maximizing cumulative reward.
Self-supervised Learning for RL. Self-supervised learning has emerged as powerful approach for learning useful representations in various domains, including reinforcement learning (RL). In RL, self-supervised learning techniques aim to leverage unlabeled or partially observed data to pre-train agents or learn representations that facilitate downstream RL tasks. Prior work mainly focuses on using self- supervied learning to get state representations [51, 38, 36, 50, 10, 34, 9, 49, 32] or world models [22, 23, 44, 27, 12, 37J. In [51], the authors evaluate a broad set of state representation learning objectives
offline datasets and demonstrate the effectiveness of contrastive self-prediction. In this work, we investigate representation learning objectives in trajectory-space with the sequence modeling tool, which enables us to explore the impact of different modalities (e.g. states, actions, goals, etc.) on trajectory representation learning.
3 Method
We revisit the role of sequence modeling in offline reinforcement learning, from the perspectives of trajectory representation learning and policy learning. Section 3.1 describes the background knowl- edge on extracting policies from offline trajectories using supervised learning. Section 3.2 introduces two-stage framework that decouples trajectory representation learning and policy learning, which serves as the basis for our investigation. In Section 3.3, we propose a specific instantiation of the two-stage framework, in which we leverage self- supervised learning objective Condi-Condttione Predictive Coding (GCPC), to acquire performant trajectory representations for policy learning.
3.1 Offline Reinforcement Learning via Supervised Learning
We follow prior work that leverages sequence modeling for offline reinforcement learning, and adopt the Reinforcement learning via Supervised learning (RvS) (e.g. [15]) setting. RvS aims to solve the offline RL problem as conditional, filtered, or weighted imitation learning. It assumes that a dataset has been collected offline, but the policies used to collect the dataset may be unknown, such as with human experts or a random policy. The dataset contains a set of trajectories:
= {r') Each trajectory 7' in the dataset is represented as {(s,. ((s..a)) ae) where H is the length of the trajectory, and 5t. refer to the state, action at timestep t, respectively. A trajectory may optionally contain the reward Fy received at timestep t.
As the trajectories are collected with unknown policies, they may not be optimal or have expert-level performance. Prior work [10, 33, 30] has shown that properly utilizing offline trajectories containing suboptimal data might produce better policies. Intuitively, the suboptimal trajectories may still contain sub-trajectories that demonstrate useful "skills", which can be composed to solve new tasks.
Given the above trajectories, we denote a goal-conditioned policy ma as a function parameterized Toss
by 8 that maps an observed trajectory (() = {sge, acc} and goal (()) to an action dz The goal (()) is computed from TrH which can be represented as either a target state configuration sampled
3

---

from St.H or an expected cumulative reward/retumn-to computed from FEs (see Appendix A.6 for details). For simplicity of notation, we write T as Tobs and glt) as g. We consider policy should be able to take any form of state or trajectory information as input and predict the next action: (1) when only the current observed state 8t and the goal g are used, the policy ág - ( g) ignores the history observations; (2) when Te is a sequence model, it can employ the whole observed trajectory to predict the next action å₂ He : To optimize the policy, commonly used objective is find the parameters that fit the mapping of observations to actions using maximum likelihood estimation:
|rl II
0* = argmax EY
no (a(arl Tabss 9)
0
3.2 Decoupled Trajectory Representation and Policy Learning
Sequence modeling can be used for decision making from two perspectives, namely trajectory representation learning and policy learning. The former aims to acquire useful representations from raw trajectory inputs, often in the form of a condensed latent representation, or the pretrained network weights themselves. The latter aims to map the observations and the goal into actions that accomplish the task. To explicitly express the trajectory representation function, we rewrite the goal-conditioned policy function as follows:
0r = [T
where f(-) is an arbitrary function, such as neural network, that computes representation from Tobs- f() can also be an identity mapping such that (-)-) directly operates on Tobs-
Motivated by the recent success of sequence modeling in NLP [11, 40] and CV [24, 14], both the trajectory learning function and the policy learning function can be implemented with Transformer neural networks as (-) and T-(-). We hypothesize that it is beneficial for fo to condense the trajectories into a compact representation using sequence modeling techniques. We also hypothesize that it is desirable to decouple the trajectory representation learning from policy learning. The decoupling not only offers flexibility on the choice of representation learning objectives, but also allows us to study the impact of sequence modeling for trajectory representation learning and policy learning independently. We thus adopt two-stage framework with a TrajNet fo(-) and a PolicyNet TT(-). TrajNet aims to learn trajectory representations with self-supervised sequence modeling objectives, such as masked autoencoding [24] or next token prediction [40]. PolicyNet aims to obtain performant policy with the supervised learning objective in Equation 1.
We now describe an overall flow of the two networks, which we will show general to represent recently proposed methods. In the first stage, we approach trajectory representation learning as masked autoencoding. TrajNet receives a trajectory 7 and an optional goal g. and is trained to reconstruct r from a masked view of the same. Optionally, TrajNet also generates a condensed trajectory representation B, which can be utilized by PolicyNet for subsequent policy learning:
4, B = fo (Masked(T).g)
During the second stage, TrajNet is applied on the unmasked observed trajectory - (Tote g), to obtain Bobs PolicyNet then predicts the action a given Tabs (or the current observed state 82), the goal g, and trajectory representation Boss:
a = n ( (Tobs Bobs: g)
(4)
Our framework provides a unified view to compare different design choices (c.g. input information, architecture, training objectives, etc.) for representation learning and policy learning, respectively. Many existing methods can be seen as special cases of our framework. For example, prior works [34, 49] instantiate TrajNet with a bi-directional Transformer pre-trained via masked prediction, and re-use it as the PolicyNet in a zero-shot manner. To implement DT [10], f(-) is set as an identity mapping function of the input trajectory and π .T.)) trained to autoregressively generate actions. These methods learn trajectory representations and the policy jointly, with training objectives inspired
by MAE [24] and GPT [40]. At last, our framework recovers RvS-G/R [15] by having the output of f(-) in Eq. 2 as the last observed state st from Tobs

---

State or Goal
Stage
Masked
Trajectory Representation Learning
Lomobble tokons
TrajNet
Masking Patterns
Bidirectional Transformer Decoder
MAB-
Bottlenckk
MAL-FF
Bidirectional Transformer Encoder
MAP-RC
MAB-ALL
Histury States
Future States
Goal
Slot tokens
Figure 1: Stage 1 Trajectory Representation Learning. For notation consistency between the two stages, we separate the input into observed (history) states and future states. Left: TrajNet. We input randomly masked history state, goal and slot tokens the transformer encoder. The decoder takes in encoded slot tokens (the bottleneck) and a sequence of masked tokens, and reconstruct the whole trajectory. With this training objective, coding, which conditioned on the goal and history states. Right: All Masking Patterns. TrajNet can be trained with different masking patterns with their corresponding reconstruction objectives. The illustration of TrajNet on the left uses "MAE-RC", which is adopted by GCPC.
encourage the bottleneck perform predictive
3.3 Goal-Conditioned Predictive Coding
Finally, introduce specific design of the two-stage framework, Noal-Conditioned Predictive Coding (GCPC), which inspired by our hypotheses introduced Section 3.2. To facilitate the transfer of trajectory representations between the two stages, compress the trajectory into latent representations using sequence modeling, which we refer to as the bottleneck. With this design, we train the bottleneck perform goal-conditioned future prediction, that the latent representations encode future behaviors toward the desired goal, which are subsequently used to guide policy learning.
In the first stage (as shown in Figure 1), include state tokens, one goal token, and a few learnable slot tokens. The action tokens are ignored. We first embed the goal token and all state tokens with separate linear encoders, then apply sinusoidal positional encoding before all inputs are sent into the Transformer Encoder.
bi-directional Transformer as TrajNet. The inputs
Stage 1
Stage Policy Leaming
Trajectory Representation Leaming
Pre-traine trained TrajNet
A:
PolicyNet
Bidirectional Transformer Decoder
MLP
Bottlenack
Coal
Bidirectional Transformer Encoder
State, A ction, Goal
History States
Coal
tokens
Leamable tokens
Figure Stage Policy Learning. We implement policy learning with simple MLP PolicyNet. We input unmasked history states and retrieve the bottleneck from pre-trained encoder. Then the bottleneck taken as the input to the policy network. We view the bottleneck generated by pre-trained encoder as goal-conditioned latent representations for the future trajectory.
While the full trajectories can be used for trajectory representation learning with TrajNet, only observed states are available during policy learning. For notation consistency, we denote the observed states "history", and the states that follow the observed states as "future". The total number of input states for stage is T k p. and for stage 2 Both and p are hyperparameters that represent history window length and future window length, respectively.
5

---

To perform condiiondt predictive coding, the entire future trajectory is masked. The tokens in the observed history are randomly masked (Figure "MAE-RC"). The bottleneck is taken as the encoded slot tokens, which condenses the masked input trajectory into a compact representation. The decoder takes the bottleneck and T masked tokens as the input, and aims to reconstruct both the history and the future states. Our intuition is that this will encourage the bottleneck to learn representations encoding the future trajectory and provide useful signals for decision making. The training objective of the TrajNet is to minimize MSE loss between the reconstructed and the ground-truth trajectory.
Figure 2 illustrates the interface between the TrajNet and the PolicyNet. PolicyNet implemented as simple MLP network. The trained TrajNet has its weights frozen, and only the TrajNet encoder is used compute the trajectory representation Boss from k unmasked history states. PolicyNet takes the current state st. goal 9, and bottleneck Ba as input, and outputs an action. The training objective of PolicyNet to minimize MSE loss between the predicted and ground-truth actions.
Discussion. Decoupling trajectory representation learning and policy learning allows us to explore different model's inputs and sequence modeling objectives. For example, recent work [2, 39] observed that the action sequences could potentially be detrimental to the learned policy in some tasks. In GCPC, we employ state-only trajectories as input and ignore the actions. Different objectives (e.g. masking patterns in the masked autoencoding objective) also have an impact on what is encoded in the bottleneck. Here we introduce five sequence modeling objectives for trajectory representation learning (as shown in Table !) and study their impacts on policy learning. When using "AE-H" or "MAE-H" the bottleneck is only motivated to summarize the history states, When using "MAE-F" or "MAE-RC", the bottleneck is asked to perform predictive coding, and thus encode the future state sequences achieve the provided goal. By default, we adopt the "MAE-RC" objective GCPC.
Table 1: Objectives for Trajectory Representation Learning
Input
Reconstruct
History
Future
History
Future
AE-H MAE-H
Unmasked
Randomly Masked
MAE-F
Unmasked
Fully Masked
MAE-RC
Randomly Masked
Fully Masked
MAE-ALL
Randomly Masked
Randomly Masked
4
Experiments
In this section, we aim to answer the following questions with empirical experiments: (1) Does sequence modeling benefit reinforcement learning via supervised learning on trajectory data, and how? (2) Is beneficial decouple trajectory representation learning and policy learning with bottlenecks? (3) What are the most effective trajectory representation learning objectives?
4.1 Experimental Setup
To answer the questions above, we conduct extensive experiments on three domains from D4RL offline benchmark suite [16]: AntMaze, FrankaKitchen and Gym Locomotion. AntMaze is class of long-horizon navigation tasks, featuring partial observability, sparse reward and datasets that consist primarily of suboptimal trajectories. In this domain, 8- BoF Ant robot needs "stitch" parts of subtrajectories and navigates to a particular goal location in partially observed mazes. In addition to three mazes from the original D4RL, also include larger maze (AntMaze- Ultra) proposed by [28]. Both large and ultra setup in AntMaze poses significant challenges due to the complex maze
Figure 3: D4RL environments used in evaluation. From left to right: AntMaze, FitnkaKitchen.
Walker 2D, Hopper, and Halfcheetah.

---

layout and long navigation horizon. FrankaKitchen is a long- horizon manipulation task, in which 9-DoF Franka robot arm is required to perform 4 subtasks in a simulated kitchen environment (e.g. open the microwave, turn on the light). In Gym Locomotion, we evaluate our approach on three continuous control tasks: Walker 2D, Hopper and Halfcheetah. Following [15], we refer to the "goal" of AntMaze and Kitchen as the target state configuration, and that of Gym as the average return-to-go.
Experimental details. For all benchmarks, we use two-layer transformer encoder and one- layer transformer decoder as the TrajNet, and a two-layer MLP as the PolicyNet (see detailed hyperparamters in A.3). During policy learning, we use the pre-trained TrajNet that achieves the lowest validation reconstruction loss to generate the bottleneck. For each evaluation run, we follow [29] to take the success rate over 100 evaluation trajectories for AntMaze tasks. For Kitchen and Gym Locomotion, we average returns over 50 and 10 evaluation trajectories, respectively. To determine the performance with a given random seed, we take the best evaluation result among the last five checkpoints (see discussion in A.2). For performance aggregation, we report the mean performance and standard deviation averaged over five seeds for each experiment.
4.2 Impact of Trajectory Representation Pretraining Objectives
In the two-stage framework, the pre-trained TrajNet generates trajectory representations in the form of the bottleneck, which is taken as an input to PolicyNet. To study the impact of trajectory representation learning objectives on the resulting policy performance, we implement five different sequence modeling objectives (as in Table 1) by varying masking patterns in the first stage pretraining.
Table 2 compares the performance of policies with different settings and pretraining objectives'. We note that MAF-F is the only effective masking pattern perform zero-shot in- ference. After decoupling representation learning and policy learning, the MLP policy consistently outperforms the zero- shot transformer policy. This suggests good objectives for two stages could be different by decoupling can get the
MAE-H
MAR-RC
100
101
Nomallied Soort
60-
40
best of both worlds. Also, we observe that removing action sequences from trajectory representation pretraining yields performance gains in this task, whereas previous single-stage Transformer policy (e.g. [10]) usually requires action inputs to function, which further demonstrates the flexibility of our decoupled framework. With properly designed objectives, sequence modeling can generate powerful trajectory represen- tations that facilitate the acquisition of performant policies. In both Table 2 and Figure 4, we observe that both MAE-F and MAE-RC outperform the other objectives both AntMaze-Large and Satchen-Partial, confirming the importance of the predictive coding objective for trajectory representation learning.
Figure 4: Policy learning curves with different pretraining objectives on Sitchen-Partial
Table 2:
Comparison of trajectory representation pretraining objectives. We evaluate five different objectives under three settings on large AntMaze environment: (1) Zero-shot: When actions are considered as part of the masked tokens, the pretrained TrajNet can be directly utilized as the policy: (2) Two-stage (w/ actions): Two-stage framework employs an MI.P as PolicyNet, with state-only trajectories as the input to TrajNet. (3) Two-stage (w/ actions): Two-stage framework employs an MLP as PolicyNet, with state-action trajectories as the input TrajNet.
Large-Play
AE-H
MAE-H
MAE-F
MAE-RC
MAE-ALL
Zero-shot
21.2 17.1
0
0
Two-stage (w/ actions)
12.6 +: 4.7
6.4 38
57.2 5.5
62.0 10.7
11.6 13
Two-stage (w/o actions)
30.2 + 6.6
36.6 + 12.6
76.2 4.0
78.2 + 32
32.0 : 13.2
4.3 The Role of Goal Conditioning in Trajectory Representation Pretraining
We also investigate whether goal conditioning (i.e. the goal input) in TrajNet is necessary beneficial for learning trajectory representations. In Table 3, we observe that the objectives without performing
For pre-training on AntMaze, when actions (i.e. the torque applied on joints) are considered. the full state space is reconstructed; when state-only trajectories are used, we reconstruct only the locations of the agent.
7

---

Large-Diverse
Kitchen-Miise
Hopper- Replay
w/ GC
GC
1.2
0.65
Validaion MSB MSB
0.0125
1.0
0.8
0.0100
0.60
0.6
0.35
0.4
0.0050
0.50
60
60
Figure 5: Reconstruction loss on the validation set during the trajectory representation learning stage, when MAE-F objective is used. "GC" refers to goal conditioning in TrajNet.
future prediction is insensitive to the goal conditioning. For example, the results of AE-H and MAE-H pretraining objectives align with the intuition that merely summarizing the history should not require the goal information. However, goal conditioning is crucial for predictive coding objectives (e.g. MAE-F and MAE-RC), Removing the goal from TrajNet might result in aimless prediction and misleading future representations, which thus harm the policy. Figure shows the curves of validation loss during pretraining with MAE-F objective. We observe that goal conditioning reduces the prediction етгоr and helps the bottleneck properly encode the expected long-term future, which would benefit the subsequent policy learning. One hypothesis that specifying the goal enables the bottleneck to perform coal-conditioned implicit planning. The planned future may provide the correct waypoints for the agent to reach the distant goal. Figure illustrates qualitative result of the latent future in AntMaze, which depicts the future states decoded from the bottleneck using the pre-trained Transformer decoder. It demonstrates that the latent future with goal conditioning helps point out the correct direction towards the target location.
Table Ablation study of goal conditioning on AntMaze-Large. Removing the goal conditioning from TrajNet would seriously affect predictive coding objectives and harm the resulting policy performance, suggesting that the properly encoded future representation provides crucial guidance for performing long-hod morghhorzzon tasks. "GC" refers to goal conditioning in TrajNet.
Large-Play
AE-H
MAE-H
MAE-F
MAE-RC
MAE-ALL
w/o GC
32.8 4.3
33.2 11.9
10.0 22
16.0
36.2 10.4
w/ GC
30.2 66
36.6 + 126
76.2 4.0
78.2 332
32.0 + 132
Goal Location
Agent Location
Latent Future for Timestop Goal)
TF
Evaluation
T F
Latent Puture for Timestep t/w/o Goal)
TG
UFF
UF
Timestep t
Figure 6: Latent future visualization on AntMaze-LLage. qualitative result comparing the latent future with and without goal conditioning. Given the goal information, the bottleneck can encode the latent future moving the desired direction.

---

4.4 Latent Future versus Explicit Future as Policy Conditioning Variables
Prior work [27, 26] has demonstrated that planning into the future is helpful for solving long- ong-horizon tasks. These work performs planning on the explicit future by sampling desirable future states (or transitions), while GCPC leverages noal-conditioned latent representations that encode the future state sequence. In this experiment, we examine how the implicit encoding of future information affects policy performance when it serves as conditioning variable of PolicyNet. Specifically, we obtain the bottleneck from a TrajNet, which is pre-trained with p-length future window. The bottleneck encodes latent representations of p future states. Correspondingly, we acquire the same number of explicit future states using the pre-trained transformer decoder (see Appendix A.4). Either the bottleneck or explicit future states are taken as auxiliary inputs to the PolicyNet. In Table 4, we evaluate the policy with y 70 in large AntMaze and p 30 in Kitchen. The results illustrate that the bottleneck is powerful future information carrier that effectively improves the policy performance. Different from previous approaches that estimate explicit future states with step-by-step dynamics models [27], using latent future representations can mitigate compounding rollout errors. Compared to diffusion- based planning methods [2], which require iterative refinement to obtain future sequence, latent future avoids the tome-consuming denoising step and reduces the decision latency.
Table Comparison between latent and explicit future. Compared to explicit future states, the latent future encoded by the bottleneck is more effective for policy learning. p is future window size.
Large-Pla Play Large-Diverse
Kitchen-Mixed
Kitchen-Partial
70
30
Explicit Future
67.9 9.3
70.0 4.6
72.4 = 4.5
73.9 + 9.3
Latent Future
78.2 322
80.6 +39
75.6 = 0.8
90.2 + 6.6
Table 5: Average normalized scores of GCPC against other baselines AntMaze. TAP's perfor- mances are taken from the original paper [28]. For other baselines, we obtain their performance by re-rumning wuthor-providee ing [29], we bold all scores within 5 percent of the maximum per task (2 0.95 max).
our replicated implementations with evaluation protocol. Follow-
Dataset
BC
CQL
IQL
DT
TAP
WGCSL
GCIQL
DWSL
RyS-G
GCPC
Umaze
63.4 9.94
88.2 23
92.8 1
55.6
1:63
90.8 24
91.6 1.40
71.2 4:4
70.4 4:4
71.2 13
Umaze-Diverse
63,4 4:44
47.4
71.2 2:0
53.4
55.6 157
88,8 2:
74.6 23
66.2 556
71.2
Medium- Play
0.6 0:5
72.8 23
75.8 1:1
78.0
63.2 133
82.6 1:44
77.6 30
71.8 1:77
70.8 1.34
Medium- Diverse
0.4 0.1
70.8
76.6
85.0
46.0
76.2
74.8
72.0
72.2
Large-Play
36.4 10.
50.0 4.77
74.0
0.6 466:11
40.0 162
15.2 =m
35.6 776
78.2 3.
Large-Diverse
4.
52.6
82.0
23
29.8
25.2
80.6
Ultra- Ultra-Play
18.0 = 13.3
21.2 4
22.0
0.2 2:
20.6 1:28
25.2 3:
25.6 6:67
56.6 1:
Ulra- Diverse
186
1:40
26.0
28.4 118
26.4 1777
54.6 18.
Average
16.0
47.4
57.3
13.6
32.4
57.3
47.8
49.2
69.4
4.5 Effectiveness of GCPC
Finally, we show the effectiveness of GCPC by evaluating it on three different domains: AntMaze, Kitchen and Gym Locomotion.
Baselines and prior methods. We compare our approach to both supervised learning methods and value-based RL methods. For the former, we consider: (1) Behavioral Cloning (BC), (2) RvS- R/G [15], a conditional imitation learning method that is conditioned on either a target state or an expected return. (3) Decision Transformer (DT) [10]. return- cetuun-condiiionedd model-free method that policy, (3) Trajectory Transformer (TT) [27]. a moodel-based method trajectory model for planning, and (4) Decision Diffuser (DD) [2], diffusion-bel based planning method that synthesizes future states with diffusion model and acts by computing the inverse dynamics. For the latter, we select methods based on dynamic programming, including (5) CQL [31] and (6) IQL [29]. Additionally, we include three goal (state)-conddtioned baselines for AntMaze and Kitchen: (7) Goal-conditioned IQL (GCIQL), (8) WGCSL [52] and (9) DWSL [25]. AntMaze- Ultra is customized environment proposed by [28], therefore we include TAP's performance on the vÓ version of AntMaze for completeness. When feasible, we re-run the baselines with our evaluation protocol for fair comparison [[].
learns a transformer-bas
that performs beam search on transformer-basse

---

Our empirical results on AntMaze and Kitchen are presented in Table 5 and Table 6. We find our methods outperform all previous methods on the average performance. In particular, on the most challenging large and ultra AntMaze environments, our method achieves significant improvements over the RvS-G baseline, demonstrating the efficacy of learning good future representations using sequence modeling in Long-horizon tasks. Table 7 shows the results on Gym Locomotion tasks. Our approach obtains competitive performance as prior methods. We also notice that compared to Decision Transformer, RvS-R can already achieve strong average performance. This suggests that for some tasks sequence modeling may not be a necessary component for policy improvement. With a large fraction of near-optimal trajectories in the dataset, simple MLP policy may provide enough capacity to handle most of the locomotion tasks.
Table 6: Average normalized scores of GCPC against other baselines on Kitchen. CQL and DD's results are taken from [2]. For other baselines, we obtain their performance by re-running author- provided or our replicated implementations with our evaluation protocol. Following [29], we bold all scores within 5 percent of the maximum per task (2 0.95 max).
Dataset
BC
CQL
IQL.
DT
DD
WGCSL
GCIQL
DWSL.
RyS-G
GCPC
Mixed 48.9 1.07
52.4
53.2 1.66
50.7 11111
65.0
77.8 36
74.6 1 19
74.6 L.6
69.4 4.2.
75.6 1 68
Partial
437
50.1
39.7=43
18.6
57.0
75.2 #64
74.7 #.1
74.0 = 58
71.7 # 7.
90.2 = 6.6
Average
45.1
51.3
56.5
49.7
61.0
76.5
74.7
74.3
70.6
82,9
Table 7: Average normalized scores of our approach against other baselines on Gym Locomotion. TT and DD's 's results are taken from [2]. For other baselines, we obtain their performance by re-running puttorr-rrvided or our replicated implementations with our evaluation protocol. Following [29], we bold all scores within 5 percent of the maximum per task (2 0.95 max).
Dataset
Environment
BC
CQL
IQL
DT
TT
DD
RvS-R
GCPC
Medium- Medium-Expert
HalfCheetah
61.9
87.7
92.4
88.8
95
90.6
93.4
94.0
Medium- Expert
Hopper
55.1 2
110.5 29
104.2 8.7
108.4 : 20
110
111.8
111.3 : 622
111.7 0.
Medium-Expert
Walker2d
100.4 + 11.4
110.4 0.6
110.2 4.77
108.6 = 03
101.9
108.8
109.6 : 0.4
109.0 #2
Medium
HallCheetah
43.0 4.4
47.1 1:03
47.7 1:03
42.9 =02
46.9
49.1
44.2 0:
44.5 1.03
Medium
Hopper
55.8 1..
70.1
69.2
67.8 =2
61.1
79.3
65.1 5
68.0
Medium
Walker2d
74.1 #:32
83.5 0.5
84.5 1.5
76,5 1.6
79
82.5
78.4 2.66
78.0 2:4
Medium-Replay
HalfChectah
37.2 1.3
45.4 0.3
44,9 0.3
37,8 = 0.9
41.9
39.3
40.2 à: 02
40.7 1:5
Medium-Replin Replay
Hopper
33.7 1.3
96.2 1M
93.9 9:9
78.0 = 11.6
91.5
88.5 à: 129
94,2 1:33
Medium Replay
Walker2d
19.2 23
79.8
78.6 5.
72.5 93
82.6
75
71.0 =
77.6
Average
53.4
81,2
80.6
75.7
78.9
81.8
78.0
79.7
5 Conclusion
In this work, we aim to investigate the role of sequence modcling learning trajectory representations and its utility in acquiring performant policies. To accomplish this, we employ a two-stage framework that decouples trajectory representation learning and policy learning. This framework unifies many existing RvS methods, enabling us to study the impacts of different trajectory representation learning objectives for sequential decision making. Within this framework, we introduce a specific design Goal-cone conlicoonii Predictive Coding (GCPC), that incorporates a compact bottleneck to transfer representations between the two stages and learns goal-conditioned latent representations modeling the future trajectory. Through extensive experiments, we observe that the bottleneck generated by GCPC properly encodes the goal-conditioned future information and brings significant improvement over some long- horizon tasks. By empirically evaluating our approach on three benchmarks, we demonstrate GCPC achieves competitive performance on a variety of tasks.
Limitations and Future work. GCPC models the future by performing maximum likelihood estimation on offline collected trajectories, which may predict overly optimistic future behaviors and lead to suboptimal actions in stochastic environments. Future work includes discovering policies that are robust to the environment stochasticity by considering multiple possible futures generated by GCPC. Another limitation of our work is that GCPC may not be sufficient to maintain high accuracy for long-term future prediction when high-dimensional states are involved, which may potentially be tackled by leveraging foundation models to acquire representations for high dimesiiie inputs.
10

---

Acknowledgments and Disclosure of Funding
We appreciate all anonymous reviewers for their constructive feedback. We would like to thank Calvin Luo and Haotian Fu for their discussions and insights, and Tian Yun for the help on this project. This work in part supported by Adobe, Honda Research Institute, Meta AI, Samsung Advanced Institute of Technology, and Richard B. Salomon Faculty Research Award for C.S.
References
[1] Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Mare Belle- mare. Deep reinforcement learning the edge of the statistical precipice. In Advances in neural information processing systems, 2021.
[2] Anurag Ajay, Yilun Du, Abhi Gupta, Joshua B. Tenenbaum, Tommi S. Jaakkola, and Pulkit Agrawal. Is conditional generative modeling all you need for decision making? International Conference on Learning Representations, 2023.
[3] Bowen Baker, 1lge Akkaya, Peter Zhokov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune. Video pretraining (vpt): Learning act by watching unlabeled online vidcos. In Advances in Neural Information Processing Systems, 2022.
[4] Hangbo Bao, Li Dong, and Furu Wei. Beit: Bert pre-training of image transformers. In International Conference on Learning Representations, 2022.
[5] Mare Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 2013.
[6] Prajjwal Bhargava, Rohan Chitnis, Alborz Geramifard, Shagun Sodhani, and Amy Zhang. Sequence modeling is robust contender for offline reinforcement learning. arXiv preprint arXiv:2305. 14550, 2023.
[7] David Brandffonrrnner, Alberto Bictti, Jacob Buckman, Romain Laroche, and Joan Bruna. When does return-oodittioeed supervised learning work for offline reinforcement learning? In Advances in Neural Information Processing Systems, 2022.
[8] Tom Brown, Benjamin Mann, Nick Ryder, Mclanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In Advances in neural information processing systems, 2020.
[9] Micah Carroll, OTT Paradise, Jessy Lin, Raluca Georgescu, Mingfei Sun, David Bignell, Stephanie Milani, Katja Hofmann, Matthew Hausknecht, Anca Dragan, and Sam Devlin. Uni[MASK]: Unified inference in sequential decision problems. In Advances in Neural Infor- mation Processing Systems, 2022.
[10] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. arXiv preprint arXiv:2106.01344 2021.
[11] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 1mXiv:1810.04880 2018.
[12] Yiming Ding, Ignasi Clavera, and Pieter Abbeel. Mutual information maximization for robust plannable representations. arXiv preprint arXiv:2005.08 14, 2020.
[13] Yiming Ding, Carlos Florensa, Pieter Abbeel, and Mariano Phielipp. Goal-conditioned imitation learning. In Advances in neural information processing systems, volume 32, 2019.
[14] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. image is worth 16x16 words: Transformers for image recognition scale, arXiv preprint 20X0v 2020.

---

[15] Scott Emmons, Benjamin Eysenbach, Ilya Kostrikov, and Sergey Levine. Rvs: What is essential for offline RL via supervised learning? In International Conference on Learning Representations, 2022.
[16] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219 2020.
[17] Yuwei Fu, Di Wu, and Benoit Boulet. A closer look at offline RL agents. Advances Neural Information Processing Systems, 2022.
[18] Scott Fujimoto and Shixiang Gu. A minimalist approach to offline reinforcement learning. In Advances in Neural Information Processing Systems, 2021.
[19] Hiroki Furuta, Yutaka Matsuo, and Shixiang Shane Gu. Generalized decision transformer for of- fline hindsight information matching, In International Conference on Learning Representations, 2022.
[20] Dibya Ghosh, Abhishek Gupta, Ashwin Reddy, Justin Fu, Coline Manon Devin, Benjamin Eysenbach, and Sergey Levine. Learning to reach goals via iterated supervised learning. In International Conference on Learning Representations, 2021.
[21] David Ha and Jürgen Schmidhuber. World models. arXiv preprint arXiv: 101222220..112,, 2018.
[22] Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. arXiv preprint arXiv:2010.02199 2020.
[23] Nicklas Hansen, Yixin Lin, Hao Su, Xiaolong Wang, Vikash Kumar, and Aravind Rajeswaran. Modem: Accelerating visual model-based reinforcement learning with demonstrations, arXiv preprint arXiv:2212.056698 2022.
[24] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
[25] Joey Hejna, Jensen Gao, and Dorsa Sadigh. Distance weighted supervised learning for offline interaction data. In Proceedings of the 40th International Conference on Machine Learning, 2023.
[26] Michael Janner, Yilun Du, Joshua Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. In Proceedings of the 39th International Conference on Machine Learning, 2022.
[27] Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. In Advances in Neural Information Processing Systems, 2021.
[28] Zhengyao jiang, Tianjun Zhang, Michael Janner, Yueying Li, Tim Rocktäschel, Edward Grefen- stette, and Yuandong Tian. Efficient planning in compact latent action space. In International Conference on Learning Representations, 2023.
[29] Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learming. arXiv preprint arXiv:2110.06169, 2021.
[30] Aviral Kumar, Joey Hong, Anikait Singh, and Sergey Levine. Should run offline reinforcement learning or behavioral cloning? In International Conference on Learning Representations, 2022.
[31] Aviral Kumar. Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. In Advances in Neural Information Processing Systems, 2020.
[32] Michael Laskin, Aravind Srinivas, and Pieter Abbeel, Curl: Contrastive unsupervised represen- tations for reinforcement learning. In International Conference on Machine Learning, pages 5639-5650. PMLR, 2020.
12

---

[33] Kuang-Huee Lee, Ofir Nachum, Sherry Yang, Lisa Lee, C. Daniel Freeman, Sergio Guadarrama, Ian Fischer. Winnie Xu, Eric Jang, Henryk Michalewski, and Igor Mordatch. Multi-game decision transformers. In Advances in Neural Information Processing Systems, 2022.
[34] Fangchen Liu, Hao Liu, Aditya Grover, and Pieter Abbeel. Masked autoencoding for scalable and generalizable decision making. In Advances in Neural Information Processing Systems, 2022.
[35] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, 2021.
[36] Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3m: universal visual representation for robot manipulation. arXiv preprint arXiv:2203. /2601, 2022.
[37] Tung D Nguyen, Rui Shu, Tuan Pham, Hung Bui, and Stefano Ermon. Temporal predictive coding for model-based planning in latent space. In International Conference on Machine Learning, pages 8130-8139, PMLR, 2021.
[38] Simone Parisi, Aravind Rajeswaran, Senthil Purushwalkam, and Abhinav Gupta. The unsur- prising effectiveness of pre-trained vision models for control. International Conference on Machine Learning, pages 17359-17371. PMLR, 2022.
[39] Alexander Pashevich, Cordelia Schmid, and Chen Sun. Episodic Transformer for Vision-and- Language Navigation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.
[40] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018
[41] Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gómez Colmenarejo, Alexander Novikov, Gabriel Barth-maron, Mai Giménez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg. Tom Eccles, Jake Bruce, Ali Razavi, Ashley Edwards, Nicolas Heess, Yutian Chen, Raia Hadsell, Oriol Vinyals, Mahyar Bordbar, and Nando de Freitas. A generalist agent. Transactions on Machine Learning Research, 2022. Featured Certification.
[42] Machel Reid, Yutaro Yamada, and Shixiang Shane Gu. Can wikipedia help offline reinforcement learning? arXiv preprint arXiv:2201. 122122 2022.
[43] Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourleenth international conference on artificial intelligence and statistics. JMLR Workshop and Conference Proceedings, 2011.
[44] Younggyo Seo, Danijar Hafner, Hao Liu, Fangchen Liu, Stephen James, Kimin Lee, and Pieter Abbeel. Masked world models for visual control. In Conference on Robot Learning, pages 1332-1344. PMLR, 2023.
[45] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
[46] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, volume 30, 2017.
[47] Kerong Wang, Hanye Zhao, Xufang Luo, Kan Ren, Weinan Zhang, and Dongsheng Li. Boot- strapped transformer for offline reinforcement learning. In Advances in Neural Information Processing Systems, 2022.
[48] Jialong Wu, Haixu Wu, Zihan Qiu, Jianmin Wang, and Mingsheng Long. Supported policy optimization for offline reinforcement learning. In Advances in Neural Information Processing Systems, 2022.
13

---

[49] Philipp Wu, Arjun Majumdar, Kevin Stone.
Yixin Lin, Igor Mordatch, Pieter Abbeel, and Aravind Rajeswaran. Masked trajectory models for prediction, representation, and control. In International Conference on Machine Learning, 2023.
[50] Tete Xiao, Ilija Radosavovic, Trevor Darrell, and Jitendra Malik. Masked visual pre-training for motor control. arXiv preprint 1rX1::2203306173, 2022.
[51] Mengjiao Yang and Ofrr Nachum. Representation matters: offline pretraining for sequential decision making. In International Conference on Machine Learning. PMLR, 2021.
[52] Rui Yang, Yiming Lu, Wenzhe Li, Hao Sun, Meng Fang. Yali Du, Xiu Li, Lei Han, and Chongjie Zhang. Rethinking coal-conditioned supervised learning and its connection to offline RL. In International Conference on Learning Representations, 2022.
14

---

A Experimental Details
A.1 Additional Baseline Details
We provide additional information on the baseline methods and their performance we present in the main paper. We compare our proposed method with (I) supervised learning methods, including Behavioral Cloning (BC), RvS, Decision Transformer (DT), Trajectory Transformer (TT), and Decision Diffuser (DD). We also compare with (2) value-based methods, including Conservative Q-Learning (CQL) and Implicit Q-Learning (IQL). DT uses a transformer encoder to model offline- RL as autoregressive sequence generation problem in model-free way, which predicts future actions based on previous states, actions, and returns-to to-go. Similarly, The model-based TT also adopts transformer structure for sequence prediction but uses beam search for planning during execution. Inspired by conditional generative model, DD adopts a diffusion model to sample future state sequence based on returns or constraints and extract actions between the states by using inverse dynamics model with relyng on value estimation. Similar to our PolicyNet, RvS-G and RyS-R take the current state and the goal (retum state) as input, and train a two-layer MLP to predict the actions via supervised learning. For value-based methods, in order to solve the problem of overestimating values induced by distributional shift, CQL adds value regularization terms the standard Bellman error objective to learn lower bound of the true Q-function. IQL approximates the policy improvement step implicitly instead of updating the Q-function with target actions sampled from the behavior policy. For the additional goal -cnddiioeod baselines, GCIQL is goal- conditioned version of IQL. WGCSL enhances GCSL with an advanced compound weight, which optimizes the lower-bound of the coal-conditioned RL objective. DWSL first models the distance between states and the goal, and extracts the policy by imitating the actions that reduce the minimum distance metric.
Baseline Performance Source. For AntMaze, we use v2 version for D4RL environments and vO version for the additional AntMazeUUlrr environments. For Gym Locomotion and Kitchen, we use v2 and v0 version of the datasets from the D4RL benchmark, respectively. For all baselines except TT, DD and TAP. we obtain their performance by re-rumning author- author-provided (CQL, IQL, DT, WGCSL, GCIQL, DWSL) or our replicated (BC, RvS-G/R) implementations with our evaluation protocol. For CQL's performance on Kitchen, we are unable to reproduce the previously reported performance with the defaaut hyperparameters, so we take the corresponding results from [2]. For additional goal conditionn baselines (i.e. WGCSL, GCIQL and DWSL), we take the implementations provided by DWSL authors and use the same goal specification (see details in A.6) for AntMaze and Kitchen as ours. For TT, DD and TAP's performance, we take the results from the original papers.
A.2 Evaluation Protocol
Many prior work [48, 18] reports their performance by taking the final checkpoint's 's evaluation results and averaging them over multiple seeds. However, in some cases we observe that this evaluation protocol can lead large oscillations in the policy performance when gradient steps (or training epochs) are slightly perturbed or the random seeds are changed (as in Figure A1). As different methods may various number of training steps, seeds and evaluation trajectories, it hampers drawing robust conclusions by comparing these results. To faithfully capture the performance trend, which crucial for ablation analysis, for each seed take the best evaluation result among the last five checkpoints as its performance, where each evaluation result is the average return over N evaluation trajectories (e.g. N 10, 50, 100) achieved by a given checkpoint, and then we calculate the mean performance and standard deviation over five seeds. Empirically we find this evaluation protocol yields results that are less sensitive to the above factors and usually have smaller error bounds, facilitating the comparisons in ablation studies. Mcanwhile, we observe that in some cases this evaluation protocol might lead to results that are higher than those reported in previous papers, especially when the policy has more drastic performance fluctuations during training. We re-run most baselines using the same evaluation protocol for fair comparisons.
To investigate the impact of different aggregation metrics, we follow [1] to compare performance aggregation using mean, median, IQM and optimality gap on the AntMaze tasks. In Figure A2, we observe that GCPC consistently outperforms compared baselines across all metrics.
'attss//eituubbco//jjjjn/////
15

---

Antmaze-Large
Antmaze-Large
Hopper Hopper-Mediune MMediun------ Mepeut-Expert
125
DWSL. GCIQL
80
100
U MMM
Nomnalized Seare
40
60
75
40
20
50
20
25
GCPC
CQL
70k
80k
IM
Figure A1: Training curves of different methods with various default gradient steps
Modian
IQM
Mcan
Optimality Gap
BC
CQL
GCIQL
DWSL
RvS-G
GCPC
20
40
60
20
30
45
60
10
20
30
Normalized Score
Figure A2: Different aggregation metrics with 95% confidence intervals on AntMaze tasks.
Apart from model-free policies, it also remains unclear whether or how the results of planning-based methods [2, 27] would be affected by the choice evaluation protocol. leave adopting more reliable and robust evaluation approaches for comprchensive analysis to the future work.
A.3 Hyperparameter
We list hyperparameters for BC and RvS-G/R replication in Table Al and GCPC implementation in Table A2. For RvS-G/R and PolicyNet in GCPC, use a two-layer feedforward MLP as the policy network, taking the current state and goal (state or return-to-go) as input, the only difference is that PolicyNet takes the bottleneck as additional input. All experiments are performed single Nvidia RTX A5000.
Table Al: Replicated BC and RvS-G/R Hyperparameter
Hyperparameters
Value
Hidden Layers
2
Embedding Dim
1024
Nonlinearity
ReLU
Epochs
80 AntMaze, Kitchen
100 Gym Locomotion
Batch Size
1024 AntMaze, Gym Locomotion
256 Kitchen
Dropout
0
Optimizer
Adam
Learning Rate
1e-3
A.4 Explicit future
Existing methods, like TT and DD, utilize future information by sampling future states (or transitions) and planning on the explicit future. TT learns a world model that autoregressively generates states, actions, rewards and return-to-go estimates, then uses beam search select future trajectories
16

---

Table A2: GCPC Hyperparameter
Hyperparameters
Values
Optimizer
Adam
TrajNet
Encoder Layers
2
(Bidirectional Transformer)
Decoder Layers
I
Attention Heads
4
Embedding Dim
256
Nonlinearity
GELU
Slot Tokens
4
(History Window k. Future Window p)
(10, 70) AntMaze
(5. 30) Kitchen
(5, 20) Locomotion
Epochs
60 AntMaze, Locomotion
20 Kitchen
Batch Size
1024
Dropout
0.1
Learning Rate
le-4
PolicyNet
Hidden Layers
2
(Multi-Layer Perceptron)
Embedding Dim
1024
Nonlincarity
RcLU
Epochs
80 AntMaze, Kitchen
100 Gym Locomotion
Batch Size
1024 AntMaze, Gym Locomotion
256 Kitchen
Dropout Learning Rate
0
1e-3 AntMaze, Gym Locomotion
1c-4 Kitchen
with highest cumulative rewards. DD leverages the diffusion process with classifier-fre guidance to generate future states that satisfy reward maximization or other constraints, and uses inverse dynamics to extract actions between future states.
In GCPC, we compress the goal-conditioned future into latent representations and use them to support policy learning. In Section 4.4, we study the effect of explicitly leveraging decoded future states. We adapt our framework as follows: During the policy learning and evaluation, we first freeze transformer encoder and decoder trained in the first stage. Instead of directly passing the encoded bottlenecks to the PolicyNet, we leverage the pre-trained transformer decoder to decode the bottleneck into goal-conditioned futures. The new PolicyNet is modified to take the observed states, the goal, and the concatenation of the decoded future states. The bottleneck is discarded and not used by the modified PolicyNet.
A.5 Future window length
We further investigate the effect of future window length on policy performance, which decides how far the bottleneck can look into the future during the first stage pretraining. In Table A3, we show that predicting further future would generally bring bigger improvements in AntMaze-Large, emphasizing that planning into long-term future can largely benefit policy performance. We also observe that the improvements brought by enlarging future window are specific both datasets and the range of future window length. The improvements become marginal once the future window expands beyond certain size.
A.6
Goal Sampling and Specification
To perform goal-conditioned imitation learning, we follow [15] to sample an outcome from the future trajectory т:: as the goal. When the goal in the form of a target state (e.g. in AntMaze and Kitchen tasks), we randomly sample a reachable future state from { and keeps the task-related subspace as g. Specifically, for AntMaze we take the ant's (æt, y) location as the goal, which are the first two dimensions of the state space; for Kitchen's 's goal, we keep the dimensions
17

---

Table A3: Future window length. In AntMaze-LLage, the farther into the future can the bottleneck see during trajectory representation pre-training, the more helpful they are for policy learning.
Future window length p
20
40
70
62.4 = 5.6
78.2 = 4.8
80.6 = 3.9
Large-Play
49.6 1:
76.6
78.2
Average
56.0
77.4
79.4
that indicate the position of the target object in all subtasks and zero out other dimensions (c.g. dimensions that indicate the robot proprioceptive state) in the 30-dimensional state space". When the goal is represented the expected return-to-go (e.g. in Gym Locomotion tasks), we use the average return achieved over some number of timesteps into the future as g (i.e. g Ø- Line To where follow [15] set H to the constant maximum episode length).
B Pseudocode of GCPC
Algorithm 1 Goal-conditioned Predictive Coding (GCPC) for RvS
I: Input: Dataset of trajectories D = {r}. History window length k, Future window length Masking ratio r, Bi-directional transformer encoder hene, Bi-directional transformer decoder hdee Learnable slot tokens Q, Policy network
2: while First Stage do
3: Sample trajectory and timestep: [(s;, ( ~ D,, [K; H]
4: Sample a goal g, obtain observed states and full states: 84 = St-k+ 1:t, 810 = 81 81-
5:
Randomly mask observed states with masking ratio r: % = Masked 8h.
6:
Encode masked observed states, goal and learnable slot tokens: Reconstruct observed and future states with the bottleneck B: S haco (
hene Q2 2,5h
7:
8: Compute loss: MSE($ (wr Sw)
9: Update parameters in hene and hase
10: end while
11: Freeze parameters in hene
12: while Second Stage do
13:
Sample trajectory and timestep: + ((8i, H
D, ~ (k; H]
14:
Sample a goal g, obtain observed states: 8h, = 8t-k+1#
15:
Encode observed states, goal and learnable slot tokens: Bobs Nemc (Q, 8hs 9)
16:
Train the policy network π with the bottleneck Bobs: â = π (Bobs: St, g) Compute loss: MSE( ac)
18:
Update parameters in π
end while
20:
return Policy network π
C Pre-training mask ratio
In the first stage of GCPC, randomly mask history states with a masking ratio We compare using fixed masking ratios (20%, 40%, 80%) and dynamic masking ratio "D" uniformly sampled from 0%. 20%, 40%. 60% and 80%. Figure A3 shows the influence of different masking ratios on AntMaze and FrankaKitchen. We found that low masking ratios (e.g. 20%) would usually perform worse except on the Kitchen-Mixed dataset, and high masking ratio (80%) turns out be more effective, which is similar to the observation in MAE [24]. However, the optimal masking ratio is not consistent across environments. order achieve the best overall performance, we adopt dynamic masking ratio "D" in our implementation.
IIn the Kitchen experiments, we found important to zero-out redundant dimensions, such as robot proprioceptive dimensions, when specifying the task goal for goal (state) conditioned methods. We use this specification for all goal conditoood methods.
18

---

Lege-rrens Lugeday
Kackenkird Katiev/atal
72
85
0.8
0.2
Rais
Figure A3: Mask ratio ablation on AntMaze and FrankaKitchen. "D" refers dynamic mask ratio randomly drawn from 0%, 20%, 40%, 60% and 80%
D Additional Experiments on FrankaKitchen
In Table A4, we present how different pretraining objectives affect policy performance on FrankaKitchen benchmark. MAE-F and MAE-RC the only valid objectives in zero-shot setting. In two-stage setting, we observe that action inputs would generally cause larger variance and harm the performance of two predictive coding objectives. Similar to observations AntMaze, predictive coding objectives can consistently generate powerful trajectory representations, which yield performance gains Kitchen tasks.
Table A4: Objective Ablation on Kitchen-Parting We report mean and standard error over seeds.
AE-H
MAE-H
MAE-F
MAE-RC
MAE-ALL
Zero-shot
0
4.4 9
4.4 + 6.5
Two-stage (w/ actions)
76.9 2399
73.6 16.6
86.4 + 12.5
80.4 #
66.8 14.5
Two-stage (w/o actions)
66.4 5.9
72.7 1 7.0
92.5 1: 4.9
90.2 1: 6.6
68.0 4.99
E Visualizing the Latent Future
In this section, visualize the latent future by decoding the bottlenecks back into the state space using the pre-trained decoder. We visualize the currently observed state and the decoded future states inside the maze environment. Figure A4 includes two examples extracted from successful evaluation rollouts. The blue part on the left shows the actual position the agent at certain timestep 8, during the evaluation and the green part the right is the latent future predicted based history states 8g1 and the goal g. We find in the latent future, the agent gradually move towards the goal, which demonstrates that bottlenecks can encode meaningful future trajectories that provide the correct waypoints as subgoals and bring the agent from the current location to the final goal.
Figure A5 illustrates how latent future vary depending different goal conditions for the same current state. By specifying different goals, the latent future would change correspondingly and point the right direction for the agent. These visualizations suggest that the bottlenecks have the ability to provide the correct intermediate waypoints based on history states and the goal, which crucial for long-horizer
goal-reaching tasks. In the second example of Figure A5, multiple paths connect the agent and the goal, but bottlenecks encode the shortest one for the agent, which also demonstrates the bottleneck's 's ability to stitch sub-optimal trajectories and compose the optimal path.
In Section 4.3, we found that removing the goal from TrajNet input would seriously harm the performance large AntMaze. In Figure A6, we visualize the latent future without goal conditioning. In both examples, bottlenecks encode latent futures that either move the opposite direction the goal or enter a dead end and cause tasks fail. It again demonstrates that the quality of the encoded
19

---

Goal Location
Agent Location
Evaluation
Latent Future for Timestep L1 (w/ Goul 1)
PF
LF
Timestep
Evaluation
Latent Future for Timestep (w/ Goal 2)
UP
GP
PE
Timestep 12
Figure A4: Two latent future examples from successful rollouts. Latent future shows the right path to the goal.
Goal Location Location Agent Location
2DF
2PE
2P6
Evalustion
Latent Future Timestep L(w/ Goal
JD F
Timestep
Latent Future for Timestep (w/ Goal
UF
UPA
2PP
Figure A5: Latent future conditioned on different goals in large AntMaze.
20

---

future would significantly affect how policy performs, and providing goal information can prevent misleading future.
Goel Location
Goal Location
Agent Location
Evaluation
OFF
Latent Future for Timestep (3/o Goal
2P6
EF
Pr
Timestep 83
Evaluation
Latent Future for Timestep (1/w Goal
علرد F
DFG
2F
Timestep M
Figure A6: Failure cases after removing goal from TrajNet input. conditioning would mislead the agent.
Latent future without goal
F
Slot tokens ablation
Table A5 shows how changing the number of slot tokens would affect policy performance on large AntMaze. We observed that using different numbers of slot tokens would not have big impacts on the performance. our implementation, we uniformly use slot tokens for both trajectory representation learning and policy learning.
Table A5: The number of slot tokens. We report mean and standard error over 3 seeds.
Slot Tokens
4
Large-Diverse Diverse
80.6 3.3
77.2 1: 4.9
80.6 3.99
Large-Play
81.2 4,4
78.8 4.7
78.2 # 3.2
21


Reviews:
[{"review_id": "asgakWYJhN", "reviewer": "Reviewer_3wNA", "summary": "The manuscript presents a novel framework for formulating decision making as a supervised learning problem on offline-collected trajectories. Contrary to previous methods, the authors propose to decompose the process into two stages: a) a trajectory representation learning stage (using sequential modeling techniques, like Transformers), and b) a policy learning stage where the trajectory representation learned in the previous stage is being used as an input to the policy. Overall, the proposed framework and specific instantiation (GCPC) is extensively evaluated and analyzed through experiments in 5 different simulation environments.", "questions": "I do not have any important questions.", "limitations": "The authors adequately discuss the limitations of their approach.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The main strengths of the manuscript are as follows:\n\n- The paper is overall well written, the framework clearly articulated and the experiments well presented.\n- I like the decoupling to two stages: it makes sense, makes things clear, and allows for better experimentation.\n- Extensive experiments and analysis.\n", "weaknesses": "I have, however, a few concerns:\n\n- Why did you use the success rate over 100 evaluation trajectories for the AntMaze and only 10 for the other scenarios?\n- \"We use the best evaluation result among the last five checkpoints as the performance for each seed.\" Is there a specific reason for using the best among the last five checkpoints and not just the last checkpoint? Did the learning have a tendency to diverge or oscillate?\n- \"For each experiment, we report the mean performance averaged over different seeds.\". Using the median and quartiles/percentiles should on average be a better metric. See this wonderful paper by colleagues on how to properly choose the number of seeds and type of metric: https://arxiv.org/abs/1806.08295\n- In section 4.3, did the authors remove both the goal and the bottleneck inputs? Or just the bottleneck ones? Or just the goal? There needs to be some clarification here in the text. If they removed both, I am not sure if this experiment has any importance.\n\n\nTypos:\n\n- Line 228: \" Other than that, We also\". W should be lowercase\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback! We address your concerns in detail below and will update our paper accordingly:\n\n>***The numbers of trajectory rollouts***\n\nIn our experiment, we regard IQL[a] as a robust value-based baseline, which has demonstrated its effectiveness across a diverse array of tasks and is widely compared by other works. We used the same number of evaluation rollouts as adopted by IQL, employing 100 rollouts for Antmaze and 10 rollouts for the other environments. We note that this setup is aligned with our empirical observations: With the same model checkpoint, the performance variance across different evaluation rollouts is often higher in Antmaze, than that of Kitchen and Locomotion.  At the earlier stage of our experiments, we applied more rollouts for Kitchen and Locomotion, and found their performances remained stable when using 10 or more rollouts.\n\n>***Why select from the last five checkpoints? Did the learning have a tendency to diverge or oscillate?***\n\nYes, we indeed observed that performance may oscillate for the AntMaze environment. Similar behavior has been repeatedly observed in earlier works (e.g. WGCSL[b], DWSL[c]).  Please find the learning curves of GCPC and multiple previous goal-conditioned methods in Figure R1 of the attached rebuttal PDF file. The curves of previous goal-conditioned methods, including GCSL[d], WGCSL, GoFar[e], GCIQL, and DWSL, were reported by the DWSL paper based on 50 rollouts.  When overlaying these curves, we can clearly see that GCPC achieves the best performance among all approaches. Our evaluation strategy more faithfully captures the performance trends, which is particularly helpful for ablation studies. In Table R4 of the rebuttal PDF, we provide the results of DWSL and WGCSL using our evaluation metrics. Our GCPC outperforms both baselines by large margins.\n\n>***Using the median and quartiles/percentiles should on average be a better metric.***\n\nThank you for the reference! We agree that better metrics, such as median and quartiles/percentiles, would make the evaluation results more fair and complete. We reported the average performance to compare with existing offline RL methods with the same metrics they have adopted. Following previous approaches (e.g. RvS[10], DD [1]), we train and evaluate our methods with multiple (e.g. 5) seeds, and calculate the mean and standard deviation across seeds for each dataset.\n\n>***In section 4.3, did the authors remove both the goal and the bottleneck inputs? Or just the bottleneck ones? Or just the goal?***\n\nThanks for pointing it out! In section 4.3, we indeed just remove the goal from TrajNet during the bottleneck computation. We retain both the bottleneck and the goal as inputs to PolicyNet. The ablation thus studies the importance of goal-conditioning for the bottleneck representation to serve as an “implicit planner”. We will update section 4.3 to clarify.\n\n\n[a] Kostrikov et al. \"Offline reinforcement learning with implicit q-learning.\" arXiv preprint arXiv:2110.06169 (2021).\n\n[b] Yang, Rui, et al. \"Rethinking Goal-Conditioned Supervised Learning and Its Connection to Offline RL.\" International Conference on Learning Representations. 2021.\n\n[c] Hejna, Joey, Jensen Gao, and Dorsa Sadigh. \"Distance Weighted Supervised Learning for Offline Interaction Data.\" arXiv preprint arXiv:2304.13774 (2023).\n\n[d] Ghosh, Dibya, et al. \"Learning to Reach Goals via Iterated Supervised Learning.\" International Conference on Learning Representations. 2020.\n\n[e] Ma, Jason Yecheng, et al. \"Offline goal-conditioned reinforcement learning via F-advantage regression.\" Advances in Neural Information Processing Systems 35 (2022): 310-323."}, {"review_id": "3hAI2nvlwb", "reviewer": "Reviewer_tnif", "summary": "The paper presents a perspective on RL via Supervised Learning (RvS) algorithms that identifies two steps—trajectory representation and policy learning—and proposes a specific algorithm for offline RL called Goal-Conditioned Predictive Coding (GCPC) based on this view. The trajectory encoder is modelled as Bidirectional Transformer (BT) that compresses trajectories into slot tokens, which are then passed to a goal-conditioned policy. Interestingly, actions are omitted from trajectories, which improves the performance. Ablation studies on the Antmaze environment on 3 seeds are conducted, including\n\n1) masking objectives for BT pertaining (=> masking entire future and inputing (randomly masked) past + future works best)\n2) impact of goal conditioning (=> conditioning helps a lot)\n3) comparing latent vs explicit future prediction (=> as per Table 3, there is no significant difference (returns within error bounds))\n4) effects of future window size (=> bigger better)\n\nComparisons against baselines on Antmaze-v2, Kitchen, and Gym Locomotion are provided. GCPC performs on the same level as Decision Diffuser (DD) (within error bounds) on Kitchen and Gym, and below IQL on Antmaze (Maze and Medium). Only on Antmaze Large GCPC outperforms all baselines.", "questions": "- Antmaze seems like a quasi-stationary environment in the sense that the policy for locomotion does not depend on where in the maze the ant is. What is the meaning of compressing a trajectory in this case? It seems wasteful.", "limitations": "yes", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- Originality: the paper presents a nice perspective on existing algorithms, that distinguishes between trajectory representation and policy learning steps. The exact proposed architecture is novel, but it is quite similar to alternatives.\n- Quality: a number of ablation studies are provided and comparisons to many baselines. However, the ablations are done on 1 environment and on 3 seeds only, and many results are within error bounds, so it is hard to make conclusions. The comparisons to baselines do not show improvement over DD/IQL except on one task: Antmaze Large.\n- Clarity: the paper is mostly clear, but improvements can be made. E.g., table captions can be more informative, many claims need to be toned down, e.g., \"significant influence\" of representation pertaining and \"superior performance\" claimed in the conclusion are not supported by the results.\n- Significance: minor. The approach performs similar to IQL/CQL/DD. Practically, there is no significant advantage to it.", "weaknesses": "The idea seems nice but the evaluations are insufficient.\n1) The ablations need more environments and seeds. It is not clear that ablations on Antmaze generalise to FrankaKitchen, for example.\n2) Performance on Kitchen and Gym is not so different from the baselines. Only Medium dataset used on Gym. No DD on Antmaze.\n3) See other comments from above about originality/quality/clarity/significance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback! We address your concerns in detail below and will update our paper accordingly:\n\n>***More seeds are needed for ablation experiments.***\n\nWe reported ablation experiments on FrankaKitchen in the supplementary materials Table A3 and A4. In Table R1, R2, and R3 of the rebuttal PDF, we report ablation experiments of Antmaze and Kitchen with 5 seeds. We observe that our observations still hold after more seeds are used: On Antmaze Large, using goal-conditioned predictive coding (i.e. MAE-F and MAE-RC) and removing actions significantly outperform the alternative sequence modeling strategies. However, as we already observed in the supplementary, the trends on FrankaKitchen are less conclusive, as most performance gaps fall within error bounds. Moreover, the performance variances tend to become larger when more seeds are used. We attribute this to the much smaller trajectory dataset size of FrankaKitchen: Compared to Antmaze datasets (nearly 1000 trajectories and 1001 transitions per trajectory), Kitchen datasets are much smaller (nearly 500 trajectories, but only 227 transitions per trajectory on average), and provide one order of magnitude fewer training examples per epoch. We thus conjecture that the TrajNet is more prone to overfitting and may fail to encode trajectory information useful for planning and decision making.\n\n>***The performance is similar to IQL / CQL / DD. What are the practical advantages of the proposed method?***\n\nAs shown in Tables 6 and 7 of the original submission and the evaluation on CALVIN[a] in our general response, GCPC outperforms both IQL[b] and CQL[c] consistently. Notably, on Antmaze Large, where most previous methods failed to solve this challenging task, GCPC achieves remarkable performance and surpasses the most performant baseline, IQL, by a large margin. While GCPC performs similarly to DD[1] on Kitchen and Gym, we note that DD is based on conditioned diffusion models, and our design is more efficient than DD during training and evaluation. On a single RTX3090 GPU, our approach achieves 3x speedup for training and **120x** speedup during evaluation on the AntMaze Large environment, when compared to the DD implementation provided by its authors. We believe the consistently competitive performance across diverse benchmarks, and the computational efficiency are the main practical advantages of our method.\n\n>***Originality of the proposed framework.***\n\nThe primary goal of this paper is to understand how sequence modeling can be properly applied to decision making, and our framework is proposed for this purpose. To investigate the influence of sequence modeling on decision-making, we dissect the process into trajectory representation learning and policy learning. In the meantime, we aim to propose a general framework to compare recent works on sequence modeling for decision making, rather than a particular unique design. We extensively examined various sequence modeling strategies under the proposed two-stage framework, and discovered a powerful objective (GCPC) for trajectory representation learning. We demonstrate the importance of future planning in long-horizon tasks and validate the feasibility of acquiring an implicit planner – the bottleneck, with sequence modeling techniques. In addition, we further demonstrate the efficacy of the “implicit planner” by conducting a comparative analysis against the plans in the form of explicit future states. We believe our insights on the impact of sequence modeling for decision making, as well as the discovered GCPC objective with strong empirical performance, are both original and significant.\n\n>***Why only Medium dataset was used on Gym? How about DD performance on Antmaze?***\n\nWe report medium dataset performance as it is a standard practice adopted by previous work, such as TT[17], and DD[1]. The rationale is that the medium dataset contains suboptimal trajectories which can be used to validate a method’s capability to learn better policies than the offline dataset.\n\nDD did not report on Antmaze and DD’s authors don't provide any hyperparameters for Antmaze either. As noted in our response on practical advantage above, we observe that training DD on Antmaze (with its default number of training steps) demands nearly 20 hours. It is thus impractical for us to perform a meaningful hyperparameter search and report its performance.\n\n>***Clarity can be improved, such as table captions. Many claims need to be toned down.***\n\nThank you for the suggestions. We will elaborate table contents in captions and adjust some statements as appropriate based on the results in the revised paper\n\n>***Antmaze seems like a quasi-stationary environment in that the locomotion policy does not depend on where in the maze the ant is. What is the meaning of compressing a trajectory in this case? It seems wasteful.***\n\nIn the Antmaze, the ants need to avoid getting stuck in the corners and choose the correct direction (i.e. by adjusting the body orientation) at each intersection point. Since Antmaze is a partially observable environment (the ant is only aware of whether a certain direction is blocked or not), the history information is needed for the ant to remember the obstacles. Our framework effectively acquires these helpful signals by compressing a trajectory into a compact representation. This compressed representation is subsequently leveraged for supporting policy learning, as it empowers the ant to gain the necessary insights for successful task completion.\n\n[a] Mees, et al. \"Calvin: A benchmark for language-conditioned policy learning for long-horizon robot manipulation tasks.\" IEEE Robotics and Automation Letters 7.3 (2022)\n\n[b] Kostrikov et al. \"Offline reinforcement learning with implicit q-learning.\" arXiv preprint arXiv:2110.06169 (2021).\n\n[c] Kumar, Aviral, et al. \"Conservative q-learning for offline reinforcement learning.\" Advances in Neural Information Processing Systems 33 (2020)\n\n\n"}, {"review_id": "Eb0Ilv5Qsz", "reviewer": "Reviewer_eFoi", "summary": "This work introduces Goal-Conditioned Predictive Coding (GCPC) – a goal conditioned policy learning method that incorporates trajectory representations learned by a bi-direction transformer. The authors introduce a two-phase framework for goal-conditioned learning, consisting of a trajectory encoding phase, and a policy learning phase. GCPC uses a bidirectional transformer with different masking schemes as the trajectory encoder, then feeds the representation from fixed bottleneck tokens to a bidirectional decoder to predict the trajectory. Then, the policy is learned using the current state, goal, and trajectory representation. The author’s primarily evaluate their approach by taking the table from RvS (Emmons et al.) and adding their method, GCPC. They also provide a number of ablation studies over different masking strategies etc. \n", "questions": "What does bn mean in table 2? Is it batch norm? If so, it seems a bit unfair to use batchnorm for your method, while none of the baselines use it. \n\nFor which datasets / environments is GCPC goal conditioned? When is it using reward? This should be in the paper. \n", "limitations": "The authors could do a better job providing examples of failure cases or downsides of GCPC, if they exist. The limitations section as is mostly contains directions for future work. \n", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "This paper presents an original two phase method for learning policies from offline data using supervised methods. The idea of decoupling trajectory representation via a bottle-necked bidirectional transformer is new, though bottlenecked representations have been studied before with other architectures, like VAEs. It's good to see this considered as well. \n\nThe author’s ablation studies on their method are extensive – they compare different masking schemes, using and not using actions, the importance of goal representations, and window size. The quantity of datasets considered is also high. \n\nI also found the paper to be well organized, though with a few gaps in writing. \n", "weaknesses": "**Experiments**\n\nMy biggest concerns with this work are regarding the experimental comparisons in Section 4.6 – the primary results tables. \n\nSpecifically, *the baseline results are copied from other papers*  (RvS, SPOT). This is not stated in the main text or in the table caption (it should be, please add it there!). While I usually think it's better when authors run their own experiments, copy-pasted tables are now prevalent in the offline RL community.  For this work, however, I believe there is more that should be done.\n\nFirst, it seems like the authors are comparing goal conditioned algorithms to unconditional algorithms. For example, in Table 5 on Ant Maze BC, DT, TD3+BC, CQL, and IQL are all not goal conditioned (the paper from which these numbers are pulled does not train goal conditioned policies). On the other hand, RvS-G and GCPC, the two best performing methods on the table are goal conditioned! This feels problematic as in Table 2 its explicitly shown that without goal conditioning on Ant Maze GCPC drops from scores of above 75 to scores below 20 (50 point difference!). The same can be said for Table 6 with the kitchen environments, which might benefit from goal-conditioning. The effect of this is that the main experimental results feel like an apples to oranges comparison. \n\nSecond, the baselines chosen for the goal-conditioned experiments seem inappropriate. For goal-conditioned environments, the authors should be testing against goal-conditioned algorithms: Goal-Conditioned IQL with hindsight relabeling, decision transformer with goal-conditioning, GSCL [1], WGCSL [2], DWSL [3]. The best performing method on many environments in the RvS paper is GCSL as it has some degree of policy improvement via relabeling, but they don’t run it on most environments!  \n\nFinally, there is no notion of how much more computationally expensive GCPC is than baselines. GCPC has two stages, and one of those stages involves training a large bidirectional transformer. Many baselines, on the other hand, train only small MLPs. \n\nTo fix these issues, I think the authors should:\n1. Make it clear they take experiment numbers from prior works. \n2. Make it clear when experiments are goal conditioned or not\n3. For experiments that are goal conditioned, run baselines that are also goal-conditioned, ie GCSL, GC-IQL, Goal Conditioned DT, DWSL, testing hindsight relabeling as appropriate.\n4. Add a (short) discussion of compute intensity.\n\n**Two-Stage Framework**\n\nThe authors attempt to cast all “transformer for offline RL” methods into a general two stage framework. While I sort of see how this works, I don’t generally buy this as a cohesive unified framework. Most prior approaches all use the same transformer network for both trajectory representation and the policy (DT, MTM, TT). Casting other methods into this framework feels a bit unnatural, as no approach except for GCPC actually uses both a separate TrajNet and PolicyNet. I think that’s totally fine! But, I think the paper would be better explained by just stating that most other approaches use the same network for sequence representation and the policy, and GCPC separates them. Section 3.2 just doesn’t feel like the unified framework its supposed to be. \n\n**Writing**\n\nI found aspects of the writing confusing, and I believed this can be improved in the next version:\n1. The definitions of each masking scheme aren’t given until Table 1 in the results section. Given that you discuss these earlier in the paper, it feels necessary to define them in Section 3. \n2. Updating Figure 1 & 2 to make it more clear that only the bottleneck states are passed between the encoder and decoder. Also, what are the other inputs to decoder? Does it have the bottle neck tokens plus all the mask tokens for each state and action?\n3. Generally many important details – experiment numbers are taken from other papers, what is bn in Table 2, etc. – are unanswered or left in the appendix. \n4. The authors use (s, a, g) tuples in their writing, but state that sometimes g is just the reward. In the experiments section, it's really unclear when g is a state goal or the reward. \n5. I would use something other than Q for slot tokens. \n\n[1] Ghosh, Dibya, et al. \"Learning to Reach Goals via Iterated Supervised Learning.\" International Conference on Learning Representations. 2020.\n\n[2] Yang, Rui, et al. \"Rethinking Goal-Conditioned Supervised Learning and Its Connection to Offline RL.\" International Conference on Learning Representations. 2021.\n\n[3] Hejna, Joey, Jensen Gao, and Dorsa Sadigh. \"Distance Weighted Supervised Learning for Offline Interaction Data.\"  ICML 2023.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback. We address your concerns in detail below and will update our paper accordingly:\n>***Make it clear when numbers are quoted from published results***\n\nWe completely agree! We will incorporate the reviewer’s suggestion and explicitly explain if a baseline’s performance is quoted from their original papers.\n>***For which datasets/environments is GCPC goal conditioned? When is it using reward?***\n\nFor the conditioning variable, we followed the setup in RvS paper: In Antmaze and Kitchen, we used the target state as goal; In Gym locomotion, we used rewards (return-to-go). We will clarify this in the final version.\n>***It seems like the authors are comparing goal-conditioned algorithms to unconditional algorithms.***\n\nWe acknowledge that additional details, such as if an algorithm is goal-conditioned, should be highlighted, and we will incorporate the change in the final version. We follow prior work (TT [17], DD [1], RvS [10]) to present a variety of methods, including supervised learning methods (e.g. conditional BC, sequence modeling, etc.) and valued-based methods, for a comprehensive comparison. In the rebuttal PDF Table R4, we evaluate two suggested goal-conditioned methods DWSL[b] and WGCSL[a]. Our proposed GCPC outperforms DWSL and WGCSL by large margins. Please find our next response for more details.\n\nIn Table 2 of our original paper, the comparisons are not intended to demonstrate the superiority of goal-conditioning compared to alternative learning paradigms. Rather, it is an ablation study to demonstrate the importance of goal-conditioning for GCPC’s trajectory representation learning. We alter the inputs to TrajNet to include or exclude goal conditioning, and pass the resulting bottleneck representation to PolicyNet. In either scenario, PolicyNet takes the current state, the bottleneck and the goal as its inputs, and is thus always goal-conditioned. We will also clarify this detail in our revised paper.\n>***The authors should be testing against more goal-conditioned algorithms.***\n\nWe thank the reviewers for suggesting highly relevant works and agree that comparisons with goal-conditioned methods are needed for both Antmaze and Kitchen. Here we’ve provided two additional supervised goal-conditioned baselines – DWSL[b] and WGCSL[a]. For both algorithms, we use the implementation provided by DWSL’s authors, set the hindsight relabeling ratio to 1.0 to recover the experimental setting adopted by both RvS-G and GCPC, set training hyperparameters as recommended in the paper appendix[d], and re-run the baseline experiments under the same evaluation setup as ours. Please find the results in Table R4 of our attached PDF file. We observed that the performances of both goal-conditioned baselines are generally lower than the numbers reported by the DWSL paper. By comparing training curves and result tables (both from the DWSL paper), we speculate that their results correspond to the best mean performance during the whole training process, whereas our setup focuses on measuring the performance achieved near the end of the training. Among all supervised goal-conditioned baselines, GCPC significantly outperforms the compared ones on average performance and has a substantial improvement over Antmaze Large and Kitchen environments.\n>***Add a (short) discussion of computing intensity.***\n\nWe will add a short discussion of computing intensity compared to planning-based methods (e.g. TT[17] and DD[1]) in the revised version, mainly focusing on the following three aspects:\n\n1) As a sequence modeling approach, TT encodes (state, action, reward) tuples as $(|S| + |A| + 1)$ tokens, where $|S$| and $|A|$ refer to the dimensions of state and action spaces number. GCPC needs only $1$ state token per timestep. Assuming that hidden dimensions of each token are the same for both methods, the complexity of TT is $(|S| + |A| + 1)^2$ times that of GCPC, which will sharply increase when scaling to high-dimensional state/action spaces.\n\n2) Both DD and TT require either iterative sampling or step-by-step prediction to obtain a sequence of future states during evaluation. By default, for each evaluation step, DD takes $200$ diffusion steps to obtain a denoised state sequence, TT needs to predict 15 future timesteps ($15 * (|S| + |A| + 1)$ tokens) autoregressively. GCPC predicts future information within $1$ model forward step, which yields a significant efficiency gain.\n\n3) In practice, we recorded the wall time of performing evaluation on Antmaze Large using GCPC and DD. We observed DD takes nearly 1 hour for 1000 evaluation steps while GCPC takes no more than 30 seconds (a **120x** speedup).\n\n>***Updating Figure 1 & 2 to make it more clear. What are the other inputs to decoder?***\n\nIn the actual implementation, the decoder will take bottleneck tokens and a sequence of placeholder tokens for the states. We will update the corresponding figures in the final version.\n>***What does bn mean in table 2?***\n\n\"bn\" refers to the bottleneck. Since in both setting PolicyNet takes bottleneck as input, this notation is redundant and we will remove it from Table 2 for clarity.\n\n>***Providing examples of failure cases or downsides of GCPC, if they exist.***\n\n During evaluation, we observe our model would generate failure behaviors that appear in the dataset (e.g. the ant would flip over caused by bad actions) and is sensitive to the out-of-distribution states, which lead to suboptimal actions. In addition, our current approach could fail when stochasticity exists in the environment dynamics since GCPC always performs greedy planning with maximum likelihood estimation.\n\n\n[a] Yang, Rui, et al. \"Rethinking Goal-Conditioned Supervised Learning and Its Connection to Offline RL.\" International Conference on Learning Representations. 2021.\n\n[b] Hejna, Joey, Jensen Gao, and Dorsa Sadigh. \"Distance Weighted Supervised Learning for Offline Interaction Data.\" arXiv preprint arXiv:2304.13774 (2023).\n"}, {"review_id": "HiZLV6CQc3", "reviewer": "Reviewer_EZxU", "summary": "Update 8/24: As discussed with the authors, I'm updating my score to \"weak accept.\"\n\n-----------------------\n\nThis paper studies the role of representation learning for goal-conditioned behavioral cloning methods, focusing on those representations produced by sequence models. The paper fines that a statewise approach, first training a sequence model to produce representations and then using those representations as an _additional_ input to an MLP, outperforms end-to-end approaches. The paper also includes detailed ablation experiments studying how those representations should be learned (don't include actions, among other findings).", "questions": "1. L133 \"trajectory representations\" -- Why are there multiple representations?\n2. L136 -- What does it mean for B to be None?\n3. Eq. 2 -- Is the policy conditioned on the entire trajectory, or just the current observation (as indicated in L14 in Alg 1, Figure 2)?\n4. Alg 1, L3 -- Is $g$ the commanded goal or the goal that was actually reached in the future?\n5. What is the motivation for the slot tokens?\n6. The caption for Fig 1 says that the decoder will take in only the encoded slot tokens. Doesn't it also take in the encoded (and masked) history states?\n7. In the caption for Figure 2, what are \"latent future representations\"?\n8. L170, \"remove the actions from pre-training inputs\" -- When training the policy in Stage 2, does the encoder take the (past) actions are input?\n9. In Table 5, the numbers for the _reproduced_ RvS-G method are a fair bit better than the original RvS paper on all but one task. What are the differences between the implementations that might explain this gain?\n10. In a number of places, the paper refers to future representations and future information (e.g., L278). I'm confused what this is referring to. Is this the goal at the end of the trajectory, or something ese?", "limitations": "The limitations section does mention some limitations. I think there might be some other limitations that are a bit more important (e.g., the proposed method is much more complex than methods that skip Stage 1); the paper doesn't offer any theoretical explanation for _why_ the representations should help (If the tasks are Markovian, why should representations from a non-Markovian model help?).", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 4, "strengths": "**Originality**: This paper brings a fresh perspective to a class of RL methods that are becoming increasingly popular. I appreciate that, rather than focus on proposing a new method (which the paper admittedly does), much of the focus is on understanding the importance of design decisions from prior work.\n\n**Quality**:\n* The experiments seem quite thorough. Rather than just aiming to show that the proposed method is better than baselines, the paper includes a thorough investigation of how the representations should be learned, and includes experiments on a range of standard benchmarks. The experimental design seems clever and useful for probing the design decisions.\n\n**Clarity**:\n* The abstract, introduction, and related work sections are well written.\n* The introduction does a great job motivating the problem.\n* Figures 1 and 2 are useful for providing an overview of the method. The transparency on the left of Fig 2 is a useful visual trick.\n\n**Significance**\n* I think the paper is studying a really interesting problem, trying to dissect exactly why sequence models can be effective for conditional behavioral cloning.", "weaknesses": "1. The main weakness of the paper is that the experiments section is unclear, making it difficult to understand the experiments. Some of the confusion is from the different masking strategies, which I didn't see explicitly defined anywhere. Evening spelling out the names would help  (I'm guessing that \"AE\" is autoencoder and \"MAE\" is masked autoencoder, but I'm unsure what the other parts mean).\n2. I'm unsure about the claims about implicit planning. I admittedly didn't understand these experiments thoroughly (see #1). To claim that a method is performing planning, it seems like we'd need to construct a setting where we'd expect planning-based and planning-free methods should make different predictions; I didn't see any experiment like that.\n\n**Minor comments**:\n* Run a spelling + grammar checker.\n* \"they serve as world models for goal-conditioned planning\" -- Why can sequence models not serve as world models for more general RL problems? (Maybe cut \"for goal conditioned planning\")\n* L65 -- I don't think \"goal-conditioned predictive coding\" has been defined yet\n* When citing multiple papers, use \"\\citep{smith,adams}\" rather than \"\\citep{smith}\\citep{adams}\". This removes the extra square brackets between the numbered citations.\n* Don't use \"etc\" in technical writing.\n* L94 Avoid using the passive voice.\n* L111 Consider citing prior work that proposed very similar methods (albeit under different names). E.g., https://arxiv.org/abs/1906.05838, https://arxiv.org/abs/1912.06088\n* L114 Avoid starting sentences with notation or numbered citations. Starting sentences with Author-year citations is fine, because they will read like a normal sentence: \"Smith showed that ...\".\n* Alg 1 L3 -- Replace \"{\" with \"(\" to indicate that the trajectory is a sequence, rather than a set.\n* Fig 1 -- Label the bottleneck.\n* L182 \"which\" -> \"Which\"\n* Fig 3 caption \"left to right\" duplicated.\n* Table 6 caption, \"best overall performance\" --> \"best average performance\". I think it's a bit misleading to say \"overall\" because, on both tasks, there is a baseline that beats the proposed method.\n* In all tables, bold the best numbers (and those that fall within the error bars).\n* Citation [18] is a duplicate of [17]", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback. We address your concerns in detail below and will update our paper accordingly:\n>***Some of the confusion is from the different masking strategies, which I didn't see explicitly defined anywhere.***\n\nWe discussed the masking strategies in section 3.3 and also briefly introduced the definitions in Table 1. We will update section 3 to offer an elucidated clarification of these strategies. Specifically, in “AE-H” and “MAE-H”, we perform (masked) autoencoding solely on history sequence. “MAE-F”, “MAE-RC” and “MAE-ALL” requires TrajNet to reconstruct the entire trajectory including both history and future sequences. The differences between them are illustrated in the table below. \n\n|     \t| MAE-F    \t| MAE-RC      \t| MAE-ALL     \t|\n|---------|--------------|-----------------|-----------------|\n| History | Unmasked \t| Randomly masked | Randomly masked |\n| Future  | Fully masked | Fully masked\t| Randomly masked |\n>***Unsure about the claims about implicit planning.***\n\nIn section 4.3,  when we remove the goal-conditioning from the TrajNet, the bottleneck tries to predict the “goal-less” future and the planning can hardly be correctly performed (but we still keep the goal in PolicyNet). We can consider this \"No condition\" setting in Table 2 as a planning-free method. By comparing “No condition” and “Cond. on goal”, we observe that goal-conditioned bottleneck representations can significantly help policy learning. It suggests that this bottleneck encodes the desired future trajectory information according to the specified goal by performing the planning implicitly. In section 4.4 we further compare the bottleneck representation with the plan expressed as a sequence of explicit future states. Employing bottleneck representation yields a distinct performance improvement, which demonstrates that the planned future information can be well stored in a compact representation, and consolidates the role of the bottleneck as an “implicit planner”. In Figure A2, A3 and A4 of the supplementary, we also visualize the future states that are encoded in the bottleneck under different goal conditioning settings and the qualitative observations also align with above results.\n>***What does it mean for B to be None?***\n\nIn our framework, the two stages can be connected via compact representations (B), or transferred architecture and weights. We consider B in the second case to be None.\n>***Is the policy conditioned on the entire trajectory or just the current observation?***\n\nIn our design, PolicyNet has the observed trajectory but without architecture assumptions. An MLP policy conditions on the current observation by default, while a Transformer sequence model policy can consider the entire trajectory. In GCPC, the MLP policy is conditioned on the goal, current observation, and bottleneck, which either summarizes history, predicts future, or handles both tasks.\n>***Is the commanded goal or the goal that was actually reached in the future?***\n\nWe follow RvS[10] to sample a reachable future state (or the return-to-go achieved in some number of future steps) as the goal. We will clarify this in both section 3.1 and Algorithm 1, and add the detail of the goal sampling process in the supplementary material.\n>***When training the policy in Stage 2, does the encoder take the (past) actions are input?***\n\nSpecifically, we remove the actions from the TrajNet inputs during bottleneck computation. Since the TrajNet is frozen during the second stage, we keep the same input formulation as in the first stage. We will clarify this in the final version.\n>***The proposed method is much more complex than methods that skip Stage 1***\n\nWe agree that our two-staged pipeline could be more computationally intensive than those policies that do not leverage sequence modeling. As shown by our experiments, properly utilizing sequence modeling would bring substantial improvements over multiple tasks. Therefore, we believe our approach is desirable for scenarios that can afford more computational costs. We also welcome further discussions and suggestions on the limitations of our approach.\n>***If the tasks are Markovian, why should representations from a non-Markovian model help?***\n\nAntmaze as a non-Markovian task due to its partial observability. In this environment, the agent relies on past experiences to infer its surroundings and formulate optimal decisions for subsequent steps. Both Kitchen and Gym Locomotion are Markovian tasks as they provide the global state information to the agent. In our design, GCPC can generalize to both scenarios, when the context length shrinks to 1, GCPC can be considered as a Markovian model. To understand how history would affect Makovian tasks, we perform ablation experiments on Kitchen by varying the context length within the “AE-H” objective (results are shown below). We observe that having history context can also benefit Kitchen tasks, which is similar to the case in DT[6]. One hypothesis is that during training, history conditioning helps the model identify the policy used to generate the corresponding actions, among a distribution of policies, thus improving the training dynamics. Moreover, sequence modeling can also be used for predicting the future. The predicted future information, such as waypoints/subgoals, has been shown to be helpful by previous works in both Makovian[a] and non-Markovian[b] tasks\n\n| AE-H  (Two-stage w/o actions) | 1          \t| 5        \t| 15        \t|\n|-------------------------------|----------------|--------------|---------------|\n| Kitchen-Mixed             \t| 62.0 (± 18.15) | 72.5 (± 3.0) | 72.5 (± 15.8) |\n\n[a] Jayaraman, Dinesh, et al. \"Time-agnostic prediction: Predicting predictable video frames.\" arXiv preprint arXiv:1808.07784 (2018).\n\n[b] Chane-Sane, Elliot, Cordelia Schmid, and Ivan Laptev. \"Goal-conditioned reinforcement learning with imagined subgoals.\" International Conference on Machine Learning. PMLR, 2021.\n"}]

Author rebuttal:
# General Response to All Reviewers

We thank all reviewers for their insightful suggestions and constructive feedback. We are glad that the reviewers found our work to be “interesting”, “novel” (EZxU, tnif, 3wNA), offers a “nice” and “fresh” perspective on existing RL methods (tnif, EzxU), and that our experiments to be “extensive”, “thorough” and “well presented” (eFoi, 3wNA, EZxU). Below we summarize our findings in the newly added experiments as requested by the reviewers, and respond to their common questions:

1. We’ve presented the results of ablation experiments over 5 seeds on Antmaze and Kitchen (Tables R1, R2, and R3). The 5-seed results are consistent with the observations reported in the main paper and supplementary materials. Specifically, we observe that action-free goal-conditioned predictive coding yields significant improvements on Antmaze Large. As in Tables A3 and A4 of the supplementary, the impacts of masking schemes and goal-conditioning on Kitchen are less distinct, and most result differences fall within error bounds. We attribute this to the small dataset size of Kitchen, which we suspect leads to the overfitting problem on TrajNet.

2. We’ve provided two additional goal-conditioned supervised learning baselines, WGCSL[a] and DWSL[b], on Antmaze and Kitchen (Table R4). We adopt the implementation and the recommended hyperparameters as shared by the authors of DWSL, and use the same evaluation setup as GCPC. We observe that GCPC outperforms both goal-conditioned baselines by large margins on both environments.

3. We’ve evaluated our method on CALVIN[c], another long-horizon manipulation environment, featuring four target subtasks and a substantial collection of task-agnostic trajectories. We use the same evaluation protocol and 15 trajectory rollouts. As shown in the table below, our method outperforms IQL[d], POR[e], and RvS-G by large margins.

|        | IQL[d]       | POR[e]        | RvS-G         | GCPC          |
|--------|--------------|---------------|---------------|---------------|
| CALVIN | 7.8 (± 17.6) | 12.4 (± 18.6) | 20.6 (± 13.4) | 48.9 (± 11.6) |

[a] Yang, Rui, et al. "Rethinking Goal-Conditioned Supervised Learning and Its Connection to Offline RL." International Conference on Learning Representations. 2021.

[b] Hejna, Joey, Jensen Gao, and Dorsa Sadigh. "Distance Weighted Supervised Learning for Offline Interaction Data." arXiv preprint arXiv:2304.13774 (2023).

[c] Mees, Oier, et al. "Calvin: A benchmark for language-conditioned policy learning for long-horizon robot manipulation tasks." IEEE Robotics and Automation Letters 7.3 (2022): 7327-7334.

[d] Kostrikov et al. "Offline reinforcement learning with implicit q-learning." arXiv preprint arXiv:2110.06169 (2021).

[e] Xu, Haoran, et al. "A policy-guided imitation approach for offline reinforcement learning." Advances in Neural Information Processing Systems 35 (2022): 4085-4098.

Meta-review: This paper investigates how supervised sequencing models can/should be used for decision making, including different representation learning strategies from different masking of sequenced, and an empirical analysis of how those representations help control performance on various simulated control tasks. This paper also proposes their own method Propose Goal conditioned predictive coding (GCPC).

This paper had mixed reviews, with valid concerns and points of praise. And after reading the paper and the reviews, I lean accept.  

This paper's main strength is in the empirical analysis, the various comparison of various masking strategies and representations towards useful to control, and comparison to relevant recent baselines, which I believe is novel and of value to the NeurIPS community. The authors have diligently responded to the many valid reviewer concerns, which have mainly been about clarity and additional experiments. I highly recommend the authors to follow through on these promised changes and include them in their camera ready, including Reviewer tnif's concerns to downplay claims of method superiority when compared performance is within error bounds. Without all these changes I would have leaned reject, so since the changes are critical towards my flip to an "accept" rating, and I will check for these changes in the camera ready copy, so please include them!

PS: line 73 typo: “reinformenct”
Decision: Accept (poster)