Waypoint Transformer: Reinforcement Learning via Supervised Learning with Intermediate Targets
Authors: Anirudhan Badrinath
Yannis Flet-Berliac
Allen Nie
Emma Brunskill
Venue: NeurIPS 2023 poster (2023)
Keywords: offline reinforcement learning
reinforcement learning via supervised learning
behavioral cloning
TL;DR: Using intermediate target goals and rewards as conditioning variables with only behavioral cloning objectives and minimal hyperparameter tuning, we can achieve state-of-the-art performance in offline reinforcement learning.

Abstract: Despite the recent advancements in offline reinforcement learning via supervised learning (RvS) and the success of the decision transformer (DT) architecture in various domains, DTs have fallen short in several challenging benchmarks. The root cause of this underperformance lies in their inability to seamlessly connect segments of suboptimal trajectories. To overcome this limitation, we present a novel approach to enhance RvS methods by integrating intermediate targets. We introduce the Waypoint Transformer (WT), using an architecture that builds upon the DT framework and  conditioned on automatically-generated waypoints. The results show a significant increase in the final return compared to existing RvS methods, with performance on par or greater than existing state-of-the-art temporal difference learning-based methods. Additionally, the performance and stability improvements are largest in the most challenging environments and data configurations, including AntMaze Large Play/Diverse and Kitchen Mixed/Partial.

Waypoint Transformer: Reinforcement Learning via Supervised Learning with Intermediate Targets
Anirudhan Badrinath
Yannis Flet-Berliae
Allen Nie
Emma Brunskill
Department of Computer Science Stanford University fabadrina, yfletberliac, anie, ebrum)@cs.stanford.eee. stanford.
Abstract
Despite the recent advancements in offline reinforcement learning via supervised learning (RvS) and the success of the decision transformer (DT) architecture various domains, DTs have fallen short in several challenging benchmarks. The root cause of this underperformance lies in their inability to seamlessly connect segments of suboptimal trajectories. To overcome this limitation, present novel approach to enhance RvS methods by integrating intermediate targets. We introduce the Waypoint Transformer (WT), using architecture that builds upon the DT framework and conditioned on auttomaticclly-genraate show significant increase in the final return compared to existing RvS methods, with performance on par or greater than existing popular temporal difference learning- based methods. Additionally, the performance and stability improvements are largest in the most challenging environments and data configurations, including AntMaze Large Play/Diverse and Kitchen Mixed/Partial.
waypoints. The results
1 Introduction
Traditionally, offline reinforcement learning (RL) methods that compete with state-of-the-art (SOTA) algorithms have relied objectives encouraging pessimism in combination with value-based methods. Notable examples this approach include Batch Conservative Q-Learning (BCQ), Conservative
Q-Learning (CQL), and Pessimistic Q-Learning (PQL) (Fujimoto al. 2019 Kumar al. 2020
Liu et al.) 2020]. However, these methods can be challenging to train and often require intricate hyperparameter tuning and various tricks to ensure stability and optimal performance across tasks.
Reinforcement learning via supervised learning (RvS) has emerged as a simpler alternative to traditional offline RL methods [Emmons et al.) 2021]. RvS approaches are based on behavioral cloning (BC), either conditional or non-conditional, to train a policy. Importantly, these methods eliminate the need for any -iifeeee-difference (TD) learning, such as fitted value or action-value functions. This results
simpler algorithmic framework based on supervised learning, allowing for progress in offline RL to build upon work in supervised learning. There are several successful applications of RyS methods, including methods conditioned on goals and returns (Kumar et al.
2019 Janner et al. 2021 Ding et al., 2019 Chen al. 2021 Emmons et al. 2021].
However, RvS methods have typically struggled in tasks where seamlessly connecting (or "stitching")
appropriate segments of suboptimal training trajectories is critical for success (Kumar et al., 2022 For example, when tasked with reaching specific locations the AntMaze maze navigation environment or completing series of tasks in the FrankaKitchen environment, RvS methods typically perform significantly worse than learning methods such as Implicit Q-Learning [Fu al, 2020 Kostrikov et al. 2021
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

In this study, we leverage the transformer architecture (Vaswani et al. 2017] to construct an RvS method. As introduced by Chen et al. [02T]. the decision transformer (DT) can perform conditional behavioral cloning in the context of offlin RI. However, similar to other RvS methods, DT proves inferior in performance across popular Gym-MuloCo benchmarks compared to other value-based offline RL methods, with 15% relative reduction in average return and lowered stability (Table
To tackle these limitations of existing RvS methods, we introduce a waypoint generation technique that produces intermediate goals and more stable, proxy rewards, which serve as guidance to steer a policy to desirable outcomes. By conditioning a transformer- -bssed RvS method on these generated targets, we obtain a trained policy that learns to follow them, leading to improved performance and stability compared to prior offline RL methods. The highlights of our proposed approach are as follows:
We propose a novel RvS method, Waypoint Transformer, using waypoint generation net- works and establish new state-of-the-art performance, in challenging tasks such as AntMaze Large and Kitchen Partial/Mixed (Fu et al.) 2020) (Table On tasks from Gym-MuJoCo, our method rivals the performance of TD learning-based methods such as Implicit Q-
Learning and Conservative Q-Learning [Kostrikov et al., 2021 Kumar et al. 2020 and
improve over existing RvS methods.
We motivate the benefit of conditioning RvS on intermediate targets using chain-MDP example and an empirical analysis of maze navigation tasks. By providing such additional guidance suboptimal datasets, show that policy optimized with behavioral cloning objective chooses more optimal actions compared to conditioning on fixed targets (as in Chen et al. [021]] Emmons et al. [) facilitating improved stitching capability.
Our work also provides practical insights for improving RvS, such as significantly reducing training time, solving the hyperparameter tuning challenge in RvS posed by Emmons et al. [202I), and notably improved stability in performance across runs.
2
Related Work
Many recent offline RL methods have used fitted value or action-value functions Liu et al. 2020
Fujimoto al., 2019 Kostrikov et al.
202 Kumar et al., 2020 Kidambi et al. 2020
Lyu al.
2022 or model-based approaches leveraging estimation of dynamics Kidambi et al. 2020
Yu et al.
2020
Argenson and Dulac-Arnold 2020 Shen et al.) 202T Rigter et al. 2022] Zhan et 2021].
RvS, as introduced in Emmons et al. [022]] avoids fitting value functions and instead leverages
behavioral cloning. In many RvS-style methods, the conditioning variable for the policy is based
on the return (Kumar et al. [2019] Srivastava et al. 2019 [Schmichuber) [019]] Chen et al.] 2021]
but other methods use goal-conditioning [Nair et al. 20188] EEmmoss al.. [02 Ding al. 2019
Ghosh et al. 2019] or leverage inverse RL [Eysenbach et al. 2020]. Recent work by Brandionbrener
al. 2022 has explored the limitations of reward conditioning in RvS. In this study, we consider
both reward and goal-conditioning.
Transformers have demonstrated the ability to generalize to vast array of tasks, such language modeling, image generation, and representation learning [Vaswani et al.] [2017] Devlin et al. 2018
2017]Devlin
He et al. 2022 Parmar et al.) 2018). In the context of offline RL, decision transformers (DT) leverage a causal transformer architecture to fit a reward-conditioned policy (Chen et al. [021]. Similarly, (Jnner et al. 2021 frame offline RL sequence modeling problem and introduce the Trajectory Transformer, a model-based offline RL. approach that uses the transformer architecture.
Algorithms building upon the DT, such online DT Zhhnng et al. 2022 prompt DT [Xu et al. 2022] and Q-Learning DT (Yamagata et al. 2022], have extended the scope of DT's usage. Furuta al. 2021] introduce a framework for hindsight information matching algorithms to unify several hindsight- based algorithms, such as Hindsight Experience Replay [Andrychowicz et al. 2017]. DT, TT, and our proposed method.
Some critical issues with DT unresolved by existing work are (a) its instability (i.e., large variability across initialization seeds) for some tasks in the offline setting (Table and (b) its relatively poor performance on some tasks due to an inability to stitch segments of suboptimal trajectories (Kumar) et al. 2022 in such settings, RyS methods are outperformed by value-bascd methods such as

---

Implicit Q-Learning (IQL) [Kostrikov et al., 2021] and Conservative Q-Learning (CQL) (Kumar al. 2020]. We address both these concerns with our proposed upproach, demonstrating notably improved performance and reduced variance across seeds for tasks compared to DT and prior Rv$ methods (
One of the areas of further research in RvS, per Emmons et al. 2021]. is to address the complex and unreliable process of tuning hyperparameters, as studied in Zhang and Jiang [202]] and Nie
et al. 2022]. We demonstrate that our method displays low sensitivity to changes in hyperparameters compared to Emmons et al. [022] (Table Further, all experiments involving our proposed method use the same set of hyperparameters in achieving SOTA performance across many tasks (Table I
Preliminaries
We assume that there exists agent interacting with Markov decision process (MDP) with states 8t S and actions a, E A with unknown transition dynamics p(5++1 82, a2) and initial state distribution p(su). The agent chooses action sampled from transformer policy a To(a st-k..; kt-k..),, parameterized by and conditioned on the known history states st-k.t and a conditioning variable -.-k..e. Compared to the standard RL framework, where the policy is modeled by π (a, 8t), we leverage policy that considers past states within a fixed context window k.
The conditioning variable O; is specification of goal or reward based target outcome At training time, w is sampled from the data, as presented in Emmons al.) [021]] At test time, we assume that can either generate or are provided global goal information for goal-conditioned tasks. For reward-conditiones tasks, we specify a target return w, following Chen et al. [202I].
To train our RvS-baser algorithm an offline dataset D consisting of trajectories with conditioning variable Ee,, we compute the output of an autoregressive transformer model based on past and current states and conditioning variable provided in each trajectory. Using negative log-likelin likelihood loss, we use gradient descent update the policy To. This procedure is summarized Algorithm
Algorithm Training algorithm for transformer-basee policy trained on offline dataset D.
Input: Training dataset D = {TI, 73 J of training trajectories. for each r (50, do, Po, $1, 01, 4 II.... in D do
Compute Th St.k.. & - for all
Calculate LT(T) E, log (ag 8t st-k..t, kt-k.1)
Backpropagate gradients w.r.t. LT(T) to update model parameters
end for
Waypoint Generation
In this section, we propose using intermediate targets (or waypoints) as conditioning variables as an alternative to fixed targets, proposed Emmmos et al. [202]] Below, we motivate the necessity for waypoints in RvS and present a practical technique to generate waypoints that can be used for goal-conditioned (Section 4.2) and reward-conditioned tasks (Section 4.3) respectively.
4.1 Illustrative Example
To motivate the benefits of using waypoints, consider horiite, determinis- tic MDP with H+1 1 states and two possible actions at non-terminal states. A graphi- cal representation the MDP is shown Figure [ For this scenario, we consider the goal-conditioned setting where the tar- get goal state during train and test time is - (H), and the episode terminates
(1))
a
(()))
(
a
(2)
s(Ch)
(2)
(
((H)
s
Figure 1: Chain MDP to motivate the benefit of inter- mediate goals for conditional BC-based policy training.
reach w.
In offline RI, the data often suboptimal for achieving the desired goal during testing. In this example, suppose we have access to a dataset D that contains an infinite number of trajectories
3

---

collected by random behavioral policy where To(a, = e suboptimal with respect to reaching in the least number of timesteps; in expectation, it takes rーx timesteps to reach g( instead of H (optimal) since the agent "stalls" the current state with probability À and moves to the next state with probability 1 X.
st) > >>>>> 0 for all 8t. Clearly, me
Consider global goal-conditioned policy (a(a, s1,w) that is optimized using behavioral cloning objective on DD Clearly, the optimal policy Tद Tz(a; (a; (8,,w) To(a, / 82) Vs, since w (HH) is constant. Hence, the global goal-conditioned policy = is suboptimal as the behavioral policy me
Instead, suppose that condition policy (ar St, 2r) on an intermediate goal state 2, 8 for some chosen K < 1-A (expected timesteps before Th executes a2), optimized using behavioral cloning objective on D. For simplicity, suppose our target intermediate goal state 4 for some current state 8; = s(h) is simply the next state D, = g(++1), Based on data D from Tb. the probability of taking action (2)) conditioned on the chosen 4, and 8, is estimated as:
s(h+1)] = Prz, (a+ = al 88+K = gl++1) 8t = s(b]]
Pr lat = (a) |st =((),, S
Pr.., |SL+K s(++1) St s(h)
(1 A)AR-1
(K) [1 - AjAK-1]
(1) K
Hence, for the optimal intermediate goal-conditioned policy Fw trained on D, the probability of choosing the optimal action (2)) is:
8t s(h), , s(h+1)) 1
Tw (a+ = (2)
K
Since (a( (2))
8: s (h) w)
X and choose K such that K > 1 A, conclude:
Tw (a: = (2) $2. 21) > (a: (2) s,,w).
The complete derivation is presented in Appendix Based on this example, conditioning the actions on reaching desirable intermediate state is more likely to result in taking the optimal action compared to a global goal-conditioned policy. Effectively, the conditioning acts as a "guide" for the policy, directing it toward desirable intermediate targets in order to reach the global goal.
4.2 Intermediate Goal Generation for Spatial Compositionality
In this section, we address RvS's inability to "stitch" subsequences of suboptimal trajectories in order to achieve optimal behaviour, based analyses in Kumar et al.) [2022). In that pursuit, we introduce a technique to generate effective intermediate targets to better facilitate stitching and to guide the policy towards desirable outcomes, focusing primarily on goal-conditioned tasks.
Critically, the ability to stitch requires considering experiences that are more relevant to achieving ap- propriate short-term goals before reaching the global goal. To illustrate this, we show maze navigation task from the AntMaze Large environment in Figure 2] where the evaluation objective is to reach target location from the start location [Fu al.) 2020).
start
stitching region
target region
region
T
Analyzing the training trajectories that pass through either the start location (blue) or target location (red), less than 5% of trajectories extend beyond the stitch- ing region into the other region, i.c., the target or start regions respectively. Since trajectories seldom pass through both the start and target regions, the policy must "stitch" " together subsequences from the blue and red trajectories within the stitching region, where the trajectories overlap the most. By providing inter- mediate targets within this region, rather than condi- tioning solely on the global goal, we can guide the policy to connect the relevant subsequences needed to reach the target effectively.
task to navigate from the start location (circle) to the target location (star). Blue and red colored lines are training trajectories passing through the start or end locations respectively.
Figure 2: mariazzelarge-pllee

---

To obtain effective intermediate targets, we propose the goal waypoint network, explicitly designed to generate short-term, intermediate goals. Similar to the illustrative example in Section 4.1 the purpose of these intermediate targets is to guide the policy network To towards states that lead to the desired global goal by facilitating stitching of relevant subsequences.
To that end, we represent the goal waypoint network We, parameterized by o, as a neural network that makes approximate K-step predictions of future observations conditioned on the current state, St, and the target goal, w. Formally, we attempt to minimize the objective in Equation I across the same offline dataset D, where Lg is a mean- saaaan--urredddeee error (MSE) loss for continuous state spaces:
arg min Σ Lo (S w), St+K).
While our approach to intermediate target generation seems simple in relation to the complex problem of modeling both the behavioral policy and transition dynamics, our goal is to provide approximate short-term goals to facilitate the downstream task of reaching the global goal w, rather than achieving perfect predictions of future states under the behavioral policy.
4.3 Proxy Reward Generation for Bias-Variance Reduction
In this section, we address the high bias and variance of conditioning variables used by prior RyS methods in reward-conditioned tasks, such as |Emmons et al.] (202T) and Chen et al. (202T). Analogously to Section 4.2 (i.e.. for goal-conditioned tasks), we propose a technique to generate intermediate reward targets for reward-conditioned tasks to mitigate these issues.
Existing methods rely on either an initial cumulative reward-to-go (desired return) or an average reward-to-go target, denoted as w. Importantly, the former is updated using rewards obtained during the rollout, while the latter remains constant over time (Emmons et al., 2021 Chen et al. 2021]. However, using these conditioning variables during evaluation gives rise to two main issues: (a) the Monte Carlo estimate of return used to compute the cumulative reward-to-go exhibits high variance and (b) the constant average reward-to-go target introduces high bias over time. Based on our analyses of the bias and variance of these approaches in Appendix C, we observe that these issues contribute to decreased performance and stability across runs when evaluating Rvš methods.
Although a potential approach to mitigate these issues to leverage TD learning, such as the Q-Learning Transformer |Yamagata et al. 2022], we restrict our work to RyS methods utilizing behavioral cloning objectives due to the inherent complexity of training value-based methods. To address the aforementioned concerns, we introduce a reward waypoint network denoted as Wa. parameterized by Φ. This network predicts the average and cumulative reward-to-go (ARTG, CRTG) conditioned on the return, and current state, 8t, using offline data D. To optimize this network, we minimize the objective shown in Equation 2
using an MSE loss:
E(F.EEZ Elarn' - W))
arg min
(2)
FED
By modeling both ARTG and CRTG, we address the high bias of constant ARTG target and reduce the variance associated with Monte Carlo estimates for CRTG. The construction of the reward waypoint network is similar in motivation and the prediction task to baseline network, used to mitigate high variance in methods like REINFORCE [Suttonn and Barto 1999]. However, the distinguishing feature of our reward waypoint network lies in its conditioning on the return, which allows for favorable performance even on suboptimal offline datasets.
5 Waypoint Transformer
4
a
We propose the waypoint transformer (WT), transformer-based offline RL method that lever- ages the proposed waypoint network Wa and a GPT-2 architecture based on multi-head at- tention [Radford et al. [019]. The WT pol- icy na is conditioned on past states 8t-k.. and waypoints (either generated goals or rewards) k Wa(
waypoint transformer
9
W
W
W
Waypoint Transformer architecture, where Φ. Wo(s,, w) represents the output of the goal or reward waypoint network.
Figure 3:
with a context win-
dow of size k, as shown in Figure 3
5

---

Table 1: Normalized scores and training time per task Gym-MuloCo, AntMaze, and Kitchen tasks, where bold highlighting indicates SOTA performance, defined by a method's average performance being contained within the interval of the method with highest average.
TDX BC
Oncecppi
CQL.
ReS-ERG
191
OUT
WT (hurs)
halfcherihh medium-2
hoppr-mofumm2?
63.
83.7
$1.
71.5+00
45.5
18.+00..
354+11
609 183
97.
64
+3,5+12.
13.9
81. 5.5
12.0
53
738
58.97:1
halichectah mediam expent-12
90.7 4.4.
13
21
14.9
U.5
mediuus exporis22
98.0
103.3
1054 1.88
91.5
101.7 16.5
wilker25 meSium
1101
113.0
108.8
= 1.0
109.0
4.99
753
uumiaae unnuee 72
75.5
64.3
14.01
W7.5 2.6
35.5
02.55
05.4 4.5
53.6
6LU T.l
untmmaze umaze divense- 12
71.4
60.7
84.0
62.2 =: 13.8
45.6
50.2
60.9
2: 2.5
42.2 5.4
71.5 7.6
unmaae medium play v2
10.6
0.5
61.2
71.3
0.0
5.4
58.1 12.7
0.0 0.0
62.8 5.8
anmaze medium diverse V2
3.0
0.0
53.7
7000 10.9
0.0
9.8
67.3 8.0
0.0 0.0
66.7
aumzze large play x2
0.2
0.0
15.8
39.6
0.0
0.0
50.4 10.5
0.0 0.0
72.5
onmmze ummzze-lrrge-ee brge-divem---2
00
0.0
14.9
47.5
0.0
6.0
16.9 4.8
0.0 0.0
72.0
mnsiiaee org V2
273
20.9
5016
03:0= K33
16.7
22.5
53.5
2: 7.2
16,0 2.1
05.4 4.9
kitchen
83.8
62.5
65.8
4.0
46.5 3.0
19.2 4.6
kitchen panial-s0
69.8
46.3
38.0
66.0
$1.4 4. 19.5
638 3.
kithhn-misso---0
51.0
51.0
40.0
60.3 9.4
28.8
70.9 2.1
5333
5500
14.5.12
Svenge
61.7
68.3
40.1
50.5
80.
3.7
45.8 7.3
71.4 2.8
Irsiting time (min)
30
80
20
10
50
150
20
For goal-conditioned and reward-conditioned tasks, we train the goal and reward waypoint network We respectively offline dataset DD independently of the policy. To train the WT policy, we use Algorithm| [ to iteratively optimize its parameters 9. During this process, the trained weights & of We are frozen to ensure the interpretability of the waypoint network's 's generated goal and reward waypoints. To further simplify the design and improve computational efficiency, the WT is not conditioned past actions Qt-k..t (i.e., unlike the DT) and we concatenate 4, with s, to produce one token per timestep t instead of multiple tokens as proposed in Chen et al. 2021
6 Experiments
We present series of evaluations of WT across tasks involving reward and goal-conditioning, with comparisons to prior offline RL methods. For this, we leverage D4RL, an open-source benchmark for offline RL, consisting of varying datasets for tasks from Gym-MuJoCo, AntMaze, and FrankaKitchen Fu et al. 2020).
Tasks in the AntMaze and FrankaKitchen environments have presented challenge for offline RL methods as they contain little to no optimal trajectories and perform eritical evaluations of a model's stitching ability (Fu et al., 2020). Specifically, in FrankaKitchen, the aim is to interact with set of kitchen items to reach a target configuration, but the partial and mixed offline datasets consist of suboptimal, undirected data, where the demonstrations are unrelated to the target configuration. Similarly, AntMaze is a maze navigation environment with sparse rewards, where the play and diverse datasets contain target locations unaligned with the evaluation task. For our experiments on these environments, we use coal-conditionin on the target goal state (i.e., w = Starget), constructing intermediate targets with the goal waypoint network.
Gym-MuloCo serves as a popular benchmark for prior work in offline RI, consisting of environments such as 2D, Hal Cheetahh and Hopper. Importantly, we evaluate across offline datasets with varying degrees of optimality by considering the medium, medium-replle datasets [Fu et al.) 2 For these tasks, use reward-conditioning given target return, construct- ing intermediate reward targets using the reward waypoint network; as noted in Emmons et al. 2021]. we find that the notion of goals in undirected locomotion tasks is ill-defined.
and medium medium-expert
Across all environments and tasks, we use the same set of hyperparameters, as reported in Appendix B. To measure the stability (i.e., variability) in our method across random initializations, we run each experiment across seeds and report the mean and standard deviation.
6.1 Comparing WT with Prior Methods
To evaluate the performance of WT, we perform comparisons with prior offline RL. methods, including conditional BC methods such as DT and RyS-R/G; value-based methods such as Onestep RL

---

Brandfonbremerr et al. 2021], TD3 BC (Fujimoto and Gu 2021]. CQL, and IQL; and standard BC baselines. For all methods except DT, we use reported results from Emmons et [2021] and Kostrikov et al. 2021]. J. We evaluate DT using the official implementation provided by Chen et al.
2021 across random initializations, though we are unable reproduce some of their results.
Table I shows the results of our comparisons to prior methods. Aggregated across all tasks, WT (71.4 + 2.8) improves upon the next best method, IQL (68.3 + 6.9), with respect average normalized score and achieves equal-best runtime. In terms of variability across seeds, there is a notable reduction compared to IQL and most other methods.
In the most challenging tasks requiring stitching, our method demonstrates performance far exceeding the next best method, IQL. On the AntMaze Large datasets, WT demonstrates substantial relative percentage improvement of 83.1 13 (play) and 51 31.6% (diverse). On Kitchen Partial and Mixed, the improvement is 37.8% and 39.0% respectively. WT's standard deviation across seeds is reduced factor of more than 2x compared to IQL for these tasks.
Similarly, reward-conditioned tasks with large performance gaps between BC and value-based methods such as hopper-medt medimm- un-replay-v2 WT demonstrates increased average performance by 105.3% compared to DT and 21.0% compared to RyS-R, with standard deviation reduced by factor of 10.0x and 5.3x respectively.
6.2 Utility of Waypoint Networks
To analyze the utility and behavior of
waypoint networks, we qualitatively eval-
uate agent's performance across roll- outs of trained transformer policies on antmaze- Larnuz-e-arge--la----a
For this anal- ysis, we consider a WT policy (using a goal waypoint network with K 30) and a global coal-conditioned transformer policy (i.c.. no intermediate goals). Across both models, the architecture and hyperparame- ters for training are identical.
Proportion of Complated Ruma 6.2 nA 0.6 B.8 1.0
No Waypoints
WT
The ant's locations across 100 rollouts WT policy (Figure 4a) and global goal- conditioned transformer policy (Figure4p)) demonstrate that WT shows notably higher ability and consistency in reaching the goal location. Specifically, without interme- diate goals, the ant occasionally turns in the wrong direction and demonstrates a lesser ability to successfully complete a turn based on the reduction of density each turn (Figure 4p). Consequently. the WT achieves more than twice the evaluation return (72.5 2.8) compared to the global goal- conditioned policy (33.0 = 10.3) and completes the task more quickly on average (Figure (t).
4:00
1000
Timestep (t)
Figure Shows the ant's location across 100 rollouts of (a) WT policy and (b) global goal-conditioned transformer policy; (c) generated intermediate goals by the waypoint network Wg. (d) the proportion of all successful runs completed by timestep f.
Based on Figure 4p. we observe that the goal waypoint network provides goals that correspond the paths traversed in Figure 4 for the WT policy. This shows that the waypoint network successfully guides the model toward the target location, addressing the stitching problem proposed in Figure While the global goal-conditioned policy is successful in passing beyond the stitching region into the target region only 45% of the rollouts, accounting for 82% of its failures reach the target, WT successful in this respect for 87% of rollouts.
6.3 Ablation Studies
Goal-Conditioned Tasks On goal-conditioned tasks, we examine the behavior of the goal waypoint network as relates to the performance of the policy at test time by ablating aspects its configuration
7

---

and training. For this analysis, we consider larpazze-arrg-ppaa---aaa a challenging task that
critically evaluates the stitching capability of offline RL techniques.
To understand the effect of the configura- tion of the goal waypoint network test performance, we ablate two variables rel-
70
Normalbed Score Scare
Normalized Seare
80
evant generating effective intermediate goals: the temporal proximity of intermedi- ate goals (K) and the validation loss of the goal waypoint network.
50
50
40
40
30
30
20
0
20
40
G0
80
0.4
0.6
0.8
1.0
1.2
The normalized score attained by the agent is shown as function of K and the vali- dation loss of the goal waypoint network in Figure [ For this environment and dataset, an ideal choice for K is around 30 timesteps. For all nonzero K, the per- formance is reduced
Figure 5: Normalized score attained by WT antmaze-largg-plae plage--2 based on varying left: the temporal proximity of generated goals, K, and right: goal waypoint network RMSE on a held-out dataset.
reasonably consistent rate either side of K 30. Importantly, when K = (i.e., no intermediate goals), there is a notable reduction in performance compared to all other choices of K: compared to the optimal K 30, the score is reduced by factor of 2.2x.
In Figure (right). the normalized score shows the negligible change for values of held-out RMSE between 0.4 and 0.6, corresponding to least 1,000 gradient steps roughly 30 sec of training, with a sharper decrease henceforth. As the RMSE increases to over 1, we observe a relative plateau in performance near an average normalized score of 35-45, roughly corresponding to performance without using& waypoint network (i.e., K = 0 in Figure (left).
Additionally, perform comparisons between the goal waypoint network and manually constructed waypoints as intermediate targets for WT, for which the methodology and results are shown in Appendix D. Based on that analysis, show that manual waypoints statistically significantly improve upon no waypoints (44.5 2.8 vs. 33.0 = 10.3), but they remain significantly worse than generated waypoints.
Reward-Condittoned Tasks On reward- conditioned tasks, we ablate the choice of different reward-conditioning techniques.
240
(CXIG)
Propartion of Ruuns
0.8
Standard Deviation
200
160
Specifically, we examine the performance of WT and variance of the reward way-
0.6
120
0.4
point network in comparison to CRTG up- dated using the rewards obtained dur- ing rollouts and a static ARTG (i.e., as
40
0
20
40
60
80
100
400
800
Normalized Score
Timestep
done in Chen et al. [2021] and Em- mons et al. [2021]. We consider the hopper-mediuw remlayrepllya-ee
Figure 6: Comparison of different reward-conditioning methods on hopper-medinee mwwiim-replayyeeeeeeeeeee Left: Per- formance profiles for transformers using ARTG, CRTG, and WT across 5 random seeds. Right: Standard devia- tion in CRTG inputted to the model when updated with at- tained rewards (CRTG, = w w-E, 7²rr)) and using predic- tions from the reward waypoint network (W(st, when average return is approximately held constant.
task for this analysis as there is (a) a large performance gap between RvS and value-based meth- ods, and (b) high instability across seeds for RvS methods (e.g., DT) as shown in Ta- [ For all examined reward-condittoned techniques, the transformer architecture and training procedure are identical, and the target (normalized) return is 95, corresponding to SOTA performance.
To examine the distribution of normalized scores across different seeds produced by each of the de- scribed reward-condition techniques, construct performance profiles, displaying the proportion of runs greater than certain normalized score AAgaral et al.) 2021]. As shown Figure 6 (left), WT demonstrates increased performance and stability across random initializations compared to the remaining conditionn techniques.
Additionally, we perform analysis to determine whether using a reward waypoint network to predict the CRTG as opposed to updating the CRTG using attained rewards as in Chen al. [00T]] affects the variability of the conditioning variable passed to the policy network (i.c., not of the

---

performance as that is examined in Figure Importantly, to account for performance differences between the policies trained with either method that may influence the variability of the attained CRTG, we sample subset of runs for both methods such that the average performance is constant. Based on Figure (right), it clear that as a function of the timestep, when accounting for difference in average performance, the standard deviation in the CRTG predicted by WT grows at slower rate compared to updating CRTG with attained rewards.
Transformer Configuration Based the work in Emmons et al. 2021 balance between expressiveness and regularization to maximize policy performance. We ablate the probability of node dropout Pdrop and the number of transformer layers L. To further examine this balance, experi- ment with conditioning on past actions - similarly to the DT, to characterize its impact on performance and computational efficiency. In this section, we consider antmaze-large law---aaage----eee----- ments.
and kitchen-menn -iieen-miiedvv) one task from each category of environ-
Based on Table 2 we observe that the sensitivity to the various ablated hyperparameters relatively low in terms of performance, and removing action conditioning results in reduced training time and increased performance, perhaps due to reduced distribution shift at evaluation. In context of prior RvS work where dropout (Parop 0.1) decreased performance compared to dropout by 1.5-3x AntMaze, the largest decrease average performance on WT only by a factor of 1.1x [Emmoon)
al. 2021].
Table 2: Ablation of transformer configuration showing normalized score on MuJoCo (v2), AntMaze (v2) and Kitchen (v0), including dropout (Prrop), transformer layers (L), and action conditioning (az), where bolded hyperparameters (c.g., 0.150) are used for final models and bolded scores are optimal.
Pdrop
hopper-mediie metiium--mmmm--- replaay
antmaze-large play
kitchen mixed
Average
0.000
75.5 + 8.3
68.3 + 5,9
72.9 + 0.5
72.2 + 4.9
0.075
89.8 2.8
70.8 4.5
71.8 + 1.2
77.5 = 2.8
0.150
88.9 + 2.4
72.5 + 2.8
70.9 - 2.1
7
0.225
75.7 9.4
72.2 2.7
71.2 + 1.0
73.0 4.4
0.300
74.7 = 10.2
73.5 + 2.5
69.2 + 2.0
72.5 + 4.9
0.600
58.4 7.5
73.8 5.2
66.5 + 2.7
66.2 5.1
L hopper-medium-replayy
antmaze-large-play
kitchen-mixed
Average
82.1 8.8
72.1 + 5.7
71.6 1.6
75.3 + 5.4
88.9 1 2.4
72.5 1 2.8
70.9 1 2.1
77.4 1 2.4
3
89.9 + 1.6
71.8 + 3.0
70.3 +
77.3 + 2.2
91.1 2.8
65.8 + 3.8
69.7 + 1.0
75.5 + 2.5
88.8 4.5
66.7 + 4.7
70.0 0.8
75.2 + 3.3
at
hopper-meddiumrrepl.aa
antmaze-large-playy
kitchen-mixed
Average
Yes
76.9 1 9.0
66.5 +: 5.6
65.2 1: 2.8
69.5 + 5.8
No
88.9 + 2.4
72.5 2.8
70.9 2.1
77.4 2.4
Discussion
In this study, we address the issues with existing conditioning techniques used in RvS, such as the "stitching" problem associated with global goals and the high bias and variance of reward-to-go targets, through the automatic generation of intermediate targets. Based on empirical evaluations, we demonstrate significantly improved performance and stability compared to existing RvS methods, often on par with or outperforming TD learning methods. Especially on challenging tasks with suboptimal dataset composition, such AntMaze Large and Kitchen Partial/Mixed, the guidance pro- vided by the waypoint network through intermediate targets (e.g., as shown in Figure significantly improves upon existing state-of-the- performance.

---

We believe that this work can present pathway forward to developing practical offline RL methods leveraging the simplicity of RyS and exploring more effective conditioning techniques, formalized by Jy|mmons et al. [02T]. In addition to state-of-the-an performance, we demonstrate several desirable practical qualities of the WT: it is less sensitive to changes in hyperparameters, significantly faster to train than prior RvS work, and more consistent across initialization seeds.
However, despite improvements across challenging tasks, WT's margin of improvement on AntMaze U-Maze and Kitchen Complete (i.e., easier tasks) is lower: its normalized scores are more comparable to DT and other RyS methods. We believe this is likely due to stitching being less necessary in such tasks compared to difficult tasks, rendering the impact of the waypoint network negligible. To further characterize the performance of the waypoint networks and WT on such tasks an interesting direction for future work. In addition, there are several limitations inherited by the usage of the RvS framework, such as manual tuning of the target return at test time for ceward-conditioned tasks using a grid search, issues with stochasticity, and an inability to learn from data with multimodal outcomes.
8 Conclusion
We propose method for reinforcement learning via supervised learning. Waypoint Transformer, conditioned on generated intermediate targets for reward and goal-conditioned tasks. We show that RvS with waypoints significantly surpasses existing RyS methods and achieves on par with or surpasses popular state-of-the-atr methods across a wide range of tasks from Gym-MuJoCo, AntMaze, and Kitchen. With improved stability across runs and competitive computational efficiency, we believe that method advances the performance and applicability of RyS within the context of offline RL.
Acknowledgments and Disclosure of Funding
This work was supported in part by NSF grant #21 12926...
We thank Scott Emmons and Ilya Kostrikov for their discussions on and contributions to providing results for prior offline RL methods.
References
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Marc Bellemare. Deep reinforcement learning at the edge of the statistical precipice. Advances in neural information processing systems, 34:29304-29322 2021.
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojeiech Zaremba. Hindsight experience replay. Advances in neural information processing systems, 30, 2017.
Arthur Argenson and Gabriel Dulac-Amold. Model-based offline planning. arXiv:2008.00 05556, 2020.
arXiv preprint
David Brandfonbrener, Will Whitney, Rajesh Ranganath, and Joan Bruna. Offline rl without off off-policy evaluation. Advances in neural information processing systems, 34:4933 4946, 2021.
David Brandfonbrener, Alberto Bietti, Jacob Buckman, Romain Laroche, and Joan Bruna. When does return- cetuu--condiiooned supervised learning work for offline reinforcement learning? arXiv preprint 1.Xiv:2206600109) 2022.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling, Advances in neural information processing systems, 50041-10841977 2021.
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 2018.
Yiming Ding, Carlos Florensa, Pieter Abbeel, and Mariano Phielipp. Goal-conditioned imitation learning. Advances in neural information processing systems, 32, 2019.
10

---

Scott Emmons, Benjamin Eysenbach, Ilya Kostrikov, and Sergey Levine. Rvs: What is essential for offline rl via supervised learning? arXiv preprint arXiv:2112.1075/ 2021.
Ben Eysenbach, Xinyang Geng, Sergey Levine, and Russ R Salakhutdinov. Rewriting history with inverse rl: Hindsight inference for policy improvement. Advances in neural information processing systems, 11114483-14999 2020.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprini arXiv:2004.07219, 2020.
Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34:20132-20111 2021.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. International conference on machine learning, pages 2052-2062. PMLR, 2019.
Hiroki Furuta, Yutaka Matsuo, and Shixiang Shane Gu. Generalized decision transformer for offline hindsight information matching. arXiv preprint arXiv:2111.10366 2021.
Dibya Ghosh, Abhishek Gupta, Ashwin Reddy, Justin Fu, Coline Devin, Benjamin Eysenbach, and Sergey Levine. Learning to reach goals via iterated supervised learning. arXiv preprint 1rXiv:1912.06088 2019.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16000-16009, 2022.
Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. Advances in neural information processing systems, 14:1273-1286, 2021.
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model- based offline reinforcement learning. Advances in neural information processing systems, 33: 21810-21823, 2020.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. arXiv preprint 1rXiv:2110.06/063 2021.
Aviral Kumar, Xue Bin Peng, and Sergey Levine Reward-conditioeed policies. arXiv preprint arXiv:1912.13465, 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1 179-1191, 2020.
Aviral Kumar, Joey Hong, Anikait Singh, and Sergey Levine. When should we prefer offline reinforcement learning over behavioral cloning? arXiv preprint arXiv:2204.056788 2022.
Hao Liu and Pieter Abbeel. Emergent agentic transformer from chain of hindsight experience. arXiv preprint arXiv:2305. 2023.
Yao Liu, Adith Swaminathan, Alekh Agarwal, and Emma Brunskill. Provably good batch off-policy
reinforcement learning without great exploration. Advances in neural information processing systems. 33:1264-1274, 2020.
Jiafei Lyu, Xiaoteng Ma, Xiu Li, and Zongqing Lu. Mildly conservative q-learning for offline reinforcement learning. arXiv preprint arXiv:2206.0474 2022.
Ashvin V Nair, Vitchyr Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine. Visual reinforcement learning with imagined goals. Advances in neural information processing systems, 31, 2018
Allen Nie, Yannis Flet-Berliac, Deon Jordan, William Steenbergen, and Emma Brunskill. Data- efficient pipeline for offline reinforcement learning with limited data, Advances Neural Informa- fion Processing Systems, 35:14810-1482 2022.

---

Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In International conference on machine learning, pages 4055-4066 PMLR, 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Marc Rigter, Bruno Lacerda, and Nick Hawes. Rambo-rl: Robust adversarial model-based offline reinforcement learning. arXiv preprint Xrxiv:2204. 12581, 2022.
Juergen Schmidhuber. Reinforcement learning upside down: Don't predict rewards-just map them to 2019.
actions. arXiv preprint arXiv:
Jian Shen, Mingcheng Chen, Zhicheng Zhang. Zhengyu Yang, Weinan Zhang, and Yong Yu. Model- based offline policy optimization with distribution correcting regularization. In Machine Learning and Knowledge Discovery in Databases. Research Track: European Conference, ECMI PKDD 2021, Bilbao, Spain, September 13-17, 2021, Proceedings, Part I I1 pages 174-189. Springer, 2021.
Rupesh Kumar Srivastava, Pranav Shyam, Filipe Mutz. Wojciech Jaskowski, and Jürgen Schmidhuber. Training agents using upside-down reinforcement learning. arXiv preprint arXiv:1912.0288 2019.
Richard Sutton and Andrew G Barto. Reinforcement learning: An introduction. Robolica, 17(2): 229-235, 1999.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
Mengdi Xu, Yikang Shen, Shun Zhang, Yuchen Lu, Ding Zhao, Joshua Tenenbaum, and Chuang Gan. Prompting decision transformer for few-shot policy generalization. International Conference on Machine Learning, pages
PMLR, 2022.
Taku Yamagata, Ahmed Khalil, and Raul Santos-Rode Rodriguez. Q-learning decision transformer: Leveraging dynamic programming for conditional sequence modelling in offline rl. arXiv preprint 2.XY:22009.0393,, 2022.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. Advances in Neural Information Processing Systems, 33:14129-1441 2020.
Xianyuan Zhan, Xiangyu Zhu, and Haoran Xu. Model-based offline planning with trajectory pruning. arXiv preprint arXiv: 105.07351, 2021.
Siyuan Zhang and Nan Jiang. Towards hyperparameter- free policy selection for offline reinforcement learning. Advances in Neural Information Processing Systems, 34: 12845, 2021.
Qinqing Zheng, Amy Zhang, and Aditya Grover. Online decision transformer. In International Conference on Machine Learning, pages 27042-27059. PMLR, 2022.
12

---

A Derivation for Illustrative Example
We provide detailed derivations based on the simple deterministic MDP shown in Section 4.1, in the context of offline dataset DD collected by random behavioural policy mh. We show that minimization of a maximum likelihood objective on TG yields Tb, the behavioural policy. Note TG St; W) = TC(a; st) as w = g(H) a constant (and as a result, a, is conditionally independent). To obtain the optimal policy Tē. maximize the following objective:
arg max E (ED (a se:w)] FG
We simplify an expectation over an infinitely large dataset D collected by Tт.
aE) llog (a st, w]] = E., ez (A log Te = ()) st,w) + (1 - )) log T = al2) [[,ww]
Since the actions conditionally independent of the states, let To (a: = a(1) w) for any
state Sy. Then:
a. (a; St, st,w)] w)] Alog p + (1 A) log(1 P)
We calculus maximize the above objective with respect to p)
d
1-
Alog A + (1 - X) logg p)] = dp
p 1-6
X(1 6) (1 X)
Ø( P)
Setting the derivative to 0:
A(1 -1 -
2p 2 0 -=
This yields an identical policy to the behavioural policy Next, consider the derivation of the probability of taking action (2) conditioned on $, and based on data DD from T..
Pr., (a: = (2) 8t s(h) SLKK = shh+11]
PT., [ai a SLLK (h++1) 81 (())
Pray [st+ ((++1)
8: = s(h)]
In the above step, we used the definition of conditional probability. To compute these probabilities,
we recognize that to end up with SH+K = slh+1) from s( the agent must take action (2)
exactly once between timestep i and + K - 1; any more implies the agent has moved beyond s (h+1) and any less implies the agent is still at sh)
The probability in the numerator can be written as a product of taking action (2) timestep t,
followed by taking action (1) timestep + to t + K 1:
(+
Pr., (at a (2) St4K = (h++1) |s s((] = =1 X)
II
t'1+1
(1 - K-1
The probability in the denominator can be written product of taking action (2)) exactly one timestep < l' < < K, followed by taking action a (1) the remaining timesteps. This can be modeled by a binomial probability where there are K slots to take action a!)), each with probability 1 >. Hence:
(1)) A)1
Pr., [s+4 = (((+1)
=s)=s(h)
1(1 -K
The overall probability is computed as:
13

---

Pra [a (2))
S s(h+1) 31
sshh]
(1 A)AK-1 K(1 1JYK-1
Prz, [s--K - glh+1)
81 = =s(h]
1
K
We can apply similar argument to show that oir (i.e., at optimum) must clone the derived prob- ability when a maximum likelihood objective applied. Hence, for the optimal intermediate goad-cond policy Tw, we know it obeys:
Tw (a: a22) st = =((h,, t1 = =((ht1)) K
Since (a (22) St A, conclude that: K
((h), ,w) To(ai a22) 8t = slh))
À and we choose
(a (2)) se. Dt) T6 = =a (e,ww
This concludes the derivation.
B Experimental Details
In this section, we provide more details about the experiments, including hyperparameter configura- tion, sources of reported results for each method, and details of each environment (i.e., version). For all experiments on WT, the proposed method, we run 5 trials with different random seeds and report the mean and standard deviation across them. On AntMaze and Kitchen, use goal-conditioning, whereas wwwarddconditioning is used for Gym-MuJoCo. For all experiments on DT, including Gym- MuJoCo, run 5 trials with random initializations using the default hyperparameters proposed in Chen et al. 2021] and used in the official GitHub repository. We are unable to reproduce some of the results demonstrated in Chen al. 202T and reported in succeeding work such as Kostrikov et al. [002
Emmons et al. [2021]
B.1
Environments and Tasks
AntMaze For AntMaze tasks, include previously reported results for all methods except RyS-G from Kostrikov et al. [202]]. The results for the RvS-G are from Emmons et al. [2021]. We run experiments for DT srwward-ooniitiondd as per Chen et al. [021]] and WT across seeds. For all reported results, including WT, AntMaze v2 is used as opposed to AntMaze vO.
FrankaKitchen On Kitchen, we include available reported results from Kostrikov al. 2021 for all methods except RyS-G and Emmons et al. [2021] for RvS-G, with results omitted for TD3 + BC and Onestep RL they are not available in other work provided by the authors. Similarly to AntMaze, we run experiments for DT and WT across 5 seeds. The target goal confi onfigurationn for WT is "all" (i.e., where all the tasks solved), per [Emmons et al. [021]] For all reported results, including WT, Kitchen v0 is used.
Gym-MuJoCo On the evaluated locomotion tasks, reported results from Kostrikov et al. [2021] for all methods except RvS-R and Emmons et al. [202T] (RvS-R). We run experiments for DT and WT across seeds. The MuJoCo v2 environments used for all methods.
B.2 WT Hyperparameters
In Table 3 we show the chosen hyperparameter configuration for WT across all experiments. Consis- tent with the neural network model in RvS-R/G with 1.IM parameters Emmons et al. [202]). the WT contains 1.1M trainable parameters. For the most part, the chosen hyperparameters align closely with default values in deep learning; for example, use the ReLU activation function and a learning rate of 0.001 with the Adam optimizer.
In Table 4 we show the chosen hyperparameter configuration for the reward and goal waypoint networks across experiments. The reward waypoint network always outputs values, the ARTG and CRTG, In general, the goal waypoint network outputs the same dimension the state since
14

---

Table 3: Hyperparameters and configuration details for WT across all experiments.
Hyperparameter
Value
Transformer Layers
Transformer Heads
16
Dropout Probability (attn)
0.15
Dropout Probability (resid)
0.15
Dropout Probability (embd)
0.0
Non-Linearity
ReLU
Learning Rate 0.001
Gradient Steps 30,000
Batch Size 1024
makes k-step predictions. Depending on the environment, the goal waypoint outputs either 2-dimensional location for AntMaze 30-dimensional state for Kitchen.
Table 4: Hyperparameters and configuration details for goal and reward waypoint networks across all experiments.
Hyperparameter Value
Number of Layers
Dropout Probability
0.0
Non-Linearity
ReLU
Learning Rate
0.001
Gradient Steps
40,000
Batch Size
1024
B.3 Evaluation Return Targets
The target return for the Gym-MuJoCo tasks are specified in Table in the form of normalized scores. These were obtained typically by performing exhaustive grid searches over 4-6 candidate target return values, following prior work [Chen al., [2021] Emmons al. [02T]. Typically, we choose the range of the grid search based on the interval close to or higher than the state-of-the-ar normalized scores on each of the tasks.
Table 5: Normalized score targets for WT on reward-conditioned tasks in Gym-Mujoco.
Task
Normalized Score Target
nopper-mediu--replay-yaa
95
hopper-medium-r/
73.3
hopper-meddum--xppert.-2
125
walkerr26-mddiii--rpply---2
90
walker2dd-medium-22
85
valleerzd-mettimm-eepprrt--
122.5
halffhhetaah-mdddu--repp------
45
halfcheetah-mediim-yy
52.5
half Malfchhetah-nnddumeegggeeee
105
C Analysis of Bias and Variance of Reward-Conditioning Variables
We analyze the bias and variance of existing reward-conditioning techniques: constant average reward-to-go (ARTG) target, as used in Emmons et al. [202]) and a cumulative return target updated with rewards collected during the episode (CRTG), in Chen et al. [2021). By analyzing the bias and variance of these techniques, we can determine the potential issues that may explain the performance of methods that condition using these techniques.
15

---

Consider the definitions of the true ARTG (Ra) and CRTG (Re) below, based on given trajectory where the length of the trajectory T. These definitions are used train the policy,
1
Rd (T,
-
(T( = Σ y/r 1/mmt
(4)
At evaluation time, it is impossible to calculate Ty for any t' 2 t. As result, we provide ARTG and CRTG targets, 0₂ and 0g respectively. At evaluation time. the values of the ARTG and CRTG are estimated and used as follows in Chen et al. 2021] and Emmons et al. [2021]:
Ra(T, 1) = 0n
(5)
Rc(T,t) = 0e
Enn
(6)
pmml
Ideally, the errors the estimated R. and R. are minimal so as to accurately approximate the true average cumulative reward-to-go respectively, but that is often infeasible. To characterize the error of each of the evaluation estimates of ARTG and CRTG, consider the decomposition of the error for a particular reward conditioning variable R and estimated evaluation A presented in Theorem C.1
Theorem C.1. The general birinvariance decomposition of the expected squared error between a true R(T, t) (e.g., Equations I or 2) and an estimator R(T, t) (e.g., Equations or 4) under the trajectory distribution induced by arbitrary policy and unknown transition dynamics p(5141 $t, at) is given by: (R((,,, - ((), - E(R(T,t) ()r,, Var(R(r, R(T, t)]
Proof. Similarly to the derivation of the standard bias-variance tradeoff, we expand terms and separate into multiple expectations using the linearity of expectation. We leverage the definition of covariance, Cov(X, Y)- [[XY] E[]]E[Y], and variance, Var[X] = E [| E|X]" in several steps. [-((R(T,t) t( R(T, (((,,) B[R(T, ( R(T, t)]
We can simplify the first term using the definition of the variance and the third term using the definition of the covariance.
E, (R((,, = Var[R(T, ()) E[ -- Var(R(r,tt)
- 1)
- R(T,t)
E[R(T, [E ([](((T,))) ()))
Similarly, we simplify the [[A t) term using the definition of the variance and collect terms.
(R((R() ( t))² (E[R(T, +)]
+ Var[f[(r, +)] 2Cov(f(FF,t), R(T, +))+
Var[R(r,ti]
E[ f((T, 1)} + Var[ ()
Var[[[T,,t]
Equivalently, since Var(X- - Y = Var X Var[Y] 2Cov(X. we can rewrite the result as follows.
E, [,((T - (R(T,t)) B(R(T,t) R((T Var(A((T,t) R(T, t)]
16

---

ARTG Target (RalT, t) 0a)
Average Reward Go
Average Reward-to-Gr
200
400
600
800
1000
200
400
800
1000
Timestep (t)
Timestep (t)
Figure 7: True average reward-to-go as function of timestep & for 200 rollouts of transformer policy on hopper-meet merr-medium medium-----ee--- iesiium-reee um-replay-v2 compared to the constant ARTG target 0, (dotted black line), for left: successful rollouts and right: unsuccessful rollouts.
This completes the derivation of the general bias- variance decomposition.
Analysis ARTG Consider the decomposition of the error per the bias-variance tradeoff of the ARTG. Trivially, the variance of the estimate in Equation 3 is zero and it is independent of R because is constant value, and result, the expected error composed entirely of the bias and irreducible variance.
(T, R. (T, Er Rr, f)} 2 Var[Ba(T, ()]
= (T, 0.1 (T,
Based on the terms that we are able to minimize, we can derive through calculus that the squared error and bias of a constant estimator are minimized by the mean, i.e., when 0, E, (T, t)]. However, the offline RL setting, we cannot easily determine this value for an arbitrary trained policy и. result, the bias of the technique during evaluation can lead to high error in estimating the true ARTG, which may consequently cause reduced performance during evaluation of the policy.
Empirically, we demonstrate that the high bias of this technique may lead to instability in achieved return across rollouts hopper-mediure re-lrpyyy--2 Specifically, suppose that rollout is classi- fied as unsuccessful if it terminates before the time limit T 1000, and analogously, successful rollout reaches 7' without termination. Based on these distinctions, we display the true ARTG across 200 successful and unsuccessful rollouts of trained transformer policy in Figure 7
Clearly, for all successful rollouts, the ARTG matches the constant target closely across most t e [1,T), with the exception of f = T. However, across most failed rollouts, the bias of the constant target for small t z 0 (i.e., when the policy is taking its first few actions) is relatively high. As t grows, the bias seems to spike upwards significantly and the episode terminates shortly thereafter, indicating unsuccessful rollout.
Hence, it is evident that when the true ARTG closely follows the prescribed constant ARTG target, the policy consistently achieves state-of the-th--art performance. However, whenever the target ARTG underestimates or overestimates the true ARTG by a margin of greater than 0.4, the rollout tends to be unsuccessful, often achieving less than half the return compared to successful rollouts.
To that end, the reward waypoint network uses a neural network formulation to estimate Rg (i.e.. without using constant estimator f,) based on the state and the target return w. By training neural network to provide less biased estimate of the ARTG, we show that we can achieve substantial performance improvements over a constant estimate of ARTG on the same task. As shown 6 RvS-R and a constant ARTG both exhibit significantly lower average performance and greater variability compared to WT (with reward waypoint network).
17

---

Table Normalized evaluation scores for different policies and ARTG estimation techniques on the hopper -edeium-e -replay- task.
Technique
Normalized Score
Transformer (constant ARTG)
66.5 + 15.6
RyS-R (constant ARTG)
73.5 12.8
WT (waypoint network)
88.9 + 2.4
Analysis of CRTG Consider similar decomposition of the error using Theorem C.1 of the CRTG. Though neither the bias nor the variance of R. are necessarily zero, it is important to note that the variance term can be large even if the bias is zero, i.e., if 0, E E
Corresponding to a best-case scenario where R. unbiased, we consider a bound of the expected error constructed using only the variance term. For simplicity, we assume that we incur the worst-case variance term with minimal covariance between R. and R... The resulting dominant quantities are the variance of Re and the irreducible variance of R..
(,((R,(T,t) = t) - R(T, Var(R(r,tt R(T,t)]
N Var Var[ .) - R(F,t))
Var o rel + Var[R(T, t)]
Prior work has shown that Monte Carlo estimates return exhibit high variance, motivating techniques such ing baseline networks for REINFORCE, k-step returns or TD(A) for TD-learning methods, etc., which aim reduce variance at the cost of potentially incurring bias Sutton and Barto [999]. In that vein, we demonstrate that reward waypoint network predicts similar quantity as a baseline network and inherits many of its desirable properties.
50000
Estimated CRTG
Predicted CRTG (WT)
40000
Varlance
30000
20000
10000
Specifically, while the baseline network predicts Etm 7'T given 8t, we additionally condition our reward waypoint network's prediction on w. In the offline RL setting where datasets are often a mixture of suboptimal and optimal trajectories, we require that the waypoint net- work can differentiate between such trajectories. Hence, by conditioning on the overall reward w, the reward waypoint network is able to differentiate between high and low return-achieving trajectories.
200 400
6ó0
8ô0
1000
Timestep (t)
Figure 8: Variance of estimated CRTG (Equation 4) and predicted CRTG by the reward waypoint network on the hopper-mediume -
Empirically, demonstrate that the predicted CRTG from reward waypoint network exhibits lesser variance on hopper- ienediiumree un- than the estimated CRTG (i.e., as in Equation 4). As shown in Figure the variance of both techniques are relatively similar until t 400, after which the variance of the estimated CRTG appears to grow superlinearly as a function of t. In the worst case, at 2 T, the variance nearly 5x larger than for the predicted CRTG.
D Additional Experiments
D.1 Analysis of Stitching Region Behavior
To add on to the analysis of the goal waypoint network presented in the main text, we analyze the "failure" regions of transformer policies with and without a goal waypoint network. That is, by determining the final locations of the agent, we can examine where the agent ended up instead of the target location. Similar to the analysis Section 6.3, this analysis can inform the stitching capability of our methods.
18

---

Figure 9: End locations for cettmaze-arre-plaee during 100 rollouts of left: WT and right:
global condicondd transformer policy.
Based on Figure 9 clear that the WT does not get "stuck" (e.g., after taking the wrong turn) as often as the policy conditioned on global-goals. Moreover, the number of ants ending up near the beginning portions of the maze (i.e., the bottom left) significantly smaller for WT, which contributes to its doubled success rate. We believe these are primarily attributable to the guidance provided by the goal waypont waypoint network through consistent set of intermediate goals to reach the target location at evaluation time.
Interestingly, we observe that WT displays an increased rate failure around the final turn relative to other regions in the maze. As there is relative lack of density in other failure regions closer to the beginning the maze, we hypothesize that some rollouts may suffer from the ant getting "stuck" at challenging critical points in the maze, as defined in Kumar et al. [2022]. This indicates an interesting direction of exploration for future work and technique combat this could result in policies with nearly 100% success rate in completing antmaze - -large largepplay-v2
D.2 Target Reward Interpolation
Unlike traditional value-based approaches, RvS methods such as DT, RvS, and WT present simple methodology to condition the policy achieve particular target turn. In theory, this allows RvS to achieve performance corresponding to the desired per- formance level, provided that it has non-zero coverage within the offline training dataset.
Achieved llornalized Return
120
To examine RyS's capability in this regard, replicate an analysis performed in Emmons et al. [202]] to examine whether RvS can interpolate between modes in the data to achieve particular target return. In this case, we compare WT to RvS-R on the walker2d-medl mmedium-expee um-expert- task, in which the data is composed of two modes of policy performance (i.e., medium and expert). The mode corre- sponding to the "medium" data is centered at normalized score of 80, whereas "expert" performance located normalized score of 10.
100
80
60
40
20
Target Normalized Return
Based on Figure 10, WT shows reasonably improved ability to interpolate in some regions than RyS-R. Where RyS-R displays failure to interpolate from normalized targets of 30-60 and WT does not, both tend to be unable to interpolate between 70-100 (i.e., between the modes of the dataset).
Figure 10: Achieved normalized score versus the target normalized score for wew-rrdddddiim-eppppeee on WT and RvS-R.
task
Although the reasons for the failure in interpolation are unclear across two RvS methods, it is worthwhile analysis for future work. We hypothesize that techniques such as oversampling may mitigate this issue this may simply be linked to low frequency in the data distribution.
19

---

D.3 Comparisons to Manual Waypoint Selection
We compare the performance of the proposed goal waypoint network with a finite set manual waypoints, hand-selected based on prior oracular knowledge about the critical points within the maze for achieving success (i.e., turns, midpoints). Based on the selected manual waypoints, shown in Figure [1] we use simple algorithm to provide intermediate targets c based on distance-based sorting approach, shown in Algorithm 2
+
+
Figure II: Manually selected waypoints (blue pluses) for antmaze- -artmazz--rrgeppaa---22 the chosen task to evaluate the proposed approach. As before, the start location is marked with maroon dot, and the target location is marked wit a gold star.
Algorithm 2 Manual waypoint selection with Wm and 8t using L2 distance and a given global goal
We t {wm ||wm - 2 < ||s, wl12] (consider waypoints that brings agent closer to w return arg ming.ew. l/ s:
With all configuration and hyperparameters identical to WT, we compare the performance of a global goal-conditioned policy, WT with manual waypoints, and WT with the goal waypoint network antmaze-large 1ar----aag---e- v2ay-v2 in Table
The results demonstrate that WT clearly outperforms manual waypoint selection in succeeding in the AntMaze Large environment. However, while comparing a global-goal conditioned policy and policy conditioned on manual waypoints, it is clear that the latter improves upon average performance and variability across initialization seeds. We believe that this illustrates that (a) waypoints, whether manual generated, tend to improve performance of the policy and (b) finer-grained waypoints provide more valuable information for the policy succeed more.
Table 7: Normalized evaluation scores for different waypoint selection techniques the antmaze- Larnze--arge-ae -2 task.
Technique
Normalized Score
No Waypoints
33.0 + 10.3
Manual Waypoints
44.5 + 2.8
Waypoint Network
72.5 + 2.8
We believe that this provides further verification and justification for both the generation of intermedi- targets and the procedure of generation through a goal waypoint network that performs k-step prediction.
D.4 Delayed Rewards
An important case to consider for reward-conditioned tasks is when the rewards are delayed (often, provided at the end of the trajectory). By providing nearly no intermediate targets, it is often chal- lenging to complete these tasks. To verify that our design choices to construct modeled intermediate
20

---

targets does not reduce performance on configurations in which such information is unavailable, we evaluate WT on MuJoCo tasks where all rewards are provided at the final timestep. Moreover, we ablate the choice of removing actions from WT,, which may have an effect the performance considering that intermediate rewards are not provided.
To compare our performance with other methods, we report results for CQL, DT, and QDT from Yamagata al. [2022] for all non-expert tasks and from Chen al. [2021] from hopper-medi um cem--eeprtv2 in Table g The results demonstrate that our method is relatively
comparable to DT in performance and that removing actions does not significantly affect performance in these cases. All BC methods significantly outperform CQL across the chosen delayed reward tasks.
Table 8: Normalized scores across 5 seeds delayed reward MuJoCo v2 tasks, where all rewards provided at the final timestep, on WT with action conditioning (a²) and without action conditioning.
Environment
CQL
DT
QDT
WT (a.)
WT
hopper-medlum-expen experttt-2222
9.0
107.3 + 3.5
103.9 3.0
104.2 + 2.4
aafechectah-mediume replayy--v
7.8 6.9
33.0 +
4.8
32.8 + 7.8
30.6 2.6
30.9 2.8
walker2d-medium-
0.0 1 0.4
69.9 1 2.0
63.7 1 6.4
71.5 1 0.9
70.8 1 1.7
E Baseline Reproduction
We reproduce the TD3+BC results reported in Fujimoto and Gu 2021 and subsequently in Kostrikov et al. [021 Emmons al. [021), which not reproduced by Liu and Abbeel (2023 For
MuJoCo tasks chosen from the same evaluation tasks as WT, we show the normalized score of
TD3+BC and the reported mean score as function of the number of iterations of training. For all of the tasks, it is evident that our reproduction aligns with the original reported result, whereas those Liu and Abbeel 2023 consistently higher. We hypothesize that these findings similar in nature to our inability to reproduce results for DT in
hallcheetah ralichectah-medlumeee exsriue--tttuu
hellcheetah-mediumme
nooger-medium- expertvv2
No Sarultoen Schi
48.5
48.0
50
47.5
Inportedin
Reporied
Reported
Reporied
40-
Reported
Beratione hupper-mesium-egpee
lieratoon hoppper-eduum-y
Beragaon
Banations walker2 dwmeeimm-eeppllyy-dd
welker met-mmmieee-eeeeee---ll- expertoy2
100
110
homalz:d Spore
Norma Soree 500g
60
40
Reported WT
30
Raportod In WT Reported AT
Reported in
Ruported in WT Reported K7
Reperted AT
Reported AT
0.0
0.2
0.4
0.5
0.8
L.0
0.0
0.2
0.4
0.6 0.8
1.0
0.0
0.2
0.6 0.8
1.0
0.0
0.2
0.4
0.5
0.8
1.0
Berations
ttenations
1:5
Pecstions
Figure 12: Reproductions of TD3+BC baseline for 8 MuloCo tasks across 5 seeds for 1M gradient steps, showing that all our reported results (blue, dotted line) lie within the reproduced confidence intervals and most of the results reported in Liu and Abbeel 2023 (red, dotted line) do not.
Moreover, to attempt to reduce the possible effect of hyperparameters on the performance difference, sweep 16 combinations of the number of actor and critic layers based on the original codebase used by the TD3+BC authors here: implementation features hardcoded 2 layer neural network for actor and critic. In 9 we show that no combination of hyperparameters for the number of actor and critic layers is sufficient to reach the performance reported by Liu and Abbeel (2023) and that as the number of layers increases beyond or decreases beyond 2, the performance decreases on hopper- -medi meediu--repa
http://gthuub.com/fffjjiim
_BC_ Note that their original
21

---

Table Average normalized score of TD3+BC baseline across seeds by varying number of actor (L a) and critic network hidden layers (Lc) on hopper hopper-medi -elar----------ly---2-
Lc
L
3
1
48.7
80.1
73.1
73.9
2
55.2
64.4
50.2
61.3
3
59.8
59.1
69.6
64.6
4
42.9
60.5
63.6
54.4
22


Reviews:
[{"review_id": "Nj0ryIllRT", "reviewer": "Reviewer_8scR", "summary": "This work proposes a novel approach for enhancing RL via supervised learning (RVS) by integrating intermediate targets. The main idea is that while standard approaches condition on only the terminal state, the proposed waypoint transformer conditions on automatically generated intermediate waypoints instead, improving spatial compositionality. An additional technique for reducing bias/variance by conditioning on predicted average and cumulative reward to go conditioned on the return and current state is proposed. Experiments show sota performance on many environments, outperforming previous RVS approaches and matching or outperforming other offline RL approaches.", "questions": "Can the authors address the weaknesses and questions above?", "limitations": "Yes the authors have adequately addressed limitations.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The proposed waypoint approach is novel, simple and effective. To me it is similar to hindsight relabelling, except the goal state is taken as an intermediate state instead of the final state. This makes sense as its more likely the actions taken are optimal with respect to a closer state than a further state. The work shows a good illustrative example to build this intuition as well.\n* The proposed proxy reward generation to model both ARTG and CRTG effectively reduce bias/varaince by essentially acting as a baseline network while still enabling performance better than the behavior policy through the return conditioning.\n* The experimental results are convincing and showcase improved performance and also stability which is great to see.", "weaknesses": "* It is unclear when the spatial vs reward waypoint network is used. It seems like the spatial waypoint can only be applied in specific environemnts and not others. But it's not clear when, and whether both can be used at the same time.\n* While intermediate spatial waypoints improve spatial compositionality, more abstract \"skill compositionality\" seems out of reach with this method.\n* I don't understand why the reward waypoint network is called a waypoint network when it doesn't seem like it has anything to do with waypoints. I could potentially see a relationship if the horizon of the reward-to-go was only up to t+K to draw relationship to the intermediate outputs, but what is actually used is the full horizon T as far as I can tell. Cand can the authors be clear how the reward-to-go target is defined and its relation ship to the predicted ARTG and CRTG? Also minor: typo in equation 2, superscript should be t' \n* Can the authors compare the behavhior cloning objective of learning the reward-to-go (essentially learning a value function of the behavior policy) compared to alternatives such as Q-Learning transformer? Whle the authors mention they restrict themselves to behavior cloning objectives  (L198), I don't see a strong justification for this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback and insightful comments. Below are our responses to the reviewer's questions.\n\n> It is unclear when the spatial vs reward waypoint network is used. It seems like the spatial waypoint can only be applied in specific environemnts and not others. But it's not clear when, and whether both can be used at the same time.\n\nApologies for the confusion. We mention in Section 6 (line 238, 242) that the AntMaze/Kitchen tasks use goal conditioning and the MuJoCo tasks use reward conditioning. Hence, we use the goal (“spatial”) waypoint network for AntMaze/Kitchen since that is used for goal-conditioned tasks (Section 4.2, line 150) and the reward waypoint network for MuJoCo since that is used for reward-conditioned tasks (Section 4.3, line 187).\n\nBoth could certainly be used at the same time by concatenating the values of the two waypoint networks, but the task would have to be amenable to having reasonable definitions of goals. For example, the MuJoCo tasks (e.g., Hopper) typically do not have any notion of a target state, which makes defining goals for those tasks challenging (Emmons et al., 2021). Similarly, if rewards are provided, then the reward waypoint network can certainly be used. \n\n>While intermediate spatial waypoints improve spatial compositionality, more abstract \"skill compositionality\" seems out of reach with this method.\n\nThis is an interesting observation, but to our knowledge, WT’s behaviour should be largely equivalent to other offline RL techniques in these cases, and depending on how one defines skill compositionality, this could potentially be desirable behaviour.\n\nIn the case that different skills are exhibited within the same or different spatial segments, then spatial compositionality should enable composing different skills within the respective spatial segments.\n\nIf you are referring to composing different skills unseen in those spatial segments in the dataset, which spatial compositionality would not allow, then that would be out of reach with our method. However, this could actually be a desirable behaviour since it is unclear whether the skill (unseen in a particular spatial segment) would be optimal for that segment since there is no data coverage. As a concrete example, an unobserved obstacle may be that the floor is slippery in regions where the behavioural policy does not employ the skill of zig-zagging; in this case, composing the skill of zig-zagging in slippery areas with other skills could result in suboptimal performance. In a similar vein, offline RL techniques such as CQL penalize out-of-distribution actions (such as executing actions for some “skill” when it is unobserved from particular states) and TD3+BC employs a BC objective for a similar reason.\n\n> I don't understand why the reward waypoint network is called a waypoint network when it doesn't seem like it has anything to do with waypoints. I could potentially see a relationship if the horizon of the reward-to-go was only up to t+K to draw relationship to the intermediate outputs, but what is actually used is the full horizon T as far as I can tell. \n\nApologies for the confusion - the definition of waypoints here may differ from the conventional  definition of a waypoint. We have defined waypoints as intermediate targets (Sec 4, line 113). In this case, the generated RTGs (even with the full horizon) are intermediate targets (i.e., a target that incorporates intermediate information about the current state, $s_t$).\n\nFor reward-conditioned tasks, CRTG can also be considered as providing intermediate targets (or waypoints) since the reward-to-go information is updated at each timestep (unlike ARTG, which is fixed, similarly to a global goal). The critical difference is that our intermediate targets are generated by a model (hence, waypoint network), which demonstrably reduces the variance of the target and improve performance in nearly all cases compared to DT which uses CRTG. \n\n>can the authors be clear how the reward-to-go target is defined and its relation ship to the predicted ARTG and CRTG? \n\nOur proposed reward-to-go target is simply the concatenation of the predicted ARTG and CRTG from the reward waypoint network. As shown in Equation 2, the quantity our reward waypoint network estimates is exactly the concatenated ARTG and CRTG (i.e., we minimize the mean squared error between the concatenated ARTG/CRTG with the output of $W_\\phi$).\n\n> Can the authors compare the behavhior cloning objective of learning the reward-to-go (essentially learning a value function of the behavior policy) compared to alternatives such as Q-Learning transformer? Whle the authors mention they restrict themselves to behavior cloning objectives (L198), I don't see a strong justification for this.\n\nThe Q-Learning DT (QDT) uses standard TD learning techniques, which unfortunately introduces all the potential training issues associated with these methods (more hyperparameter tuning, training instability, etc. as stated in Sec 1, lines 19-20) and forfeits many of the advantages associated with BC-style algorithms.\n\nMoreover, QDT shows notably worse performance than simply using an off-the-shelf Q-Learning algorithm like CQL (i.e., skipping training the DT entirely) or DT across nearly all environments. Unfortunately, the QDT only improves upon reference DT’s mean score in 1 of 6 tested MuJoCo tasks, with statistically significantly worse performance than DT in 2/6 tasks. Similarly, on MuJoCo tasks with delayed rewards, QDT demonstrates a higher score on only 1/6 tasks.\n\nHence, we believe that adding Q-Learning to DT complicates the training procedure notably more, without much added value in terms of performance. We add a simpler BC objective whose training is rapid and stable across all tasks without much tuning (Figure 5, line 321) and demonstrate equal or improved performance compared to DT on all tasks.\n\nPlease let us know if you have any more feedback or would like to discuss more!\n"}, {"review_id": "rT6jqD8Dve", "reviewer": "Reviewer_751f", "summary": "This paper proposed a novel, straightforward approach for generating intermediate waypoints in transformer-based sequence modeling. The waypoint networks are supervised with offline datasets and trained to minimize the mean squared error between the targets and predictions. In addition to the improvements shown in the main results, the authors conducted thorough ablation studies, revealing the correlation between waypoints and RL performance.", "questions": "- Since the waypoint networks are trained to minimize the MSE, imagining multiple paths equally distributed in the dataset between two states A to B, what kind of waypoint would the network predict at state A?\n- Have the author thought about extending the idea to the visual domain?", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "This paper is well-written and easy to follow. The proposed method is novel and straightforward. The analysis experiment is well-designed and informative.", "weaknesses": "My main concern relates to the baseline model's performance.\n\n- Have the authors tuned the baseline model thoroughly? They are much lower than the results reported in AT[1]. For example, the TD3+BC results are different from either AT or the original paper. 10% BC result is also much lower than those reported in AT.\n- Ablation results from more tasks is beneficial. The authors only choose one task, so the current results are unconvincing.\n\n\n[1]: Emergent agentic transformer from chain of hindsight experience. H Liu, P Abbeel - arXiv preprint arXiv:2305.16554, 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback and insightful comments. Below are our responses to the reviewer's questions.\n\n> Have the authors tuned the baseline model thoroughly? They are much lower than the results reported in AT[1]. For example, the TD3+BC results are different from either AT or the original paper. \n\nOur TD3+BC results are reported from the v2 results in the original paper (**Table 9** in Fujimoto et al., 2021), and the same results are reported by Kostrikov et al. (2021) and Emmons et al. (2021). Unfortunately, the AT paper does not report most of the hyperparameters they used for TD3+BC (nor the code), and it is unclear whether they use the v0 or v2 versions of D4RL/MuJoCo. It is also worth noting that they only used 3 seeds. We suspect this discrepancy in results could be due to either of the aforementioned reasons.\n\nWe are able to replicate all of *our* results for TD3+BC in Table 1 on the v2 environments using the original implementation from Fujimoto et al. (2021) (please see Figure 1 in our rebuttal PDF). Compared to the AT paper’s results, our reproductions are quite different, by around 25-30% in the worst case. To verify that hyperparameters aren’t likely the issue, we search over 16 hyperparameter choices on `hopper-medium-replay-v2` (Table 2 in rebuttal PDF) but are still not close to the AT paper’s reported performance.\n\nOur hypothesis is that the AT paper [1] may have similar evaluation issue(s) as DT since we were similarly unable to reproduce results for DT (Sec 6.1, line 252). Note that DT has similar reported scores in both the AT [1] and DT papers for `hopper-medium-replay-v2` (88.7 and 82.7), yet these are vastly different from our reproduction (43.3) on v2. Others have reported this type of discrepancy in DT as well; please see issue 42 on the official DT GitHub.\n>10% BC result is also much lower than those reported in AT.\n\nSimilarly, the hyperparameters/code used for 10% BC are not provided in the AT paper. We verify with the implementation of BC that is used in Kostrikov et al. (2021) and Emmons et al. (2021) and obtain similar numbers as we reported (within 3-4%). We believe this is due to the same reason as before.\n\n> Ablation results from more tasks is beneficial. The authors only choose one task, so the current results are unconvincing.\n\nWe are happy to address this. We have expanded our ablations to include `kitchen-mixed-v0` and `hopper-medium-replay-v2`, which represent one task from each of our environment types (Table 1, in rebuttal PDF). We have additionally tested more values of dropout. The results demonstrate that only the lack of dropout, too much dropout (>0.3, whereas 0.1-0.15 is typically standard for a transformer), or having a 1-layer transformer may yield reductions in performance, which is relatively unsurprising.\n\n**Questions**:\n\n> Since the waypoint networks are trained to minimize the MSE, imagining multiple paths equally distributed in the dataset between two states A to B, what kind of waypoint would the network predict at state A?\n\nGiven our simple implementation of the goal waypoint network as a feedforward neural network, it would likely take the mean of the future states within both sets of trajectories, which is certainly suboptimal. Unfortunately, since DT (and most BC methods) minimize the MSE in the action space, it would likely suffer from the same kind of issue as well. That is, it would optimize to output the mean of two actions that are equally distributed from one state with the same outcome condition (e.g., return or goal). The waypoint network would not change/fix this inherent issue.\n\nThat said, in practise, we have observed that this hasn’t been (empirically) an issue for two potential reasons:\n1. The chosen $K$ is quite small, which means that the maximum error magnitude due to this averaging/mixing issue can be quite small.\n2. It is uncommon to have perfectly equally distributed paths. Consider a weighted average of the first set of paths consisting of some proportion $\\alpha > 0.5$ and with the second set of paths consisting of $1 - \\alpha$ proportion of the dataset. In this case, the optimal value per a BC objective is to weight future states of the first set of paths by $\\alpha$. Since $\\alpha > 0.5$, the waypoints are likely to drift towards the first set of paths successively more and more.\n\nFinally, as mentioned in Sec 4.2 (line 179), the technique is intended and designed to be quite simple (i.e., easy to setup and train) to provide rough intermediate targets. More complex techniques could mitigate this issue, such as multimodal models using a mixture of Gaussians, for example.\n\n> Have the author thought about extending the idea to the visual domain?\n\nCould you please clarify what you mean by the visual domain? This idea could certainly be applied to Atari or similar environments which have visual input, but the existing DT approach has already shown strong performance in that domain (equals or outperforms CQL for 3/4 games), so our approach is likely less needed. In more recent work, Lee et al. (2022) have demonstrated significantly improved multi-game performance from multi-game DT compared to CQL, BC, and DQN by roughly at least a factor of 2x. We wanted to focus primarily on challenging areas in which the DT lagged behind existing offline RL techniques (AntMaze/Kitchen) or where we could not replicate the official results (MuJoCo v2 tasks).\n\nPlease let us know if you have any more feedback or would like to discuss more!\n\n*Lee, Kuang-Huei, et al. \"Multi-game decision transformers.\" Advances in Neural Information Processing Systems 35 (2022): 27921-27936.*\n"}, {"review_id": "k1nNA444Dl", "reviewer": "Reviewer_QWFP", "summary": "This paper proposes Waypoint Transformer for offline reinforcement learning. The novel Waypoint Transformer architecture dynamically generates waypoints for guiding the policy, either in a goal state or a return manner. The authors empirically show that the Waypoint Transformer exhibits better performance over previous reinforcement learning via supervised learning (RvS) algorithms. Meanwhile, numerous ablations are presented in the main text.", "questions": "(a) in section 4.1, the horizon is fixed. I think the analysis does not apply when the agent encounter the last few states? For example, if we set $K=5$, and suppose $H=100$, then the conditioned intermediate goal state is invalid if $t\\ge 96$, since we cannot get $s_{101}$. Any clarifications here?\n\n(b) in line 223, the authors write that \"the WT is not conditioned on past actions $a_{t-k..t}$ (unlike DT)\". I actually cannot get the meaning of this sentence. Does the authors mean that DT receive action as input while WT do not? Meanwhile, based on DT paper, it seems DT only receives the $a_t$ as input, while not the past action sequences\n\n(c) based on the ablations in Table 2, it seems it is better not to receive action as input. Any explanation here? Why is it?\n\n(d) how should we choose $K$ in practice, or in real-world applications? How should we determine which to condition, return or goal state?\n\n(e) in appendix B.2, the authors use different gradient steps in Table 1 and 2, and a large batch size 1024. Do these parameters have a large influence on the performance of the waypoint transformer", "limitations": "The authors discuss the limitations and future work in the main text", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "(a) this paper is easy to read and follow. I appreciate that the idea is quite straightforward, and simple. As far as the reviewer can tell, the presented idea in this paper is novel\n\n(b) it is good to see that sequence modeling algorithms can finally achieve meaningful performance on tasks like antmaze, kitchen. I believe this is the most critical contribution of this work\n\n(c) the authors provide many illustrations to aid the readers understand the claims\n\n(d) the experimental results show that Waypoint Transformer is better than many baseline algorithms", "weaknesses": "(a) some claims from the authors are not proper. For example, the authors write that \"with performance on par or greater than existing state-of-the-art temporal difference learning-based methods\". With the fast advances in the field of offline RL, the chosen baselines in this paper are no longer state-of-the-art. There are many model-free or model-based offline RL algorithms that can achieve far better (often saturated) performance on MuJoCo or other datasets (e.g., ensemble-based algorithms like EDAC).\n\n(b) Table 1 is chaotic, with multiple bolded and highlighted numbers in each row. It is hard to decode whether the waypoint transformer can achieve the best performance on numerous datasets. It seems currently that the performance of waypoint transformer is close to IQL, but I think this is already a big step. Another concern from the reviewer is that the authors do not compare against Q-learning transformer (although cited in the paper). It seems that the authors fits the return-to-go (in Equation 2), and this is similar to fit a value function. I believe this shares some similarities to Q-learning transformer.\n\n(c) thought the authors claim that they use fixed hyperparameters across all of the experiments, they actually use different training paradigms for different datasets (i.e., return conditioned for MuJoCo datasets, and goal conditioned for antmaze and kitchen datasets). This reduces the generality of the waypoint transformer. The reviewer wonders how the waypoint transformer behaves on MuJoCo datasets when conditioned on the goal state.\n\n(d) in ablations, the authors only provide $p_{drop}\\in\\\\{0,0.075,0.15\\\\}$ and $L\\in\\\\{1,2,3\\\\}$. I believe this is slightly insufficient to show the sensitivity to the hyperparameters. It is unclear whether waypoint transformer can achieve better performance with larger $p_{drop}$\n\n(e) the minor issues are listed below\n\n - in Algorithm 1, w.r.t ==> w.r.t.\n\n - line 119, consider ==> we consider\n\n - the gradient step of waypoint transformer seems to be much fewer than DT or other algorithms. I believe mroe gradient steps are expected since the performance of many offline RL algorithms are known to collapse with longer gradient steps", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable feedback. Below are our responses to your questions.\n\n>There are many model-free or model-based offline RL algorithms that can achieve far better (often saturated) performance… (e.g., ensemble-based algorithms like EDAC).\n\nWe should clarify that we are not referring to or competing with ensemble methods or online fine-tuning, and we have added this to the text. That said, we do not dispute that there may be a better state-of-the-art. Based on that, we propose rephrasing to: “performance on par or greater than contemporary/commonly used TD learning methods”. To our knowledge, algorithms like CQL, IQL, TD3+BC, etc. are commonly used due to performance and usability. Our method is simpler to train, as computationally efficient, and has on-par or greater performance than such methods.\n\n> … the authors do not compare against Q-learning transformer. \n\nWe have added this comparison to our results. We initially omitted it since QDT has worse performance than a Q-Learning algorithm like CQL (i.e., skipping training the DT entirely) or DT across nearly all tasks. Specifically, QDT only improves upon reference DT’s score in 1/6 tested MuJoCo tasks. On MuJoCo tasks with delayed rewards, QDT demonstrates a higher score on only 1/6 tasks.\n\n> they actually use different training paradigms for different datasets… This reduces the generality of the [WT]. The reviewer wonders how the [WT] behaves on MuJoCo datasets when conditioned on the goal state.\n\nWe believe that the option of having goal or reward conditioning increases the generality of WT compared to traditional offline RL methods. Importantly, our method can work even if there are no rewards, unlike many other techniques. If rewards are provided, reward conditioning can be used, and if the task has a specific target state, then goal conditioning can be used.\n\nFor MuJoCo, the tasks don’t lend themselves to being formulated as goal-reaching tasks since there is no target (e.g., Kitchen has a target configuration) - Emmons et al. (2021) made the same observation. However, if the tasks were reformulated with a target, we could use goal conditioning with a goal waypoint network. In fact, Ant is an undirected locomotion task that is part of MuJoCo, and AntMaze wraps the ant locomotion task with a clear objective or goal, which allows for goal conditioning.\n\n> It is unclear whether waypoint transformer can achieve better performance with larger p_drop\n\nThank you for the suggestion. The lower dropout values were chosen largely since it is more standard to have lower dropout values (0.1-0.15) for transformers and we wished to restrict ourselves to more default hyperparameter values whenever possible. We have run ablations on a more broad range of dropout values (Table 1 in rebuttal PDF). For some environments, increasing dropout to be large leads to a significant reduction in performance.\n\n>I believe mroe gradient steps are expected since the performance of many offline RL algorithms are known to collapse with longer gradient steps\n\nCould you clarify? We did not observe any collapses in performance with WT with 100K gradient steps, but the performance was slightly lower (67.3 $\\pm$ 5.0 on antmaze-large-play-v2).\n\n> …the conditioned intermediate goal state is invalid if $t \\ge 96$, since we cannot get. Any clarifications here?\n\nApologies - you are absolutely correct. Eqn 1 should be:\n$$\n\\arg \\min_\\phi \\sum_{\\tau \\in \\mathcal{D}} \\sum_{t = 1}^{|\\tau| - K} L_\\phi(W_\\phi(s_t, \\omega), s_{t + K})\n$$\n>Does the authors mean that DT receive action as input while WT do not? … it seems DT only receives the as input, while not the past action sequences\n\nWT does not receive any actions as input. For DT, the causal transformer architecture is able to aggregate information from all past actions since the preceding tokens will all be unmasked during the multi-head attention procedure. That is, it will be able to incorporate information from all states, actions, and rewards preceding that current timestep and the current state. \n\n>based on the ablations in Table 2, it seems it is better not to receive action as input. Any explanation here?\n\nAt train time, the actions that will be passed into the transformer will be taken by the behaviour policy. Though we attempt to conditionally mimic the behaviour policy, we will ultimately learn a different policy, i.e., distribution shift. At evaluation time, we instead pass in actions taken by the learned policy, which result in additional distribution shift in the input sequence. We hypothesize this causes performance reduction. \n>how should we choose K in practice… which to condition, return or goal state?\n\nFor $K$, our ablations suggest a small fraction of the horizon (e.g., 1-5% of $H$). If rewards are provided, then we can certainly condition on the return. If goals are clearly defined for the task, then it would be valuable to use goal-conditioning. For example, the Gym-MuJoCo tasks don’t permit goal conditioning since there is no “target” per se, but AntMaze has a clear target.\n>…the authors use different gradient steps in Table 1 and 2, and a large batch size 1024. Do these parameters have a large influence on the performance of the waypoint transformer?\n\nThe choice of large batch size is based on the findings of Emmons et al. (2021). Our experiments show that the choice of batch size of WT does affect performance but not significantly (512: 66.6 $\\pm$ 4.7, 1024: 72.5 $\\pm$ 2.8, 2048: 67.5 $\\pm$ 5.6 on antmaze-large-play-v2). Training for 100K steps yielded a score of 67.3 $\\pm$ 5.0.\n\nAs for the different number of training steps for the waypoint network, we chose 40K and 30K gradient steps respectively as they roughly correspond to similar training times. In Sec 6.3, we comment that even 1,000 gradient steps on the waypoint network can achieve a score of ~65 on antmaze-large-play-v2 (line 321).\n\nPlease let us know if you have any more feedback or would like to discuss more!\n"}, {"review_id": "pOTzFHsuUJ", "reviewer": "Reviewer_j7or", "summary": "This paper analyzes the limitation of the existing decision transformer architecture, i.e., the “stitching” problem associated with global goals and the high bias and variance of reward-to-go targets. To solve this limitation, this paper presents Waypoint Transformer, which predicts the constant average and the cumulative reward-to-go as short-term and intermediate goals, and then guide the transformer to generate trajectories to reach these goals. This method achieves good performance on various benchmarks and significantly reduce training time.", "questions": "> WT is not conditioned on past actions. How does WT solve the ‘delayed reward ’case, i.e., an action receives a reward after many interactions?\n\n> Why waypoint network can generate short-term and intermediate goals? How can we ensure that its output can guide the policy towards to ‘stitching region’?\n\n> Why does the waypoint network predict a TD(n) target to achieve the trade-off between a constant ARTG target and the MC estimates for CRTG? Is it a more intuitive and direct way to solve the high bias and the variance? \n", "limitations": "See Weaknesses.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "> This paper analyzes the limitation of the existing decision transformer architecture – the “stitching” problem, i.e., it cannot stitch segments of suboptimal trajectories into an optimal sequence. To solve this problem, this paper connects the relevant subsequences by providing intermediate targets, which is implemented by predicting the constant and the cumulative reward-to-go.\n\n> This paper achieves SOTA performance in many benchmarks compared with offline RL methods. In addition, it found that we can reduce training time and improve the performance by removing action conditions in original decision transformer.\n\n> The method is easy to be implemented.\n", "weaknesses": "> The novelty of the method is limited. The constant condition, i.e., the average reward-to-go, and the cumulative reward-to-go are widely used in offline reinforcement learning in recent work.\n\n> The presentation of this paper can be improved, e.g., it is difficult to see the average reward-to-go is a constant but the cumulative reward-to-go is updated using rewards obtained during the rollout in Eq. (2). In addition, the relationship between short-term goals and reward-to-go is not clear.\n\n>The authors do not provide a detailed analysis of the failure cases or limitations of their method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback and insightful comments. Below are our responses to your questions.\n\n>The novelty of the method is limited. The constant condition, i.e., the average reward-to-go, and the cumulative reward-to-go are widely used…\n\nWe do not dispute that ARTG/CRTG are commonly used (they serve as our baselines), and we cite relevant literature that use these quantities. However, to our knowledge, no work has investigated the issues this presents with regards to bias and variance of the conditioning variable itself and proposed a solution that shows performance competitive with TD learning methods on most tasks. We also believe that conditioning on **predicted** RTGs/goals in offline RL is novel.\n\nWhile our method is quite intuitive/simple, we provide a new perspective to enable more useful conditioning variables for offline RvS methods and present significant improvements in performance over DT/RvS, on-par or surpassing IQL/CQL/etc.\n>The presentation of this paper can be improved, e.g., it is difficult to see the average reward-to-go is a constant but the cumulative reward-to-go is updated using rewards obtained during the rollout in Eq. (2). In addition, the relationship between short-term goals and reward-to-go is not clear.\n\nWe apologize that the presentation is unclear, but we respectfully disagree with your analysis of Eq 2. Eq 2 shows the training objective for the reward waypoint network, where we train against the true concatenated ARTG and CRTG over training trajectories. It does not show that the cumulative RTG is updated since it minimizes a loss. Since the CRTG being updated with rewards increases variance significantly (Sec 4.3, line 192), we do not perform those updates in WT. The ARTG is set to be a constant at evaluation in techniques such as RvS but not in WT (Sec 4.3, line 193).\n\nShort-term goals are defined in the context of goal conditioning (Sec 4.2), where we do not consider rewards. Reward-to-go is only considered in the context of reward conditioning (Sec 4.3). We consider them both as intermediate targets for the paradigms of reward and goal conditioning.\n\n> The authors do not provide a detailed analysis of the failure cases or limitations of their method.\n\nWe discuss some of the concrete limitations of the method in Sec 7 based on our results (lines 383-388), but we agree that these are not detailed analyses due to space limitations. There are certainly limitations that continue to exist due to the BC objective, including with distribution shift, multimodality, stochastic environments, and random data. We are happy to add such analyses to the paper.\n\n>WT is not conditioned on past actions. How does WT solve the ‘delayed reward ’case, i.e., an action receives a reward after many interactions?\n\nThis is an interesting point, but it is unclear why action conditioning should notably improve performance in this case. Empirically, it seems to have a negligible effect (Table 3 in rebuttal PDF).\n\nSuppose that the rewards are provided at the end of the sequence and we condition on CRTG, which is equal to the return throughout the sequence. The larger the return of the training trajectory, the more likely those actions are to be imitated during evaluation time when we set the target return to be as large. \n\nA transformer policy (conditioned on actions) trains on sequences containing the actions of the behavioural policy. However, during evaluation, the actions placed in the sequence are of the learned transformer policy (not behavioural) which generated those actions, i.e., effectively duplicating info.\n\nIf we do not condition on the learned policy’s actions at evaluation time, the hidden states of the transformer should anyway contain information about these past actions (since the transformer policy produced them). Hence, we do not lose much information about the learned policy’s actions.\n\nIn Table 3 (rebuttal PDF), we show that including actions has negligible impact across 3 tasks, and we are happy to perform more experiments to further verify.\n\n>Why waypoint network can generate short-term and intermediate goals? How can we ensure that its output can guide the policy towards to ‘stitching region’?\n\nAs described in Sec 4.2, the training objective (Eq 1) minimizes error in $K$-step predictions of state $s_{t + K}$ based on the current state $s_t$ and eventual outcome $\\omega$ under the behavioural policy. Importantly, since $K$ is chosen to be relatively small compared to the time horizon $H$, the predictions will reflect intermediate states $s_{t+K}$ that should be reached between the current state $s_t$ and the eventual outcome $\\omega$. We cannot guarantee it will lead the policy to a stitching region, but it will provide short-term goals that will help provide intermediate guidance on how to achieve the global goal (and that is typically through a stitching region).\n\n>Why does the waypoint network predict a TD(n) target to achieve the trade-off between a constant ARTG target and the MC estimates for CRTG? Is it a more intuitive and direct way to solve the high bias and the variance?\n\nWe use a simple BC objective to predict/generate the CRTG and ARTG of trajectories (instead of fixing them to a constant value or updating using rewards). We do not leverage bootstrapping of a value function in any way (i.e., we predict a Monte Carlo TD($\\lambda = 1$) target).\n\nAs mentioned in Sec 4.3 (lines 204-208), an analogous approach is training a baseline network, which is known to reduce variance with a similarly simple BC objective. We believe this is an intuitive and simple way to reduce variance of PG methods, and we have simply adapted it for usage in offline RL by specifying a target return (since we want to perform conditional BC) and also predicting ARTG (which is scaled by the number of timesteps). To our knowledge, there is not a simpler way proposed to solve this.\n\nPlease let us know if you have any more feedback or would like to discuss more!"}]

Author rebuttal:
We would like to thank all the reviewers again for their valuable insights and feedback. We have carefully considered all comments, addressed each concern to the best of our ability, and believe that the new figures/tables (attached as PDF) and revisions significantly improve the thoroughness of the experiments and clarity of writing.

Based on Reviewer 751f’s remark that some of our baseline methods’ performance do not match with results in Liu et al. (2023), we have reproduced results for these baseline methods, namely TD3+BC (Figure 1, rebuttal PDF) and BC. While we can replicate all of *our* reported results (which match with the original TD3+BC results in Fujimoto et al. (2021), as well as Emmons et al. (2021) and Kostrikov et al. (2021)), we are still unable to achieve the results reported by Liu et al. (2023). We suspect that this discrepancy arises since they used only 3 seeds and may have (partially/completely) used the vastly different, older v0 D4RL/MuJoCo environments. Further, since Liu et al. (2023) do not provide hyperparameters/code for their baselines, we have performed experiments across 16 hyperparameter sets on TD3+BC (Table 2, rebuttal PDF) and verified that no combination yields even close-to-comparable performance to the reported results in Liu et al. (2023).

Moreover, based on feedback from Reviewers 751f, QWFP, and j7or, we have significantly expanded the scope of the ablation studies to include 3 environments from Gym-MuJoCo, AntMaze and Kitchen, with 3 additional tested values for dropout for each environment (Table 1, rebuttal PDF). We have also performed ablations across 3 tasks for the delayed rewards case (Table 3, rebuttal PDF). These expanded ablations reinforce that our initial hyperparameter selections were appropriate.

To clarify points raised by Reviewers j7or and QWFP about the cases in which each of the reward and goal waypoint networks are applicable (e.g., a common query was why goal conditioning was not used for MuJoCo, which is also addressed in past RvS work, Emmons et al. (2021)), we have expanded the descriptions of the use of the waypoint networks in Section 4.2/4.3. In Section 6.3, we have further elaborated on why we hypothesize action conditioning generally does not improve performance. Finally, based on feedback from Reviewers 8scR, j7or, and QWFP, we have extended the results with reported performance from Q-Learning Decision Transformer and enriched our discussion with an in-depth analysis of model limitations.

References: *Liu, Hao, and Pieter Abbeel. "Emergent agentic transformer from chain of hindsight experience." arXiv preprint arXiv:2305.16554 (2023).*

Meta-review: Reviewer opinions are split on this work -- with two recommendations for accept and two for reject. After reviewing the paper and discussion, the AC recommends acceptance. The paper presents a relatively straightforward idea, is well-written, and can achieve meaningful performance in difficult tasks settings where behavior cloning techniques for offline RL have so far faltered.

**Summary of Dissenting Opinions.** One reviewer recommends rejection due to discrepancies in reported performance for the TD3 + BC baseline. The reviewer notes that performance reported in Liu et al. 2023 (Table 2) does not match those presented in the submitted work. Authors responded that the D4RL dataset version was not clearly specified in Liu et al. Consistency with results in Fujimoto et al. (Table 2) and the details in the appendix (C.3) suggest the results in Liu et al. may be on D4RLv0. The experiments in the submitted work use D4RLv2 and are thus comparable to Table 9 from Fujimoto et al., with which they are reasonable close. The AC finds these arguments convincing and does not see this as a reason to recommend rejection. The other reviewer recommending rejection did not participate in the author discussion or update their review. Some concerns expressed were requests for clarification or improved presentation. To the AC's judgement, the authors response seems to satisfy their stated concerns.
Decision: Accept (poster)