Reinforcement Learning Gradients as Vitamin for Online Finetuning Decision Transformers
Authors: Kai Yan
Alex Schwing
Yu-Xiong Wang
Venue: NeurIPS 2024 spotlight (2024)
Keywords: Decision transformer
reinforcement learning
Primary area: reinforcement_learning
TL;DR: We found that introducing RL gradient would significantly improve online decision transformer's performance during online finetuning.

Abstract: Decision Transformers have recently emerged as a new and compelling paradigm for offline Reinforcement Learning (RL), completing a trajectory in an autoregressive way. While improvements have been made to overcome  initial shortcomings, online finetuning of decision transformers has been surprisingly under-explored. The widely adopted state-of-the-art Online Decision Transformer (ODT) still struggles when pretrained with low-reward offline data. In this paper, we theoretically analyze the online-finetuning of the decision transformer,  showing that the commonly used Return-To-Go (RTG) that's far from the expected return hampers the online fine-tuning process. This problem, however, is well-addressed by the value function and advantage of standard RL algorithms. As suggested by our analysis, in our experiments, we hence find that simply adding TD3 gradients to the finetuning process of ODT effectively improves the online finetuning performance of ODT, especially if ODT is pretrained with low-reward offline data. These findings provide new directions to further improve decision transformers.

Reinforcement Learning Gradients as Vitamin for Online Finetuning Decision Transformers
Kai Yan
Alexander G. Schwing Yu-Xiong Wang
University of Illinois Chbama-Champaign
[kaiyan3, aschwing, yxs)@illinois.
for Saaeaaiinno___Olleeeeeeeee
Abstract
Decision Transformers have recently emerged as a new and compelling paradigm for offline Reinforcement Learning (RL), completing trajectory in an autoregres- sive way. While improvements have been made to overcome initial shortcomings, online finetuning of decision transformers has been surprisingly under-explored. The widely adopted state-of-the-arr Online Decision Transformer (ODT) still strug- gles when pretrained with low-reward offline data. In this paper, we theoretically analyze the monly used Return-To-Ge (RTG) that's far from the expected return hampers the online fine-tuning process. This problem, however, is well-addressed by the value function and advantage of standard RI. algorithms. As suggested by our analysis, in our experiments, we hence find that simply adding TD3 gradients to the fine- tuning process of ODT effectively improves the online finetuning performance ODT, especially if ODT is pretrained with low-reward offline data. These findings provide new directions further improve decision transformers.
of the decision transformer, showing that the com-
1 Introduction
While Reinforcement Learning (RL) has achieved great success in recent years [55] 31], it is known to struggle with several shortcomings, including training instability when propagating Temporal Difference (TD) error along long trajectories [14], low data efficiency when training from seratch [67], and limited benefits from more modern neural network architectures [12]. The latter point differs significantly from other parts of the machine learning community such as Computer Vision [7] and
Natural Language Processing [11].
To address these issues, Decision Transformers (DTs) have been proposed as an emerging paradigm for RI, introducing more modern transformer architectures into the literature rather than the still widely used Multi-Layer Perceptrons (MLPs). Instead evaluating state and state-action pairs, a DT considers the whole trajectory as sequence complete, and trains offline data in a supervised, auto-regressive way. Upon inception, DT' have been improved in various ways, mostly dealing with architecture changes (37], the token to predict other than return-to-go 22), addressing the problem of being overly optimistic [46), and the inability to stitch together trajectories [5). Significant and encouraging improvements have been reported on those aspects.
However, one fundamental issue has been largely overlooked by the community: offline-to-online RI using decision transformers, i.e., finetuning of decision transformers with online interactions. Offline-to-onee -oninee---onliee RI.
772 41 is widely studied sub-field RI, which combines offline RI. learning from given, fixed trajectory data and online RL data from interactions with the environment. By first training on offline data and then finetuning, the agent can learn policy with much greater data efficiency, while calibrating the out-of-diitribution error from the offline dataset. Unsurprisingly, this sub-field has become popular in recent years.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

While there are numerous works in the dffline-to-nnlin RL sub-field [35 28 62 surprisingly few works have discussed the offline-to-online finetuning ability of decision transformers. While there is work that discusses finetuning of decision transformers predicting encoded future trajectory information [64], and work that finetunes pretrained decision transformers with PPO multi-agent RL (38), the current widely adopted state-of-the-an is the Online Decision Transformer (ODT) [74]: the decision transformer training is continued on online data following the same supervised-learnin paradigm as in offline RL. However, this method struggles with low-reward data, as well as with reaching expert-level performance due to suboptimal trajectories 41 (also see Sec. 4p.
To address this issue and enhance online finetuning of decision transformers, we theoretically analyze the decision transformer based on recent results 77. showing that the commonly used conditioning on a high Return-To To-Go (RTG) that's s far from the expected return hampers results. To fix, we explore the possibility of using tried-ond-true RI. gradients. Testing on multiple environments, we find that simply combining TD3 (21) gradients with the original auto-regressive ODT training paradigm is surprisingly effective: it improves results of ODT, especially if ODT is pretrained with low-reward offline data.
Our contributions are summarized follows:
1) We propose a simple yet effective method to boost the performance of online finetuning of decision transformers, especially if offline data is of medium-to-lue quality;
2) We theoretically analyze the online decision transformer, explain its "policy update" mechanism when using the commonly applied high target RTG, and point out its struggle to work well with online finetuning:
3) We conduct experiments on multiple enviromments, and find that ODT aided by TD3 gradients (and sometimes even the TD3 gradient alone) are surprisingly effective for online finetuning of decision transformers.
2 Preliminaries
Markov Decision Process. Markov Decision Process (MDP) the basic framework of sequential Decision-making An MDP is characterized by five components: the state space S, the action space A, the transition function p. the reward r, and either the discount factor 7 or horizon H. MDPs involve an agen/ making decisions in discrete steps / E {0, 1, J. On step I, the agent receives the current state 8t E S, and samples an action a, A according to its stochustic policy I ((ee))) E A(A), where A is the probability simplex over A, or its deterministic policy ju(;;) E A. Executing the action yields a reward r(2t, dt) E R. and leads to the evolution of the MDP to a new state st+1+ governed by the MDP's transition function p(5t41|5e. dz). The goal of the agent is to maximize the total reward Σ, 7'r(s,,, az), discounted by the discount factor 7 E [o, 1] for infinite steps, or
Σ'= r(s,, az) for finite steps. When the agent ends complete run, it finishes an episode, and the state(-action) data collected during the run is referred to as a trajectory т.
Offline and Online RL. Based on the source of learning data, RL can be roughly categorized into offline and online RL. The former learns from a given finite dataset of state-action- trajectories, while the latter learns from trajectories collected online from the environment. The effort of combining the two is called offline- offline-to-online RL, which first pre-trains policy using offline data, and then continues to finetune the policy using online data with higher efficiency. Our work falls into the category of offline-to-online RL. We focus on improving the decision transformers, instead of g-learning-baser methods which are commonly used in offline-to-onlin RL.
Decision Transformer (DT). The decision transformer represents new paradigm of offline RL, going beyond a TD-error framework. It views trajectory T as a sequence to be auto-regressively completed. The sequence interleaves three types of tokens: returns-to-ge (RTG, the target total return), states, and actions. At step f, the past sequence of context length K is given as the input, i.e., the input is (RTG,Kk. model, which is usually implemented with a GPT-like architecture [II]. The model is trained via supervised learning, considering the past K steps of the trajectory along with the current state and the current return-to-go the feature, and the sequence of all actions a in a segment as the labels. At evaluation time, a desired return RTG specified, since the ground truth future return RTG isn't t known in advance.
RTG,, st), and an action is predicted by the regr-segres
2

---

Online Decision Transformer (ODT). ODT has two stages: offline pre-training which is identical to classic DT training, and online finetuning where trajectories are iteratively collected and the policy is updated via supervised learning. Specifically, the action a, at step L during rollouts is computed by the deterministic policy (T -, RT: T Teval, RTG = RTG or sampled from the stochastic policy xD' (ag]::-TTa T,-T:-1: R T Tovat, RTG = RTG Here, T is the context length (which is Toaal in evaluation), and RTG E IR is the target return-to-go. The data buffer, initialized with offline data, is gradually replaced by online data during finetuning.
When updating the policy, the following loss (we use the deterministic policy as an example, and thus omit the entropy regularizer) is minimized:
Tim
DT (50::: 00.t-1, RTGo:, RTG = RTG T = =
(I)
Note, Timin is the training context length and RTG is the real return-to-go. For better readability, we denote 62+2 Sy 1. 2. y E N Say (ie.,, left exclusive and right inclusive), and similarly { 4.422-... ay } as aazy and (RTG RTG,,,,,,,,,,?! } as RTGay. Specially, index y represents an empty sequence. For example, when / 1, a0:0 is an empty action sequence as the decision transformer is not conditioned on any past action.
One important observation: the decision transformer is inherently eff-policy (the exact policy dis- tribution varies with the sampled starting point, context length and return-to-go), which effectively guides our choice of RL gradients to off off-policy algorithms (see Appendix
C for more details).
TD3. Twin Delayed Deep Deterministic Policy Gradient (TD3) 21 is a state-of-the-ar online off off-policy RL algorithm that learns deterministic policy م (s). It is an improved version of an actor-critic (DDPG (2)) with three adjustments to improve its stability: 1) Clipped double Q- learning, which maintains two crities (estimators for expected return) Qω1 Q+a |S] x |A] - R and uses the smaller of the two values (i.e., min (Q+1: Qø2) ) to form the target for TD-error minimization. Such design prevents overestimation of the Q-value; 2) Policy smoothing, which adds noise when calculating the Q-value for the next action to effectively prevent overfitting; and 3) Delayed update, which updates ft RL less frequently than Qo Qa, to benefit from better Q-value landscape when updating the actor. TD3 also maintains a set of target networks storing old parameters of the actor and critics that are soft-updated with slow exponential moving average updates from the current, active network. In this paper, we adapt this algorithm to fit the decision transformer architecture so that it can be used as auxiliary objective in online finetuning process.
3 Method
This section is organized as follows: we will first provide intuition why RL gradients aid online finetuning of decision transformers (Sec. [.I. and present our method of adding TD3 gradients (Sec. (.2).. To further justify our intuition, we provide a theoretical analysis on how ODT fails to improve during online finetuning when pre-trained with low-reward data (Sec. 3.3
3.1 Why RL Gradients?
In order to understand why RL gradients aid online finetuning of decision transformers, let us consider
an MDP which only has single state sp, one step, one dimensional action a E [-1, 1] (i.e., a bandit with continuous action space) and a simple reward function r(a) (a + 1)² if a < 0 and r(a) = 2a otherwise, as illustrated in Fig. [ In this case, a trajectory can be represented effectively by scalar, which is the action. If the offline dataset for pretraining is of low quality, i.e., all actions in the dataset are either close to -1 or 1, then the decision transformer will obviously not generate trajectories with high RTG after offline training. As consequence, during online finetuning. the new rollout trajectory is very likely to be uninformative about how to reach RTG since is too far from RTG Worse still, it cannot improve locally either, which requires 8a the decision transformer yields exactly the inverse, i.c., ØरTG Since the transformer is not invertible (and even if the transformer is invertible, often the ground truth RTG(a) itself is not), we cannot
ØRTG However,
da
'ODT uses different RTGs for evaluation and online rollouts, but we refer to both as RTG as they are both expert-level expected returns.
3

---

ODT
ODT TD3
Offline data
Action
Target
RTG
True return
Target
True retur
aRTi
Returns-To--e (RTG)
Action a
Unreachable target RTG
Local improvement by Q-function
An overview our work, illustrating why ODT fails to improve with low-return offline data and RL gradients such as TD3 could help. The decision transformer yields gradient BRTG but local policy improvement requires the opposite. i.c.. da current policy conditioning on high target RTG does not actually lead to high real RTG, which is very likely when the target RTG is too far from the pretrained policy and out-of out-of-distribution. By adding small coefficient for RL gradients, the agents can improve locally, which leads to better performance.
Figure 1:
øKIG Therefore, the agent cannot recover if the
30T-DC0C
DD
mocttd HEG
Breund Suth
DDPS
100
150 20 de
38
(a) Reward Curves
(b) Rollout Distribution
Critic
(d) DT Policy
Figure 2: An illustration of simple MDP, showing how RL can infer the direction for improvement, while online DT fails. Panels (a) and (b) show, DDPG and ODT+DDPG manage to maximize reward and find the correct optimal action quickly, while ODT fails do Panel (c) shows how a DDPG/ODT+
critic (from light blue/orange to dark blue/red) manages to fit ground truth reward (green eurve). Panel (d) shows that the ODT policy (changing from light gray to dark) fails discover the hidden reward peak near 0 between two low-reward areas (near- -1 and respectively) contained in the offline data. Meanwhile, ODT+ succeeds in finding the reward peak.
easily estimate the former from the latter. Thus, the hope for policy improvement relies heavily on the generalization of RTG, i.e., policy yielded by high RTGeval indeed leads better policy without any data as evidence, which is not the case with our constructed MDP and dataset.
In contrast, applying traditional RL for continuous action spaces to this setting, either learn value function Q (sg, a)) IR + R. which effectively gives us a direction of action improvement
32 TD3 (21)), an advantage A(so, a) that highlights whether AWAC 44). IQL (28). Either way provides direction which suggests how to change the action locally in order to improve the (estimated) return. In our experiment illustrated in Fig. (see Appendix [ for details), we found that easily solve the aforementioned MDP while ODT fails.
aQ(=g.a)
Ca. (e.g., SAC (25] DDPG
focusing on action improves or worsens the policy (e.g., AWR
RL algorithms like DDPG 32]
Thus, adding RL gradients aids the decision transformer improve from given low RTG trajectories. While one may argue that the celf-supervised training paradigm of ODT [74] can do the same by "prompting" the decision transformer generate a high RTG trajectory, such paradigm still unable to effectively improve the policy of the decision transformer pretrained on data with low RTGs. We provide a theoretical analysis for this in Sec. 3.3 In addition, we also explore the possibility of fixing this problem using other existing algorithms, such as JSRL [59] and slowly growing RTG (i.e., curriculum learning). However, we found that those algorithms cannot address this problem well. See Appendix G.8 for ablations.

---

3.2 Adding TD3 Gradients to ODT
In this work, we mainly consider TD3 [2 as the RL gradient for online finetuning. There are Two reasons for selecting TD3. First, TD3 is a more robust off-policy RL algorithm compared to other off-policy RI algorithms (54). Second, the success of TD3+BC [20) indicates that TD3 is good candidate when combined with supervised learning. A more detailed discussion and empirical comparison to other RL algorithms can be found in Appendix C
Generally, we simply add a weighted standard TD3 actor loss to the decision transformer objective. do this, we follow classic TD3 and additionally train two critic networks Qoi Qaz S x A + R. parameterized by $1 O2 respectively. In the offline pretraining stage, we use the following objective for the actor:
Timis
min EynD ur
Pe aQ b (86s p (S0:;; 00:t-1: RTG RTG RTG = =())+
T
train
(2)
| PT (so.ts d0:-1 RTGo,t, RTG RTG T = acl [
Here, a E {0, 0.1} } is hyperparameter, and the loss sums over the trajectory segment. For critics Qe Qos, the standard TD3 critic loss
T
min EYND 01.00
(Qo, (st, az) - Qmin.e)² + (Q (st, az) - Qmin,))) () with
1=1
Qmin,t r1 7(1 di)
min Qeien (st, clip (war + clip(e. -c, c), alow ohigh) (E(1,2)
where (m0:Tmm++1. +1, G0:T
RTGo:T do:Tmn To:Tmi RTG RTGreal J is the trajectory seg- ment sampled from buffer D that stores the offline dataset. Further, d, indicates whether the trajectory ends on the 1-th step (true is 1, false 0), Qaii the target to fit, Q is produced by the target net- work (stored old parameter), 2; is the context for "next state" at step l. RL is the target network for the actor (i.e.. decision transformer). For an n n-dimensional action, clip(a, xr, V), E IR" E R" I means clip a; to yi 2:] for E (1, 2........,, n). bound for every dimension respectively.
Clow E R. and Chigh E IR" are the lower and upper
To demonstrate the impact on aiding the exploration of a decision transformer, in this work we choose the simplest form of a critic, which is reflective, i.e., only depends on the current state. This essentially makes the Q-value an average of different context lengths sampled from near-uniform distribution (see Appendix D for the detailed reason and distribution for this). The choice is based on the fact that training a transformer based value function estimator quite hard (45) due increased input complexity (i.e., noise from the environment) which leads to reduced stability and slower convergence. In fact, to avoid this difficulty, many recent works on Large Language Models (LLMs) [13 and vision models 488 which finetune with RI. adopt a policy-based algorithm instead of an actor-critic, despite a generally lower variance of the latter. our experiments, we also found such a critic to be much more stable than recurrent critic network (see Appendix G for ablations).
During online finetuning, we again use Eq. and Eq. 33) but always use 0.1 for Eq.
While the training paradigm resembles that of TD3+BC, our proposed method improves upon TD3+BC in the following two ways: 1) Architecture. While TD3+BC uses MLP networks for single steps, we leverage a decision transformer, which is more expressive and can take more context into account when making decisions. 2) Selected instead of indiscrininated behavior cloning. Behavior cloning mimics all data collected without regard to their reward, while the supervised learning process of decision transformer prioritizes trajectories with higher return by conditioning action generation on higher RTG. See Appendix G.9 for an ablation.
3.3
Why Does ODT Fail to Improve the Policy?
As mentioned in Sec. 3.1 is the goal of ODT "prompt" a policy with a high RTG, i.e to improve policy by conditioning high RTG during online rollout. However, beyond the intuition provided
5

---

in Sec. 3.c.3.1 in this section, we will analyze more formally why such a paradigm is unable to improve the policy given offline data filled with low-RTG trajectories.
Our analysis is based on the performance bound proved by Brandfonbrener et al. 7 Given a dataset
drawn from underlying policy /3 and given its RTG distribution P3 (either continuous or discrete), under assumptions (see Appendix E. we have the following right performance bound for a decision transformer with policy T (a|s, RTGeval) conditioned on eTG
Se (< +2 11? af
RTGeval Br=(s1, (......,............. ) (als.RTG,.. [RTGrea]
Here, oz = inf,, PR real = eTGeaal for every initial state $1, > 0 is constant, H is the horizon of the M Based on this tight performance bound, we will show that with high probability, ay grows superlinearly with respect to RTG true, then the RTG term (i.e., the actual return from online rollouts) must decrease to fit into the tight bound, as RTG grows.
To show this, we take a two-step approach: First, we prove that the probability mass of the RTG distribution is concentrated around low RTGs, i.e., event probubility Præ (RTG - Eg 2 cls) for c > O decreases superlinearly with respect to c. For this, we apply the Chebyshev inequality, which yields a bound of O (+) However, without knowledge on PR (s). the variance can be made arbitrarily large by high RTG outliers, hence making the bound meaningless.
Fortunately, we have knowledge about the RTG distribution Pa (RTG|s) from the collected data. If we refer to the maximum RTG in the dataset via RTG and if we assume all rewards are non-negative then all trajectory samples have an RTG in [0, fmax] Thus, with adequate prior distribution, we can state that with high probability 1 - 8, the probability mass is concentrated in the low RTG area. Based on this, we can prove the following lemma:
Lemma 1. (Informal) Assume rewards r(s, a) are bounded in [0, Rmaa] and RT 2 RTG 9max- Then with probability at least 1 5, we have the probability of event Prş bounded as follows:
Prg (RTGeval V 2 cls) SO
R2, T2
(5)
where 5 depends on the number of trajectories in the datasel and prior distribution (see Appendix for a concrete example and more accurate bound). (s(s) is the value function of the underlying policy (a]ss) that generates the dataset, for which we have (1) = Eg(HTG])s)
P
The second step uses the bound of probability mass Pra (RTG 2 cls) to derive the bound for ay. For the discrete case where the possibly obtained RTGs are finite or countably infinite (note, state and action space can still be continuous), this is simple, we have
P9 (RTG- = (s) + cls) = Prş (RTG = (s) + cls) < Prę (RTG 2 +P(s))cls))
(6)
Thus ay inf,, (RTG|(1)) can be conveniently bounded by Lemma For the continuous case, the proof is more involved as probability density P3 (RTG|s) can be very high on an extremely short interval of RTG, making the total probability mass arbitrarily small. However, assuming that Pa is Lipschitz when RTG 2 RTG 9max (i.e., RTG area not covered by dataset), combined with the discrete distribution case, we can still get the following (see Appendix For proof):
Corollary (Informal) If the RTG distribution is discrete (i.e., number of possible different RTGs are at most countably infinite), then with probability at least- 1 - 8, 1
aq grows on the order of (RTG (
with respect RTG For continuous RTG distributions satisfying Lipschitz continuous RTG density pa- aq
grows on the order of D (.)
Here, 2(-) refers to the big- Omega notation (asymptotic lower bound).
4
Experiments
In this section, aim to address the following questions: a) Does our proposed solution for decision transformers indeed improve its ability to cope with low-reward pretraining data. b) Is improving
FFq. 4 is very informal. See Appendix for a more rigorous description. Note use "max" instead of "8max" this is property of the environment and not the dataset.

---

what to predict, while still using supervised learning, the correct way to improve the finetuning ability of decision transformers? e) Does the transformer architecture, combined with RL gradients, work better than TD3+BC? d) Is it better to combine the use of RI and supervised learning, or better to simply abandon the supervised loss in online finetuning? e) How does online decision transformer with TD3 gradient perform compared to other offline RI. algorithms? f) How much does TD3 improve over DDPG which was used in Fig. 2
Baselines. In this section, we mainly compare to six baselines: the widely recognized state-of-the-art DT for online finetuning, Online Decision Transformer (ODT) 74 PDT, baseline improving over ODT by predicting future trajectory information instead of return-to go; TD3+BC [20]. MLP offline RL baseline; TD3, an ablated version of our proposed solution where we use TD3 gradients only for decision transformer finetuming (but only use supervised learning of the actor for offline pretraining);
IQL.
28 one of the most popular offine RI. algorithms that can be used for online finetuning: DDPG 32 +ODT, which is the same as our approach but with DDPG instead of TD3 gradients (for
ablations using SAC 225 IQL. (28] PPO [52). AWAC [40] and AWR 47 see Appendix C.. Each of the baselines corresponds to one of the questions a). b), c). d), e) and f) above.
Metries. We use the normalized average reward (same as D4RL's standard (T9] as the metric, where higher reward indicates better performance. If the final performance is similar, the algorithm with fewer online examples collected to reach that level of performance is better. We report the reward curve, which shows the change of the normalized reward's 's mean and standard deviation with 5 different seeds, with respect to the number of online examples collected. The maximum number of steps collected is capped at 500K (for mujoco) or 1M (for other environments). We also report evaluation results using the rliable 33 library Fig. 7f Appendix B
Experimental Setup. We use the same architecture and hyperparameters such as learning rate (see Appendix F2 for details) ODT (7).. The architecture is transformer with 4 layers and 4 heads in each layer. This translates to around 13M parameters in total. For the critic, we use Multi-Layer Perceptrons (MLPs) with width 256 and two hidden layers and ReLU M activation function. Specially, for the random dataset. we collect trajectories until the total number of steps exceeds 1000 in every epoch, which differs from ODT, where only trajectory per epoch is collected. This is because many random environments, such as hopper, have very short episodes when the agent does not perform well, which could lead to overfitting if only a single trajectory collected per epoch. For fairness, we use this modified rollout for ODT in our experiments as well. Not doing so does not affect ODT results since it does generally not work well on random datasets, but will significantly increase the time to reach certain number of online transitions. After rollout, we train the actor for 300 gradient steps and the critic for 600 steps following TD3's delayed update trick.
4.1 Adroit Environments
Environment and Dataset Setup. We test on four difficult robotic manipulation tasks [49], which are the Pen, Hammer, Door and Relocate environment. For each environment, we test three different datasets: expert, cloned and human, which are generated by finetuned RL policy, an imitation learning policy and human demonstration respectively, See Appendix F.T for details.
Results. Fig. shows the performance of each method on Adroit before and after online finetuning. TD3+BC fails on almost all tasks and often diverges with extremely large Q-value during online finetuning. ODT and PDT perform better but still fall short of the proposed method, TD3+ODT. Note, IQL, TD3 and TD3+ODT all perform decently well (with similar average reward as shown in Tab. 2 in Appendix B. However, we found that TD3 often fails during online finetuning, probably because the environments are complicated and TD3 struggles to recover from a poor policy generated during online exploration (i.e., it has a catastrophic fortgropiicffoogttningg issue). To see whether there is simple fix, in Appendix G.7, we ablate whether an action regularizer pushing towards a pretrain policy similar to TD3+BC helps, but find it to hinder performance increase in other environments. IQL is overall much more stable than TD3, but improves much less during online finetuning than TD3+ODT. ODT can achieve good performance when pretrained on expert data, but struggles with datasets of lower quality, which validates our motivation. DDPG+ODT starts out well in the online finetuning stage but fails quickly, probably because DDPG is less stable compared to TD3.
7

---

Penrexpert-v
Hammenaspertsi
exppett-
Deorexpert-vl
Pen-cloned-ot
Hammer-dooned-11
Doon-cioned-11
Pan-hhumanne
Hammmes--hum-aa annol
Relocatellamanee
Normal ted Reward
Dooi-humanwwl
10
0.9
Transitinn
TD3 + BC
IQL
ODT
PDT
TD3
DDPG+OD1 ODT
TD3 ODD (ours)
Figure 3: Results Adroit 49 environments. The proposed method, TD3+ODT, improves upon baselines. Note that TD3, IQL, and TD3+ODT all perform decently at the beginning of online finetuning, but TD3 fails while TD3+ODT improves much more than IQL. during online finetuning.
Artimaze-umaze-22
Artraze-umaaa-divrrreed
Antmaze-meddum-playyw
Antrmaze- maw.maaeemeeiuu-iiveee--ee
100
75
50
25
03
62
3.6
0.6
60
8.0 82
04
06
86
1.0
3.
02
04
3.6
0.
62
04
0.5
0.0
L5
Online Transitions
+BC
ODT
DDPG-ODT
TD3-ODT [ours)
Figure 4: Reward curves for each method in Antmaze environments. IQL works best the large maze, while our proposed method works the best on the medium maze and umaze. DDPG+ODT works worse than our method and IQL but much better than the rest of the baselines, which again validates our motivation that adding RL gradients to ODT is helpful.
4.2 Antmaze Environments
Environment and Dataset Setup. We further test harder version of the Maze2D environment in D4RL 19 where the pointmass is substituted by a robotic ant. We study six different variants, which umaze, umaze-diverse, medium-play, medium-diverse, large-play and large-diverse.
Results. Fig. lists the results of each method on umaze and medium maze before and after online finetuning (see Appendix for for reward curves and Appendix B for results summary large antmaze). TD3+ODT works the best on umaze and medium maze, and significantly outperforms TD3. This shows that RL gradients alone not enough for offline-to-one
RL of the decision transformer. Though TD3+ODT does not work on large maze, we found that IQL+ODT works decently well. However, choose TD3+ODT in this work because IQL+ODT does work well the random datasets. This is probably because IQL. aims to address the
(OOD) estimation problem (28). which makes it better utilizing offline data but worse at online exploration. See Appendix C for detailed discussion and results. DDPG+ODT works worse than TD3+ODT but much better than baselines except IQL.
4.3 MuJoCo Environments
Environment and Dataset Setup. We further test on four widely recognized standard environ- ments (88) which are the Hopper, Halfcheetah, Walker2d and Ant environment. For each environ- ment, we study three different datasets: medium, medium-replay, and random. The first and second one contain trajectories of decent quality, while the last one is generated with random agent.

---

TD3+BC
IQL
ODT
PDT
TD3
DDPGHODT
after TD3+ODT (ours)
Ho-M-v2
60.24(+4.4)
44.72(2113)
48.44(48.69)
74.48(722.2)
28.98(299.25
41.7G13.18)
(9.07(225:97)
Ho-MR-v2
99.07(+33.39)
62.76(-7.63)
83.29(463.17)
84.53(882.23)
93.72(455.00
32.36(49.9)
95.65(+65.80
Ho-R-v2
8.36(-0.35)
20.42(+12.36)
29.08(26.922)
35,9(+34.67)
75.68473.69
25.12(423,14)
76.13(+74.15)
Ha-M-v2
51.29(+2.73)
37.12/-10.35)
42.27(+19.23))
39.35(439.55)
70.90429.59)
35.69(+14.71)
76.91(+35.3)
Ha-MR-v2
56.5(+13.07)
49.97(46.84)
41.45(+26.77)
31.47(33188
40.599
53.71(+24.91)
73.27(+43:98)
Ha-R-v2
0.74;+0.9)
66.55(+66.3)
59.35(-57.1)
Wa M-v2
(5.34(++349)
65.55(
15.57(+18.47)
63.37(+63.3)
90.49 74)
2.01()69.54)
97.86(+27.06
Wa-MR-v2
83.28(+0.0)
98.99(+28.78)
77.2(+12446)
54.49)454.18)
(00.88(+332554
(04(-60.59)
100.6(+42.54))
Wa-R-v2
6.99(+5.86)
10.67(+44.6)
14.12(+9.82)
15.47(+15.32)
69.91(+66.31)
2.91(22.47)
57.86(453.27)
An-M-v2
129.1
(10.36(114226)
88.1(00.11)
$2.08(+48.47)
125.67((337.55)
10.81(-75.52)
132.0(+41.42))
An-MR-v2
(29.33((41.03)
113. (6(424244)
64(44444)
36.92(432.41) 41)
(33.58(+51.17)
4.05(-87.7)
130.23(+
An-R-v2
67.89(333)47)
12.28(40.97)
24.96(-6.44)
14.38()10.38)
63,47(432.02)
4.93(-26.55)
(1.69(440.31)
Average
68.52(+14.6)
$5.9(47.8)
55.14(+18.58)
41.97(+00.44)
87.644444.95)
22.87(119.22)
86.38(+46.59)
Table 1: Average reward for each method MuloCo environments before and after online finetuning.
The best performance for each environment is highlighted in bold font, and any result 90% of the best performance is underlined. To save space, the name of the environments and datasets are abbreviated as follows: for the environments Ho=Hopper, Ha=HalfCheetah. Wa=Walker2d, An=Ant; for the datasets M-Medium, MR=Medium-Repe R=Random. The format is finetuning)". The proposed solution performs well.
Results. Fig. 6 shows the results of each method on MuJoCo before and after online finetuning. We observe that autoregressive-based algorithms, such as ODT and PDT, fail to improve the policy on MuJoCo environments, especially from low-reward pretraining with random datasets. With RL gradients, TD3+BC and IQL can improve the policy during online finetuning, but less than decision transformer (TD3 and TD3+ODT). In particular, we found IQL to struggle on most random datasets, which are well-solved by decision transformers with TD3 gradients. TD3+ODT still outperforms TD3 with an average final reward of 88.51 vs. 84.23. See Fig.
Appendix for reward curves.
Ablations on a. Fig. (a) shows the result of using different a (i.e., RI. coefficients) on different environments. We observe an increase of a to improve the online finetuning process. However, if is too large, the algorithm may get unstable.
A blations evaluation context length Tevat Fig. 5 (b) shows the result of using different Teval
on halfcheetah- sarrchhmeiium-eee medium-replay-v2 replay-v2 and hammer-cloned--ve The result shows that Teval needs to be balanced between more information for decision- decision-making and potential training instability due to longer context length. As shown in the malfcheetaahmedding coaddduu--pplayyeee replay-v2 result, Teval too long or too short can both lead to performance drops. More ablations are available in Appendix G
Heltcheetsh replyum-rppoy-22
Halfcheetah mwd.mmmmmmmmmiliie
Hammer-cinnd
223
23
2
64
68
04
08
15
Tynacc]
(a) Ablations on a
(b) Ablations on Teval
Figure 5: Panel (a) shows ablations on RL coefficient a. While higher a aids exploration as shown in the pafichettah-mmeduu--llllla the hammer-human-v I case. Panel (b) shows ablations on Tevat. Teval balances training stability and more information for decision-making.
case, it may sometimes introduce instability, which is shown in
5 Related Work
Online Finetuning of Decision Transformers. While there are many works on generalizing decision transformers (e.g.. predicting waypoints (5), goal, or encoded future information instead of return-to-
go [22] 36)), improving the architecture [7 16 53 65] or addressing the overly wvellyoppimmistic [46]
or trajectory stitching issue (33) ) there is surprisingly little work beyond online decision transformers that deals with online finetuning of decision transformers. There is some loosely related literature:
MADT 31 proposes finetune pretrained decision transformers with PPO. PDT 64 also studies
online finetuning with the same training paradigm ODT [74]. QDT [66
uses an offline RL

---

algorithm to re-label returns- to-go for offline datasets. AFDT 76 and STG 75 use decision transformers offline to generate an auxiliary reward and aid the training of online RL algorithms. A few works study in-context learning [33 and meta-learning 600 30 of decision transformers, where improvements with evaluations on new tasks are made possible. However, none of the papers above focuses on addressing the general online finetuning issue of the decision transformer.
Transformers as Backbone for RL. Having witnessed the impressive success of transformers in Computer Vision (CV) 17 and Natural Language Processing (NLP) [.], numerous works also studied the impact of transformers in RL either as model for the agent model 39 50]
38 or as a world However, large portion of state-of-the-aie art work in RL is still based on simple Multi- Layer Perceptrons (MLPs) [35 [28]. This largely because transformers are significantly harder to train and require extra effort 45]. making their ability to better memorize long trajectories [42] harder to realize compared ML.Ps. Further, there are works on using transformers as feature extractors for trajectory 37 45 and works that leverage the common sense of transformer-hacee Large Language Model's for RL priors [T0
9 70
In contrast, our work focuses on improving the new "RL. via paradigm, aiming to merge this paradigm with the benefits of
Supervised learning" (RvS)
classic RL training.
Offline-to-nniinn RL. Offline-to-online RL bridges the gap between offline RL, which heavily de- pends on the quality of existing data while struggling with which requires many interactions and is of low data efficiency. Mainstream tonin-eo--nnlne RL meth-
policies, and online RL,
ods include teacher-student [5T 59
72 and out-of-distribution handling (regularization 21 29 62).
avoidance [28 23 ensembles 2 15 24 ). There are also works on pessimistic Q-value initializa-
tion [69], confidence bounds [66] and a mixture of offiine and online training [56 73). However, all
the aforementioned works are based on Q-learning and don't 't consider decision transformers.
6 Conclusion
In this paper, we point out an under-explored problem in the Decision Transformer (DT) community, ie., online finetuning. To address online finetuning with decision transformer, examine the current ofate-of-the-art, online decision transformer, and point out an issue with low-reward, sub-optimal pretraining. To address the issue, propose mix TD3 gradients with decision transformer training. This combination permits to achieve better results in multiple testbeds. Our work is complement to the current DT literature, and calls out new aspect of improving decision transformers.
Limitations and Future Works. While work theoretically analyzes an ODT issue, the conclusion relies on several assumptions which we expect to remove in future work. Empirically, in this work we propose simple solution orthogonal to existing efforts like architecture improvements and predicting future information rather than return-to-oo. To explore other ideas that could further improve online finetuning of decision transformers, next steps include the study of other environments and other ways to incorporate RL gradients into decision transformers. Other possible avenues for future research include testing our solution image-based environments, and decreasing the additional computational cost compared to ODT (an analysis for the current time cost is provided in Appendix H))
Acknowledgments
This work was supported in part by NSF under Grants 2008387, 2045586, 2106825, MRI 1725729, NIFA Award 8020-67021-32111 search Institute, and the Jump ARCHES endowment through the Health Care Engineering Systems Center at Illinois and the OSF Foundation.
the IBM-Illinoi, Discovery Accelerator Institute, the Toyota Re-
10

---

References
[1] Agarap, A. F. Deep learning using rectified linear units (relu). arXiv preprint arXiv:/803.000 2018.
[2] Agarwal, R., Schuurmans, D., and Norouzi, M. An optimistic perspective on offline reinforce- ment learning. In ICMI, 2020.
[3] Agarwal, R., Schwarzer, M., Castro, P. S., Courville, A., and Bellemare, M. G. Deep reinforce- ment learning at the edge of the statistical precipice. In NewrIPS, 2021.
[4] Ba, J. L., Kiros, J. R., and Hinton, G. E. Layer normalization. arXiv preprint arXiv:1607.06450, 2016.
[5] Badrinath, A., Flet- Flet-Berliac, Y., Nie, A., and Brunskill, E. Waypoint transformer: Reinforcement
learning via supervised learning with intermediate targets. In NeurIPS, 2023.
[6] Bastani, O., Pu, Y., and Solar-Lezama, A. Verifiable reinforcement learning via policy extraction. In NeurlPS, 2018.
[7] Brandfonbrener, D., Bietti, A., Buckman, J., Laroche, R., and Bruna, J. When does return- conditioned supervised learning work for offline reinforcement learning? In NeurlPS, 2022.
[8] Broekman, G., Cheung, V., Pettersson, I.., Schneider, J., Schulman, J., Tang, J., and Zaremba, W. Openai gym. arXiv preprint arXiv:/606.01540. 2016.
[9] Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Chen, X., Choromanski, K., Ding, T., Driess, D., Dubey, A., Finn, C., et al. Rt-2: Vision-language-aaen models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15818, 2023.
[10] Brohan, A., Chebotar, Y., Finn, C., Hausman, K., Herzog, A., Ho, D., Ibarz, J., Irpan, A., Jang, E., Julian, R., et al. Do as can, not as say: Grounding language in robotic affordances. In CoRL, 2023.
[11] Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, D., Dhariwal, P., Neelakantan, A.. Shyam, P.. Sastry, G., Askell, A., et al. Language models are few-shot learners. In NeurIPS, 2020.
[12] Chebotar, Y., Vuong, Q., Hausman, K., Xia, F., Lu, Y., Irpan, A., Kumar, A., Yu, T., Herzog, A., Pertsch, K., et al. Q-transformer: Scalable offline reinforcement learning via autoregressive q-functions. In CoRL, 2023.
[13] Chen, C., Wang, X., Jin, Y., Dong, V. Y., Dong. L., Cao, J.. Liu, Y., and Yan, R. Semi-offlin reinforcement learning for optimized text generation. In /CML, 2023.
[14] Chen, L., Lu, K., Rajeswaran, A., Lee, K., Grover, A., Laskin, M., Abbeel, P., Srinivas, A., and Mordatch, Decision transformer: Reinforeement learning via sequence modeling. In NeurlPS, 2021.
[15] Chen, X., Wang, C., Zhou, Z., and Ross, K. Randomized ensembled double q-lennnin:: Learning fast without model. In ICLR, 2021.
[16] David, S. B., Zimerman, I., Nachmani, E., and Wolf, L. Decision s4: Efficient sequence-based rl via state spaces layers. In ICLR, 2022.
[17] Dosovitskiy, A., Beyer, I.., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
[18] Emmons, S., Eysenbach, B., Kostrikov, I., and Levine, S. Rys: What is essential for offline via supervised learning? In ICLR, 2022.
[19] Fu, J., Kumar, A., Nachum, O., Tucker, G., and Levine, D4rl: Datasets for deep data-driven reinforcement learning, arXiv preprint arXiv:2004.07211 2020.

---

[20] Fujimoto, S. and Gu, S. S. A minimalist approach to offline reinforcement learning. In NeurIPS, 2021.
[21] Fujimoto, Hoof, H., and Meger, D. Addressing function approximation error in actor-critic methods. In JCML, 2018.
[22] Furuta, H., Matsuo, Y., and Gu, S. S. Generalized decision transformer for offline hindsight information matching. In ICLR, 2022.
[23] Garg, D., Hejna, J., Geist, M., and Ermon, Extreme q-learning: Maxent rl without entropy. In ICLR, 2023.
[24] Ghasemipour, S. K. S., Schuurmans, D., and Gu, S. S. Emaq: Expected-max q-learning operator for simple yet effective offline and online rl. ICML, 2021.
[25] Haarnoja, T., Zhou, A.. Abbeel, P., and Levine, Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In ICML, 2018.
[26] Hong, J., Kumar, A., and Levine, S. Confidence-condttionee value functions for offline reinforcement learning. In ICLR, 2023.
[27] Kingma, D. and Ba, J. Adam: A method for stochastic optimization. In ICLR, 2015.
(28] Kostrikov, I., Nair, A., and Levine, S. Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:2 /0,06169, 2021.
[29] Kumar, A., Zhou, A., Tucker, G., and Levine, S. Conservative q-learning for offline reinforce- ment learning. In NeurlPS, 2020.
[30] Lee, J., Xie, A., Pacchiano, A., Chandak, Y., Finn, C., Nachum, O., and Brunskill, E. In-context makiin from supervised pretraining. In JCML Workshop on New Frontiers in Learning. Control, and Dynamical Systems, 2023.
[31] Lee, K.-H., Nachum, O., Yang, M. S., Lee, L.., Freeman, D., Guadarrama, S., Fischer, I,, Xu, W., Jang, E., Michalewski, H., et al. Multi-game decision transformers. In NeurlPS, 2022.
[32] Lillicrap. T. P., Hunt, J. J., Pritzel, A., Heess, N. M. O., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. Continuous control with deep reinforcement learning. In ICLR, 2016.
[33] Lin, L., Bai, Y., and Mei, S. Transformers decision makers: Provable rentoormmet learning via supervised pretraining. In /CLR, 2024.
[34] Liu, H. and Abbeel, Emergent agentic transformer from chain of hindsight experience. In ICML, 2023.
[35] Lyu, J., Ma, X., Li, X., and Lu, Z. Mildly conservative q-learning for offline reinforcement learning. In NeurIPS, 2022.
[36] Ma. Y., Xiao, C., Liang, H., and Hao, Rethinking decision transformer via hierarchical reinforcement learning. arXiv preprint arXiv:2311.00267 2023.
[37] Mao, H., Zhao, R., Chen, H., Hao, J., Chen, Y., Li. D., Zhang, J., and Xiao, Z. Transformer in transformer as backbone for deep reinforcement learning. In AAMAS, 2024.
[38] Meng, L., Wen, M., Yang, Y., Le, C., Li, X., Zhang, W., Wen, Y., Zhang, H., Wang, J., and Xu, B. Offline pre-trained multi-agent decision transformer: One big sequence model tackles all smac tasks. arXiv preprint arXiv:2112 02845, 2021.
[39] Micheli, V., Alonso, E., and Fleuret, Transformers are sample efficient world models. In ICLR, 2023.
[40] Nair, A., Dalal, M., Gupta, A., and Levine, S. Accelcrating online reinforcement learning with offline datasets. arXiv preprint Ttt::2006009355,) 2020.
[41] Nakamoto, M., Zhai, Y., Singh, A., Mark, M. S., Ma, Y., Finn, C., Kumar, A., and Levine, Cal-ql: Calibrated offline rl pre-training for efficient online fine-tuning, In NeurIPS, 2023.
12

---

[42] Ni, T., Ma, M., Eysenbach, B., and Bacon, P.-L. When do transformers shine in rl? decoupling memory from credit assignment. In NeurlPS, 2023.
[43] Oh, J., Guo, Y., Singh, S., and Lee, H. Self-imitation learning. In ICML, 2018.
[44] Ouyang, L.,, Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P. F., Leike, J., and Lowe, R. Training language models to follow instructions with human feedback. In NeurlPS, 2022.
[45] Parisotto, E., Song, F., Rae, J., Pascanu, R., Gulcehre, C., Jayakumar, S., Jaderberg, M., Kaufman, R. I.., Clark, A., Noury, S., et al. Stabilizing transformers for reinforcement learning. In ICML, 2020.
[46] Paster, K., Mcllraith, S., and Ba, J. You can't count luck: Why decision transformers and fail in stochastic environments, In NeurIPS, 2022.
[47] Peng, X. B., Kumar, A., Zhang, G., and Levine, S. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00177 2019.
[48] Pinto, S., Kolesnikov, A., Shi, Y., Beyer, L.,, and Zhai, X. Tuning computer vision models with task rewards. In ICML,, 2023.
[49] Rajeswaran, A., Kumar, V., Gupta, A., Vezzani, G., Schulman, J., Todorov, E., and Levine, Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demon- strations. RSS, 2018.
[50] Robine, J., Höftmann, M., Uelwer, T., and Harmeling, S. Transformer-based world models happy with 100k interactions. In ICLR, 2023.
[51] Schmitt, S., Hudson, J. Zidek, A., Osindero, S., Doersch, C., Czamecki,, W. M., Leibo, J. Z., Kuttler, H., Zisserman, A., Simonyan, K., et al. Kickstarting deep reinforcement learning. arXiv preprint arXiv: 100::8803000 03835, 2018.
[52] Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimiza- tion algorithms. arXiv preprint arXiv: 1777v77707.06347, 2017.
[53] Shang, J., Kahatapitiya, K., Li, X., and Ryoo, M. S. Starformer: Transformer with state-action- reward representations for visual reinforcement learning. In ECCV, 2022.
[54] Sharif, A. and Marijan, D. Evaluating the robustness of deep reinforcement learning for autonomous and adversarial policies multi-agent urban driving environment. QRS, 2021.
[55] Silver, D., Hubert, T., Schrittwieser, J., Antonoglou, I., Lai, M., Guez, A., Lanctot, M., Sifre, L.,
Kumaran, D., Graepel. T., Lillicrap, T. P., Simonyan, K., and Hassabis, D. Mastering chess and shogi by self- play with general reinforcement learning algorithm. Science, 2018.
[56] Song. Y., Zhou, Y.. Sekhari, A., Bagnell, J. A., Krishnamurthy, A., and Sun, W. Hybrid rl: Using both offline and online data make rl efficient. In ICLR, 2023.
[57] Sudhakaran, S. and Risi, S. Skill decision transformer. Foundation Models for Decision Making Workshop al NeurIPS, 2022.
[58] Todorov, E., Erez, T., and Tassa, Y. Mujoco: A physics engine for model- -ased control. In IROS, 2012.
[59] Uchendu, I., Xiao, T., Lu, Y.. Zhu, B., Yan, M., Simon, J., Bennice, M., Fu, C., Ma, C., Jiao, J., al. Jump-start reinforcement learning. In ICMI., 2023.
[60] Wang, Z., Wang, H., and Qi, J. T3gdt: Three-tier tokens guide decision transformer for offline meta reinforcement learning. In Robot Learning Workshop: Pretraining, Fine- nanniTnnnng and Generalization with Large Scale Models in NeurIPS, 2023.
13

---

[61] Wolczyk, M., Cupiat, B., Ostaszewski, M., Bortkiewicz, M., Zajae, M., Pascanu, R., Kuciñski, E., and Milos, Fine-tuning reinforcement learning models is secretly forgetting mitigation problem. In ICMI,, 2024.
[62] Wu, J., Wu, H., Qiu, Z., Wang. J., and Long, M. Supported policy optimization for offline reinforcement learning. In NeurIPS, 2022.
[63] Wu, Y.H.,, Wang, X., and Hamaya, M. Elastic decision transformer. In NeurIPS, 2023.
[64] Xie, Z., Lin, Z., Ye, D., Fu, Q. Wei, Y., and Li, S. Future-conditioned unsupervised pretraining for decision transformer. ICML, 2023.
[65] Xu, M.. Lu, Y.. Shen, Y., Zhang. S., Zhao, D.. and Gan, C. Hyper-decision transformer for efficient online policy adaptation. In ICLR, 2023.
[66] Yamagata, T., Khalil, A., and Santos Rodriguez, R. Q-learning decision transformer: Leveraging dynamic programming for conditional sequence modelling in offline rl. /CML, 2023.
[67] Yan, K. Schwing, A., and Wang. Y.-.. Ceip: Combining explicit and implicit priors for reinforcement learning with demonstrations. In NeurlPS, 2022.
[68] You, Y., Li, J., Reddi, S., Hseu, J., Kumar, S., Bhojanapalli, S., Song. X., Demmel, J., Keutzer, K., and Hsich, C.-J. Large batch optimization for deep learning: Training bert in 76 minutes. In ICLR, 2020.
[69] Yu, Z. and Zhang, Actor-critic alignment for offline-t--online reinforcement learning. In ICMI., 2023.
[70] Yuan, H., Zhang, C., Wang, H., Xie, F., Cai, P., Dong, H., and Lu, Z. Plan4MC: Skill reinforcement learning and planning for open-world Minecraft tasks. In Foundation Models for Decision Making Workshop at NeurIPS 2023, 2023.
[71] Yue, Y., Lu, R., Kang, B., Song, S., and Huang, G. Understanding, predicting and better resolving q-value divergence in offline-rl. NeurIPS, 2024.
[72] Zhang, H., Xu, W., and Yu, H. Policy expansion for bridging offline-to-online reinforcement learning. In ICLR, 2023.
[73] Zheng, H., Luo, X., Wei, P., Song, X., Li, D., and Jiang, J. Adaptive policy learning for offline-to-online reinforcement learning. In AAAI, 2023.
[74] Zheng, Q., Zhang, A., and Grover, A. Online decision transformer. In ICML, 2022.
[75] Zhou, B., Li, K., Jiang, J., and Lu, Z. Learning from visual observation via offline pretrained state-to-go go transformer. In NeurlPS, 2023.
[76] Zhu, D., Wang, Y., Schmidhuber, J., and Elhoseiny, M. Guiding online reinforcement learning with action-free offline pretraining, arXiv preprint arXiv:2301.12870 2023.
14

---

Appendix: Reinforcement Learning Gradients as Vitamin for Online Finetuning Decision Transformers
The Appendix is organized as follows. In Sec. A we discuss the potential positive and negative social impact of the paper. Then, we summarize the performance shown the main paper Sec. After this, we will explain our choice of RI gradientss in the paper in Sec. C and why our critic serves as an average of policies generated by different context lengths in Sec. D We then provide rigorous statements for the theroetical analysis appearing in the paper in Sec. and list the environment details and hyperparameters in Sec. F We then present more experiment and ablation results in Sec. G Finally, we list our computational resource usage and licenses of related assets in Sec. and Sec. respectivvey.
A Broader Societal Impacts
Our work generally helps automation of geeision-making by improving the use of online interaction data of pretrained decision transformer agent. While this effort improves the efficiency of decision- makers and has the potential to boost variety of real-life applications such as robotics and resource allocation, it may also cause several negative social impacts, such as potential job losses, human de-skilling (making humans less capable of making decisions without Al), and misuse of technology (e.g., military).
B Performance Summary
In this section, we summarize the average reward achieve by each method on different environments and datasets, where the result for Adroit is shown in Tab. / and the result for Antmaze is shown in Tab. 3 As the summary table for MuJoCo is already presented in Sec. 4 we show the reward curves In Fig. 6 For more rigorous evaluation, we also report other metrics including the median, InterQuartile Mean (IQM) and optimality gap using the rliable (3] library. See Fig. for details. Breakdown analysis for each enviromment can be downloaded by browsing to https:t:s//. //kaiaanaan9
github. 10/ //////ets/bbeaaddoo... wtw.raaadoorrrraalleeeeeeeeeeeee www.bllleeeee.e rar
TD3-BC
IQL
ODT
PDT
TD3
DDPG+ODT
TD3+DDT (ours)
P.E.
47.88(84.22 84,23)
149.65(3.63)
121,82(+5.48)
25.07(+25.56)
69.74)
120.65( 1165((111.900
P-C-v
3.75(-108)
78.12(+25.01)
22.88(-24.0)
14.05(+12.15)
18.04(-17.39)
-1.4(-81.87)
(23.77(+58..05)
P-H-v
26.77(43.19)
96.5(427.7))
27.55(-13.91)
4.03(40.38)
38.58(-57.71)
2.09(-92.56)
107.1(+1187)
H-E-v1
3.11(-0.2))
126.54(+13.28))
123.07(+12.79)
98.95(498.94)
93.99(331441
-0.24(-177.05)
129.8(+6.34)
HCC-1
0,33(40.03)
2.27(+0.58)
0.84(+0.32)
0.66(40.66)
0.07(-0.69)
0.12(-0.83)
226.39(+124..55
0.17(-0.3)
16.12(+14.18)
0.97(0.13)
32.76(432.75 72))
0.03(-1.11)
0.06(-1.02)
+16.83(+115.88
D.E.v
-0.4(-0.01)
97.57(-7.92)
50.26(+50.14 14)
59.48(199.41)
76.92(-25.56)
26.48()7.772
(03.13(-1.94)
D-C-rl
-0,36(00.01
23.8(+21 66)
5.45(45.37)
(.38(+1.54)
0.17(-488
-0.01(4.45)
+8.23(+53.31)
DH-1I
40.33-0.11
34.64(-29.65)
10.61(+6.69)
0.05(+0.22)
-0.14(-9.22)
65.24(+55.94)
RE
-1.37(+0,22)
105.78(+2811
10L-6(+2.11)
66.57(+66.7)
0.44(-106.67)
10664-006.88)
91.38(-16. 16.19)
R.C.l
0.3(40.0)
L.1(00.97)
0.06(+0.08)
0.03(-0.04)
0.19(.0.29)
0.12(0..32)
0.36(+0.26)
R-H.v
1).8(40.D)
16(+1.5)
0.04(40.05)
07044+0.1)
-..17(00288)
-0.1(-0.27)
1,19900.99)
Average
6.6(7.66)
61,14(10:49)
38.73(3.755
25(4247))
31.3((29.63)
3,1((32996
87.84(33.09)
Table 2: Average reward for each method in Adroit Environments before and after online finetuning.
The best result for each setting is marked in bold font and all results > 90% of the best performance underlined. To save space, the name of the environments and datasets abbreviated follows: P=Pen, H=Hammer, D=Door, R=Relocate for environment, and E=Expert, C=cloned, H=Human for the dataset. It is apparent that while both IQL, TD3 and TD3+ODT perform decently well before online finetuning, our proposed solution significantly outperforms all baselines on the adroit testbed. DDPG+ODT starts out well in the online stage, but fails probably due DDPG's training instability compared to TD3.
C Why Do We Choose TD3 to Provide RL Gradients?
In this section, we provide an ablation analysis on which RL gradient fits the decision transformer architecture best. Fig. Billustrates the result of using a pure RL gradient for online finetuning of pretrained decision transformer (for those RL algorithms with stochastic policy, adopt the same
15

---

TD3+BC
IQL
ODT
PDT
TD3
DDPG-ODT
ID3-ODT fours)
U-v2
0.2((+0.07)
95.99(46.59)
15.921-38.08)
29.94(+29.94)
0.0(+0.0)
95.62(+433622)
-9.59(+83.59)
UD-v2
0.53(+0.33)
44.83-18.177
0.01-52.0)
23.41(114559 25.55(+25.55)
80.0(+42.0)
MP-v2
0,0c+0.0)
91.59(221.59)
0.0(+0.0)
0.01+0.01
70.55(+70.55)
25.66(-14,34)
96.79(+42.79)
MD-v2
0.01(+0.01)
88.63(423.43)
0.0(+0.0)
0.0(+0.0)
20.24(+20.24)
29.14(-16.86)
96.31(+36.31)
LP-22
0.0(+8.0)
51,44996)
0.0(+0.0)
0.0(+0.01
0.01+0.0)
0.0(+0.0)
0.0(+0.0)
LD-v2
0,0(+0.0)
70.6(+32.6)
0.0(+0.0)
0.0(+0.00
Average
0.13(+0.07)
13.83)+12.61)
2.65(-15.0))
4.99(+4.99)
19. 19.39)
28,
62. (((44))
Avg. (U+M)
0.19(+0.08)
80.26(+8.36)
3.98(-22.52)
7.49(+7-49)
29.08(+29.08)
43.466-0.544
93.17(+51.17)
Table 3: Average reward for each method in Antmaze Environments before and after online linetuning. The best result is marked in bold font and all results > 90% fo the best performance are underlined.
To save space, the name of the environments and datasets are abbreviated as follows: U=Umaze, UD=Umaze- Diverse, MP=Medium-PPe Play, MD=Medium-Diie Diverss, LP=Large-Plas and LD=Large- Diverse. U+M=Umaze and Medium maze. Our method performs the best on umaze and medium maze, while IQL performs the best on large maze. Both methods are much better than the rest on average. TD3+BC diverges on antmaze in our experiments.
Hopper-mediumaa
Ant- Ant-medium-22
Walker2d 2ilke-2ddmeduum-22
Hbllichettah-mediie
178
100
56-
Hesper-nddim--mply
Ant-medimm-mplay..
Vaker2d-meelim--mppee
Hallcheetah- meneetat-maduu--pplllee
300
1
Hogpat-random-42
Ant-random-12
Walker26-andom-2
Halkchetth-rondom-2
114
Online Transilions
TD3+BC
IQL
ODT
PDT
TD3
DDPG+ODT
TD3+ODT (ours)
Environments. The TD3 gradient significantly improves the overall performance of the decision transformer; autoregressive algorithms, such as ODT and PDT, fails improve policy in most cases (especially random dataset), while TD3+BC and IQL's improvement during finetuning is generally limited.
Figure 6: Results MuJoCo 58
architecture as ODT which outputs a squashed Gaussian distribution with trainable mean and standard deviation). It is apparent that TD3 [21] and SAC [25] are the RI. algorithms that suit the decision transformer best. Fig. further shows the performance comparison between decision transformer with SAC+ODT mixed gradient and TD3+ODT mixed gradient (both with coefficient 0.1). The result shows that TD3 the better choice when paired with supervised learning.
Note, While PPO is generally closely related with transformers (e.g., Reinforcement Learning from Human Feedback (RLHF) 444) ), and was used in some prior work for online finetuning of decision transformers with small, discrete action space (38), in our experiments, we find PPO generally does not work with the decision transformer architecture. The main reason for this the importance sampling issue: PPO has the following objective for an actor To parameterized by 0:
To(a|s) A TMu (s, a), clip
Ta(a)s) Te.. (als) 10,11-11,
max min [E( Te.. (als)
c, (11+ A (Tta((s)
(7)
Here, New is the policy at the beginning of the training for the current epoch. Normally, the denominator of the red part, Tee (a]s), would be reasonably large, as the data is sampled from that distribution. However, because of the offline nature caused by different RTGs and context lengths at rollout and training time, the denominator for the red part in Eq. 7 could be very small in training, which will lead to very small loss if A (8)a) > O. This introduces significant instability during the training process. Fig. TO illustrates the instability and degrading performance PPO-finetuned decision transformer.
16

---

Mear
Optimaiiy
TDT loursi
PDT
GD1
TD3-BC
(a) Adroit
Optimality
TD3+00T jourst-
DorG4CD cor
FDT
GDT
(b) MuloCo
Median
IOM
Mean
Optmmaity
TDS tourst
DDPG+03T
00T
10L
20
50
120
40
60
100
400
60
190
40
60
50
100
(c) Antmaze (umaze and medium)
Median
IOM
Mean
Optimaliyy Cap
tourst
DOrG4CDT
TD3
10
ID3-BC.
40
80
120
186
200
40
80
120
180
200
150
175
200
225
250
275
15
20
30
46
50
(d) Maze2d
Figure Our main final results re-evaluated using the rliable library with 10000 bootstrap replications. The x-axes are normalized scores (optimality gap is Jo Pr(reward & x)dr). Our method indeed outperforms all baselines Adroit, MuJoCo and antmaze (umaze and medium).
In contrast, RLHE, does exhibit such problem: it does length in evaluation and training. Thus RLHF does not encounter the problem described above.
different return-to-go and context
Besides the RL gradients mentioned above, IQL works well on large Antmazes, also explore the possibility of using IQL as the RL gradient for decision transformer instead of TD3. We found that IQL gradients, when applied the decision transformer, indeed lead much better results antmaze-large large. However, IQL. fails to improve the policy when the offline dataset subsumes very low reward trajectories, which does not conform with motivation. This is probably because IQL, as an offline RL algorithm, aims to address out-of-distribution evaluation issue, which a much more important source of improvement exploration in the online case. Thus, choose TD3 the RL. gradient applied to decision transformer finetuning in this work. Fig. [T shows the result adding TD3 gradient vs. adding IQL gradient on taimazz-large-lla--aee and hopper-random-
D
Why Our Critic Serves an Average of Policies Generated by Different Context Lengths?
As we mentioned in Sec.
When updating a deterministic DT policy, the following loss minimized:
T
(Soct 00:-11 RTGo:1, RTG RTG T t) ag
(8)
where Tirin is the training context length and RTG is the real return-to-go.
17

---

Hoppermediumi2?
unt-medlum-v2
Walker2d medium--c x22
Haltcheetah mellcheetahmmelumv mediummv2
106
26-
24
Nosuer-meduumrrgiinnn
Ant mottmmdiummnno replay
Walker2d wewerrdd-eddu-mmmeee replay v2
Halfcheetah replay v2
106
-10
Happer- randomm-
Ant-random--z
Waller2d.-andemm-
Haftchettah-rnddo-vz
150
2RACE JOARS
22662 2000 Junce
1925e
Online Transitions
ODT
TD3
SAC
AWAC
AWR
PPO
Figure 8: Performance comparison of different, pure RL gradients for online finetuning standard D4RL benchmarks, with TD3 (21), SAC [25), AWAC [40], AWR [47] and PPO [52]. We also plot
ODT's performance as reference. The result shows that generally, TD3 and SAC work the best, while PPO does not work at all.
Hopper medium-e
mediumm?
120
106
Hopper-meduum-reee
Ant madtmmedmmmmme raplay
Malker2d-medumm-mplaaya
140
100
Shtpper-andomwye
Ant: ant.raadom-- V2
Walker2d randomia
Haffcheetah randooh-radddd
106
secco Reward
230006 200000 J0O0E ROCEO 593006
130088 200000 300088 AC0000 53306
10000 20060 260000 40008 5c0300
103006 200090 269006 40669 568000
Online Transitions
TD3+ODT
SAC+ODT
Figure 9: Performance comparison between SAC+ODT and TD3+ODT on standard D4RL bench- marks. TD3+ODT significantly outperforms SAC+ODT.
However, if we consider particular action a, in some trajectory of the dataset, during training (both offline pretraining and online finetuning), the policy generated by the decision transformer fitting a, will be
a1 1 Ot-Tit11, RTG,
RTG RTG
T' is actually sampled from a distribution (1(1, Tirain) over integers between and Tiain inclusive; this distribution U' is introduced by the randomized starting step of the sampled trajectory segments, and is almost a uniform distribution on integers, except that a small asymmetry is created because the context length will be capped at the beginning of each trajectory. See Fig. 12 for illustration.
Therefore, online decision transformers (and plain decision transformers) are actually trained to predict with every context length between 1 and Tiain- During the training process, the context length randomly sampled according to U', and a critic is trained predict an "average value" for the policy generated with context length sampled from U'.
18

---

61.5
1300Grediat
163
200
200
500
(a) Reward
(b) The ratio
(a|s]
Tewd (a)s)
Figure 10: An illustration the training instability and the corresponding performance of PPO-
finetuned decision transformer. The r axis is 300x the number of gradient steps.
Hoppen random-22
Antmaae lange play.22
Normelized Reward
Antraze diverse
106
20
26-
19000
sedion
avices
6/2
IQL+ODT
T03+00T
166
Online Transitions
Figure 11: Performance comparison between IQL+ODT and TD3+ODT. While IQL gradient is good both large antmaze environments, is much easier fall into local minima low-reward offline dataset such as Hopper-.
E
Mathematical Proofs
In this section, will state the theoretical analysis summarized Sec. 3.3 more rigorously. We will first provide an explanation on how the decision transformer improves its policy during online finetuning, linking it to an existing RL method in Sec. E.1 and Sec. E.2 performance in Sec.
We will then bound its
E.1 Preliminaries
Advantage- Weighted Actor Critic (AWAC) [40 is an -ffline-to-online RL algorithm, where the replay buffer is filled with offline data during offline pretraining and then supplemented with online experience during online finetuning. AWAC uses standard Q Q-learning to train the critic Q |S] x |A] + R, and update the actor using weighted behavior cloning, where the weight exponentiated constant).
advantage (i.e., exp
A(s,a) where À
E.2 Connection between Decision Transformer and AWAC
We denote 3 as the underlying policy of the dataset, and P3 as the distribution over states, actions or returns induced by B. Note such P3 can be either discrete continuous. By prior work (7), for decision transformer policy DT we have the following formula holds for any return-to-go RTG e R of the future trajectory:
Pa(aass B(lss) PRTGT))
Ps P3
n (als,, RTG) Pa(als, RTG)
(10)
Ps(RTG]s)
Based on Eq. 11). we have the following lemma:
Lemma For state-action pair (8, a) in an MDP, RTG E TR assume distributions of return-to-ge (RTG) Pp (RTG)s a) and PRTGG 8) are Laplace distributions with scale then for any RTG large
19

---

Step Tiain
Step 2
Train
Causal Transformer
Terais
Tirain
RTG
F2
G
S
Rs
RTG
C
Trajectory
(a) Context length during training
(b) Distribution U' of
Figure 12: a) illustrates the context length T2 during training; Teval the context length of a2 upon sampling and evaluation. It is easy to see that T2 is randomized during training due to the left endpoint of the sampled trajectory segment. b) shows the distribution U' of T2; while Týj for step j 2 Tuain is uniformly sampled between 1 and Tirain because the start of the segment is uniformly sampled, T2 for start of the trajectory. Thus U' not exactly uniform.
step i < Train will be capped
(R(RGGls, PR(RTG]s)
(RTG Q% (s.
PR(RTG)
exp
RTG (s.a)
exp(A (s,
(9(s)
(s.
RTG
(RTG] (RRGG))))
Figure 13: An illustration of how decision transformer policy update RTG is related to AWAC. Though the assumption strong to form the exact same formula, it shows the basic idea of how we link pd (RTG) to its distance to 8 (s) and Q* (8, a) in this paper.
enough (more rigorously, RTG > max(Q (*())) R(G) is updated in the same way as AWAC
Proof. If return-to-go are Laplace distributions, then by the symmetric property of such distributions, the mean of RTG given or (s, a) would be the expected future return, which by definition are value functions for 3, i.e., (s) for (RTG|s and Q (s, a) for Ps(RTG s, a). See Fig. 13 as an illustration. As RTGeval 2 max maxx(Q((s (x, a), V (s)), we have
1
RTG Q2 (9(
PR a) = (RTG) 8, a, exp
2a
(11)
1
RTG - V8(s))
(RTG))) = P. (RTG (,TGGss00 (8,a) = exp
2x
And thus we have (KTG)s.a) (RTG
exp (*"(s.a-VV
exp( (9 (x.(
) where A is the advantage
function.
While the Laplace distribution assumption in this lemma is strict and impractical for real-life applica- tions, it gives us three crucial insights for decision transformer:
For any state state-action pair (s, a), we have E, [RTG s,
E,
(s), which is important property of P3 RTG;
'Decision transformers have sequence of past RTGs as input, but the past sequence can be augmented into the state space to fit into such form.
20

---

For decision transformer, the ability to improve the policy by collecting rollouts with high RTG, similar AWAC, is closely related advantage. In the above lemma, for example, the two Laplace distributions have different scales oy, oQ, we will have the ratio (g(RTG) s.a) (RTG(()))
being exp (2.0)
(=(()) if the distributions are Gaussian, we will get similar results
oq
but with quadratic terms of Qa and V
Different from AWAC, the "policy improvement" of online decision transformers heavily relies the global property of the return-to-go RTG moves further away from Q" and V If the return-to-go is far away from the support of the data, we will have almost no data evaluate P3. and its estimation can be very uncertain (let alone ratios). In this case, it is very unlikely for the decision transformer to collect rollouts with high RTG and get further improvement. This is partly supported by the corollary of Brandfonbrener al. (7]. where the optimal conditioning return they found on evaluation satisfies RTG = V the initial state $1- This is also supported by our single-state MDP experiment discussed in Sec. 3.1 and illustrated in Fig.
Those important insights lead to the intuition that decision transformers, when finetuned online, lack the ability to improve "locally" from low RTGme data, and encourages to study the scenario where P& (s(TTG]s) and Pa a) are small.
E.3 Failure of ODT Policy Update with Low Quality Data
As mentioned in Sec. F.2 we study the performance of decision transformers in online finetuning when Pp (RTGss) and Ps (RTG]s, a) is small. Specially, in this section, r(s, a) is reward function, but a reward distribution conditioned on (s, a); Y2 ~ r(s;, a;) is the reward obtained on i-th step. Such notation takes the noise of reward into consideration and forms more general framework. Also, as the discrete or continuous property of 3 is important in this section, we will use Prş represent probability mass (for discrete distribution or cumulative distribution for continuous distribution) and Pa to represent probability density (for probability density function for continuous distribution).
By prior work 77,, we have the following performance bound fight up to a constant factor for decision transformer for every iteration of updates:
Theorem E.1. For any MDP with transition function p(-|s, a) and reward random variable r(s, a),
and any condition function f, assume the following holds:
Return coverage: Pa(g = [()))ss)) 2 a. for any initial state 81;
Near determinism: for any state-action pair (8,a), a). 3 such that Pr(s'(s, 2 and
3 rross, a) such that Pr(r(s, a) ro(s, a)) 2 -
f(s) ) + r for all when transiting to next state s'. Consistency of f: f()) f
Then we have
H Σ
S (a) +2) H², (12)
Eg Esiepa [f(11]] E
E Enerissaaii
(+))
where a >> 0, 0 are constants, Pini the initial state distribution, and H is the horizon of the MDP. n is the learned policy by Eq. (10)
Proof. See Brandfonbrener et al.
In our case, we define f(s) as follows:
Definition F.2. [(1)) = RTG for all initial states 81, [(8+1)) = fs:) -r; for the (i 1)-th step following i-th step (i {1,2..... 1}).
21

---

Further, we enforce the third assumption in Thm. E. by including the cumulative reward so far in the state space (as described in the paper of Brandforbrener et al. Z Under such definition, we have tight bound on the regret between our target RTG and the true return-to-go RTG Σ Ti by our learned policy at optimal, based on current replay buffer in online finetuning.
We will now prove that under certain assumptions, grows superlinearly with respect to RTG; as the bound is tight, the expected cumulative return term will be decreasing to meet the bounds.
E By B
To do this, start with the following assumptions: Assumption F.3. We assume the following statements be true:
Bounded reward) We assume the reward is bounded in [0, Rmax] for any state-action pairs.
(High evaluation RTG) RTGeval RTG where RTG (max the largest RTG in the dataset of n trajectories generated by S.
(Beta prior) We assign the prior distribution of RTG generated by policy 3 to be Beta distribution (eta 1) for the binomial likelihood of RTG falling on [o, (max] or [RTG (max, Tmma).
Remark E.4. The Beta distribution be changed to any reasonable distribution; distribution only for convenient derivation. Considering the fact that by common sense, trajectories with high return are very hard to obtain, can further strengthen the conclusion by changing the prior distribution.
Beta
We then prove the following lemma:
Lemma 3. Under the Assumption E.3 given underlying policy B of the dataset, for any state with value function (s) and any state-action pair (s, a) with Q-function Q (s, a), any e 2 0 and RTG E R. with probability 6, we have
(1 - RTG Smax + +RRaan T - [s))
Pra(RTG (8(() 2 c(s) S
(13)
(1 R)G Bimax
e R T2
[Q (s.
Pra - Q (s. 2 cls,a)
e²
where 8 1 CDF 1, and the CDF is the cumulative distribution function.
Proof.
With the beta prior assumption in Assumption
E.3 we know that with n samples where
RTG < RTG we have the posterior distribution to be Beta(n + 1, 1), iee, with probability 1 CDF CDFFaa((+LLI))e we have (R 2 RTG (mGx) S e for 0.
Thus, by Chebyshev inequality, we know
Eg RTG - E3 RTG
Pro (RTG- ($( 2 cls) x
c²
(14)
RTR 9mux + RR T [s)]
and a similar conclusion holds for Prş (RTG (s, 2 a). Thus, the probability decays
superlinearly with respect to RTG.
Given this lemma, it remains to connect the bound of Pr, 2 op) to PRT = cp) on RTG, on R. For discrete distribution, the connection is straightforward: Prş (RTG 2 on) 2 Pra <p) = P3 (RTG = co) for any condition & or (8, a).
Thus, immediately get the following corollary:
22

---

Corollary 2. Assume the reward is bounded in [0, Rmar) for for any state and action, and the number of possible different return-to-ge go one can get is finite or countably infinite. Then for the f condition func- tion defined in Def. E.2 with probability of at least 1 1- we have a I (1-e)RTG +e (+)
(RTG-V
i.e, - grows in the order of ( os Remark E.5. While the limitation on return-to- seems strong theoretically, it is very easy to satisfy such assumption practice because it has no requirement on the discreteness of state and action space. Such corollary can be applied on reward discretization with arbitrary precision (including implicit ones by float precision).
For continuous distribution, to bound pa (RTG = co) with (RTG 2 co) RTG, we would need to assume that "peak" does not exist (see Fig. for illustration), i.e. there does not exist cases where Ps is large but Рræ is small. Thus, made the following assumption:
Assumption E.6. (Lipschitzness on uncovered RTG distribution) (RTG)) is Ky-Lipschitz where RTG is larger than RTG 9max.
Remark E.7. The assumption is reasonable because when RTG is larger than any of the RTG in the dataset, we have no data coverage for the performance of the underlying policy / under such RTG, and thus can choose any inductive bias for B.
Remark E.8. Note the Lipschitzness of pa does not rely on the Lipschitzness of the reward function. For example, consider single-state, single-step MDP where we have a uniformly random policy (0, 1) with reward r(a) = 2 1. The reward is clearly not Lipschitz on (0, 1). but the distribution of r is (ro) (rrr)
ar- (ro)
which is Lipschitz on (-o0, 1) as
dr
2y-
E (0, 1).
With such assumption, we have the following corollary:
Corollary 3. Under assumption E.6 for the condition function defined in Def. E.2
we have
of S V2Kv (RTGGul)) -1.5) ie., ay grows in the order of (2RTGG))) eval) with probability of at least 1-
Proof. Under assumption E.6 by Lipschitzness, for any RTG where pa pR(GG)))) > Pos we have (RTG + cls) > Pa Kv for any E [0, Po.
Thus, we know that if RTGg RTG 8max such that pe (RTGols) Po. then we have (RTG 2 RsG > conclusion, we know that
(See Fig. 14 for an illustration). By the contra-positive statement of the above
2ky
Pro (RTG 2 RTGols) S
P 2KV
VRTG 2 RTG,, Pa (RTG|s) S po:
(15)
and RTG is applicable to the inequality above by the high evaluation RTG assumption in Assump-
tion E.3 We then apply the proof lemma 3 but apply the inequality P (x - E[x] N c) B(x-E(x))
instead of Chebyshev inequality which leads to the conclusion.
F Experimental Details
F1 Environment and Dataset Details
E.1.1 Single-State MDP
The single-state MDP studied in Sec. 3.1 has a single state, a single action a E 1, 1]. and a reward function r(a) = (a + 1)2 if a S 0 and r(a) - 2a otherwise.
motivates why RL gradients are useful for online finetuning.
Datasets. The dataset has size of 128, with 100 actions uniformly sampled (-1, 0.95). and the remaining 28 actions uniformly sampled in (0.5, 1). The dataset is designed to conceal the reward peak in the middle. DDPG and ODT+ successfully recognized the reward peak but ODT failed.
23

---

Pp(RTG]s)
(s)
Probability mass pd. 28y
Po
RTG
vP (s)
Figure 14: An illustration of how Lipschitzness the distribution of Ps (RTG|) could link the bound between Pr(RTG (*) cls) and pa (RTG|s)) Note we do not take the left-hand side probability mass of Po into account because the triangle of probability mass could be truncated by (s).
F11.2 Adroit Environments
Environments. Adroit is set of more difficult benchmark than Mujoco in D4RL, and is becoming
increasingly popular in recent offline and offlinett-online RL works 28 23 We test four environ-
ments in adroit in our experiments: pen, hammer, door and relocate. Fig. 15 shows an illustration of the four environments.
(a) Pen
(b) Hammer
(c) Door
(d) Relocate
Figure 15: Illustration of Adroit environments used in Sec. based OpenAl Gym
and
D4RL 19
1. Pen. Pen is locomotion environment where the agent needs control a robotic hand to manipulate pen, such that its orientation matches the target. It has 24-dimensional action space, each of which controls joint the wrist fingers. The state space is 45-dimensional, which contains the pose of the palm, the angular position of the joints, and the pose of the target and current pen.
2. Hammer. Hammer is an environment where the agent needs to control robotic hand to pick up a hammer and use it to drive nail into a board. The action space 26-dimensional, each of which corresponds to a joint the hand. The state space is 46-dimensional, which describes the angular position of the fingers, the pose of the palm, and the status of hammer and nail.
3. Door. In the door environment, the agent needs use robotic hand to open door by undoing the latch and swinging it. The environment has 28-dimensional action space, which are the absolute angular positions of the hand joints. It also has 39-dimensional observation space which describes each joint, the pose of the palm, and the door with its latch.
4. Relocate. In the relocate environment, the agent needs to control robotic hand to move ball from its initial location towards a goal, both of which are randomized in the environment. The environment has a 30-dimensional action space which describes the angular position of the joints on the hand, and -i-dimmesioonl space which describes the hand well as the ball and target.

---

Dataset
Size
Normalized Reward
Pen Pen-expert-vi
499106
107.40 = 55.65
Pen-cloned-vl
499886
108.63 + 122.43
Pen-human-vl
4800
202.69 + 154.48
Hammer-expert-vl
999800
96.95 + 50.65
Hammer-cloned-vi
999872
8.11 + 23.35
Hammer-human-vl
10948
23.80 3
Door-expert-vl
999800
101.19 16.31
16.31
Door-cloned-yl
999939
12.29 = 18.35
Door-human-vl
6504
28.35 13.88
Relocate-expert-v
999800
102.25 19.83
Relocate-cloned-vl
999724
28.99 = 42.88
Relocate-human-Vl
9614
87.22 + 21.28
Table 4: The size and the average and standard deviation of the normalized reward of the Adroit datasets from D4RL 19 used in our experiments.
Datasets. For each of the four environments, we test our method across three different qualities of datasets: expert, cloned and human, all of which provided by the DAPG [49] repository. The expert dataset is generated by fine-tuned RL policy; the cloned dataset is collected from an imitation policy on the demonstrations from the other two datasets; and the human dataset is collected from human demonstrations. Tab. shows the size and average reward of each dataset.
F.1.3 Antmaze Environments
Environments. Antmaze is a more difficult version of Maze2D, where the agent controls robotic ant instead of point mass through the maze. has 27 dimensional-stall space and 8-dimensional action space. We test our method on six variants of antmaze: Umaze, Umaze-Diverse, Medium-Plas Medium-Diverse Large-Play and Large-Diverse, where "Umaze". "Medium" and "Large" describes the size of the maze (see Fig. [6 for illustration), and the "Diverse" and "Play" describes the type of the dataset. More specifically, "Diverse" means that in the offline dataset, the starting point and the goal of the agent are randomly generated, while "Play" means that the goal is generated by handcraft design. "Umaze" without suffix is the simplest environment where both starting point and the goal are fixed.
+
Open
(b) Umaze
(c) Medium
(d) Large
Illustration of mazes in antmaze and maze2D environment, where the red point is the goal and the green point is the current location of the agent.
Figure 16:
Datasets. Similar to Adroit and MuloCo, we test our method on datasets provided by D4RL. Tab. 5 shows the size and normalized reward of each dataset. Note, following IQL (28) and CQL 29 we conduct reward shaping: subtracting from all rewards in the dataset and environment the value 1 during training both our method and baselines to provide denser reward signal all antmaze environments. However, still count original sparse reward when comparing the performance.
F.1.4 MuJoCo
Environments. We test our method on four widely used environments: Hopper, Halfcheetah, Walker2d and Ant. Fig. shows illustration of the four environments.
25

---

Dataset
Size
Normalized Reward
Antmaze- Umaze-v2
998573
86.14 34.55
Ontmaze-Umaze-Divernee
999000
3.48 + 18.32
Autmaze-Mediiu-Pllay-22
999000
90.85 28.83
Autnare-Meddun--Divrssee-
999000
66.29 47.27
Antmaze-1arge-Playy-2
999000
92.73 25.97
Antmaze-Large-iverrse
999000
86.17 34.52
Table 5: The size and the average and standard deviation of the normalized reward of the Antmaze
datasets from D4RL 19 used in our experiments.
(a) Hopper
(b) Halfcheetah
(c) Ant
(d) Walker2d
Figure 17: Illustration of MuJoCo environments used Sec. based on OpenAl Gym D4RL
and
19.
1. Hopper. Hopper is locomotion task on a 2D vertical plane, where the agent manipulates single-legged robot to hop forward. Its state is I1-dimensional, which describes the angle and velocity for the robot's joints. Its action is 3-dimensional, which corresponds to the torques applied on the three joints for the current time step respectively.
2. Halfcheetah. Halfcheetah also 2D environment which requires the agent control cheetah-like robot to run forward. The states are 17-dimensional, containing the coordinate and velocity of the joints The actions are 6-dimensional, which control the torques on the joints of the robot.
Ant. In Ant, agent controls a four-legged 8-DoF robotic ant to walk 3D environment and tries to move forward. It has 111-dimensional state space describing the coordinates and velocities of the joints.
4. Walker2d. Walker2d is 2D environment in which the agent needs to manipulate 8-DoF two-legged robot to walk forward under the agent's control. Its state space 27 27-dimensional.
Datasets. We test method across three different qualities of datasets: medium, medium-replay and random. The medium dataset contains trajectories collected by an agent trained with RI., but early-stopped medium-level performance. The medium-replay dataset is the collection of trajectories sampled in the training process of the agent mentioned above. The random dataset contains trajectories collected by agent with random policy, Tab, []shows the size and normalized reward of each dataset.
E.1.5 Maze2D Environments
Environments. Maze2D is another set of D4RL environment, where the agent needs to control a point mass to navigate through 2D maze and arrive fixed goal. It has a 4-dimensional state space describing its coordinate and velocity, and a 2-dimensional action describing its acceleration. The reward is determined by its current distance to the goal. We test method four variants of maze: Open, Umaze, Medium and Large with increasing difficulty. The map of each maze is illustrated in Fig. [6 Maze2D environment is tested in Sec. G.2
Datasets. We again test our method on datasets provided by D4RI.. Tab. shows the size and normalizeed reward of each dataset.
26

---

Dataset
Size
Normalized Reward
Hopper medium-v2
999906
44.32 = 12.27
402000
14.98 16.32
Hopper-random--2
999996
1.19 1.16
HalfCheeetah-mecduu--z
1000000
40.68 5.12
MaiCCheetal-meddiuu--eeg----
202000
HallChecetah-randdom-22
1000000
0.07
Walker2d-mediume
999995
62.09 23.83
302000
14.84 19.48
Walker2d
malker2d-medium mediim--rndoom- random-v2
999997
0.01 = 0.09
Ant- aet-medium--2
999946
80.30 - 35.82
302000
30.95 + 31.66
Ant-medium-randome
999930
6.36 + 10.07
Table The size and the average and standard deviation the normalized reward of the MuJoCo datasets from D4RL [9 used in our experiments.
Dataset
Size
Normalized Reward
Maze2D-Open-e
999999
30.08 + 50.17
999869
-12.55 9.82
MaazeDDMMddimm Maze2D-Large-vl
1999733
-3.46 = 3.95
3999692
-1.71 2.87
Table 7: The size and the average and standard deviation of the normalized reward of the Maze2D datasets from D4RL 19 used our experiments.
F.2 Hyperparameters
F2211 Single-State MDP
For all networks, we use a simple MDP with two hidden layers of width 128, and ReLU [D] as activation function. We add Tanh activation function to limit the output for ODT and the actor of DDPG to [-1, 1]. For both methods, we use Adam (27 as the optimizer, and the learning rate is set to 10-3 We pretrain epochs offline data (20 gradient steps) and 16 epochs for online finetuning, with a batch size of 32 for gradient update and collect 64 new rollout states for each epoch (thus train 2n + 4 steps for the n-th online finetuning epoch). RTG is set at 1, which serves the (constant) input for ODT rollout and DDPG actor. Both DDPG and ODT uses deterministic actor with exploration noise uniform in [-0.01, during online rollouts.
F.2.2 Other Experiments
Tab. 8 summarizes hyperparameters that are common across all environments, and Tab. Summmarizes hyperparameters that are different across environments. For environments that exist ODT 77,, we follow the hyperparameters from ODT medium environments. We did not use positional embedding as suggested by ODT (74). Specially, for antmaze, we remove most (all but 10) 1-step trajectories, because the size of the replay buffer for decision transformers is controlled by the number of trajectories, and antmaze dataset contains a large number of I-step trajectories due to its data generation mechanism (immediately terminate episode when the agent is close the goal, but do not reset the agent location). Also, we add Layernorm 4 after each hidden layer of the critic for Adroit, Maze and Antmaze environments, according to Yue et al. '7l's advice. We found that such practice stabilizes the training process (see Sec. G.5 for ablation).
For ODT and TD3 baseline, we use the same code our TD3+ODT, while setting coefficients for RL and supervised gradients accordingly. For PDT baseline, we use the default hyperparameter PDT paper, and pretrain PDT for 40K steps for all experiments. For TD3+BC and IQL., we use the default hyperparameter their codebase, and pretrain them for 1M steps for all experiments (remaining the same as that in the codebase).
27

---

Hyperparam
Value
# dim of embedding dimensions
5T2
# of attention heads per layer # of transformer layers Dropout
4
4
0.1
Actor Optimizer
LAMB
668
# steps collected per epoch
1000 (random dataset), 1 trajectory (others)
Actor activation function
ReLU
Scheduler
10a steps, linear warmup
Critic layer
2
Critic width
256
Critic activation function
ReLU
Batch size
256
actor update per epoch
300
Online exploration noise
0.1
TD3 policy noise
0.2
TD3 noise clip
0.5
TD3 target update ratio
0.005
Table The common hyperparameters across all environments used in our experiments.
Tem
Tiat
RTG.a
RTGatt
Pretmin Online
Ir..
In
Weight decay # pretrain steps Buffer size
Hupper
20
5
361X)
7200
0.1
0.99
10"
10"
0.0005
5K
1K
HallCheetah
20
5
6000
12000
0.1
0.99
10
10-4
10"4
5K
1K
Walker2d
3
Pen
12000
12000
0.0002
40K
Hammer
16000
16000
0.1
0.99
0.0002
10-
10
40K
5K
Door
4000)
4000
0.1
0.99
0,0002
10"
10-4
40K
5K
Relocate
5000
5000
0.1
0.99
0.0002
10
10
40K
5K
Maze2D Maze2D.Open
120
120
0.99
0.0002
10-4
10
40K
5K
-Umaze
-Medium
60
60
0,0002
40K
5K
-Lagg
60
60
0,0002
10
40K
5K
Antmaze-U Umaazzze-Uaazee
-100
100
0.998
0,0002
10
10
40K
2K
-Medium
200
200
0.1
0.1
0.998
0.0002
10-
10-4
200K
2K
-Large
500
5500
0.998
0.0002
10-4
10-4
200K
2K
Table 9: Environment-specinn hyperparameters, where T Tiain and Teval stands for training and eval-
uation context length, RTG and RTGeatiee represents RTG during evaluation and online rollout respectively, a is the coefficient for RL gradient, is the discount factor, Ir. is the critic learning rate, and Ir, is the actor learning rate. Buffer size counted the number of trajectories. Note RTGs of antmaze have been modified according to our reward shaping.
G
More Results
G.1 Delayed Reward
Though we have tested MuloCo environments in Sec. 4 it is worth noting that many offline RL algorithms have addressed the MuloCo benchmark quite well [ 23 Thus, also tested settings where RL struggles to obtain good performance to further analyze the performance of using RI. gradients for decision transformers.
Environment and Experimental Setup. In this experiment, we use the same experiment and dataset as in Sec. 4 exxeept for one major difference: the rewards are not given immediately after each step. Instead, the cumulative reward during a short period M steps is given only at the end of the period, while the rewards observed by the agents within a period are all zero. We adopt such setting from prior influential work [33) which creates a rpward setting where RL algorithms struggle. We test M in this experiment.
Results. We use the same baselines as that in Sec. Tab. 10 summarizes the performance of each method. Generally, DT with TD3 gradient still works very well, much better than ODT. While
28

---

TD3-BC
IQL
ODT
PDT
1D3
ID3+ODT (ours)
Ho-M-v2
50.26-1.3))
43.091-21.288)
92.01+43.03
83.50481.73)
82.1((+179)
82.9[(18.661
Ho-MR-v2
99.85(+42.33)
79.63(-4.72)
85.07(467.6)
82.2(-80.21)
90.81(+18.79)
98.55(+67.1)
Ho-R-v2
8.8(+0.3)
18.49(+10.62)
30,58(+28.37)
25. 25.1(-23,86)
75.55(+73,57)
52.38(450.4)
50.45(+2.56)
40.31(66.8)
42.07(+20.499
38.71(338.66)
65,92(+24.91)
(6.33(+26.1)
Hn-MR-v2
83.59))
47.01((3.46)
19,45(+22.30)
32.55(+32.76)
28.42)+
49.95(+20.22
Ho-R-v2
42.11(4228775 75)
342.34
2.166-0.07)
-0.85(+0.96)
52.24(+-49.99)
48. 18.791+46.33 S4)
Wa va
84.73(+2.19)
62.01(-16.59)
76.21(+3.65)
59.59()
86.42(+18.88)
(9..77)
61.1(-118)
86.67()13.93)
76.96(45.25)
53.73(37.177
(6.38(+344422)
91.93(+27.92)
Wa-R-v2
6.78(+4.37)
7.13(+1.22)
7.93(+3.73)
18.35(+1182)
57.43(+53.32)
56.72(+51.72)
An.M-v2
(14.21(+18.09)
103.34(+6.4)
80.29( 0.83)
49.89(+45.92)
(18.21(+300111
(11.67(221.55))
An-MR-v2
322.11(+33.56)
103.5(+2491)
84.61(-2.13)
42.58( 39.17)
112.2((23.98) (3)9))
(16.6(228.99)
An-R-v2
77.41(+21.34)
10.77(40.3)
22.55(-8.82)
17.37(+13.69)
49.3(+17.87)
(3.00((+)1554
Average
64.28(+12.17)
53.07(3.311)
53.4(+15.22)
41.92(+39.52)
78.68(+35.15)
76.31(+33.51)
Table 10: Average reward for each method in MuJoCo Environments before and after online finetuning with delayed rewards. To save space, the name of the environments and datasets are compressed, where Ho=Hopper, Ha=HalfCheetah, Wa=Walker2d, An=Ant for the environment. and M=Medium, MR=Medium Replayy R=Random for the dataset. The format is 'fmaa(+increasee) The best result for each setting highlighted in bold font, and any result 90% of the best performance underlined.
Hopper-medium--2
Art-medium-22
Walker Walker2d-eedlum-2
Halfcheetah-mddimm>
300
Hopper-medlum-npplee
Ant:medium-ropaie repp-y-aplly
Wbiker22-mediummmpppl
Halttheetal-meduum roplay-v2
100
sce
12
20-
Hopper-andomm-2
Art-random-22
Waller2ddrandomm22
Hallcheetah-randomy
170
66-
vaice
Online Transitions
TD3+BC
IQL
ODT
PDT
TD3
TD3+ODT (ours)
Figure 18: Reward curves for MuloCo environments with delayed reward.
TD3+BC works well in several scenarios, it struggles random environments. See Fig. 18 for
reward curves.
G.2 Maze2D Environment
We test on navigation tasks in D4RL 19 where the agents need to control a pointmass through four different mazes: Open, Umaze, Medium and Large with different dataset respectively. Fig. lists the performance of each method on Maze2D before and after online finetuning, and Tab. 11 summarizes the performance before and after online finetuning. The result shows that our method again significantly outperforms autoregressive-basee algorithms such as ODT and PDT, which validates our motivation Sec. 3.1 DDPG+ODT works similarly well as TD3+ODT in this environment with simple state and action space.
TD3+BC
IQL
ODT
PDT
TD3
DDPG+ODT
TD3+ODT (ouns)
Open-v0
350.58-0.22)
576.06(24.588)
574.4(4306.60)
515.97(48599)
430.6I(-73.36)
574.03(496.02)
574.32((29.57)
Umaze-v2
90.91(-63.15)
43.19(46.37)
139.45(113.4)
140.1(-107.76
Medium-v2
96.91(463.2)
177.61497.31)
26.11((12.77)
80.36(+81.49)
184.55(+138.57)
133.03(495.2))
136.63(491 69)
Large-v2
132.66(438.21) 21)
214.02(-143.21)
20.37(+17.25)
38.744439.14)
232.9++218.22)
170.09(+151444)
189.71(+170.17
Average
167.77(441.09)
281.92(+94.97)
(66.02(485.74)
199.98)1190.02)
252.53(+103.9)
254.15((114.05)
260 260.19(499: 19(19988
Table II: Average reward for each method in Maze2D Environments before and after online finetuning.
Our method works slightly worse than IQL but better than all other baselines.
29

---

Maze2d-open-v0
Mazeedduumaze-1
Mazezd-medium-vl
Mazezd-large-wl
200
20-
146
Online Transitions
TD3 BC
ODT PDT
TD3
DDPG CDT
TD3+ODT (ours)
Figure 19: Reward curve for each method in Maze2D Environments. Again, autoregressive algorithms such ODT and PDT does not perform well in this case.
Hammer-cloned-vl
Normalized Reward
100
0.0
0.2
0.4
0.6
0.8
Online Transitions
1ee
traainn20
T_train=10
T_train=5
T_train=3
T. T_train=2
T_train=1
Figure 20: The reward curves on hammer-clonedee with different T Tirain and Toaal While longer Timn leads to faster convergence in this environment, runs with too long Tman are also unstable.
G.3 Ablations on training context length Tirain
Fig. 20 shows the result of using different context lengths on hammer-cloned-vl environment (in this experiment, we use Teval = 1 to demonstrate the effect of more different Turinn.. It is shown from the experiment that the selection account and training stability; while longer Timin brings faster convergence when growing from 1 to the reward curves with T Train E {10, 20} oscillates more than that with Tiain = 5.
Tiain needs balanced between more information taken into
G.4 Longer Training Process
In some environments, such as hopper-random-v2. walker2d-random-yy and ant-random-v2, our proposed method still seems to be improving after 500K online samples. In Fig. 21 we show the finetuning result of our proposed method with more online transitions, which effectively shows that method has greater potential in online finetuning when finetuned for more gradient steps.
G.5 The Effect of Layernorm
As have mentioned in Sec. F.2 suggested by Yue al. 71 apply Layernorm 4 critic networks for environment other than MuloCo for better stability in training. In our experiment, we found that it greatly stabilizes the critic complicated environments such Adroit, but makes online policy improvement less efficient on casier MuJoCo environments. Fig. 22 shows the performance and critic MSE loss comparison shown that layernorm helps stabilizes online finetuning in some cases such as pen-cloned-vl, but hinders performance increase on other environments such halfcheetah-mediune
some environments with and without layernorm;
clearly
G.6 Recurrent Critic
As mentioned in Sec. 3.2, we use reflexive critics (i.e., critics that only take the current state nand action) to add RL gradients to decision transformer, and this creates average effect among policies generated by different context lengths (see Sec. D in the appendix for detail). this section, we explore recurrent critic by substituting the MIP critic using a LSTM, such that for trajectory segment, the evaluation for the i-th action a; is based on all state-action pairs (81, a1)...... (8:-1, ax-1) and found that recurrent critics much less stable than
current state s;. As shown in Fig. 23
30

---

Happer-random-22
Walker2ddraaddmm22
Ant-random-v2
Normalized Reward
75-
20-
25-
146
Transitions
Lec
OD3
Figure 21: The reward curves of method when finetuned for more steps (we only report curves until the black line Fig. 6. is clearly shown that our method has greater potential for improvement when finetuned for more steps.
Nomakeel tenves
20
Halfcheetah medium 12
600
Hafthhetat- med.um realoy-2
Nimalcol Nowe
150
Pen clonediv
m
Pen-llone v]
420
202
Online Tansttion
Figure 22: The reward curve and critic MSE loss comparison between runs with and without layer- norm. Layernorm effectively stabilizes online finetuning in pen-cloned-vl, but hinders performance increase in halfcheetah Ialfchectah-medium-y
reflexive critics, and the instability increases as the training context length Turin grows; on the contrary, reflexive critic can well-handle the case where Tirin is long.
G.7 Regularizer for Pure TD3 Gradients
In the Adroit environment results discussed in Sec. 4 we found that the baseline of ODT finetuned using pure TD3 gradients struggles due to catastrophic forgetting. Inspired by Wolezyk al. [6], we test whether adding KL regularizer can fix the forgetting problem. Though our policy is deterministic, we can approximately interpret the policy as Gaussian with very small variance. Thus, a KL regularizer can be simply added using co ||a 4ntall3 where a is the current action and aold is the action predicted by the pretrained policy. We set co 0.05 and test this method on the Adroit cloned and expert dataset. We illustrate the result in Fig. 24 We find that the KL regularizer effectively addresses the issue on expert environments for both TD3 and TD3+ODT. But it can sometimes hinder the policy improvement of TD3+ODT with low return during online finetuning.
Normalized Reward
60-
60-
20320
40359
80003
$5002
130300
Online Transitions
T_train
T_train
T_train 20
Refrexive (T_train 20)
Figure 23: The performance of reflexive critic recurrent critic hopper-medium-vv. It clearly shown that recurrent critic is much harder to train, and its performance decreases as the training context length Train grows.
31

---

Parv clonediv1
Hammer cloned v1
Relecate cloned v2
Door dooned
100
140
66-
60-
20-
Pen-expertwv
expertvvl
Relocate ep
Deorexppetivv
Nomm licoo Romain
125
15
60-
28-
0.6
04
0.6
Transitions
DDT-JSRL (orode
GBT
T03+KL
T03+-O0T (OUPS)+R)
GBT (ruminnum))
TD3
TD3+000T (CUPS)
The result of ODT with better exploration (only in cloned dataset) and TD3/TD3+ODT forgetting mitigation. The result shows that 1) ODT with curriculum RTG does not work; 2) even with exploration supported by an oracle, ODT can still fail on some environments such as hammer; 3) JSRL with the pretrained policy does not work for forgetting mitigation; 4) KL regularizer effectively addresses the issue on expert environments for both TD3 and TD3+ODT, but it can hinder the improvement of TD3+ODT with low return.
Figure 24:
G.8 Other possible exploration improvement techniques
In Sec. 3.1
we state that ODT cannot explore the high-RTG region when pretrained with low-quality offline data, and we ran simple experiment to verify this (Fig. 2}. In this section, we test two potential alternatives for addressing the exploration problem: JSRL and curriculum learning.
For JSRL, an expert policy is used for the first steps in an episode, before ODT takes over. We set n 100 (100% max episode length for adroit pen, 50% max episode length for other adroit environments) initially, and apply exponential decay rate of 0.99 for every episode. We test two settings of ISRL: the expert policy being the offline pretrained policy, and the expert policy being oracle, i.e., an IQL policy trained on the Adroit expert dataset.
For currieulum learning, we ODT with a gradually increasing target RTG with the current RTG for rollouts being RTGeval - 0.99-V (RTGeval RTGama) Here, is the number of episodes sampled in online stage, and RTGána is the average RTG of the offline dataset.
Results are summarized in Fig. 24 the task is too hard and cannot be improved by random exploration without gradient guidance. Further, even with oracle exploration, ODT is not guaranteed to succeed: fails on the hammer environment where TD3+ODT succeeds, probably because of insufficient expert-level data and inability to improve with random exploration.
We found that curriculum RTG does not work, probably because
G.9 Ablations the Architecture
In this section, we further examine the source of the performance gain of our method compared to TD3+BC. There are two key differences as stated Sec. (.2 The architecture and RL via Supervised (RvS) learning [18 We can hence assess two more baselines: TD3+BC with our transformer architecture and TD3+RvS using TD3+BC's architecture. We present the ablation result on the Adroit cloned environment in Fig. 25 The result shows that only TD3+BC with our architecture works (albeit still worse than our method). We hypothesize that this is because a simple MLP is hard to model the complicated policy which takes both RTG and state into account.
To further assess simply adding more layers to the MLP works, we conduct an ablation on number of layers for TD3+RvS. The result is illustrated in Fig. 26 It shows that simply adding few layers to the MI.P does not aid performance. We speculate that it is probably the transformer architecture that helps modeling the
policy.
32

---

Par clr-clonddvv1
Hammen cloned v/
Relecate clonediv1
Moon-doone--g vt
Leet Reward
125
115-
30-
03
0.2
044
6.6
6.6
1.3
0.8
6.2
3.6
6.0
60
62
34
0.5
0.0
Online Transitions
007 TD3+BC
1D3 PVS
TD3+ BC Gour (ch)
TD3+001 (ours)
Figure 25: The result ODT and TD3+BC ablations (TD3+RVS, DDPG+ODT, TD3+BC with our architecture and curriculum RTG for ODT) Adroit environments. The result shows that only TD3+BC with our architecture works. However, it remains worse than our method.
Pen-expert-v1
Harmieer-eeppe
melocate--xxperiie
Doon-expert-vl
100
135
00-
50-
Online Transition
TD3+-R5
T01-RV5
00T
D3 (oum)
Figure 26: Results of adding more layers to TD3+RvS. The result shows that simply adding MLP layers does not help TD3+RvS match the performance of ODT and TD3+ODT.
H Computational Resources
We conduct all experiments with single NVIDIA RTX 2080Ti GPU on an Ubuntu 18.04 server cquipped with 72 Intel Xeon Gold 6254 CPUs 3. 10GHz. Mujoco experiments takes about 6 hours, and the bottleneck is the gradient update: about 50% time is spent backpropagation and update of parameters. Our critic appended to ODT only takes up about 20% time to train, which 90% of the critic training time is spent decision transformer inference to get action for "next state". For the actor, the training overhead of our method is negligible since only contains an MLP critic inference get the Q-value. Therefore, overall our method only uses 20% extra time compared to ODT for training, but attains much better results.
Dataset and Algorithm Licenses
Our code is developed upon multiple algorithm repositories and environment testbeds.
Algorithm Repositories. We implement our method on the basis of online decision transformer repository, which has CC BY-NC 4.0 license. We also refer to IQL. [28] PDT 64 and TD3+BC (0)
repository when running baselines, all of which have MIT licenses.
Environment Testbeds. We utilize OpenAl gym 8 MuJoCo [58) and D4RL 11 as testbed, which have an MIT license, an Apache-2.0 license, and an Apache-2.0 license respectively.
33

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Please refer to the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: See Sec. d
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
34

---

Justification:
See Appendix
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: See Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
35

---

Answer: [Yes]
Justification: See httpp://githuu: com/KaiiYn229//RL http:://iiiiiio....///a/aa3////2 _s____________________________ Vitamin, aetamin_for_.. fwr__oooooneeooo
Decision_Transtia Transforrrrss for the code. The data available in public D4RL dataset.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm ec/ public/guidese https//ooooSuuuusssssooooooo
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips. //nips.cc/ppl wwcppblic/guticcccccccceee.oo.pe /eeggudddsseCeebbbiniinn ssiiisoliccyyyyyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: See Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: See Sec. and Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
36

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
Our research in the paper fully conforms with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
See Appendix
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
37

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: See Appendix
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
38

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We attach our code in the supplementary material, and has pub- lished it Github httpa///ggttuub..oooaaaaaaaaaaaa
_a____a______aa__ Vitamin_foor _0nlne_ne_____
Transfoorres
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
39


Reviews:
[{"review_id": "tNm55imB3V", "reviewer": "Reviewer_AdzZ", "summary": "This paper addresses the challenge of online finetuning of online decision transformers (ODT). Theoretical results are provided to show the target return-to-go can hamper finetune. The authors propose to have TD3 gradients added to ODT finetune and improve its performance especially when pretrained with low-reward data. Extensive empirical results are provided to show the proposed method can achieve stronger or competitive results across a large number of environments.", "questions": "- Will alternative strategies that improve exploration at finetune phase help ODT to the same extent? \n- The random datasets are not commonly tested in previous papers, when the baselines are tested on these datasets, are the hyperparameters tuned for these datasets?", "limitations": "Some limitations are discussed in section 6.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "**originality**\n- The idea to combine TD3 gradient and ODT training is quite novel. The theoretical results can also be considered novel findings. \n\n**quality**\n- Overall good quality, the paper is well-written. Figures are easy to read and extensive experiments and ablations are provided. \n\n**clarity**\n- Overall clear and easy to follow. \n\n**significance**\n- The empirical results help improve our understanding of DT methods and how to better finetune them. The results are quite strong. The extensive experiments across different benchmarks make the results more convincing. The theoretical results add to significance.", "weaknesses": "- Maybe I missed something but I feel it might be good to have ODT baselines that try to tackle online finetuning with alternative (and more naive) methods. For example, authors argue that a main difficulty when finetuning DT is that when pretrained with low-reward data, the target RTG at finetuning stage is simply hard to obtain. But if we provide a realistic target RTG initially (based on pretrain data), and gradually increase it, will that help performance? Another thing is if we have a better exploration policy, will that help ODT finetune to the same extent? \n- Adding the TD3 component can make the training more complex and slower.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for appreciating our work. Below are responses to questions:\n\n**Q1. test ODT baselines that tackle online finetuning with alternative methods, e.g., better exploration policy and gradually increasing target RTG.**\n\nGreat suggestions! Currently, ODT explores due to the stochasticity of its policy. We enhance ODT in two ways and provide results in the global pdf: 1) ODT+gradually increasing target RTG (which is similar to curriculum learning in RL); and 2) a **cheating** baseline ODT with **oracle exploration**, i.e., jump-start RL [2], using as the guide policy an expert policy trained via IQL on expert data. Concretely, the expert policy is used for the first $n$ steps in an episode, before ODT takes over. We set $n=100$ (100% max episode length for adroit pen, 50% max episode length for other adroit environments) initially, and apply an exponential decay rate of $0.99$ for every episode.\n\nResults are summarized in Fig. 1 of the global pdf. Curriculum RTG does not work, probably because the task is too hard and cannot be improved by random exploration without gradient guidance. Also, even with oracle exploration, ODT is not guaranteed to succeed: it fails on the hammer environment where TD3+ODT succeeds, probably because of insufficient expert-level data and an inability to improve with random exploration. \n\n**Q2. TD3 slows down training.**\n\n**Our method is only 20% slower than ODT without TD3 gradients.** In Appendix H, we discussed the impact of the introduced RL gradient on training: the critic only takes about 20% of the time to train, while the actor overhead is negligible (since it only contains an MLP critic inference to get the Q-value). We found that on our machine, for the MuJoCo experiment, ODT requires about 6 hours to train. We will add part of the discussion in Appendix H to the limitation section of the revised paper.\n\n**Q3. Are the hyperparameters tuned on the random datasets?**\n\nNo, as suggested in Tab. 9 in the appendix, we use the same hyperparameters for every variant (e.g., medium / medium-replay / random, expert / human / cloned) on the same environment. Note, our parameters on the random datasets are aligned with the ODT medium environment and not tuned further.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "c5v31NK0JD", "reviewer": "Reviewer_1gFB", "summary": "The authors introduce a novel framework for improving the performance of Online Decision Transformer through adding TD3 gradients to the fine-tuning ODT objective. This is motivated by a theoretical analysis of ODT, that highlights an issue with low-reward, sub-optimal pretraining. The authors also provide an extensive empirical investigation.", "questions": "* Can the authors comment or perform an ablation with ODT+DDPG for all the main experiments as well?\n* What is the impact of context length on your approach?\n* For the main experiments how many random seed runs are each experimental result over? E.g. in Figure 2.", "limitations": "Limitations are adequately discussed in section 6.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Paper is clearly written, and well motivated. The detailed explanation on the preliminaries was useful to the reader.\n* The contribution of TD3+ODT appears novel, original, and of high significance to the community.", "weaknesses": "* Overclaim of the contribution of “We propose a simple yet effective method to boost the performance of online finetuning of decision transformers.” The empirical results of relocate-expert-v1 show the opposite, where ODT outperforms TD3+ODT. Perhaps the authors can discuss this result and/or refine the claim to environments with particular properties where the expectation is that TD3+ODT outperforms ODT.\n* Minor: Section 3.1 could highlight the conceptual figure of the plot of Figure 1, c, to help the reader understand the graph at the beginning of section 3.1.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for appreciating our work. Below are responses to questions:\n\n**Q1. Overclaim of “boosting performance”, and the need to refine claims with properties where TD3+ODT should outperform ODT.**\n\nThanks for pointing this out. While TD3+ODT is generally better than ODT, we are happy to rephrase our claim to reflect that there are some cases where ODT performs better than TD3+ODT. To be specific, we expect ODT to struggle (either fail completely or fail to further improve) with medium-to-low quality offline data. This is supported by our result on adroit environments with cloned/human dataset, antmaze, and MuJoCo (especially medium-replay and random where there are many trajectories with low ground truth RTGs). In contrast, if offline data has good quality (e.g., adroit expert), or the reward signal is sparse (e.g., delayed reward in Appendix G.1) so that RL struggles, then we expect ODT to work as good or better.\n\n**Q2. [minor] Figure in Section 3.1 could be improved for better understanding.**\n\nThanks a lot for the advice! We will modify the figure accordingly in the revised version.\n\n**Q3. Ablations on DDPG+ODT.**\n\nGreat suggestion! Unfortunately, due to the rebuttal time limit, we can only provide ODT+DDPG results on some environments for now. We will provide results of ODT+DDPG on all additional environments in the revised version. The current results are shown in Fig. 2 of the global pdf. We find DDPG+ODT does not work. We speculate that this is due to the rougher landscape of the estimated Q-value without smoothing, delayed update, and double Q-learning by TD3.\n\n**Q4. Impact of context length on the approach.**\n\nWe conducted ablations on both the context length at training time ($T_{\\text{train}}$) and evaluation / rollout time ($T_{\\text{eval}}$). The former is shown in Fig. 18 of Appendix G.3, and the latter is shown in Fig. 4 (b). Generally, we found that there is a tradeoff between more information and training stability: while longer context length allows the decision transformer to utilize more information, it also causes increased input complexity, i.e., noise from the environment leads to reduced stability and slower convergence in training. Note, this differs from LLM works, where context length improves generalization thanks to extremely high expressivity, huge amounts of data, and less noisy evaluation. This is especially the case when bootstrapping is involved, as the fitting target depends on its own output. In fact, in prior work, Parisotto et al. [4] suggest in their Sec. 3.1 that if a recurrent agent can first learn a Markovian policy and value function at the start of its training, then the training process could be stabilized.\n\n\n**Q5. Number of random seeds.**\n\nWe use 3 seeds for all experiments (a few ablations without standard deviation use only 1 seed). Note, even without ablations this amounts to a large number of experiments: 1) we use 46 different datasets for our main results [4 for maze, 24 for mujoco ({random, medium-replay, medium}x{hopper, halfcheetah, walker2d, ant}x{normal, delayed reward}), 12 for adroit ({pen, hammer, door, relocate}x{expert, cloned, human}), and 6 for antmaze]; and 2) we test 6 methods. This amounts to 800+ runs (46 datasets, 6 methods, 3 seeds), each of which requires 6-8h on average (IQL and TD3+BC need less time, but PDT and other methods on complex environments such as adroit require more).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BESzyvBEkz", "reviewer": "Reviewer_gVzs", "summary": "This paper presents a method for fine-tuning decision transformers (DT) online. The proposal is to integrate TD3 with DT such that the gradients from TD3 can help the online policy to explore highly rewarding trajectories, hence further improving the agent performance. It can be seen as a variant of TD3+BC with the MLP being replaced by a transformer. The experimental results show the proposed method outperforms several baselines, and is effective even when the offline dataset is of low quality, a setting which traditional online DT cannot do well.", "questions": "- Could you please explain more about the point (2) in line 182: how does DT prioritize trajectories with higher return?", "limitations": "- Compared to TD3+BC, the proposed method uses transformer instead of MLP. It costs much more computation to gain better performance. Such limitation is not mentioned in the paper.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The topic of improving DT for online fine-tuning is worth exploring. The way this paper approaches the topic is reasonable.\n- The empirical results are good.", "weaknesses": "- The TD3 gradients for policy update are not clear. For example, in equation 2, it seems that $Q$ is a constant if you optimize $\\mu^{RL}$? Or is $a_t$ predicted by $\\mu^{RL}$?\n- Some notations are a bit confusing. For example, there are RTG, RTG_real, RTG_eval. Better to clearly define them first.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for appreciating our work. Below are responses to questions:\n\n**Q1. TD3 gradient for policy update is not clear.**\n\nThanks for pointing this out! There is a typo in Eq. (2) which should read as follows:\n\n$$\\min\\_{\\mu^{\\text{DT}}}\\mathbb{E}\\_{\\tau\\sim D}\\Bigg[\\frac{1}{T\\_{\\text{train}}}\\sum\\_{t=1}^{T\\_{\\text{train}}}\\left[-\\alpha Q\\_{\\phi\\_1}(s\\_t,\\mu^{\\text{DT}}(s\\_{0:t},a\\_{0:t-1},\\text{RTG}\\_{0:t},\\text{RTG}=\\text{RTG}\\_{\\text{real}})) + \\|\\|\\mu^{\\text{DT}}(s\\_{0:t},a\\_{0:t-1},\\text{RTG}\\_{0:t},\\text{RTG}=\\text{RTG}\\_{\\text{real}})-a\\_t\\|\\|\\_2^2\\right]\\Bigg],$$\n\ni.e., the $a_t$ in Q should be the action generated by the decision transformer.\n\n\n\n**Q2. Some notations are confusing.**\n\nWe introduced RTG in line 82-83 (“RTG, the target total return”), RTG_eval in line 88 (“a desired return RTG_eval is specified”) and line 96, and RTG_real in line 100 (the real return-to-go). We understand that they are not introduced concurrently, which makes it a bit confusing. We will modify this in the revised version.\n\n**Q3. How does DT prioritize trajectories with higher return as mentioned in point (2) in line 182?**\n\nDT prioritizes trajectories because the actions are learned to be generated conditioned on the Return-To-Go (RTG), and we use high RTG as the condition in evaluation. Consider as an example training with two sets of trajectories extending from the same state: one set with ground truth RTG being 0 and the other set with ground truth RTG being 1. If inference asks DT to generate trajectories conditioned on RTG=1, then the generated trajectory will be more similar to the latter set of training trajectories. \n\n**Q4. The limitation that the use of transformers requires more computational cost is not mentioned.**\n\nThanks for pointing this out. We are happy to include a discussion in the limitation section in the revised version. Meanwhile, we would like to mention two points:\n\n1. Our work aims to analyze and improve ODT. We thus compare computational cost primarily to ODT instead of MLP-based solutions;\n\n2. In Appendix H we stated that our proposed solution is only marginally (20%) slower than ODT: while the use of RL gradients slows down training, the actor training overhead is negligible (since it only contains an MLP critic inference to get the Q-value), and the critic only takes about 20% time to train. We found that on our machine, for the MuJoCo experiment, ODT requires about 6 hours to train.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "a7tlEGvZGP", "reviewer": "Reviewer_NWdD", "summary": "The authors consider the problem of online fine-tuning decision transformers. Current approaches for this problem do not work well when the offline data is low-quality. The authors analyze the online DT algorithm theoretically and show why DT-induced policy update would not work if the RTG used for conditioning is much higher than RTG seen in the dataset. They then propose combining the ODT update with RL gradients, in particular, the ones obtained from the TD3 model. This approach was then tested through Adroit, Antmaze, and Mujoco experiments, where the method obtained strong results. Finally, the authors perform additional analyses and make improvements to their proposed approach.", "questions": "Currently, I would be inclined to accept the paper if not for the problems with the evaluation. As such, I would like to ask the authors:\n- Please discuss and address the questions about the evaluation setting, as raised in the Weaknesses section above. If this is sufficiently resolved, I'd be happy to increase the score.\n- Please consider introducing stronger baselines. This is not as crucial, but it would make the paper stronger.\n\nLess important but interesting to discuss:\n- The authors say: \"training a transformer-based value function estimator is quite hard.\" I agree with this opinion based on my own experiences, but I don't think there has been a thorough discussion of this in the literature. The papers referenced by the authors do not really address this issue well in my opinion. So, it's not a weakness of the paper, but I would like to ask the authors where they think the problems of value learning in transformers come from (or maybe even their online training in general).", "limitations": "Although the authors do discuss the limitations of their work, I think there are at least two more points that should be discussed:\n* The method has not been tested on image-based datasets and environments.\n* There is an increased computational cost as compared to a regular ODT, since we have to train the value functions.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The work considers the important problem of finetuning decision transformers. I see this as a significant issue, as we need strong online fine-tuning methods to better leverage pre-trained models.\n- The proposed method seems to achieve quite good performance in the presented experiments, at least when not taking the statistical significance of the results into account. In my opinion, the authors also consider a sufficient set of benchmarks and baselines.\n- The study is backed by theoretical results.\n- Related work is well-described to the best of my knowledge.\n- The appendix includes many additional results and extended analysis that might be of interest to the reader.", "weaknesses": "- The empirical results seem to be noisy, and a more careful approach to statistical significance is needed. Looking at, e.g., Figure 5, I'm not sure if TD3+ODT is better than TD3 as the plots overlap quite a lot, but the corresponding Table 1 makes it look like TD3+ODT is a clear winner. \n    - This includes bolding \"the best solution\" in some of the tables. If one of the solutions achieves a score of 101 and the other of 100, in the RL regime, it is usually not possible to tell which one is statistically better and the fair thing to do is to bold both. \n    - I didn't find the information on how many seeds are used for each experiment, and this is very important in the RL context.\n    - In summary, I would suggest the authors adopt the recommended practices of [1] to have a clearer picture of the results.  \n- I think the baselines considered here should be made stronger\n    - The TD3 baseline collapses right after fine-tuning starts in many cases. It looks like it might be caused by forgetting which was shown to be a problem in RL fine-tuning [2]. I wonder if combining TD3 with some forgetting mitigation technique would help.\n    - The authors point out that their method improves on TD3+BC in two ways: architecture (MLP -> DT) and objective (BC -> RvS). I think these two axes of improvement should be investigated separately. That is, introducing two additional baselines would show us where the improvements come from: TD3+BC+Transformer and TD3+RvS (with MLP rather than DT). \n\n[1] Agarwal, Rishabh, et al. \"Deep reinforcement learning at the edge of the statistical precipice.\" Advances in neural information processing systems 34 (2021): 29304-29320. \\\n[2] Wolczyk, Maciej, et al. \"Fine-tuning Reinforcement Learning Models is Secretly a Forgetting Mitigation Problem.\" Forty-first International Conference on Machine Learning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for appreciating our work. Here are our responses:\n\n**Q1. Results are noisy, and a more careful approach to statistical significance is needed.**\n\nThanks for pointing this out! We understand the importance of reliable evaluation and statistical significance. For this we reported standard deviations in our learning curves in all main results (Fig. 2,3,5) and most ablations (Fig. 6-7, 9, 16-17, 19). We provided Tab. 1 so that readers can focus on the final performance and its gain over pretraining.  \n\nTo make our reported results more reliable, according to the reviewer’s advice, we evaluated all main results (MuJoCo, adroit, maze, antmaze) using the rliable library, as suggested by the paper mentioned in the review. Results are provided in Fig. 3 of the global pdf, which shows: our method indeed outperforms all baselines on adroit, MuJoCo, and umaze/medium antmaze. Also, we will update Tab. 1-3, 10-11 by highlighting scores >=90% of the best reward with bold-face following MAHALO [1].\n\n**Q2. Number of random seeds.**\n\nThanks for pointing this out. We use 3 seeds for all experiments (a few ablations without standard deviation use only 1 seed). Note, even without ablations this amounts to a large number of experiments: 1) we use 46 different datasets for our main results [4 for maze, 24 for MuJoCo ({random, medium-replay, medium}x{hopper, halfcheetah, walker2d, ant}x{normal, delayed reward}), 12 for adroit ({pen, hammer, door, relocate}x{expert, cloned, human}), and 6 for antmaze]; and 2) we test 6 methods. This amounts to 800+ runs (46 datasets, 6 methods, 3 seeds), each of which requires 6-8h on average (IQL and TD3+BC need less time, but PDT and other methods on complex environments such as adroit require more).\n\n**Q3. Try adding forgetting mitigation methods onto TD3 baseline.**\n\nGreat suggestion! The paper mentioned by the reviewer is pretty recent. It discusses 4 types of forgetting mitigation methods: Behavioral Cloning (BC), Kick Starting (KS), Episodic Memory (EM), and Elastic Weight Consolidation (EWC). Our TD3 baseline and the MLP TD3+BC baseline already use EM as a mitigation method (quote, “simply keeping the examples from the pre-trained task in the replay buffer when training on the new task”). \n\nNote, since TD3 is deterministic without log-likelihood, adapting EWC is out of scope because it requires the Fisher information matrix, which in turn needs the gradient of the log-likelihood. This is also true for BC and KS, which are auxiliary losses of KL-divergences between policies on states drawn from different samplers. However, since BC/KS are essentially soft constraints that prevent the current policy from being too different from the pretrained policy, we can implement BC/KS on continuous action space for deterministic policies by substituting KL divergence with Euclidean distance between actions predicted by the current and pretrained policies. Here, we add a $0.05\\cdot\\|\\|a_{\\text{new}}-a_{\\text{old}}\\|\\|_2^2$ penalty term to the actor loss, which is a gradual shift from BC (sampled from offline dataset) to KS (sampled from online rollouts). \n\nTo see whether other forgetting mitigation methods work, we also implemented jump-start RL [2]. Concretely, the pretrained policy is used for the first $n$ steps in an episode, before ODT takes over. We set $n=100$ (100% max episode length for pen, 50% max episode length for other adroit environments) initially, and apply an exponential decay rate of $0.99$ for every episode.\n\nWe evaluate both methods on adroit environments with the expert dataset (where the forgetting issue is serious) and the cloned dataset (where the agent needs to improve from low-return policies). Results are summarized in Fig. 1. We find that 1) jump-start RL struggles (possibly because it does not directly prevent out-of-distribution updates), and 2) while BC/KS effectively improves both TD3 and TD3+ODT on expert datasets, it also hinders online policy improvement on most cloned datasets.\n   \n**Q4. Try other baselines such as TD3+BC+transformer and TD3+RvS.**\n\nWe would like to point out that our work is focused on analyzing and improving ODT. Thus the main baseline for this work is ODT instead of TD3+BC. That being said, we agree that TD3+BC+transformer and TD3+RvS are interesting baselines to test. The results are shown in Fig. 2 in the global pdf. We test the methods on adroit environments with the cloned dataset. The results show that TD3+RvS slightly improves upon TD3+BC, while TD3+BC+transformer improves more. We speculate that an MLP is not expressive enough to model the policy change over different RTGs. Note, both suggestions still fall short of our method.\n\n**Q5. Where does the problem of value learning come from?**\n\nGood question! Generally, we found that there is a tradeoff between more information and training stability: while longer context length allows the decision transformer to utilize more information, it also causes increased input complexity, i.e., noise from the environment leads to reduced stability and slower convergence in training. Note, this differs from LLM works, where context length improves generalization thanks to extremely high expressivity, huge amounts of data, and less noisy evaluation. This is especially the case with bootstrapping involved, as the fitting target depends on its own output. In fact, Parisotto et al. [4] suggest that if a recurrent agent can first learn a Markovian policy and value function, then the following training could be stabilized.\n\n**Q6. Some limitations are left out, e.g., lack of image-based testbed and increased computational cost.**\n\nThanks for pointing these out! Those are important questions, and we will discuss those in a revised limitation section. Regarding computational cost, we discussed the impact of the RL gradient on training in Appendix H: we found that our method is only marginally (20%) slower than ODT. For the MuJoCo experiment, ODT needs ~6 hours to train on our machine.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers, ACs and SACs for valuable feedback. We are delighted that our idea was appreciated as novel (AdzZ, 1gFB), well-motivated (1gFB), valuable (NWdD, gVzs, 1gFB) and backed by theoretical foundations (AdzZ, NWdD), the literature review was appreciated as well-described (NWdD), and results were referred to as strong (AdzZ, gVzs) and sufficiently extensive (AdzZ, NWdD). Also, reviewers unanimously rate the presentation of our work positively (AdzZ gives “excellent”, all other reviewers give “good”).

We answer common questions here:

**Q1. Number of random seeds (1gFB, NWdD).** We use 3 seeds for all experiments (a few ablations without standard deviation use only 1 seed). Note, even without ablations this amounts to a large number of experiments: 1) we use 46 different datasets for our main results [4 for maze, 24 for mujoco ({random, medium-replay, medium}x{hopper, halfcheetah, walker2d, ant}x{normal, delayed reward}), 12 for adroit ({pen, hammer, door, relocate}x{expert, cloned, human}), and 6 for antmaze]; and 2) we test 6 methods. This amounts to 800+ runs (46 datasets, 6 methods, 3 seeds), each of which requires 6-8h on average (IQL and TD3+BC need less time, but PDT and other methods on complex environments such as adroit require more).

**Q2. Our design slows down training (AdzZ, gVzs, NWdD).** We thank the reviewers for pointing this out, and we are happy to include the discussion in the limitation section. Meanwhile, we note two points:

1. Our work aims to analyze and improve ODT. We thus compare computational cost primarily to ODT instead of MLP-based solutions;

2. In Appendix H we stated that our proposed solution is only marginally (20%) slower than ODT: while the use of RL gradients slows down training, the actor training overhead is negligible (since it only contains an MLP critic inference to get the Q-value), and the critic only takes about 20% time to train. We found that on our machine, for the MuJoCo experiment, ODT requires about 6 hours to run. 

**Q3. Additional ablations (AdzZ, 1gFB, NWdD).** We also provide a variety of new experimental ablations, which are summarized in the global pdf (experiment 1 and 5 mentioned below are in Fig. 1; experiment 2, 3, 6 are in Fig. 2; experiment 4 is in Fig. 3).

**1. Better exploration by “cheating”: letting an expert take over in early steps (reviewer AdzZ).** Even with an oracle (“cheating”) exploration strategy, ODT can still fail in some cases where our method prevails, likely because of insufficient expert-level data and an inability to improve with random exploration.  

**2. ODT baselines which gradually increase target RTG (reviewer AdzZ).** We find that such curriculum learning does not work, probably because the task is too hard and cannot be improved by random exploration without gradient guidance.

**3. ODT+DDPG (reviewer 1gFB).** We find that ODT+DDPG does not work. We speculate that this is due to a less stable Q-value landscape during training.

**4. Better evaluation using rliable library (reviewer NWdD).** We re-evaluate all our main results using the rliable library, and find that our method still generally outperforms other baselines, especially in adroit environments.

**5. TD3 with forgetting mitigation issues implemented via KL regularizer and jump-start RL where the guide policy is the pretrained policy (reviewer NWdD).** Jump-start RL with the pretrained policy does not work well, probably because it does not directly prevent out-of-distribution policy updates. KL regularizer effectively mitigates forgetting for both TD3 and TD3+ODT, but it also hinders policy improvement of the TD3+ODT policy pretrained on low-RTG offline data.

**6. TD3+BC+transformer and TD3+RvS (reviewer NWdD).** We find that TD3+BC+transformer works well in the adroit environment, albeit still worse than our proposed method. TD3+RvS does not work: it only slightly outperforms TD3+BC. We speculate that this is because an MLP is not expressive enough to model the policy change over different RTGs.

Finally, we list the papers referred to in our responses:

**References**

[1] A. Li et al. MAHALO: Unifying Offline Reinforcement Learning and Imitation Learning from Observations. In ICML, 2023.

[2] I. Uchendu et al. Jump-Start Reinforcement Learning. In ICML, 2023.

[3] S. Emmons et al. RVS: What is Essential for Offline RL via Supervised Learning? In ICLR, 2022.

[4] E. Parisotto et al. Stabilizing Transformers for Reinforcement Learning. In ICML, 2020.

Meta-review: This paper has been unanimously recommended for acceptance. It tackles an interesting problem—the offline-to-online problem in reinforcement learning—applied to the decision transformer and provides some theoretical and empirical results. In one way or another, the rebuttal addressed most of the authors' concerns, and I'm recommending the acceptance of this paper.
Decision: Accept (spotlight)