Title: Reinforcement Learning Gradients as Vitamin for Online Finetuning Decision Transformers
Venue: NeurIPS 2024
Accepted: yes
Keywords: Decision transformer

reinforcement learning
Abstract: Decision Transformers have recently emerged as a new and compelling paradigm for offline Reinforcement Learning (RL), completing a trajectory in an autoregressive way. While improvements have been made to overcome  initial shortcomings, online finetuning of decision transformers has been surprisingly under-explored. The widely adopted state-of-the-art Online Decision Transformer (ODT) still struggles when pretrained with low-reward offline data. In this paper, we theoretically analyze the online-finetuning of the decision transformer,  showing that the commonly used Return-To-Go (RTG) that's far from the expected return hampers the online fine-tuning process. This problem, however, is well-addressed by the value function and advantage of standard RL algorithms. As suggested by our analysis, in our experiments, we hence find that simply adding TD3 gradients to the finetuning process of ODT effectively improves the online finetuning performance of ODT, especially if ODT is pretrained with low-reward offline data. These findings provide new directions to further improve decision transformers.

Reinforcement Learning Gradients as Vitamin for Online Finetuning Decision Transformers

Abstract

Decision Transformers have recently emerged as a new and compelling paradigm for offline Reinforcement Learning (RL), completing a trajectory in an autoregressive way. While improvements have been made to overcome initial shortcomings, online finetuning of decision transformers has been surprisingly under-explored. The widely adopted state-of-the-art Online Decision Transformer (ODT) still struggles when pretrained with low-reward offline data. In this paper, we theoretically analyze the online-finetuning of the decision transformer, showing that the commonly used Return-To-Go (RTG) that's far from the expected return hampers the online fine-tuning process. This problem, however, is well-addressed by the value function and advantage of standard RL algorithms. As suggested by our analysis, in our experiments, we hence find that simply adding TD3 gradients to the finetuning process of ODT effectively improves the online finetuning performance of ODT, especially if ODT is pretrained with low-reward offline data. These findings provide new directions to further improve decision transformers.

1 Introduction

While Reinforcement Learning (RL) has achieved great success in recent years [55, 31], it is known to struggle with several shortcomings, including training instability when propagating a Temporal Difference (TD) error along long trajectories [14], low data efficiency when training from scratch [67], and limited benefits from more modern neural network architectures [12]. The latter point differs significantly from other parts of the machine learning community such as Computer Vision [17] and Natural Language Processing [11].

To address these issues, Decision Transformers (DTs) [14] have been proposed as an emerging paradigm for RL, introducing more modern transformer architectures into the literature rather than the still widely used Multi-Layer Perceptrons (MLPs). Instead of evaluating state and state-action pairs, a DT considers the whole trajectory as a sequence to complete, and trains on offline data in a supervised, auto-regressive way. Upon inception, DTs have been improved in various ways, mostly dealing with architecture changes [37], the token to predict other than return-to-go [22], addressing the problem of being overly optimistic [46], and the inability to stitch together trajectories [5]. Significant and encouraging improvements have been reported on those aspects.

However, one fundamental issue has been largely overlooked by the community: offline-to-online RL using decision transformers , i.e., finetuning of decision transformers with online interactions. Offline-to-online RL [72, 41] is a widely studied sub-field of RL, which combines offline RL learning from given, fixed trajectory data and online RL data from interactions with the environment. By first training on offline data and then finetuning, the agent can learn a policy with much greater data efficiency, while calibrating the out-of-distribution error from the offline dataset. Unsurprisingly, this sub-field has become popular in recent years.

While there are numerous works in the offline-to-online RL sub-field [35, 28, 62], surprisingly few works have discussed the offline-to-online finetuning ability of decision transformers. While there is work that discusses finetuning of decision transformers predicting encoded future trajectory information [64], and work that finetunes pretrained decision transformers with PPO in multi-agent RL [38], the current widely adopted state-of-the-art is the Online Decision Transformer (ODT) [74]: the decision transformer training is continued on online data following the same supervised-learning paradigm as in offline RL. However, this method struggles with low-reward data, as well as with reaching expert-level performance due to suboptimal trajectories [41] (also see Sec. 4).

To address this issue and enhance online finetuning of decision transformers, we theoretically analyze the decision transformer based on recent results [7], showing that the commonly used conditioning on a high Return-To-Go (RTG) that's far from the expected return hampers results. To fix, we explore the possibility of using tried-and-true RL gradients . Testing on multiple environments, we find that simply combining TD3 [21] gradients with the original auto-regressive ODT training paradigm is surprisingly effective: it improves results of ODT, especially if ODT is pretrained with low-reward offline data.

Our contributions are summarized as follows:

1) We propose a simple yet effective method to boost the performance of online finetuning of decision transformers, especially if offline data is of medium-to-low quality;

2) We theoretically analyze the online decision transformer, explain its 'policy update' mechanism when using the commonly applied high target RTG, and point out its struggle to work well with online finetuning;

3) We conduct experiments on multiple environments, and find that ODT aided by TD3 gradients (and sometimes even the TD3 gradient alone) are surprisingly effective for online finetuning of decision transformers.

2 Preliminaries

Markov Decision Process. A Markov Decision Process (MDP) is the basic framework of sequential decision-making. An MDP is characterized by five components: the state space S , the action space A , the transition function p , the reward r , and either the discount factor γ or horizon H . MDPs involve an agent making decisions in discrete steps t ∈ { 0 , 1 , 2 , . . . } . On step t , the agent receives the current state s t ∈ S , and samples an action a t ∈ A according to its stochastic policy π ( a t | s t ) ∈ ∆( A ) , where ∆( A ) is the probability simplex over A , or its deterministic policy µ ( s t ) ∈ A . Executing the action yields a reward r ( s t , a t ) ∈ R , and leads to the evolution of the MDP to a new state s t +1 , governed by the MDP's transition function p ( s t +1 | s t , a t ) . The goal of the agent is to maximize the total reward ∑ t γ t r ( s t , a t ) , discounted by the discount factor γ ∈ [0 , 1] for infinite steps, or ∑ H t =1 r ( s t , a t ) for finite steps. When the agent ends a complete run, it finishes an episode , and the state(-action) data collected during the run is referred to as a trajectory τ .

Offline and Online RL. Based on the source of learning data, RL can be roughly categorized into offline and online RL. The former learns from a given finite dataset of state-action-reward trajectories, while the latter learns from trajectories collected online from the environment. The effort of combining the two is called offline-to-online RL, which first pre-trains a policy using offline data, and then continues to finetune the policy using online data with higher efficiency. Our work falls into the category of offline-to-online RL. We focus on improving the decision transformers, instead of Q-learning-based methods which are commonly used in offline-to-online RL.

Decision Transformer (DT). The decision transformer represents a new paradigm of offline RL, going beyond a TD-error framework. It views a trajectory τ as a sequence to be auto-regressively completed. The sequence interleaves three types of tokens: returns-to-go (RTG, the target total return) , states, and actions. At step t , the past sequence of context length K is given as the input, i.e., the input is ( RTG t -K , s t -k , a t -k , . . . , RTG t , s t ) , and an action is predicted by the auto-regressive model, which is usually implemented with a GPT-like architecture [11]. The model is trained via supervised learning, considering the past K steps of the trajectory along with the current state and the current return-to-go as the feature, and the sequence of all actions a in a segment as the labels. At evaluation time, a desired return RTGeval is specified, since the ground truth future return

RTGreal isn't known in advance.

Online Decision Transformer (ODT). ODT has two stages: offline pre-training which is identical to classic DT training, and online finetuning where trajectories are iteratively collected and the policy is updated via supervised learning. Specifically, the action a t at step t during rollouts is computed by the deterministic policy µ DT ( s t -T : t , a t -T : t -1 , RTG t -T : t , T = T eval , RTG = RTGeval ) , 1 or sampled from the stochastic policy π DT ( a t | s t -T : t , a t -T : t -1 , RTG t -T : t , T = T eval , RTG = RTGeval ) . Here, T is the context length (which is T eval in evaluation), and RTGeval ∈ R is the target return-to-go. The data buffer, initialized with offline data, is gradually replaced by online data during finetuning.

When updating the policy, the following loss (we use the deterministic policy as an example, and thus omit the entropy regularizer) is minimized:

Note, T train is the training context length and RTGreal is the real return-to-go. For better readability, we denote { s x +1 , s x +2 , . . . , s y } , x, y ∈ N as s x : y (i.e., left exclusive and right inclusive ), and similarly { a x +1 , a x +2 , . . . , a y } as a x : y and { RTG x +1 , . . . , RTG y } as RTG x : y . Specially, index x = y represents an empty sequence. For example, when t = 1 , a 0:0 is an empty action sequence as the decision transformer is not conditioned on any past action.

One important observation: the decision transformer is inherently off-policy (the exact policy distribution varies with the sampled starting point, context length and return-to-go), which effectively guides our choice of RL gradients to off-policy algorithms (see Appendix C for more details).

TD3. Twin Delayed Deep Deterministic Policy Gradient (TD3) [21] is a state-of-the-art online off-policy RL algorithm that learns a deterministic policy a = µ RL ( s ) . It is an improved version of an actor-critic (DDPG [32]) with three adjustments to improve its stability: 1) Clipped double Qlearning , which maintains two critics (estimators for expected return) Q ϕ 1 , Q ϕ 2 : | S | × | A |→ R and uses the smaller of the two values (i.e., min( Q ϕ 1 , Q ϕ 2 ) ) to form the target for TD-error minimization. Such design prevents overestimation of the Q -value; 2) Policy smoothing , which adds noise when calculating the Q -value for the next action to effectively prevent overfitting; and 3) Delayed update , which updates µ RL less frequently than Q ϕ 1 , Q ϕ 2 to benefit from a better Q -value landscape when updating the actor. TD3 also maintains a set of target networks storing old parameters of the actor and critics that are soft-updated with slow exponential moving average updates from the current, active network. In this paper, we adapt this algorithm to fit the decision transformer architecture so that it can be used as an auxiliary objective in an online finetuning process.

3 Method

This section is organized as follows: we will first provide intuition why RL gradients aid online finetuning of decision transformers (Sec. 3.1), and present our method of adding TD3 gradients (Sec. 3.2). To further justify our intuition, we provide a theoretical analysis on how ODT fails to improve during online finetuning when pre-trained with low-reward data (Sec. 3.3).

3.1 Why RL Gradients?

In order to understand why RL gradients aid online finetuning of decision transformers, let us consider an MDP which only has a single state s 0 , one step, a one dimensional action a ∈ [ -1 , 1] (i.e., a bandit with continuous action space) and a simple reward function r ( a ) = ( a +1) 2 if a ≤ 0 and r ( a ) = 1 -2 a otherwise, as illustrated in Fig. 1. In this case, a trajectory can be represented effectively by a scalar, which is the action. If the offline dataset for pretraining is of low quality, i.e., all actions in the dataset are either close to -1 or 1 , then the decision transformer will obviously not generate trajectories with high RTG after offline training. As a consequence, during online finetuning, the new rollout trajectory is very likely to be uninformative about how to reach RTGeval, since it is too far from RTGeval. Worse still, it cannot improve locally either, which requires ∂ RTG ∂a . However, the decision transformer yields exactly the inverse , i.e., ∂a ∂ RTG . Since the transformer is not invertible (and even if the transformer is invertible, often the ground truth RTG ( a ) itself is not), we cannot

Unreachable target RTG

RTG

Target

Action

𝑎

𝝏𝑹𝑻𝑮

𝝏𝒂

!

Local improvement by Q-function

ODT

Returns-To-Go (RTG)

Action

𝑎

Target

𝝏𝒂

𝝏𝑹𝑻𝑮

?

Offline data

50

100

150

200

250

300

Gradient Steps

0.0

0.2

0.4

0.6

0.8

1.0

Average Reward

ODT

DDPG

ODT+DDPG

(a) Reward Curves

0

2

4

6

8

10

12

14

Epoch

1.0

0.8

0.6

0.4

0.2

0.0

0.2

Action

ODT

DDPG

ODT+DDPG

(b) Rollout Distribution

1.00

0.75

0.50

0.25 0.00

0.25

0.50

0.75

1.00

Action

1.0

0.5

0.0

0.5

1.0

Estimated RTG

Ground Truth

DDPG

ODT+DDPG

(c) Critic

1.00

0.75

0.50

0.25 0.00

0.25

0.50

0.75

1.00

Action

0.0

0.2

0.4

0.6

0.8

1.0

Estimated RTG

ODT

ODT+DDPG

(d) DT Policy

easily estimate the former from the latter. Thus, the hope for policy improvement relies heavily on the generalization of RTG, i.e., policy yielded by high RTGeval indeed leads to better policy without any data as evidence, which is not the case with our constructed MDP and dataset.

In contrast, applying traditional RL for continuous action spaces to this setting, we either learn a value function Q ( s 0 , a ) : R → R , which effectively gives us a direction of action improvement ∂Q ( s 0 ,a ) ∂a (e.g., SAC [25], DDPG [32], TD3 [21]), or an advantage A ( s 0 , a ) that highlights whether focusing on action a improves or worsens the policy (e.g., AWR [47], AWAC [40], IQL [28]). Either way provides a direction which suggests how to change the action locally in order to improve the (estimated) return. In our experiment illustrated in Fig. 2 (see Appendix F for details), we found that RL algorithms like DDPG [32] can easily solve the aforementioned MDP while ODT fails.

Thus, adding RL gradients aids the decision transformer to improve from given low RTG trajectories. While one may argue that the self-supervised training paradigm of ODT [74] can do the same by 'prompting' the decision transformer to generate a high RTG trajectory, such paradigm is still unable to effectively improve the policy of the decision transformer pretrained on data with low RTGs. We provide a theoretical analysis for this in Sec. 3.3. In addition, we also explore the possibility of fixing this problem using other existing algorithms, such as JSRL [59] and slowly growing RTG (i.e., curriculum learning). However, we found that those algorithms cannot address this problem well. See Appendix G.8 for ablations.

3.2 Adding TD3 Gradients to ODT

In this work, we mainly consider TD3 [21] as the RL gradient for online finetuning. There are two reasons for selecting TD3. First, TD3 is a more robust off-policy RL algorithm compared to other off-policy RL algorithms [54]. Second, the success of TD3+BC [20] indicates that TD3 is a good candidate when combined with supervised learning. A more detailed discussion and empirical comparison to other RL algorithms can be found in Appendix C.

Generally, we simply add a weighted standard TD3 actor loss to the decision transformer objective. To do this, we follow classic TD3 and additionally train two critic networks Q ϕ 1 , Q ϕ 2 : S × A → R parameterized by ϕ 1 , ϕ 2 respectively. In the offline pretraining stage, we use the following objective for the actor:

Here, α ∈ { 0 , 0 . 1 } is a hyperparameter, and the loss sums over the trajectory segment. For critics Q ϕ 1 , Q ϕ 2 , we use the standard TD3 critic loss

where τ = { s 0: T train +1 , a 0: T train , RTG 0: T train +1 , d 0: T train , r 0: T train , RTG = RTGreal } is the trajectory segment sampled from buffer D that stores the offline dataset. Further, d t indicates whether the trajectory ends on the t -th step (true is 1 , false is 0 ), Q min is the target to fit, Q ϕ i, tar is produced by the target network (stored old parameter), z t is the context for 'next state' at step t . µ RL tar is the target network for the actor (i.e., decision transformer). For an n -dimensional action, clip ( a, x, y ) , a ∈ R n , y ∈ R n , z ∈ R n means clip a i to [ y i , z i ] for i ∈ { 1 , 2 , . . . , n } . a low ∈ R n and a high ∈ R n are the lower and upper bound for every dimension respectively.

To demonstrate the impact on aiding the exploration of a decision transformer, in this work we choose the simplest form of a critic, which is reflective, i.e., only depends on the current state. This essentially makes the Q -value an average of different context lengths sampled from a near-uniform distribution (see Appendix D for the detailed reason and distribution for this). The choice is based on the fact that training a transformer-based value function estimator is quite hard [45] due to increased input complexity (i.e., noise from the environment) which leads to reduced stability and slower convergence. In fact, to avoid this difficulty, many recent works on Large Language Models (LLMs) [13] and vision models [48] which finetune with RL adopt a policy-based algorithm instead of an actor-critic, despite a generally lower variance of the latter. In our experiments, we also found such a critic to be much more stable than a recurrent critic network (see Appendix G for ablations).

During online finetuning, we again use Eq. (2) and Eq. (3), but always use α = 0 . 1 for Eq. (2).

While the training paradigm resembles that of TD3+BC, our proposed method improves upon TD3+BC in the following two ways: 1) Architecture . While TD3+BC uses MLP networks for single steps, we leverage a decision transformer, which is more expressive and can take more context into account when making decisions. 2) Selected instead of indiscriminated behavior cloning. Behavior cloning mimics all data collected without regard to their reward, while the supervised learning process of a decision transformer prioritizes trajectories with higher return by conditioning action generation on higher RTG. See Appendix G.9 for an ablation.

3.3 Why Does ODT Fail to Improve the Policy?

As mentioned in Sec. 3.1, it is the goal of ODT to 'prompt' a policy with a high RTG, i.e., to improve a policy by conditioning on a high RTG during online rollout. However, beyond the intuition provided

in Sec. 3.1, in this section, we will analyze more formally why such a paradigm is unable to improve the policy given offline data filled with low-RTG trajectories.

Our analysis is based on the performance bound proved by Brandfonbrener et al. [7]. Given a dataset drawn from an underlying policy β and given its RTG distribution P β (either continuous or discrete), under assumptions (see Appendix E), we have the following tight performance bound for a decision transformer with policy π DT ( a | s, RTGeval ) conditioned on RTGeval:

Here, α f = inf s 1 P β ( RTGreal = RTGeval | s 1 ) for every initial state s 1 , ϵ > 0 is a constant, H is the horizon of the MDP. 2 Based on this tight performance bound, we will show that with high probability, 1 α f grows superlinearly with respect to RTGeval . If true, then the RTGreal term (i.e., the actual return from online rollouts) must decrease to fit into the tight bound, as RTGeval grows.

To show this, we take a two-step approach: First, we prove that the probability mass of the RTG distribution is concentrated around low RTGs, i.e., event probability Pr β ( RTG -E β ( RTG | s ) ≥ c | s ) for c > 0 decreases superlinearly with respect to c . For this, we apply the Chebyshev inequality, which yields a bound of O ( 1 c 2 ) . However, without knowledge on P β ( RTG | s ) , the variance can be made arbitrarily large by high RTG outliers, hence making the bound meaningless.

Fortunately, we have knowledge about the RTG distribution P β ( RTG | s ) from the collected data. If we refer to the maximum RTG in the dataset via RTG β max and if we assume all rewards are non-negative, then all trajectory samples have an RTG in [0 , RTG β max ] . Thus, with adequate prior distribution, we can state that with high probability 1 -δ , the probability mass is concentrated in the low RTG area. Based on this, we can prove the following lemma:

Lemma 1. (Informal) Assume rewards r ( s, a ) are bounded in [0 , R max ] , 3 and RTGeval ≥ RTG β max. Then with probability at least 1 -δ , we have the probability of event Pr β bounded as follows:

where δ depends on the number of trajectories in the dataset and prior distribution (see Appendix E for a concrete example and a more accurate bound). V β ( s ) is the value function of the underlying policy β ( a | s ) that generates the dataset, for which we have V β ( s ) = E β ( RTG | s ) .

The second step uses the bound of probability mass Pr β ( RTG ≥ c | s ) to derive the bound for α f . For the discrete case where the possibly obtained RTGs are finite or countably infinite (note, state and action space can still be continuous), this is simple, as we have

Thus α f = inf s 1 P β ( RTG | s 1 ) can be conveniently bounded by Lemma 1. For the continuous case, the proof is more involved as probability density P β ( RTG | s ) can be very high on an extremely short interval of RTG, making the total probability mass arbitrarily small. However, assuming that P β ( RTG | s ) is Lipschitz when RTG ≥ RTG β max (i.e., RTG area not covered by dataset), combined with the discrete distribution case, we can still get the following (see Appendix E for proof):

Corollary 1. (Informal) If the RTG distribution is discrete (i.e., number of possible different RTGs are at most countably infinite), then with probability at least1 -δ , 1 α f grows on the order of Ω( RTG 2 eval ) with respect to RTGeval. For continuous RTG distributions satisfying a Lipschitz continuous RTG density p β , 1 α f grows on the order of Ω( RTG 1 . 5 eval ) .

Here, Ω( · ) refers to the big-Omega notation (asymptotic lower bound).

4 Experiments

In this section, we aim to address the following questions: a) Does our proposed solution for decision transformers indeed improve its ability to cope with low-reward pretraining data. b) Is improving

what to predict, while still using supervised learning, the correct way to improve the finetuning ability of decision transformers? c) Does the transformer architecture, combined with RL gradients, work better than TD3+BC? d) Is it better to combine the use of RL and supervised learning, or better to simply abandon the supervised loss in online finetuning? e) How does online decision transformer with TD3 gradient perform compared to other offline RL algorithms? f) How much does TD3 improve over DDPG which was used in Fig. 2?

Baselines. In this section, we mainly compare to six baselines: the widely recognized state-of-the-art DT for online finetuning, Online Decision Transformer ( ODT ) [74]; PDT , a baseline improving over ODTby predicting future trajectory information instead of return-to-go; TD3+BC [20], a MLP offline RL baseline; TD3 , an ablated version of our proposed solution where we use TD3 gradients only for decision transformer finetuning (but only use supervised learning of the actor for offline pretraining); IQL [28], one of the most popular offline RL algorithms that can be used for online finetuning; DDPG [32]+ODT , which is the same as our approach but with DDPG instead of TD3 gradients (for ablations using SAC [25], IQL [28], PPO [52], AWAC [40] and AWR [47], see Appendix C). Each of the baselines corresponds to one of the questions a), b), c), d), e) and f) above.

Metrics. We use the normalized average reward (same as D4RL's standard [19]) as the metric, where higher reward indicates better performance. If the final performance is similar, the algorithm with fewer online examples collected to reach that level of performance is better. We report the reward curve, which shows the change of the normalized reward's mean and standard deviation with 5 different seeds, with respect to the number of online examples collected. The maximum number of steps collected is capped at 500K (for mujoco) or 1M (for other environments). We also report evaluation results using the rliable [3] library in Fig. 7 of Appendix B.

Experimental Setup. We use the same architecture and hyperparameters such as learning rate (see Appendix F.2 for details) as ODT [74]. The architecture is a transformer with 4 layers and 4 heads in each layer. This translates to around 13 M parameters in total. For the critic, we use Multi-Layer Perceptrons (MLPs) with width 256 and two hidden layers and ReLU [1] activation function. Specially, for the random dataset, we collect trajectories until the total number of steps exceeds 1000 in every epoch, which differs from ODT, where only 1 trajectory per epoch is collected. This is because many random environments, such as hopper, have very short episodes when the agent does not perform well, which could lead to overfitting if only a single trajectory is collected per epoch. For fairness, we use this modified rollout for ODT in our experiments as well. Not doing so does not affect ODT results since it does generally not work well on random datasets, but will significantly increase the time to reach a certain number of online transitions. After rollout, we train the actor for 300 gradient steps and the critic for 600 steps following TD3's delayed update trick.

4.1 Adroit Environments

Environment and Dataset Setup. We test on four difficult robotic manipulation tasks [49], which are the Pen, Hammer, Door and Relocate environment. For each environment, we test three different datasets: expert, cloned and human, which are generated by a finetuned RL policy, an imitation learning policy and human demonstration respectively. See Appendix F.1 for details.

Results. Fig. 3 shows the performance of each method on Adroit before and after online finetuning. TD3+BC fails on almost all tasks and often diverges with extremely large Q -value during online finetuning. ODT and PDT perform better but still fall short of the proposed method, TD3+ODT. Note, IQL, TD3 and TD3+ODT all perform decently well (with similar average reward as shown in Tab. 2 in Appendix B). However, we found that TD3 often fails during online finetuning, probably because the environments are complicated and TD3 struggles to recover from a poor policy generated during online exploration (i.e., it has a catastrophic forgetting issue). To see whether there is a simple fix, in Appendix G.7, we ablate whether an action regularizer pushing towards a pretrain policy similar to TD3+BC helps, but find it to hinder performance increase in other environments. IQL is overall much more stable than TD3, but improves much less during online finetuning than TD3+ODT. ODT can achieve good performance when pretrained on expert data, but struggles with datasets of lower quality, which validates our motivation. DDPG+ODT starts out well in the online finetuning stage but fails quickly, probably because DDPG is less stable compared to TD3.

0

50

100

150

Pen-expert-v1

0

50

100

Hammer-expert-v1

0

25

50

75

100

Relocate-expert-v1

0

25

50

75

100

125

Door-expert-v1

0

50

100

150

Pen-cloned-v1

0

25

50

75

100

125

Hammer-cloned-v1

0.5

0.0

0.5

1.0

1.5

2.0

Relocate-cloned-v1

0

20

40

60

Door-cloned-v1

0.0

0.2

0.4

0.6

0.8

1.0

1e6

0

50

100

Pen-human-v1

0.0

0.2

0.4

0.6

0.8

1.0

1e6

0

50

100

Hammer-human-v1

0.0

0.2

0.4

0.6

0.8

1.0

1e6

0

1

2

3

Relocate-human-v1

0.0

0.2

0.4

0.6

0.8

1.0

Online Transitions

1e6

0

20

40

60

80

Normalized Reward

Door-human-v1

TD3+BC

IQL

ODT

PDT

TD3

DDPG+ODT

TD3+ODT (ours)

0.0

0.2

0.4

0.6

0.8

1.0

1e6

0

25

50

75

100

Antmaze-umaze-v2

0.0

0.2

0.4

0.6

0.8

1.0

1e6

0

25

50

75

100

Antmaze-umaze-diverse-v2

0.0

0.2

0.4

0.6

0.8

1.0

1e6

0

25

50

75

100

Antmaze-medium-play-v2

0.0

0.2

0.4

0.6

0.8

1.0

Online Transitions

1e6

0

25

50

75

100

Normalized Reward

Antmaze-medium-diverse-v2

TD3+BC

IQL

ODT

PDT

TD3

DDPG+ODT

TD3+ODT (ours)

4.2 Antmaze Environments

Environment and Dataset Setup. We further test on a harder version of the Maze2D environment in D4RL [19] where the pointmass is substituted by a robotic ant. We study six different variants, which are umaze, umaze-diverse, medium-play, medium-diverse, large-play and large-diverse.

Results. Fig. 4 lists the results of each method on umaze and medium maze before and after online finetuning (see Appendix C for reward curves and Appendix B for results summary on large antmaze). TD3+ODT works the best on umaze and medium maze, and significantly outperforms TD3. This shows that RL gradients alone are not enough for offline-to-online RL of the decision transformer. Though TD3+ODT does not work on large maze, we found that IQL+ODT works decently well. However, we choose TD3+ODT in this work because IQL+ODT does not work well on the random datasets. This is probably because IQL aims to address the Out-Of-Distribution (OOD) estimation problem [28], which makes it better at utilizing offline data but worse at online exploration. See Appendix C for a detailed discussion and results. DDPG+ODT works worse than TD3+ODT but much better than baselines except IQL.

4.3 MuJoCo Environments

Environment and Dataset Setup. We further test on four widely recognized standard environments [58], which are the Hopper, Halfcheetah, Walker2d and Ant environment. For each environment, we study three different datasets: medium, medium-replay, and random. The first and second one contain trajectories of decent quality, while the last one is generated with a random agent.

Results. Fig. 6 shows the results of each method on MuJoCo before and after online finetuning. We observe that autoregressive-based algorithms, such as ODT and PDT, fail to improve the policy on MuJoCo environments, especially from low-reward pretraining with random datasets. With RL gradients, TD3+BC and IQL can improve the policy during online finetuning, but less than a decision transformer (TD3 and TD3+ODT). In particular, we found IQL to struggle on most random datasets, which are well-solved by decision transformers with TD3 gradients. TD3+ODT still outperforms TD3 with an average final reward of 88 . 51 vs. 84 . 23 . See Fig. 6 in Appendix B for reward curves.

Ablations on α . Fig. 5 (a) shows the result of using different α (i.e., RL coefficients) on different environments. We observe an increase of α to improve the online finetuning process. However, if α is too large, the algorithm may get unstable.

Ablations on evaluation context length T eval . Fig. 5 (b) shows the result of using different T eval on halfcheetah-medium-replay-v2 and hammer-cloned-v1. The result shows that T eval needs to be balanced between more information for decision-making and potential training instability due to a longer context length. As shown in the halfcheetah-medium-replay-v2 result, T eval too long or too short can both lead to performance drops. More ablations are available in Appendix G.

0

1

2

3

4

5

1e5

20

40

60

Halfcheetah-medium-replay-v2

0.0

0.2

0.4

0.6

0.8

1.0

Online Transitions

1e6

2.10

2.15

2.20

Normalized Reward

Hammer-human-v1

10

1

0.1

0.01

0.001

(a) Ablations on

α

0

1

2

3

4

5

1e5

20

40

60

Halfcheetah-medium-replay-v2

T_eval=20

T_eval=10

T_eval=5

T_eval=1

0.0

0.2

0.4

0.6

0.8

1.0

Online Transitions

1e6

0

50

100

Normalized Reward

Hammer-cloned-v1

T_eval=1

T_eval=2

T_eval=3

T_eval=5

(b) Ablations on

T

eval

5 Related Work

Online Finetuning of Decision Transformers. While there are many works on generalizing decision transformers (e.g., predicting waypoints [5], goal, or encoded future information instead of return-togo [22, 5, 57, 36]), improving the architecture [37, 16, 53, 65] or addressing the overly-optimistic [46] or trajectory stitching issue [63]), there is surprisingly little work beyond online decision transformers that deals with online finetuning of decision transformers. There is some loosely related literature: MADT [31] proposes to finetune pretrained decision transformers with PPO. PDT [64] also studies online finetuning with the same training paradigm as ODT [74]. QDT [66] uses an offline RL

algorithm to re-label returns-to-go for offline datasets. AFDT [76] and STG [75] use decision transformers offline to generate an auxiliary reward and aid the training of online RL algorithms. A few works study in-context learning [33, 34] and meta-learning [60, 30] of decision transformers, where improvements with evaluations on new tasks are made possible. However, none of the papers above focuses on addressing the general online finetuning issue of the decision transformer.

Transformers as Backbone for RL. Having witnessed the impressive success of transformers in Computer Vision (CV) [17] and Natural Language Processing (NLP) [11], numerous works also studied the impact of transformers in RL either as a model for the agent [45, 38] or as a world model [39, 50]. However, a large portion of state-of-the-art work in RL is still based on simple MultiLayer Perceptrons (MLPs) [35, 28]. This is largely because transformers are significantly harder to train and require extra effort [45], making their ability to better memorize long trajectories [42] harder to realize compared to MLPs. Further, there are works on using transformers as feature extractors for a trajectory [37, 45] and works that leverage the common sense of transformer-based Large Language Model's for RL priors [10, 9, 70]. In contrast, our work focuses on improving the new 'RL via Supervised learning' (RvS) [7, 18] paradigm, aiming to merge this paradigm with the benefits of classic RL training.

Offline-to-Online RL. Offline-to-online RL bridges the gap between offline RL, which heavily depends on the quality of existing data while struggling with out-of-distribution policies, and online RL, which requires many interactions and is of low data efficiency. Mainstream offline-to-online RL methods include teacher-student [51, 6, 59, 72] and out-of-distribution handling (regularization [21, 29, 62], avoidance [28, 23], ensembles [2, 15, 24]). There are also works on pessimistic Q-value initialization [69], confidence bounds [26], and a mixture of offline and online training [56, 73]. However, all the aforementioned works are based on Q-learning and don't consider decision transformers.

6 Conclusion

In this paper, we point out an under-explored problem in the Decision Transformer (DT) community, i.e., online finetuning. To address online finetuning with a decision transformer, we examine the current state-of-the-art, online decision transformer, and point out an issue with low-reward, sub-optimal pretraining. To address the issue, we propose to mix TD3 gradients with decision transformer training. This combination permits to achieve better results in multiple testbeds. Our work is a complement to the current DT literature, and calls out a new aspect of improving decision transformers.

Limitations and Future Works. While our work theoretically analyzes an ODT issue, the conclusion relies on several assumptions which we expect to remove in future work. Empirically, in this work we propose a simple solution orthogonal to existing efforts like architecture improvements and predicting future information rather than return-to-go. To explore other ideas that could further improve online finetuning of decision transformers, next steps include the study of other environments and other ways to incorporate RL gradients into decision transformers. Other possible avenues for future research include testing our solution on image-based environments, and decreasing the additional computational cost compared to ODT (an analysis for the current time cost is provided in Appendix H).

Acknowledgments

This work was supported in part by NSF under Grants 2008387, 2045586, 2106825, MRI 1725729, NIFA Award 2020-67021-32799, the IBM-Illinois Discovery Accelerator Institute, the Toyota Research Institute, and the Jump ARCHES endowment through the Health Care Engineering Systems Center at Illinois and the OSF Foundation.

References

Agarap, A. F. Deep learning using rectified linear units (relu). arXiv preprint arXiv:1803.08375 , 2018.

Agarwal, R., Schuurmans, D., and Norouzi, M. An optimistic perspective on offline reinforcement learning. In ICML , 2020.

Agarwal, R., Schwarzer, M., Castro, P. S., Courville, A., and Bellemare, M. G. Deep reinforcement learning at the edge of the statistical precipice. In NeurIPS , 2021.

Ba, J. L., Kiros, J. R., and Hinton, G. E. Layer normalization. arXiv preprint arXiv:1607.06450 , 2016.

Badrinath, A., Flet-Berliac, Y ., Nie, A., and Brunskill, E. Waypoint transformer: Reinforcement learning via supervised learning with intermediate targets. In NeurIPS , 2023.

Bastani, O., Pu, Y ., and Solar-Lezama, A. Verifiable reinforcement learning via policy extraction. In NeurIPS , 2018.

Brandfonbrener, D., Bietti, A., Buckman, J., Laroche, R., and Bruna, J. When does returnconditioned supervised learning work for offline reinforcement learning? In NeurIPS , 2022.

Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W. Openai gym. arXiv preprint arXiv:1606.01540 , 2016.

Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Chen, X., Choromanski, K., Ding, T., Driess, D., Dubey, A., Finn, C., et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15818 , 2023.

Brohan, A., Chebotar, Y., Finn, C., Hausman, K., Herzog, A., Ho, D., Ibarz, J., Irpan, A., Jang, E., Julian, R., et al. Do as i can, not as i say: Grounding language in robotic affordances. In CoRL , 2023.

Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. In NeurIPS , 2020.

Chebotar, Y., Vuong, Q., Hausman, K., Xia, F., Lu, Y., Irpan, A., Kumar, A., Yu, T., Herzog, A., Pertsch, K., et al. Q-transformer: Scalable offline reinforcement learning via autoregressive q-functions. In CoRL , 2023.

Chen, C., Wang, X., Jin, Y., Dong, V. Y., Dong, L., Cao, J., Liu, Y., and Yan, R. Semi-offline reinforcement learning for optimized text generation. In ICML , 2023.

Chen, L., Lu, K., Rajeswaran, A., Lee, K., Grover, A., Laskin, M., Abbeel, P., Srinivas, A., and Mordatch, I. Decision transformer: Reinforcement learning via sequence modeling. In NeurIPS , 2021.

Chen, X., Wang, C., Zhou, Z., and Ross, K. Randomized ensembled double q-learning: Learning fast without a model. In ICLR , 2021.

David, S. B., Zimerman, I., Nachmani, E., and Wolf, L. Decision s4: Efficient sequence-based rl via state spaces layers. In ICLR , 2022.

Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR , 2021.

Emmons, S., Eysenbach, B., Kostrikov, I., and Levine, S. Rvs: What is essential for offline rl via supervised learning? In ICLR , 2022.

Fu, J., Kumar, A., Nachum, O., Tucker, G., and Levine, S. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219 , 2020.

Fujimoto, S. and Gu, S. S. A minimalist approach to offline reinforcement learning. In NeurIPS , 2021.

Fujimoto, S., Hoof, H., and Meger, D. Addressing function approximation error in actor-critic methods. In ICML , 2018.

Furuta, H., Matsuo, Y., and Gu, S. S. Generalized decision transformer for offline hindsight information matching. In ICLR , 2022.

Garg, D., Hejna, J., Geist, M., and Ermon, S. Extreme q-learning: Maxent rl without entropy. In ICLR , 2023.

Ghasemipour, S. K. S., Schuurmans, D., and Gu, S. S. Emaq: Expected-max q-learning operator for simple yet effective offline and online rl. In ICML , 2021.

Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In ICML , 2018.

Hong, J., Kumar, A., and Levine, S. Confidence-conditioned value functions for offline reinforcement learning. In ICLR , 2023.

Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. In ICLR , 2015.

Kostrikov, I., Nair, A., and Levine, S. Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:2110.06169 , 2021.

Kumar, A., Zhou, A., Tucker, G., and Levine, S. Conservative q-learning for offline reinforcement learning. In NeurIPS , 2020.

Lee, J., Xie, A., Pacchiano, A., Chandak, Y., Finn, C., Nachum, O., and Brunskill, E. In-context decision-making from supervised pretraining. In ICML Workshop on New Frontiers in Learning, Control, and Dynamical Systems , 2023.

Lee, K.-H., Nachum, O., Yang, M. S., Lee, L., Freeman, D., Guadarrama, S., Fischer, I., Xu, W., Jang, E., Michalewski, H., et al. Multi-game decision transformers. In NeurIPS , 2022.

Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N. M. O., Erez, T., Tassa, Y ., Silver, D., and Wierstra, D. Continuous control with deep reinforcement learning. In ICLR , 2016.

Lin, L., Bai, Y ., and Mei, S. Transformers as decision makers: Provable in-context reinforcement learning via supervised pretraining. In ICLR , 2024.

Liu, H. and Abbeel, P. Emergent agentic transformer from chain of hindsight experience. In ICML , 2023.

Lyu, J., Ma, X., Li, X., and Lu, Z. Mildly conservative q-learning for offline reinforcement learning. In NeurIPS , 2022.

Ma, Y., Xiao, C., Liang, H., and Hao, J. Rethinking decision transformer via hierarchical reinforcement learning. arXiv preprint arXiv:2311.00267 , 2023.

Mao, H., Zhao, R., Chen, H., Hao, J., Chen, Y., Li, D., Zhang, J., and Xiao, Z. Transformer in transformer as backbone for deep reinforcement learning. In AAMAS , 2024.

Meng, L., Wen, M., Yang, Y., Le, C., Li, X., Zhang, W., Wen, Y., Zhang, H., Wang, J., and Xu, B. Offline pre-trained multi-agent decision transformer: One big sequence model tackles all smac tasks. arXiv preprint arXiv:2112.02845 , 2021.

Micheli, V., Alonso, E., and Fleuret, F. Transformers are sample efficient world models. In ICLR , 2023.

Nair, A., Dalal, M., Gupta, A., and Levine, S. Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359 , 2020.

Nakamoto, M., Zhai, Y., Singh, A., Mark, M. S., Ma, Y., Finn, C., Kumar, A., and Levine, S. Cal-ql: Calibrated offline rl pre-training for efficient online fine-tuning. In NeurIPS , 2023.

Ni, T., Ma, M., Eysenbach, B., and Bacon, P.-L. When do transformers shine in rl? decoupling memory from credit assignment. In NeurIPS , 2023.

Oh, J., Guo, Y., Singh, S., and Lee, H. Self-imitation learning. In ICML , 2018.

Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P. F., Leike, J., and Lowe, R. Training language models to follow instructions with human feedback. In NeurIPS , 2022.

Parisotto, E., Song, F., Rae, J., Pascanu, R., Gulcehre, C., Jayakumar, S., Jaderberg, M., Kaufman, R. L., Clark, A., Noury, S., et al. Stabilizing transformers for reinforcement learning. In ICML , 2020.

Paster, K., McIlraith, S., and Ba, J. You can't count on luck: Why decision transformers and rvs fail in stochastic environments. In NeurIPS , 2022.

Peng, X. B., Kumar, A., Zhang, G., and Levine, S. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00177 , 2019.

Pinto, A. S., Kolesnikov, A., Shi, Y., Beyer, L., and Zhai, X. Tuning computer vision models with task rewards. In ICML , 2023.

Rajeswaran, A., Kumar, V., Gupta, A., Vezzani, G., Schulman, J., Todorov, E., and Levine, S. Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations. In RSS , 2018.

Robine, J., Höftmann, M., Uelwer, T., and Harmeling, S. Transformer-based world models are happy with 100k interactions. In ICLR , 2023.

Schmitt, S., Hudson, J. J., Zidek, A., Osindero, S., Doersch, C., Czarnecki, W. M., Leibo, J. Z., Kuttler, H., Zisserman, A., Simonyan, K., et al. Kickstarting deep reinforcement learning. arXiv preprint arXiv:1803.03835 , 2018.

Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017.

Shang, J., Kahatapitiya, K., Li, X., and Ryoo, M. S. Starformer: Transformer with state-actionreward representations for visual reinforcement learning. In ECCV , 2022.

Sharif, A. and Marijan, D. Evaluating the robustness of deep reinforcement learning for autonomous and adversarial policies in a multi-agent urban driving environment. In QRS , 2021.

Silver, D., Hubert, T., Schrittwieser, J., Antonoglou, I., Lai, M., Guez, A., Lanctot, M., Sifre, L., Kumaran, D., Graepel, T., Lillicrap, T. P., Simonyan, K., and Hassabis, D. Mastering chess and shogi by self-play with a general reinforcement learning algorithm. Science , 2018.

Song, Y., Zhou, Y., Sekhari, A., Bagnell, J. A., Krishnamurthy, A., and Sun, W. Hybrid rl: Using both offline and online data can make rl efficient. In ICLR , 2023.

Sudhakaran, S. and Risi, S. Skill decision transformer. Foundation Models for Decision Making Workshop at NeurIPS , 2022.

Todorov, E., Erez, T., and Tassa, Y. Mujoco: A physics engine for model-based control. In IROS , 2012.

Uchendu, I., Xiao, T., Lu, Y., Zhu, B., Yan, M., Simon, J., Bennice, M., Fu, C., Ma, C., Jiao, J., et al. Jump-start reinforcement learning. In ICML , 2023.

Wang, Z., Wang, H., and Qi, Y. J. T3gdt: Three-tier tokens to guide decision transformer for offline meta reinforcement learning. In Robot Learning Workshop: Pretraining, Fine-Tuning, and Generalization with Large Scale Models in NeurIPS , 2023.

Wołczyk, M., Cupiał, B., Ostaszewski, M., Bortkiewicz, M., Zaj ˛ ac, M., Pascanu, R., Kuci´ nski, Ł., and Miło´ s, P. Fine-tuning reinforcement learning models is secretly a forgetting mitigation problem. In ICML , 2024.

Wu, J., Wu, H., Qiu, Z., Wang, J., and Long, M. Supported policy optimization for offline reinforcement learning. In NeurIPS , 2022.

Wu, Y.-H., Wang, X., and Hamaya, M. Elastic decision transformer. In NeurIPS , 2023.

Xie, Z., Lin, Z., Ye, D., Fu, Q., Wei, Y ., and Li, S. Future-conditioned unsupervised pretraining for decision transformer. In ICML , 2023.

Xu, M., Lu, Y., Shen, Y., Zhang, S., Zhao, D., and Gan, C. Hyper-decision transformer for efficient online policy adaptation. In ICLR , 2023.

Yamagata, T., Khalil, A., and Santos-Rodriguez, R. Q-learning decision transformer: Leveraging dynamic programming for conditional sequence modelling in offline rl. In ICML , 2023.

Yan, K., Schwing, A., and Wang, Y.-X. Ceip: Combining explicit and implicit priors for reinforcement learning with demonstrations. In NeurIPS , 2022.

You, Y., Li, J., Reddi, S., Hseu, J., Kumar, S., Bhojanapalli, S., Song, X., Demmel, J., Keutzer, K., and Hsieh, C.-J. Large batch optimization for deep learning: Training bert in 76 minutes. In ICLR , 2020.

Yu, Z. and Zhang, X. Actor-critic alignment for offline-to-online reinforcement learning. In ICML , 2023.

Yuan, H., Zhang, C., Wang, H., Xie, F., Cai, P., Dong, H., and Lu, Z. Plan4MC: Skill reinforcement learning and planning for open-world Minecraft tasks. In Foundation Models for Decision Making Workshop at NeurIPS 2023 , 2023.

Yue, Y., Lu, R., Kang, B., Song, S., and Huang, G. Understanding, predicting and better resolving q-value divergence in offline-rl. NeurIPS , 2024.

Zhang, H., Xu, W., and Yu, H. Policy expansion for bridging offline-to-online reinforcement learning. In ICLR , 2023.

Zheng, H., Luo, X., Wei, P., Song, X., Li, D., and Jiang, J. Adaptive policy learning for offline-to-online reinforcement learning. In AAAI , 2023.

Zheng, Q., Zhang, A., and Grover, A. Online decision transformer. In ICML , 2022.

Zhou, B., Li, K., Jiang, J., and Lu, Z. Learning from visual observation via offline pretrained state-to-go transformer. In NeurIPS , 2023.

Zhu, D., Wang, Y., Schmidhuber, J., and Elhoseiny, M. Guiding online reinforcement learning with action-free offline pretraining. arXiv preprint arXiv:2301.12876 , 2023.

Appendix: Reinforcement Learning Gradients as Vitamin for Online Finetuning Decision Transformers

The Appendix is organized as follows. In Sec. A, we discuss the potential positive and negative social impact of the paper. Then, we summarize the performance shown in the main paper in Sec. B. After this, we will explain our choice of RL gradients in the paper in Sec. C, and why our critic serves as an average of policies generated by different context lengths in Sec. D. We then provide rigorous statements for the theroetical analysis appearing in the paper in Sec. E, and list the environment details and hyperparameters in Sec. F. We then present more experiment and ablation results in Sec. G. Finally, we list our computational resource usage and licenses of related assets in Sec. H and Sec. I respectively.

A Broader Societal Impacts

Our work generally helps automation of decision-making by improving the use of online interaction data of a pretrained decision transformer agent. While this effort improves the efficiency of decisionmakers and has the potential to boost a variety of real-life applications such as robotics and resource allocation, it may also cause several negative social impacts, such as potential job losses, human de-skilling (making humans less capable of making decisions without AI), and misuse of technology (e.g., military).

B Performance Summary

In this section, we summarize the average reward achieve by each method on different environments and datasets, where the result for Adroit is shown in Tab. 2, and the result for Antmaze is shown in Tab. 3. As the summary table for MuJoCo is already presented in Sec. 4, we show the reward curves in Fig. 6. For a more rigorous evaluation, we also report other metrics including the median, InterQuartile Mean (IQM) and optimality gap using the rliable [3] library. See Fig. 7 for details. Breakdown analysis for each environment can be downloaded by browsing to https://kaiyan289. github.io/assets/breakdown_rliable.rar .

Table 2: Average reward for each method in Adroit Environments before and after online finetuning. The best result for each setting is marked in bold font and all results > 90% of the best performance are underlined. To save space, the name of the environments and datasets are abbreviated as follows: P=Pen, H=Hammer, D=Door, R=Relocate for environment, and E=Expert, C=cloned, H=Human for the dataset. It is apparent that while both IQL, TD3 and TD3+ODT perform decently well before online finetuning, our proposed solution significantly outperforms all baselines on the adroit testbed. DDPG+ODT starts out well in the online stage, but fails probably due to DDPG's training instability compared to TD3.

C Why Do We Choose TD3 to Provide RL Gradients?

In this section, we provide an ablation analysis on which RL gradient fits the decision transformer architecture best. Fig. 8 illustrates the result of using a pure RL gradient for online finetuning of a pretrained decision transformer (for those RL algorithms with stochastic policy, we adopt the same

0

25

50

75

100

Hopper-medium-v2

0

50

100

0

25

50

75

100

Walker2d-medium-v2

0

20

40

60

80

Halfcheetah-medium-v2

0

25

50

75

100

Hopper-medium-replay-v2

0

50

100

150

0

25

50

75

100

Walker2d-medium-replay-v2

0

20

40

60

Halfcheetah-medium-replay-v2

0

100000 200000 300000 400000 500000

0

20

40

60

80

100

Hopper-random-v2

0

100000 200000 300000 400000 500000

0

20

40

60

80

100

Ant-random-v2

0

100000 200000 300000 400000 500000

0

20

40

60

80

Walker2d-random-v2

0

100000 200000 300000 400000 500000

Online Transitions

0

20

40

60

Normalized Reward

Halfcheetah-random-v2

TD3+BC

IQL

ODT

PDT

TD3

DDPG+ODT

TD3+ODT (ours)

architecture as ODT which outputs a squashed Gaussian distribution with trainable mean and standard deviation). It is apparent that TD3 [21] and SAC [25] are the RL algorithms that suit the decision transformer best. Fig. 9 further shows the performance comparison between a decision transformer with SAC+ODT mixed gradient and TD3+ODT mixed gradient (both with coefficient 0 . 1 ). The result shows that TD3 is the better choice when paired with supervised learning.

Note, While PPO is generally closely related with transformers (e.g., Reinforcement Learning from Human Feedback (RLHF) [44]), and was used in some prior work for online finetuning of decision transformers with a small, discrete action space [38], in our experiments, we find PPO generally does not work with the decision transformer architecture. The main reason for this is the importance sampling issue: PPO has the following objective for an actor π θ parameterized by θ :

Here, π θ old is the policy at the beginning of the training for the current epoch. Normally, the denominator of the red part, π θ old ( a | s ) , would be reasonably large, as the data is sampled from that distribution. However, because of the offline nature caused by different RTGs and context lengths at rollout and training time, the denominator for the red part in Eq. (7) could be very small in training, which will lead to a very small loss if A π θ old ( s, a ) > 0 . This introduces significant instability during the training process. Fig. 10 illustrates the instability and degrading performance of a PPO-finetuned decision transformer.

0

20

40

60

80

100

TD3+BC

IQL

ODT

PDT

TD3

DDPG+ODT

TD3+ODT (ours)

Median

0

20

40

60

80

100

IQM

20

40

60

80

Mean

30

45

60

75

90

Optimality Gap

(a) Adroit

15

30

45

60

75

90

TD3+BC

IQL

ODT

PDT

TD3

DDPG+ODT

TD3+ODT (ours)

Median

15

30

45

60

75

90

IQM

30

45

60

75

90

Mean

15

30

45

60

75

Optimality Gap

(b) MuJoCo

0

20

40

60

80

100

TD3+BC

IQL

ODT

PDT

TD3

DDPG+ODT

TD3+ODT (ours)

Median

0

20

40

60

80

100

IQM

20

40

60

80

100

Mean

20

40

60

80

100

Optimality Gap

(c) Antmaze (umaze and medium)

40

80

120

160

200

TD3+BC

IQL

ODT

PDT

TD3

DDPG+ODT

TD3+ODT (ours)

Median

40

80

120

160

200

IQM

150

175

200

225

250

275

Mean

0

10

20

30

40

50

Optimality Gap

(d) Maze2d

In contrast, RLHF, does not exhibit such a problem: it does not use different return-to-go and context length in evaluation and training. Thus RLHF does not encounter the problem described above.

Besides the RL gradients mentioned above, as IQL works well on large Antmazes, we also explore the possibility of using IQL as the RL gradient for decision transformer instead of TD3. We found that IQL gradients, when applied to the decision transformer, indeed lead to much better results on antmaze-large. However, IQL fails to improve the policy when the offline dataset subsumes very low reward trajectories, which does not conform with our motivation. This is probably because IQL, as an offline RL algorithm, aims to address out-of-distribution evaluation issue, which is a much more important source of improvement in exploration in the online case. Thus, we choose TD3 as the RL gradient applied to decision transformer finetuning in this work. Fig. 11 shows the result of adding TD3 gradient vs. adding IQL gradient on Antmaze-large-play-v2 and hopper-random-v2.

D Why Our Critic Serves as an Average of Policies Generated by Different Context Lengths?

As we mentioned in Sec. 2, When updating a deterministic DT policy, the following loss is minimized:

where T train is the training context length and RTGreal is the real return-to-go.

0

25

50

75

100

Hopper-medium-v2

0

50

100

0

20

40

60

80

100

Walker2d-medium-v2

0

20

40

60

80

Halfcheetah-medium-v2

0

25

50

75

100

Hopper-medium-replay-v2

50

0

50

100

0

20

40

60

80

100

Walker2d-medium-replay-v2

0

20

40

60

80

Halfcheetah-medium-replay-v2

0

100000 200000 300000 400000 500000

0

20

40

60

80

100

Hopper-random-v2

0

100000 200000 300000 400000 500000

25

0

25

50

75

Ant-random-v2

0

100000 200000 300000 400000 500000

0

20

40

60

80

Walker2d-random-v2

0

100000 200000 300000 400000 500000

Online Transitions

0

20

40

60

Normalized Reward

Halfcheetah-random-v2

ODT

TD3

SAC

AWAC

AWR

PPO

40

60

80

100

120

Hopper-medium-v2

0

50

100

20

40

60

80

100

120

Walker2d-medium-v2

0

20

40

60

80

Halfcheetah-medium-v2

0

25

50

75

100

Hopper-medium-replay-v2

80

100

120

140

0

25

50

75

100

Walker2d-medium-replay-v2

20

40

60

Halfcheetah-medium-replay-v2

0

100000 200000 300000 400000 500000

0

20

40

60

80

100

Hopper-random-v2

0

100000 200000 300000 400000 500000

0

20

40

60

80

100

Ant-random-v2

0

100000 200000 300000 400000 500000

0

20

40

60

80

Walker2d-random-v2

0

100000 200000 300000 400000 500000

Online Transitions

0

20

40

60

Normalized Reward

Halfcheetah-random-v2

TD3+ODT

SAC+ODT

However, if we consider a particular action a t in some trajectory τ of the dataset, during training (both offline pretraining and online finetuning), the policy generated by the decision transformer fitting a t will be

T is actually sampled from a distribution U ′ (1 , T train ) over integers between 1 and T train inclusive; this distribution U ′ is introduced by the randomized starting step of the sampled trajectory segments, and is almost a uniform distribution on integers, except that a small asymmetry is created because the context length will be capped at the beginning of each trajectory. See Fig. 12 for an illustration.

Therefore, online decision transformers (and plain decision transformers) are actually trained to predict with every context length between 1 and T train. During the training process, the context length is randomly sampled according to U ′ , and a critic is trained to predict an 'average value' for the policy generated with context length sampled from U ′ .

(a) Reward

(b) The ratio

π

θ

(

a

|

s

)

π

θ

(

a

|

s

)

evaluation/return_mean_gm

training/ratios

Reward

Ratio

1200

Rewar

4e+8

1000

800

3e+8

600

2e+8

400

200

1e+8

0

X300-Gradient Steps

X300 GradientSteps

0

100

200

300

400

500

100

200

300

400

500

old

0

100000

200000

300000

400000

500000

0

20

40

60

80

100

Hopper-random-v2

0.0

0.2

0.4

0.6

0.8

1.0

1e6

0

20

40

60

Antmaze-large-play-v2

0.0

0.2

0.4

0.6

0.8

1.0

Online Transitions

1e6

0

10

20

30

40

50

Normalized Reward

Antmaze-large-diverse-v2

IQL+ODT

TD3+ODT

E Mathematical Proofs

In this section, we will state the theoretical analysis summarized in Sec. 3.3 more rigorously. We will first provide an explanation on how the decision transformer improves its policy during online finetuning, linking it to an existing RL method in Sec. E.1 and Sec. E.2. We will then bound its performance in Sec. E.3.

E.1 Preliminaries

Advantage-Weighted Actor Critic (AWAC) [40] is an offline-to-online RL algorithm, where the replay buffer is filled with offline data during offline pretraining and then supplemented with online experience during online finetuning. AWAC uses standard Q -learning to train the critic Q : | S | × | A |→ R , and update the actor using weighted behavior cloning, where the weight is exponentiated advantage (i.e., exp ( A ( s,a ) λ ) where λ > 0 is some constant).

E.2 Connection between Decision Transformer and AWAC

We denote β as the underlying policy of the dataset, and P β as the distribution over states, actions or returns induced by β . Note such P β can be either discrete or continuous. By prior work [7], for decision transformer policy π DT , we have the following formula holds for any return-to-go RTG ∈ R of the future trajectory:

Based on Eq. (10), we have the following lemma:

Lemma 2. For state-action pair ( s, a ) in an MDP, RTG ∈ R , assume the distributions of return-to-go (RTG) P β ( RTG | s, a ) and P β ( RTG | s ) are Laplace distributions with scale σ , then for any RTG large

RTG

1

𝑠𝑠1

𝑎𝑎1

RTG

2

𝑠𝑠2

𝑎𝑎2

RTG

3

𝑠𝑠3

𝑎𝑎3

RTG

4

𝑎𝑎4

𝑅𝑅

eval

𝑅𝑅

train

𝑅𝑅2

𝑎𝑎2

𝑎𝑎3

𝑎𝑎4

(a) Context length T 2 during training

𝑅𝑅2

′

𝑅𝑅2

Trajectory

𝜏𝜏

𝑅𝑅

train

Step

𝑗𝑗 ≥ 𝑅𝑅train

𝑅𝑅

train

Step

𝑖𝑖

<

𝑅𝑅

train

𝑅𝑅

train

(b) Distribution

U

′

of

T

2

)

𝑉 𝛽 (𝑠)

𝑅𝑇𝐺

𝑃𝛽(𝑅𝑇𝐺)

𝑄 𝛽

(𝑠, 𝑎)

𝑃𝛽(𝑅𝑇𝐺|𝑠)

𝑃𝛽(𝑅𝑇𝐺|𝑠, 𝑎)

𝑃𝛽

𝑅𝑇𝐺 𝑠, 𝑎

𝑃𝛽(𝑅𝑇𝐺|𝑠)

∝ exp

𝑅𝑇𝐺 - 𝑄 𝛽

𝑠, 𝑎

𝑅𝑇𝐺 - 𝑉 𝛽

𝑠, 𝑎

= exp(𝐴 𝛽

𝑠, 𝑎

𝑅𝑇𝐺

enough (more rigorously, RTG ≥ max { Q β ( s, a ) , V β ( s ) } ), π ( a | s, RTG ) is updated in the same way as AWAC. 4

Proof. If return-to-go are Laplace distributions, then by the symmetric property of such distributions, the mean of RTG given s or ( s, a ) would be the expected future return, which by definition are value functions for β , i.e., V β ( s ) for P β ( RTG | s ) and Q β ( s, a ) for P β ( RTG | s, a ) . See Fig. 13 as an illustration. As RTGeval ≥ max { Q β ( s, a ) , V β ( s ) } , we have

And thus we have P β ( RTG | s,a ) P β ( RTG | s ) = exp ( Q β ( s,a ) -V β ( s ) σ ) = exp( A β ( s,a ) σ ) , where A β is the advantage function.

While the Laplace distribution assumption in this lemma is strict and impractical for real-life applications, it gives us three crucial insights for a decision transformer:

For any state s or state-action pair ( s, a ) , we have E β [ RTG | s, a ] = Q β ( s, a ) , E β [ RTG | s ] = V β ( s ) , which is an important property of P β on RTG;

For decision transformer, the ability to improve the policy by collecting rollouts with high RTG, similar to AWAC, is closely related to advantage. In the above lemma, for example, if the two Laplace distributions have different scales σ V , σ Q , we will have the ratio P β ( RTG | s,a ) P β ( RTG | s ) being exp ( Q β ( s,a ) σ Q -V β ( s ) σ V ) ; if the distributions are Gaussian, we will get similar results but with quadratic terms of Q β and V β .

Different from AWAC, the 'policy improvement' of online decision transformers heavily relies on the global property of the return-to-go as RTGeval moves further away from Q β and V β . If the return-to-go is far away from the support of the data, we will have almost no data to evaluate P β , and its estimation can be very uncertain (let alone ratios). In this case, it is very unlikely for the decision transformer to collect rollouts with high RTGtrue and get further improvement. This is partly supported by the corollary of Brandfonbrener et al. [7], where the optimal conditioning return they found on evaluation satisfies RTGeval = V β ( s 1 ) at the initial state s 1 . This is also supported by our single-state MDP experiment discussed in Sec. 3.1 and illustrated in Fig. 2.

Those important insights lead to the intuition that decision transformers, when finetuned online, lack the ability to improve 'locally' from low RTGtrue data, and encourages to study the scenario where P β ( RTG | s ) and P β ( RTG | s, a ) are small.

E.3 Failure of ODT Policy Update with Low Quality Data

As mentioned in Sec. E.2, we study the performance of decision transformers in online finetuning when P β ( RTG | s ) and P β ( RTG | s, a ) is small. Specially, in this section, r ( s, a ) is not a reward function, but a reward distribution conditioned on ( s, a ) ; r i ∼ r ( s i , a i ) is the reward obtained on i -th step. Such notation takes the noise of reward into consideration and forms a more general framework. Also, as the discrete or continuous property of β is important in this section, we will use Pr β to represent probability mass (for discrete distribution or cumulative distribution for continuous distribution) and p β to represent probability density (for probability density function for continuous distribution).

By prior work [7], we have the following performance bound tight up to a constant factor for decision transformer for every iteration of updates:

Theorem E.1. For any MDP with transition function p ( ·| s, a ) and reward random variable r ( s, a ) , and any condition function f , assume the following holds:

Return coverage: P β ( g = f ( s 1 ) | s 1 ) ≥ α f for any initial state s 1 ;

Near determinism: for any state-action pair ( s, a ) , ∃ s ′ such that Pr( s ′ | s, a ) ≥ 1 -ϵ , and ∃ r 0 ( s, a ) such that Pr( r ( s, a ) = r 0 ( s, a )) ≥ 1 -ϵ ;

Consistency of f : f ( s ) = f ( s ′ ) + r for all s when transiting to next state s ′ .

Then we have

where α f > 0 , ϵ > 0 are constants, p ini is the initial state distribution, and H is the horizon of the MDP. π DT is the learned policy by Eq. (10) .

In our case, we define f ( s ) as follows:

Definition E.2. f ( s 1 ) = RTGeval for all initial states s 1 , f ( s i +1 ) = f ( s i ) -r i for the ( i +1) -th step following i -th step ( i ∈ { 1 , 2 , . . . , T -1 } ).

Further, we enforce the third assumption in Thm. E.1 by including the cumulative reward so far in the state space (as described in the paper of Brandforbrener et al. [7]). Under such definition, we have a tight bound on the regret between our target RTGeval and the true return-to-go RTGtrue = ∑ H i =1 r i by our learned policy at optimal, based on current replay buffer in online finetuning.

1

We will now prove that under certain assumptions, α f grows superlinearly with respect to RTG; as the bound is tight, the expected cumulative return term E τ =( s 1 ,a 1 ,...,s H ,a H ) ∼ β [ ∑ H i =1 E r i ∼ r ( s i ,a i ) r i ] will be decreasing to meet the bounds.

To do this, we start with the following assumptions:

Assumption E.3. We assume the following statements to be true:

( Bounded reward ) We assume the reward is bounded in [0 , R max ] for any state-action pairs.

( High evaluation RTG ) RTGeval ≥ RTG β max, where RTG β max is the largest RTGtrue in the dataset of n trajectories generated by β .

( Beta prior ) We assign the prior distribution of RTG generated by policy β to be a Beta distribution Beta (1 , 1) for the binomial likelihood of RTG falling on [0 , RTG β max ] or [ RTG β max , T R max ] .

Remark E.4 . The Beta distribution can be changed to any reasonable distribution; we use Beta distribution only for a convenient derivation. Considering the fact that by common sense, trajectories with high return are very hard to obtain, we can further strengthen the conclusion by changing the prior distribution.

We then prove the following lemma:

Lemma 3. Under the Assumption E.3, given underlying policy β of the dataset, for any state s with value function V β ( s ) and any state-action pair ( s, a ) with Q-function Q β ( s, a ) , any c ≥ 0 and RTGeval ∈ R , with probability 1 -δ , we have

where δ = 1 -CDFBeta ( n +1 , 1) ( ϵ ) , and the CDF is the cumulative distribution function.

Proof. With the beta prior assumption in Assumption E.3, we know that with n samples where RTG ≤ RTG β max, we have the posterior distribution to be Beta ( n + 1 , 1) , i.e., with probability 1 -CDFBeta ( n +1 , 1) ( ϵ ) , we have Pr β ( RTG ≥ RTG β max ) ≤ ϵ for ϵ > 0 .

Thus, by Chebyshev inequality, we know

and a similar conclusion holds for Pr β ( RTG -Q β ( s, a ) ≥ c | s, a ) . Thus, the probability decays superlinearly with respect to RTG.

Given this lemma, it remains to connect the bound of Pr β ( RTG ≥ c 0 ) to P β ( RTG = c 0 ) on RTG, c 0 ∈ R . For discrete distribution, the connection is straightforward: Pr β ( RTG ≥ c 0 ) ≥ Pr β ( RTG = c 0 ) = P β ( RTG = c 0 ) for any condition s or ( s, a ) .

Thus, we immediately get the following corollary:

Corollary 2. Assume the reward is bounded in [0 , R max ] for any state and action, and the number of possible different return-to-go one can get is finite or countably infinite. Then for the f condition function defined in Def. E.2, with probability of at least 1 -δ , we have α f ≤ (1 -ϵ ) RTG β max + ϵR 2 max T 2 -[ V β ( s )] 2 ( RTG -V β ( s )) 2 , i.e., 1 α f grows in the order of Ω( RTG 2 eval ) .

Remark E.5 . While the limitation on return-to-go seems strong theoretically, it is very easy to satisfy such assumption in practice because it has no requirement on the discreteness of state and action space. Such corollary can be applied on reward discretization with arbitrary precision (including implicit ones by float precision).

For continuous distribution, to bound p β ( RTG = c 0 ) with Pr β ( RTG ≥ c 0 ) on RTG, we would need to assume that 'peak' does not exist (see Fig. 14 for illustration), i.e. there does not exist cases where p β is large but Pr β is small. Thus, we made the following assumption:

Assumption E.6. ( Lipschitzness on uncovered RTG distribution ) p β ( RTG | s ) is K V -Lipschitz where RTG is larger than RTG β max.

Remark E.7 . The assumption is reasonable because when RTG is larger than any of the RTGtrue in the dataset, we have no data coverage for the performance of the underlying policy β under such RTG, and thus we can choose any inductive bias for β .

Remark E.8 . Note the Lipschitzness of p β does not rely on the Lipschitzness of the reward function. For example, consider a single-state, single-step MDP where we have a uniformly random policy a ∼ U (0 , 1) with reward r ( a ) = 2 -1 a . The reward is clearly not Lipschitz on a ∈ (0 , 1) , but the distribution of r is p r ( r 0 ) = p a ( r -1 ( r 0 )) · ∂r -1 ( r 0 ) ∂r = 1 ( r -2) 2 , which is Lipschitz on ( -∞ , 1) as a ∈ (0 , 1) .

With such assumption, we have the following corollary:

Corollary 3. Under assumption E.6, for the f condition function defined in Def. E.2, we have α f ≤ √ 2 K V Ω ( RTG -1 . 5 eval ) , i.e., 1 α f grows in the order of Ω( RTG 1 . 5 eval ) with probability of at least 1 -δ .

Proof. Under assumption E.6, by Lipschitzness, for any RTG where p β ( RTG | s ) > p 0 , we have p β ( RTG + c | s ) > p 0 -K V · c for any c ∈ [0 , p 0 K V ] .

Thus, we know that if ∃ RTG 0 > RTG β max such that p β ( RTG 0 | s ) > p 0 , then we have Pr β ( RTG ≥ RTG 0 | s ) > p 2 0 2 K V (See Fig. 14 for an illustration). By the contra-positive statement of the above conclusion, we know that

and RTGeval is applicable to the inequality above by the high evaluation RTG assumption in Assumption E.3. We then apply the proof lemma 3, but apply the inequality P ( x -E [ x ] ≥ c ) ≤ E ( x -E [ x ]) 3 c 3 instead of Chebyshev inequality which leads to the conclusion.

F Experimental Details

F.1 Environment and Dataset Details

F.1.1 Single-State MDP

The single-state MDP studied in Sec. 3.1 motivates why RL gradients are useful for online finetuning. It has a single state, a single action a ∈ [ -1 , 1] , and a reward function r ( a ) = ( a +1) 2 if a ≤ 0 and r ( a ) = 1 -2 a otherwise.

Datasets. The dataset has a size of 128 , with 100 actions uniformly sampled in ( -1 , 0 . 95) , and the remaining 28 actions uniformly sampled in (0 . 5 , 1) . The dataset is designed to conceal the reward peak in the middle. DDPG and ODT+DDPG successfully recognized the reward peak but ODT failed.

𝑝0

𝑉 𝛽

𝑠 +𝑐

𝑉 𝛽 (𝑠)

𝑅𝑇𝐺

𝑝𝛽(𝑅𝑇𝐺|𝑠)

Probability mass

𝑝 0

2

2𝐾𝑉

F.1.2 Adroit Environments

Environments. Adroit is a set of more difficult benchmark than Mujoco in D4RL, and is becoming increasingly popular in recent offline and offline-to-online RL works [28, 23]. We test four environments in adroit in our experiments: pen, hammer, door and relocate. Fig. 15 shows an illustration of the four environments.

(a) Pen

(b) Hammer

(c) Door

(d) Relocate

Pen. Pen is a locomotion environment where the agent needs to control a robotic hand to manipulate a pen, such that its orientation matches the target. It has a 24 -dimensional action space, each of which controls a joint on the wrist or fingers. The state space is 45 -dimensional, which contains the pose of the palm, the angular position of the joints, and the pose of the target and current pen.

Hammer. Hammer is an environment where the agent needs to control a robotic hand to pick up a hammer and use it to drive a nail into a board. The action space is 26 -dimensional, each of which corresponds to a joint on the hand. The state space is 46 -dimensional, which describes the angular position of the fingers, the pose of the palm, and the status of hammer and nail.

Door. In the door environment, the agent needs to use a robotic hand to open a door by undoing the latch and swinging it. The environment has a 28 -dimensional action space, which are the absolute angular positions of the hand joints. It also has 39 -dimensional observation space which describes each joint, the pose of the palm, and the door with its latch.

Relocate. In the relocate environment, the agent needs to control a robotic hand to move a ball from its initial location towards a goal, both of which are randomized in the environment. The environment has a 30 -dimensional action space which describes the angular position of the joints on the hand, and a 39 -dimensional space which describes the hand as well as the ball and target.

Datasets. For each of the four environments, we test our method across three different qualities of datasets: expert, cloned and human, all of which provided by the DAPG [49] repository. The expert dataset is generated by a fine-tuned RL policy; the cloned dataset is collected from an imitation policy on the demonstrations from the other two datasets; and the human dataset is collected from human demonstrations. Tab. 4 shows the size and average reward of each dataset.

F.1.3 Antmaze Environments

Environments. Antmaze is a more difficult version of Maze2D, where the agent controls a robotic ant instead of a point mass through the maze. It has a 27 dimensional-state space and a 8 -dimensional action space. We test our method on six variants of antmaze: Umaze, Umaze-Diverse, Medium-Play, Medium-Diverse, Large-Play and Large-Diverse, where 'Umaze', 'Medium' and 'Large' describes the size of the maze (see Fig. 16 for an illustration), and the 'Diverse' and 'Play' describes the type of the dataset. More specifically, 'Diverse' means that in the offline dataset, the starting point and the goal of the agent are randomly generated, while 'Play' means that the goal is generated by a handcraft design. 'Umaze' without suffix is the simplest environment where both the starting point and the goal are fixed.

(a) Open

(b) Umaze

(c) Medium

(d) Large

田

Datasets. Similar to Adroit and MuJoCo, we test our method on datasets provided by D4RL. Tab. 5 shows the size and normalized reward of each dataset. Note, following IQL [28] and CQL [29], we conduct reward shaping: subtracting from all rewards in the dataset and environment the value 1 during training of both our method and baselines to provide denser reward signal for all antmaze environments. However, we still count original sparse reward when comparing the performance.

F.1.4 MuJoCo

Environments. We test our method on four widely used environments: Hopper, Halfcheetah, Walker2d and Ant. Fig. 17 shows an illustration of the four environments.

(a) Hopper

(b) Halfcheetah

(c) Ant

(d) Walker2d

Hopper. Hopper is a locomotion task on a 2D vertical plane, where the agent manipulates a single-legged robot to hop forward. Its state is 11 -dimensional, which describes the angle and velocity for the robot's joints. Its action is 3 -dimensional, which corresponds to the torques applied on the three joints for the current time step respectively.

Halfcheetah. Halfcheetah is also a 2D environment which requires the agent to control a cheetah-like robot to run forward. The states are 17 -dimensional, containing the coordinate and velocity of the joints The actions are 6 -dimensional, which control the torques on the joints of the robot.

Ant. In Ant, the agent controls a four-legged 8 -DoF robotic ant to walk in a 3D environment and tries to move forward. It has a 111 -dimensional state space describing the coordinates and velocities of the joints.

Walker2d. Walker2d is a 2D environment in which the agent needs to manipulate a 8 -DoF two-legged robot to walk forward under the agent's control. Its state space is 27 -dimensional.

Datasets. We test our method across three different qualities of datasets: medium, medium-replay and random. The medium dataset contains trajectories collected by an agent trained with RL, but early-stopped at medium-level performance. The medium-replay dataset is the collection of trajectories sampled in the training process of the agent mentioned above. The random dataset contains trajectories collected by an agent with random policy. Tab. 6 shows the size and normalized reward of each dataset.

F.1.5 Maze2D Environments

Environments. Maze2D is another set of D4RL environment, where the agent needs to control a point mass to navigate through a 2D maze and arrive at a fixed goal. It has a 4 -dimensional state space describing its coordinate and velocity, and a 2 -dimensional action describing its acceleration. The reward is determined by its current distance to the goal. We test our method on four variants of maze: Open, Umaze, Medium and Large with increasing difficulty. The map of each maze is illustrated in Fig. 16. Maze2D environment is tested in Sec. G.2.

Datasets. We again test our method on datasets provided by D4RL. Tab. 7 shows the size and normalizeed reward of each dataset.

F.2 Hyperparameters

F.2.1 Single-State MDP

For all networks, we use a simple MDP with two hidden layers of width 128 , and ReLU [1] as activation function. We add a Tanh activation function to limit the output for ODT and the actor of DDPG to [ -1 , 1] . For both methods, we use Adam [27] as the optimizer, and the learning rate is set to 10 -3 . We pretrain 5 epochs on offline data ( 20 gradient steps) and 16 epochs for online finetuning, with a batch size of 32 for gradient update and collect 64 new rollout states for each epoch (thus we train 2 n +4 steps for the n -th online finetuning epoch). RTGeval is set at 1 , which serves as the (constant) input for ODT rollout and DDPG actor. Both DDPG and ODT uses deterministic actor with an exploration noise uniform in [ -0 . 01 , 0 . 01] during online rollouts.

F.2.2 Other Experiments

Tab. 8 summarizes hyperparameters that are common across all environments, and Tab. 9 summarizes hyperparameters that are different across environments. For environments that exist in ODT [74], we follow the hyperparameters from ODT medium environments. We did not use positional embedding as suggested by ODT [74]. Specially, for antmaze, we remove most (all but 10 ) 1-step trajectories, because the size of the replay buffer for decision transformers is controlled by the number of trajectories, and antmaze dataset contains a large number of 1-step trajectories due to its data generation mechanism (immediately terminate an episode when the agent is close to the goal, but do not reset the agent location). Also, we add Layernorm [4] after each hidden layer of the critic for Adroit, Maze and Antmaze environments, according to Yue et al. [71]'s advice. We found that such practice stabilizes the training process (see Sec. G.5 for ablation).

For ODT and TD3 baseline, we use the same code as our TD3+ODT, while setting coefficients for RL and supervised gradients accordingly. For PDT baseline, we use the default hyperparameter in PDT paper, and pretrain PDT for 40K steps for all experiments. For TD3+BC and IQL, we use the default hyperparameter in their codebase, and pretrain them for 1M steps for all experiments (remaining the same as that in the codebase).

G More Results

G.1 Delayed Reward

Though we have tested MuJoCo environments in Sec. 4, it is worth noting that many offline RL algorithms have addressed the MuJoCo benchmark quite well [28, 23]. Thus, we also tested settings where RL struggles to obtain good performance to further analyze the performance of using RL gradients for decision transformers.

Environment and Experimental Setup. In this experiment, we use the same experiment and dataset as in Sec. 4 except for one major difference: the rewards are not given immediately after each step. Instead, the cumulative reward during a short period of M steps is given only at the end of the period , while the rewards observed by the agents within a period are all zero. We adopt such a setting from prior influential work [43], which creates a sparse-reward setting where RL algorithms struggle. We test M = 5 in this experiment.

Results. We use the same baselines as that in Sec. 4; Tab. 10 summarizes the performance of each method. Generally, DT with TD3 gradient still works very well, much better than ODT. While

Table 10: Average reward for each method in MuJoCo Environments before and after online finetuning with delayed rewards. To save space, the name of the environments and datasets are compressed, where Ho=Hopper, Ha=HalfCheetah, Wa=Walker2d, An=Ant for the environment, and M=Medium, MR=Medium-Replay, R=Random for the dataset. The format is "final(+increase)". The best result for each setting is highlighted in bold font, and any result > 90% of the best performance is underlined.

0

25

50

75

100

Hopper-medium-v2

0

50

100

150

0

20

40

60

80

100

Walker2d-medium-v2

0

20

40

60

Halfcheetah-medium-v2

0

25

50

75

100

Hopper-medium-replay-v2

0

25

50

75

100

125

0

25

50

75

100

Walker2d-medium-replay-v2

0

20

40

60

Halfcheetah-medium-replay-v2

0

100000 200000 300000 400000 500000

0

20

40

60

80

100

Hopper-random-v2

0

100000 200000 300000 400000 500000

0

20

40

60

80

Ant-random-v2

0

100000 200000 300000 400000 500000

0

20

40

60

80

Walker2d-random-v2

0

100000 200000 300000 400000 500000

Online Transitions

0

20

40

60

Normalized Reward

Halfcheetah-random-v2

TD3+BC

IQL

ODT

PDT

TD3

TD3+ODT (ours)

TD3+BC works well in several scenarios, it struggles on random environments. See Fig. 18 for reward curves.

G.2 Maze2D Environment

We test on navigation tasks in D4RL [19] where the agents need to control a pointmass through four different mazes: Open, Umaze, Medium and Large with different dataset respectively. Fig. 19 lists the performance of each method on Maze2D before and after online finetuning, and Tab. 11 summarizes the performance before and after online finetuning. The result shows that our method again significantly outperforms autoregressive-based algorithms such as ODT and PDT, which validates our motivation in Sec. 3.1. DDPG+ODT works similarly well as TD3+ODT in this environment with simple state and action space.

0.0

0.2

0.4

0.6

0.8

1.0

1e6

0

200

400

600

Maze2d-open-v0

0.0

0.2

0.4

0.6

0.8

1.0

1e6

0

50

100

150

Maze2d-umaze-v1

0.0

0.2

0.4

0.6

0.8

1.0

1e6

0

50

100

150

200

Maze2d-medium-v1

0.0

0.2

0.4

0.6

0.8

1.0

Online Transitions

1e6

0

100

200

Normalized Reward

Maze2d-large-v1

TD3+BC

IQL

ODT

PDT

TD3

DDPG+ODT

TD3+ODT (ours)

0.0

0.2

0.4

0.6

0.8

1.0

Online Transitions

1e6

0

50

100

Normalized Reward

Hammer-cloned-v1

T_train=20

T_train=10

T_train=5

T_train=3

T_train=2

T_train=1

G.3 Ablations on training context length T train

Fig. 20 shows the result of using different context lengths on hammer-cloned-v1 environment (in this experiment, we use T eval = 1 to demonstrate the effect of more different T train). It is shown from the experiment that the selection of T train needs to be balanced between more information taken into account and training stability; while longer T train brings faster convergence when growing from 1 to 5 , the reward curves with T train ∈ { 10 , 20 } oscillates more than that with T train = 5 .

G.4 Longer Training Process

In some environments, such as hopper-random-v2, walker2d-random-v2 and ant-random-v2, our proposed method still seems to be improving after 500K online samples. In Fig. 21, we show the finetuning result of our proposed method with more online transitions, which effectively shows that our method has greater potential in online finetuning when finetuned for more gradient steps.

G.5 The Effect of Layernorm

As we have mentioned in Sec. F.2, as suggested by Yue et al. [71], we apply Layernorm [4] to critic networks for environment other than MuJoCo for better stability in training. In our experiment, we found that it greatly stabilizes the critic on complicated environments such as Adroit, but makes online policy improvement less efficient on easier MuJoCo environments. Fig. 22 shows the performance and critic MSE loss comparison on some environments with and without layernorm; it is clearly shown that layernorm helps stabilizes online finetuning in some cases such as pen-cloned-v1, but hinders performance increase on other environments such as halfcheetah-medium-v2.

G.6 Recurrent Critic

As mentioned in Sec. 3.2, we use reflexive critics (i.e., critics that only take the current state nand action) to add RL gradients to decision transformer, and this creates an average effect among policies generated by different context lengths (see Sec. D in the appendix for detail). In this section, we explore recurrent critic by substituting the MLP critic using a LSTM, such that for a trajectory segment, the evaluation for the i -th action a i is based on all state-action pairs ( s 1 , a 1 ) , . . . , ( s i -1 , a i -1 ) and current state s i . As shown in Fig. 23, we found that recurrent critics are much less stable than

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

1e6

0

20

40

60

80

100

120

Hopper-random-v2

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

1e6

0

20

40

60

80

100

120

Walker2d-random-v2

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

Online Transitions

1e6

0

25

50

75

100

125

Normalized Reward

Ant-random-v2

TD3+ODT

0

1

2

3

4

5

1e5

40

50

60

70

Normalized Reward

Halfcheetah-medium-replay-v2

0

1

2

3

4

5

1e5

0

200

400

600

Critic MSE Loss

Halfcheetah-medium-replay-v2

0

1

2

3

4

5

1e5

0

50

100

150

Normalized Reward

Pen-cloned-v1

0

1

2

3

4

5

Online Transitions

1e5

0

1

2

3

Critic MSE Loss

1e6

Pen-cloned-v1

No Layernorm

Layernorm

reflexive critics, and the instability increases as the training context length T train grows; on the contrary, reflexive critic can well-handle the case where T train is long.

G.7 Regularizer for Pure TD3 Gradients

In the Adroit environment results discussed in Sec. 4, we found that the baseline of ODT finetuned using pure TD3 gradients struggles due to catastrophic forgetting. Inspired by Wołczyk et al. [61], we test whether adding a KL regularizer can fix the forgetting problem. Though our policy is deterministic, we can approximately interpret the policy as Gaussian with a very small variance. Thus, a KL regularizer can be simply added using c 0 · ∥ a -a old ∥ 2 , where a is the current action and a old is the action predicted by the pretrained policy. We set c 0 = 0 . 05 and test this method on the Adroit cloned and expert dataset. We illustrate the result in Fig. 24. We find that the KL regularizer effectively addresses the issue on expert environments for both TD3 and TD3+ODT. But it can sometimes hinder the policy improvement of TD3+ODT with low return during online finetuning.

0

20000

40000

60000

80000

100000

Online Transitions

0

20

40

60

80

100

120

Normalized Reward

T_train = 1

T_train = 2

T_train = 20

Reflexive (T_train = 20)

0

25

50

75

100

125

150

Pen-cloned-v1

0

20

40

60

80

100

120

140

Hammer-cloned-v1

0

10

20

30

40

50

60

Relocate-cloned-v1

0

20

40

60

80

100

Door-cloned-v1

0.0

0.2

0.4

0.6

0.8

1.0

1e6

0

25

50

75

100

125

150

Pen-expert-v1

0.0

0.2

0.4

0.6

0.8

1.0

1e6

0

20

40

60

80

100

120

140

Hammer-expert-v1

0.0

0.2

0.4

0.6

0.8

1.0

1e6

0

20

40

60

80

100

Relocate-expert-v1

0.0

0.2

0.4

0.6

0.8

1.0

Online Transitions

1e6

0

20

40

60

80

100

Normalized Reward

Door-expert-v1

ODT+JSRL(oracle)

ODT (curriculum)

ODT

TD3

TD3+KL

TD3+JSRL(pretrain)

TD3+ODT (ours)+KL

TD3+ODT (ours)

G.8 Other possible exploration improvement techniques

In Sec. 3.1, we state that ODT cannot explore the high-RTG region when pretrained with low-quality offline data, and we ran a simple experiment to verify this (Fig. 2). In this section, we test two potential alternatives for addressing the exploration problem: JSRL [59] and curriculum learning.

For JSRL, an expert policy is used for the first n steps in an episode, before ODT takes over. We set n = 100 (100% max episode length for adroit pen, 50% max episode length for other adroit environments) initially, and apply an exponential decay rate of 0 . 99 for every episode. We test two settings of JSRL: the expert policy being the offline pretrained policy, and the expert policy being oracle , i.e., an IQL policy trained on the Adroit expert dataset.

For curriculum learning, we use ODT with a gradually increasing target RTG with the current RTG for rollouts being RTGeval -0 . 99 N ( RTGeval -RTGdata ) . Here, N is the number of episodes sampled in online stage, and RTGdata is the average RTG of the offline dataset.

Results are summarized in Fig. 24. We found that curriculum RTG does not work, probably because the task is too hard and cannot be improved by random exploration without gradient guidance. Further, even with oracle exploration, ODT is not guaranteed to succeed: it fails on the hammer environment where TD3+ODT succeeds, probably because of insufficient expert-level data and an inability to improve with random exploration.

G.9 Ablations on the Architecture

In this section, we further examine the source of the performance gain of our method compared to TD3+BC. There are two key differences as stated in Sec. 3.2: The architecture and RL via Supervised (RvS) learning [18]. We can hence assess two more baselines: TD3+BC with our transformer architecture and TD3+RvS using TD3+BC's architecture. We present the ablation result on the Adroit cloned environment in Fig. 25. The result shows that only TD3+BC with our architecture works (albeit still worse than our method). We hypothesize that this is because a simple MLP is hard to model the complicated policy which takes both RTG and state into account.

To further assess if simply adding more layers to the MLP works, we conduct an ablation on the number of layers for TD3+RvS. The result is illustrated in Fig. 26. It shows that simply adding a few layers to the MLP does not aid performance. We speculate that it is probably the transformer architecture that helps modeling the state-and-RTG-conditioned policy.

0.0

0.2

0.4

0.6

0.8

1.0

1e6

0

25

50

75

100

125

150

Pen-cloned-v1

0.0

0.2

0.4

0.6

0.8

1.0

1e6

0

20

40

60

80

100

120

140

Hammer-cloned-v1

0.0

0.2

0.4

0.6

0.8

1.0

1e6

0.0

0.5

1.0

1.5

2.0

Relocate-cloned-v1

0.0

0.2

0.4

0.6

0.8

1.0

Online Transitions

1e6

0

20

40

60

Normalized Reward

Door-cloned-v1

ODT

TD3+BC

TD3+RVS

TD3+BC (our arch)

TD3+ODT (ours)

0.0

0.2

0.4

0.6

0.8

1.0

1e6

0

25

50

75

100

125

150

Pen-expert-v1

0.0

0.2

0.4

0.6

0.8

1.0

1e6

40

60

80

100

120

140

Hammer-expert-v1

0.0

0.2

0.4

0.6

0.8

1.0

1e6

0

20

40

60

80

100

Relocate-expert-v1

0.0

0.2

0.4

0.6

0.8

1.0

Online Transitions

1e6

0

20

40

60

80

100

Normalized Reward

Door-expert-v1

TD3+RVS (1 layer)

TD3+RVS (2 layers)

TD3+RVS (3 layers)

ODT

TD3+ODT (ours)

H Computational Resources

We conduct all experiments with a single NVIDIA RTX 2080Ti GPU on an Ubuntu 18.04 server equipped with 72 Intel Xeon Gold 6254 CPUs @ 3.10GHz. Mujoco experiments takes about 6 -8 hours, and the bottleneck is the gradient update; about 50% time is spent on backpropagation and update of parameters. Our critic appended to ODT only takes up about 20% time to train, in which 90% of the critic training time is spent on decision transformer inference to get action for 'next state'. For the actor, the training overhead of our method is negligible since it only contains an MLP critic inference to get the Q-value. Therefore, overall our method only uses 20% extra time compared to ODT for training, but attains much better results.

I Dataset and Algorithm Licenses

Our code is developed upon multiple algorithm repositories and environment testbeds.

Algorithm Repositories. We implement our method on the basis of online decision transformer repository, which has a CC BY-NC 4.0 license. We also refer to IQL [28], PDT [64] and TD3+BC [20] repository when running baselines, all of which have MIT licenses.

Environment Testbeds. We utilize OpenAI gym [8], MuJoCo [58], and D4RL [19] as testbed, which have an MIT license, an Apache-2.0 license, and an Apache-2.0 license respectively.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Please refer to the abstract and introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: See Sec. 6.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: See https://github.com/KaiYan289/RL_as_Vitamin_for_Online_ Decision_Transformers for the code. The data is available in public D4RL dataset.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: See Sec. 4 and Appendix G.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our research in the paper fully conforms with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We attach our code in the supplementary material, and has published it on Github ( https://github.com/KaiYan289/RL_as_Vitamin_for_Online_ Decision_Transformers ).

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors consider the problem of online fine-tuning decision transformers. Current approaches for this problem do not work well when the offline data is low-quality. The authors analyze the online DT algorithm theoretically and show why DT-induced policy update would not work if the RTG used for conditioning is much higher than RTG seen in the dataset. They then propose combining the ODT update with RL gradients, in particular, the ones obtained from the TD3 model. This approach was then tested through Adroit, Antmaze, and Mujoco experiments, where the method obtained strong results. Finally, the authors perform additional analyses and make improvements to their proposed approach.\n\nSTRENGTHS:\n- The work considers the important problem of finetuning decision transformers. I see this as a significant issue, as we need strong online fine-tuning methods to better leverage pre-trained models.\n- The proposed method seems to achieve quite good performance in the presented experiments, at least when not taking the statistical significance of the results into account. In my opinion, the authors also consider a sufficient set of benchmarks and baselines.\n- The study is backed by theoretical results.\n- Related work is well-described to the best of my knowledge.\n- The appendix includes many additional results and extended analysis that might be of interest to the reader.\n\nWEAKNESSES:\n- The empirical results seem to be noisy, and a more careful approach to statistical significance is needed. Looking at, e.g., Figure 5, I'm not sure if TD3+ODT is better than TD3 as the plots overlap quite a lot, but the corresponding Table 1 makes it look like TD3+ODT is a clear winner. \n    - This includes bolding \"the best solution\" in some of the tables. If one of the solutions achieves a score of 101 and the other of 100, in the RL regime, it is usually not possible to tell which one is statistically better and the fair thing to do is to bold both. \n    - I didn't find the information on how many seeds are used for each experiment, and this is very important in the RL context.\n    - In summary, I would suggest the authors adopt the recommended practices of [1] to have a clearer picture of the results.  \n- I think the baselines considered here should be made stronger\n    - The TD3 baseline collapses right after fine-tuning starts in many cases. It looks like it might be caused by forgetting which was shown to be a problem in RL fine-tuning [2]. I wonder if combining TD3 with some forgetting mitigation technique would help.\n    - The authors point out that their method improves on TD3+BC in two ways: architecture (MLP -> DT) and objective (BC -> RvS). I think these two axes of improvement should be investigated separately. That is, introducing two additional baselines would show us where the improvements come from: TD3+BC+Transformer and TD3+RvS (with MLP rather than DT). \n\n[1] Agarwal, Rishabh, et al. \"Deep reinforcement learning at the edge of the statistical precipice.\" Advances in neural information processing systems 34 (2021): 29304-29320. \\\n[2] Wolczyk, Maciej, et al. \"Fine-tuning Reinforcement Learning Models is Secretly a Forgetting Mitigation Problem.\" Forty-first International Conference on Machine Learning.\n\nQUESTIONS:\nCurrently, I would be inclined to accept the paper if not for the problems with the evaluation. As such, I would like to ask the authors:\n- Please discuss and address the questions about the evaluation setting, as raised in the Weaknesses section above. If this is sufficiently resolved, I'd be happy to increase the score.\n- Please consider introducing stronger baselines. This is not as crucial, but it would make the paper stronger.\n\nLess important but interesting to discuss:\n- The authors say: \"training a transformer-based value function estimator is quite hard.\" I agree with this opinion based on my own experiences, but I don't think there has been a thorough discussion of this in the literature. The papers referenced by the authors do not really address this issue well in my opinion. So, it's not a weakness of the paper, but I would like to ask the authors where they think the problems of value learning in transformers come from (or maybe even their online training in general).\n\nLIMITATIONS:\nAlthough the authors do discuss the limitations of their work, I think there are at least two more points that should be discussed:\n* The method has not been tested on image-based datasets and environments.\n* There is an increased computational cost as compared to a regular ODT, since we have to train the value functions.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-11"}

comments: SUMMARY:
This paper presents a method for fine-tuning decision transformers (DT) online. The proposal is to integrate TD3 with DT such that the gradients from TD3 can help the online policy to explore highly rewarding trajectories, hence further improving the agent performance. It can be seen as a variant of TD3+BC with the MLP being replaced by a transformer. The experimental results show the proposed method outperforms several baselines, and is effective even when the offline dataset is of low quality, a setting which traditional online DT cannot do well.

STRENGTHS:
- The topic of improving DT for online fine-tuning is worth exploring. The way this paper approaches the topic is reasonable.
- The empirical results are good.

WEAKNESSES:
- The TD3 gradients for policy update are not clear. For example, in equation 2, it seems that $Q$ is a constant if you optimize $\mu^{RL}$? Or is $a_t$ predicted by $\mu^{RL}$?
- Some notations are a bit confusing. For example, there are RTG, RTG_real, RTG_eval. Better to clearly define them first.

QUESTIONS:
- Could you please explain more about the point (2) in line 182: how does DT prioritize trajectories with higher return?

LIMITATIONS:
- Compared to TD3+BC, the proposed method uses transformer instead of MLP. It costs much more computation to gain better performance. Such limitation is not mentioned in the paper.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-12

comments: SUMMARY:
The authors introduce a novel framework for improving the performance of Online Decision Transformer through adding TD3 gradients to the fine-tuning ODT objective. This is motivated by a theoretical analysis of ODT, that highlights an issue with low-reward, sub-optimal pretraining. The authors also provide an extensive empirical investigation.

STRENGTHS:
* Paper is clearly written, and well motivated. The detailed explanation on the preliminaries was useful to the reader.
* The contribution of TD3+ODT appears novel, original, and of high significance to the community.

WEAKNESSES:
* Overclaim of the contribution of “We propose a simple yet effective method to boost the performance of online finetuning of decision transformers.” The empirical results of relocate-expert-v1 show the opposite, where ODT outperforms TD3+ODT. Perhaps the authors can discuss this result and/or refine the claim to environments with particular properties where the expectation is that TD3+ODT outperforms ODT.
* Minor: Section 3.1 could highlight the conceptual figure of the plot of Figure 1, c, to help the reader understand the graph at the beginning of section 3.1.

QUESTIONS:
* Can the authors comment or perform an ablation with ODT+DDPG for all the main experiments as well?
* What is the impact of context length on your approach?
* For the main experiments how many random seed runs are each experimental result over? E.g. in Figure 2.

LIMITATIONS:
Limitations are adequately discussed in section 6.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the challenge of online finetuning of online decision transformers (ODT). Theoretical results are provided to show the target return-to-go can hamper finetune. The authors propose to have TD3 gradients added to ODT finetune and improve its performance especially when pretrained with low-reward data. Extensive empirical results are provided to show the proposed method can achieve stronger or competitive results across a large number of environments.\n\nSTRENGTHS:\n**originality**\n- The idea to combine TD3 gradient and ODT training is quite novel. The theoretical results can also be considered novel findings. \n\n**quality**\n- Overall good quality, the paper is well-written. Figures are easy to read and extensive experiments and ablations are provided. \n\n**clarity**\n- Overall clear and easy to follow. \n\n**significance**\n- The empirical results help improve our understanding of DT methods and how to better finetune them. The results are quite strong. The extensive experiments across different benchmarks make the results more convincing. The theoretical results add to significance.\n\nWEAKNESSES:\n- Maybe I missed something but I feel it might be good to have ODT baselines that try to tackle online finetuning with alternative (and more naive) methods. For example, authors argue that a main difficulty when finetuning DT is that when pretrained with low-reward data, the target RTG at finetuning stage is simply hard to obtain. But if we provide a realistic target RTG initially (based on pretrain data), and gradually increase it, will that help performance? Another thing is if we have a better exploration policy, will that help ODT finetune to the same extent? \n- Adding the TD3 component can make the training more complex and slower.\n\nQUESTIONS:\n- Will alternative strategies that improve exploration at finetune phase help ODT to the same extent? \n- The random datasets are not commonly tested in previous papers, when the baselines are tested on these datasets, are the hyperparameters tuned for these datasets?\n\nLIMITATIONS:\nSome limitations are discussed in section 6.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}