Reinforcement Learning Gradients as Vitamin for Online Finetuning Decision Transformers

Authors: kai  yan
alexander g schwing
yu-xiong  wang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-08 22:06:01

Abstract: Decision Transformers have recently emerged as a new and compelling paradigm for offline Reinforcement Learning (RL), completing a trajectory in an autoregressive way. While improvements have been made to overcome  initial shortcomings, online finetuning of decision transformers has been surprisingly under-explored. The widely adopted state-of-the-art Online Decision Transformer (ODT) still struggles when pretrained with low-reward offline data. In this paper, we theoretically analyze the online-finetuning of the decision transformer,  showing that the commonly used Return-To-Go (RTG) that's far from the expected return hampers the online fine-tuning process. This problem, however, is well-addressed by the value function and advantage of standard RL algorithms. As suggested by our analysis, in our experiments, we hence find that simply adding TD3 gradients to the finetuning process of ODT effectively improves the online finetuning performance of ODT, especially if ODT is pretrained with low-reward offline data. These findings provide new directions to further improve decision transformers.

Main paper: 
 introduction: 
 While Reinforcement Learning (RL) has achieved great success in recent years [55, 31] , it is known to struggle with several shortcomings, including training instability when propagating a Temporal Difference (TD) error along long trajectories [14] , low data efficiency when training from scratch [67] , and limited benefits from more modern neural network architectures [12] . The latter point differs significantly from other parts of the machine learning community such as Computer Vision [17] and Natural Language Processing [11] .To address these issues, Decision Transformers (DTs) [14] have been proposed as an emerging paradigm for RL, introducing more modern transformer architectures into the literature rather than the still widely used Multi-Layer Perceptrons (MLPs). Instead of evaluating state and state-action pairs, a DT considers the whole trajectory as a sequence to complete, and trains on offline data in a supervised, auto-regressive way. Upon inception, DTs have been improved in various ways, mostly dealing with architecture changes [37] , the token to predict other than return-to-go [22] , addressing the problem of being overly optimistic [46] , and the inability to stitch together trajectories [5] . Significant and encouraging improvements have been reported on those aspects. However, one fundamental issue has been largely overlooked by the community: offline-to-online RL using decision transformers, i.e., finetuning of decision transformers with online interactions. Offline-to-online RL [72, 41] is a widely studied sub-field of RL, which combines offline RL learning from given, fixed trajectory data and online RL data from interactions with the environment. By first training on offline data and then finetuning, the agent can learn a policy with much greater data efficiency, while calibrating the out-of-distribution error from the offline dataset. Unsurprisingly, this sub-field has become popular in recent years.While there are numerous works in the offline-to-online RL sub-field [35, 28, 62] , surprisingly few works have discussed the offline-to-online finetuning ability of decision transformers. While there is work that discusses finetuning of decision transformers predicting encoded future trajectory information [64] , and work that finetunes pretrained decision transformers with PPO in multi-agent RL [38] , the current widely adopted state-of-the-art is the Online Decision Transformer (ODT) [74] : the decision transformer training is continued on online data following the same supervised-learning paradigm as in offline RL. However, this method struggles with low-reward data, as well as with reaching expert-level performance due to suboptimal trajectories [41] (also see Sec. 4).To address this issue and enhance online finetuning of decision transformers, we theoretically analyze the decision transformer based on recent results [7] , showing that the commonly used conditioning on a high Return-To-Go (RTG) that's far from the expected return hampers results. To fix, we explore the possibility of using tried-and-true RL gradients. Testing on multiple environments, we find that simply combining TD3 [21] gradients with the original auto-regressive ODT training paradigm is surprisingly effective: it improves results of ODT, especially if ODT is pretrained with low-reward offline data.Our contributions are summarized as follows:1) We propose a simple yet effective method to boost the performance of online finetuning of decision transformers, especially if offline data is of medium-to-low quality;2) We theoretically analyze the online decision transformer, explain its "policy update" mechanism when using the commonly applied high target RTG, and point out its struggle to work well with online finetuning;3) We conduct experiments on multiple environments, and find that ODT aided by TD3 gradients (and sometimes even the TD3 gradient alone) are surprisingly effective for online finetuning of decision transformers.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: Please refer to the abstract and introduction.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
preliminaries: 
 Markov Decision Process. A Markov Decision Process (MDP) is the basic framework of sequential decision-making. An MDP is characterized by five components: the state space S, the action space A, the transition function p, the reward r, and either the discount factor γ or horizon H. MDPs involve an agent making decisions in discrete steps t ∈ {0, 1, 2, . . . }. On step t, the agent receives the current state s t ∈ S, and samples an action a t ∈ A according to its stochastic policy π(a t |s t ) ∈ ∆(A), where ∆(A) is the probability simplex over A, or its deterministic policy µ(s t ) ∈ A. Executing the action yields a reward r(s t , a t ) ∈ R, and leads to the evolution of the MDP to a new state s t+1 , governed by the MDP's transition function p(s t+1 |s t , a t ). The goal of the agent is to maximize the total reward t γ t r(s t , a t ), discounted by the discount factor γ ∈ [0, 1] for infinite steps, or Online Decision Transformer (ODT). ODT has two stages: offline pre-training which is identical to classic DT training, and online finetuning where trajectories are iteratively collected and the policy is updated via supervised learning. Specifically, the action a t at step t during rollouts is computed by the deterministic policy µ DT (s t-T :t , a t-T :t-1 , RTG t-T :t , T = T eval , RTG = RTG eval ), 1 or sampled from the stochastic policy π DT (a t |s t-T :t , a t-T :t-1 , RTG t-T :t , T = T eval , RTG = RTG eval ). Here, T is the context length (which is T eval in evaluation), and RTG eval ∈ R is the target return-to-go. The data buffer, initialized with offline data, is gradually replaced by online data during finetuning.When updating the policy, the following loss (we use the deterministic policy as an example, and thus omit the entropy regularizer) is minimized: Ttrain t=1 µ DT (s 0:t , a 0:t-1 , RTG 0:t , RTG = RTG real , T = t) -a t 2 2 .(1)Note, T train is the training context length and RTG real is the real return-to-go. For better readability, we denote {s x+1 , s x+2 , . . . , s y }, x, y ∈ N as s x:y (i.e., left exclusive and right inclusive), and similarly {a x+1 , a x+2 , . . . , a y } as a x:y and {RTG x+1 , . . . , RTG y } as RTG x:y . Specially, index x = y represents an empty sequence. For example, when t = 1, a 0:0 is an empty action sequence as the decision transformer is not conditioned on any past action.One important observation: the decision transformer is inherently off-policy (the exact policy distribution varies with the sampled starting point, context length and return-to-go), which effectively guides our choice of RL gradients to off-policy algorithms (see Appendix C for more details).TD3. Twin Delayed Deep Deterministic Policy Gradient (TD3) [21] is a state-of-the-art online off-policy RL algorithm that learns a deterministic policy a = µ RL (s). It is an improved version of an actor-critic (DDPG [32] ) with three adjustments to improve its stability: 1) Clipped double Qlearning, which maintains two critics (estimators for expected return) Q ϕ1 , Q ϕ2 : |S| × |A| → R and uses the smaller of the two values (i.e., min (Q ϕ1 , Q ϕ2 )) to form the target for TD-error minimization. Such design prevents overestimation of the Q-value; 2) Policy smoothing, which adds noise when calculating the Q-value for the next action to effectively prevent overfitting; and 3) Delayed update, which updates µ RL less frequently than Q ϕ1 , Q ϕ2 to benefit from a better Q-value landscape when updating the actor. TD3 also maintains a set of target networks storing old parameters of the actor and critics that are soft-updated with slow exponential moving average updates from the current, active network. In this paper, we adapt this algorithm to fit the decision transformer architecture so that it can be used as an auxiliary objective in an online finetuning process.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: See Sec. 6.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
method: 
 This section is organized as follows: we will first provide intuition why RL gradients aid online finetuning of decision transformers (Sec. 3.1), and present our method of adding TD3 gradients (Sec. 3.2). To further justify our intuition, we provide a theoretical analysis on how ODT fails to improve during online finetuning when pre-trained with low-reward data (Sec. 3.3).In order to understand why RL gradients aid online finetuning of decision transformers, let us consider an MDP which only has a single state s 0 , one step, a one dimensional action a ∈ [-1, 1] (i.e., a bandit with continuous action space) and a simple reward function r(a) = (a + 1) 2 if a ≤ 0 and r(a) = 1 -2a otherwise, as illustrated in Fig. 1 . In this case, a trajectory can be represented effectively by a scalar, which is the action. If the offline dataset for pretraining is of low quality, i.e., all actions in the dataset are either close to -1 or 1, then the decision transformer will obviously not generate trajectories with high RTG after offline training. As a consequence, during online finetuning, the new rollout trajectory is very likely to be uninformative about how to reach RTG eval , since it is too far from RTG eval . Worse still, it cannot improve locally either, which requires ∂RTG ∂a . However, the decision transformer yields exactly the inverse, i.e., ∂a ∂RTG . Since the transformer is not invertible (and even if the transformer is invertible, often the ground truth RTG(a) itself is not), we cannot ∂RTG , but local policy improvement requires the opposite, i.e., ∂RTG ∂a . Therefore, the agent cannot recover if the current policy conditioning on high target RTG does not actually lead to high real RTG, which is very likely when the target RTG is too far from the pretrained policy and out-of-distribution. By adding a small coefficient for RL gradients, the agents can improve locally, which leads to better performance. show, DDPG and ODT+DDPG manage to maximize reward and find the correct optimal action quickly, while ODT fails to do so. Panel (c) shows how a DDPG/ODT+DDPG critic (from light blue/orange to dark blue/red) manages to fit ground truth reward (green curve). Panel (d) shows that the ODT policy (changing from light gray to dark) fails to discover the hidden reward peak near 0 between two low-reward areas (near -1 and 1 respectively) contained in the offline data. Meanwhile, ODT+DDPG succeeds in finding the reward peak.easily estimate the former from the latter. Thus, the hope for policy improvement relies heavily on the generalization of RTG, i.e., policy yielded by high RTG eval indeed leads to better policy without any data as evidence, which is not the case with our constructed MDP and dataset.In contrast, applying traditional RL for continuous action spaces to this setting, we either learn a value function Q(s 0 , a) : R → R, which effectively gives us a direction of action improvement ∂Q(s0,a) ∂a (e.g., SAC [25] , DDPG [32] , TD3 [21] ), or an advantage A(s 0 , a) that highlights whether focusing on action a improves or worsens the policy (e.g., AWR [47] , AWAC [40] , IQL [28] ). Either way provides a direction which suggests how to change the action locally in order to improve the (estimated) return. In our experiment illustrated in Fig. 2 (see Appendix F for details), we found that RL algorithms like DDPG [32] can easily solve the aforementioned MDP while ODT fails. Thus, adding RL gradients aids the decision transformer to improve from given low RTG trajectories. While one may argue that the self-supervised training paradigm of ODT [74] can do the same by "prompting" the decision transformer to generate a high RTG trajectory, such paradigm is still unable to effectively improve the policy of the decision transformer pretrained on data with low RTGs. We provide a theoretical analysis for this in Sec. 3.3. In addition, we also explore the possibility of fixing this problem using other existing algorithms, such as JSRL [59] and slowly growing RTG (i.e., curriculum learning). However, we found that those algorithms cannot address this problem well. See Appendix G.8 for ablations.In this work, we mainly consider TD3 [21] as the RL gradient for online finetuning. There are two reasons for selecting TD3. First, TD3 is a more robust off-policy RL algorithm compared to other off-policy RL algorithms [54] . Second, the success of TD3+BC [20] indicates that TD3 is a good candidate when combined with supervised learning. A more detailed discussion and empirical comparison to other RL algorithms can be found in Appendix C.Generally, we simply add a weighted standard TD3 actor loss to the decision transformer objective. To do this, we follow classic TD3 and additionally train two critic networks Q ϕ1 , Q ϕ2 : S × A → R parameterized by ϕ 1 , ϕ 2 respectively. In the offline pretraining stage, we use the following objective for the actor:min µ DT E τ ∼D 1 T train Ttrain t=1 -αQ ϕ1 (s t , µ DT (s 0:t , a 0:t-1 , RTG 0:t , RTG = RTG real , T = t))+ ∥µ DT (s 0:t , a 0:t-1 , RTG 0:t , RTG = RTG real , T = t) -a t ∥ 2 2 . (2)Here, α ∈ {0, 0.1} is a hyperparameter, and the loss sums over the trajectory segment. For critics Q ϕ1 , Q ϕ2 , we use the standard TD3 critic lossEQUATIONwhere τ = {s 0:Ttrain+1 , a 0:Ttrain , RTG 0:Ttrain+1 , d 0:Ttrain , r 0:Ttrain , RTG = RTG real } is the trajectory segment sampled from buffer D that stores the offline dataset. Further, d t indicates whether the trajectory ends on the t-th step (true is 1, false is 0), Q min is the target to fit, Q ϕi,tar is produced by the target network (stored old parameter), z t is the context for "next state" at step t. µ RL tar is the target network for the actor (i.e., decision transformer). For an n-dimensional action, clip(a, x, y), a ∈ R n , y ∈ R n , z ∈ R n means clip a i to [y i , z i ] for i ∈ {1, 2, . . . , n}. a low ∈ R n and a high ∈ R n are the lower and upper bound for every dimension respectively.To demonstrate the impact on aiding the exploration of a decision transformer, in this work we choose the simplest form of a critic, which is reflective, i.e., only depends on the current state. This essentially makes the Q-value an average of different context lengths sampled from a near-uniform distribution (see Appendix D for the detailed reason and distribution for this). The choice is based on the fact that training a transformer-based value function estimator is quite hard [45] due to increased input complexity (i.e., noise from the environment) which leads to reduced stability and slower convergence. In fact, to avoid this difficulty, many recent works on Large Language Models (LLMs) [13] and vision models [48] which finetune with RL adopt a policy-based algorithm instead of an actor-critic, despite a generally lower variance of the latter. In our experiments, we also found such a critic to be much more stable than a recurrent critic network (see Appendix G for ablations).During online finetuning, we again use Eq. ( 2) and Eq. ( 3), but always use α = 0.1 for Eq. ( 2).While the training paradigm resembles that of TD3+BC, our proposed method improves upon TD3+BC in the following two ways: 1) Architecture. While TD3+BC uses MLP networks for single steps, we leverage a decision transformer, which is more expressive and can take more context into account when making decisions. 2) Selected instead of indiscriminated behavior cloning.Behavior cloning mimics all data collected without regard to their reward, while the supervised learning process of a decision transformer prioritizes trajectories with higher return by conditioning action generation on higher RTG. See Appendix G.9 for an ablation.As mentioned in Sec. 3.1, it is the goal of ODT to "prompt" a policy with a high RTG, i.e., to improve a policy by conditioning on a high RTG during online rollout. However, beyond the intuition provided in Sec. 3.1, in this section, we will analyze more formally why such a paradigm is unable to improve the policy given offline data filled with low-RTG trajectories.Our analysis is based on the performance bound proved by Brandfonbrener et al. [7] . Given a dataset drawn from an underlying policy β and given its RTG distribution P β (either continuous or discrete), under assumptions (see Appendix E), we have the following tight performance bound for a decision transformer with policy π DT (a|s, RTG eval ) conditioned on RTG eval :EQUATIONHere, α f = inf s1 P β (RTG real = RTG eval |s 1 ) for every initial state s 1 , ϵ > 0 is a constant, H is the horizon of the MDP. 2 Based on this tight performance bound, we will show that with high probability, 1 α f grows superlinearly with respect to RTG eval . If true, then the RTG real term (i.e., the actual return from online rollouts) must decrease to fit into the tight bound, as RTG eval grows.To show this, we take a two-step approach: First, we prove that the probability mass of the RTG distribution is concentrated around low RTGs, i.e., event probability Pr β (RTG -E β (RTG|s) ≥ c|s) for c > 0 decreases superlinearly with respect to c. For this, we apply the Chebyshev inequality, which yields a bound of O 1 c 2 . However, without knowledge on P β (RTG|s), the variance can be made arbitrarily large by high RTG outliers, hence making the bound meaningless.Fortunately, we have knowledge about the RTG distribution P β (RTG|s) from the collected data. If we refer to the maximum RTG in the dataset via RTG βmax and if we assume all rewards are non-negative, then all trajectory samples have an RTG in [0, RTG βmax ]. Thus, with adequate prior distribution, we can state that with high probability 1 -δ, the probability mass is concentrated in the low RTG area. Based on this, we can prove the following lemma: Lemma 1. (Informal) Assume rewards r(s, a) are bounded in [0, R max ], 3 and RTG eval ≥ RTG βmax . Then with probability at least 1 -δ, we have the probability of event Pr β bounded as follows:EQUATIONwhere δ depends on the number of trajectories in the dataset and prior distribution (see Appendix E for a concrete example and a more accurate bound). V β (s) is the value function of the underlying policy β(a|s) that generates the dataset, for which we have V β (s) = E β (RT G|s).The second step uses the bound of probability mass Pr β (RTG ≥ c|s) to derive the bound for α f . For the discrete case where the possibly obtained RTGs are finite or countably infinite (note, state and action space can still be continuous), this is simple, as we haveEQUATIONThus α f = inf s1 P β (RTG|s 1 ) can be conveniently bounded by Lemma 1. For the continuous case, the proof is more involved as probability density P β (RTG|s) can be very high on an extremely short interval of RTG, making the total probability mass arbitrarily small. However, assuming that P β (RTG|s) is Lipschitz when RTG ≥ RTG βmax (i.e., RTG area not covered by dataset), combined with the discrete distribution case, we can still get the following (see Appendix E for proof): Corollary 1. (Informal) If the RTG distribution is discrete (i.e., number of possible different RTGs are at most countably infinite), then with probability at least-1 -δ, 1 α f grows on the order of Ω(RTG 2 eval ) with respect to RTG eval . For continuous RTG distributions satisfying a Lipschitz continuous RTG density p β , 1 α f grows on the order of Ω(RTG 1.5 eval ).Here, Ω(•) refers to the big-Omega notation (asymptotic lower bound).Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [Yes]• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
experiments: 
 In this section, we aim to address the following questions: a) Does our proposed solution for decision transformers indeed improve its ability to cope with low-reward pretraining data. b) Is improving what to predict, while still using supervised learning, the correct way to improve the finetuning ability of decision transformers? c) Does the transformer architecture, combined with RL gradients, work better than TD3+BC? d) Is it better to combine the use of RL and supervised learning, or better to simply abandon the supervised loss in online finetuning? e) How does online decision transformer with TD3 gradient perform compared to other offline RL algorithms? f) How much does TD3 improve over DDPG which was used in Fig. 2 ?Baselines. In this section, we mainly compare to six baselines: the widely recognized state-of-the-art DT for online finetuning, Online Decision Transformer (ODT) [74] ; PDT, a baseline improving over ODT by predicting future trajectory information instead of return-to-go; TD3+BC [20] , a MLP offline RL baseline; TD3, an ablated version of our proposed solution where we use TD3 gradients only for decision transformer finetuning (but only use supervised learning of the actor for offline pretraining); IQL [28] , one of the most popular offline RL algorithms that can be used for online finetuning; DDPG [32] +ODT, which is the same as our approach but with DDPG instead of TD3 gradients (for ablations using SAC [25] , IQL [28] , PPO [52] , AWAC [40] and AWR [47] , see Appendix C). Each of the baselines corresponds to one of the questions a), b), c), d), e) and f) above.Metrics. We use the normalized average reward (same as D4RL's standard [19] ) as the metric, where higher reward indicates better performance. If the final performance is similar, the algorithm with fewer online examples collected to reach that level of performance is better. We report the reward curve, which shows the change of the normalized reward's mean and standard deviation with 5 different seeds, with respect to the number of online examples collected. The maximum number of steps collected is capped at 500K (for mujoco) or 1M (for other environments). We also report evaluation results using the rliable [3] library in Fig. 7 of Appendix B.Experimental Setup. We use the same architecture and hyperparameters such as learning rate (see Appendix F.2 for details) as ODT [74] . The architecture is a transformer with 4 layers and 4 heads in each layer. This translates to around 13M parameters in total. For the critic, we use Multi-Layer Perceptrons (MLPs) with width 256 and two hidden layers and ReLU [1] activation function. Specially, for the random dataset, we collect trajectories until the total number of steps exceeds 1000 in every epoch, which differs from ODT, where only 1 trajectory per epoch is collected. This is because many random environments, such as hopper, have very short episodes when the agent does not perform well, which could lead to overfitting if only a single trajectory is collected per epoch. For fairness, we use this modified rollout for ODT in our experiments as well. Not doing so does not affect ODT results since it does generally not work well on random datasets, but will significantly increase the time to reach a certain number of online transitions. After rollout, we train the actor for 300 gradient steps and the critic for 600 steps following TD3's delayed update trick.Environment and Dataset Setup. We test on four difficult robotic manipulation tasks [49] , which are the Pen, Hammer, Door and Relocate environment. For each environment, we test three different datasets: expert, cloned and human, which are generated by a finetuned RL policy, an imitation learning policy and human demonstration respectively. See Appendix F.1 for details.Results. Fig. 3 shows the performance of each method on Adroit before and after online finetuning. TD3+BC fails on almost all tasks and often diverges with extremely large Q-value during online finetuning. ODT and PDT perform better but still fall short of the proposed method, TD3+ODT. Note, IQL, TD3 and TD3+ODT all perform decently well (with similar average reward as shown in Tab. 2 in Appendix B). However, we found that TD3 often fails during online finetuning, probably because the environments are complicated and TD3 struggles to recover from a poor policy generated during online exploration (i.e., it has a catastrophic forgetting issue). To see whether there is a simple fix, in Appendix G.7, we ablate whether an action regularizer pushing towards a pretrain policy similar to TD3+BC helps, but find it to hinder performance increase in other environments. IQL is overall much more stable than TD3, but improves much less during online finetuning than TD3+ODT. ODT can achieve good performance when pretrained on expert data, but struggles with datasets of lower quality, which validates our motivation. DDPG+ODT starts out well in the online finetuning stage but fails quickly, probably because DDPG is less stable compared to TD3. Figure 4 : Reward curves for each method in Antmaze environments. IQL works best on the large maze, while our proposed method works the best on the medium maze and umaze. DDPG+ODT works worse than our method and IQL but much better than the rest of the baselines, which again validates our motivation that adding RL gradients to ODT is helpful.Environment and Dataset Setup. We further test on a harder version of the Maze2D environment in D4RL [19] where the pointmass is substituted by a robotic ant. We study six different variants, which are umaze, umaze-diverse, medium-play, medium-diverse, large-play and large-diverse.Results. Fig. 4 lists the results of each method on umaze and medium maze before and after online finetuning (see Appendix C for reward curves and Appendix B for results summary on large antmaze). TD3+ODT works the best on umaze and medium maze, and significantly outperforms TD3. This shows that RL gradients alone are not enough for offline-to-online RL of the decision transformer. Though TD3+ODT does not work on large maze, we found that IQL+ODT works decently well. However, we choose TD3+ODT in this work because IQL+ODT does not work well on the random datasets. This is probably because IQL aims to address the Out-Of-Distribution (OOD) estimation problem [28] , which makes it better at utilizing offline data but worse at online exploration. See Appendix C for a detailed discussion and results. DDPG+ODT works worse than TD3+ODT but much better than baselines except IQL.Environment and Dataset Setup. We further test on four widely recognized standard environments [58] , which are the Hopper, Halfcheetah, Walker2d and Ant environment. For each environment, we study three different datasets: medium, medium-replay, and random. The first and second one contain trajectories of decent quality, while the last one is generated with a random agent. Table 1 : Average reward for each method in MuJoCo environments before and after online finetuning.The best performance for each environment is highlighted in bold font, and any result > 90% of the best performance is underlined. To save space, the name of the environments and datasets are abbreviated as follows: for the environments Ho=Hopper, Ha=HalfCheetah, Wa=Walker2d, An=Ant; for the datasets M=Medium, MR=Medium-Replay, R=Random. The format is "final(+increase after finetuning)". The proposed solution performs well.Results. Fig. 6 shows the results of each method on MuJoCo before and after online finetuning. We observe that autoregressive-based algorithms, such as ODT and PDT, fail to improve the policy on MuJoCo environments, especially from low-reward pretraining with random datasets. With RL gradients, TD3+BC and IQL can improve the policy during online finetuning, but less than a decision transformer (TD3 and TD3+ODT). In particular, we found IQL to struggle on most random datasets, which are well-solved by decision transformers with TD3 gradients. TD3+ODT still outperforms TD3 with an average final reward of 88.51 vs. 84.23. See Fig. 6 in Appendix B for reward curves.Ablations on α. Fig. 5 (a) shows the result of using different α (i.e., RL coefficients) on different environments. We observe an increase of α to improve the online finetuning process. However, if α is too large, the algorithm may get unstable.Ablations on evaluation context length T eval . Fig. 5 (b) shows the result of using different T eval on halfcheetah-medium-replay-v2 and hammer-cloned-v1. The result shows that T eval needs to be balanced between more information for decision-making and potential training instability due to a longer context length. As shown in the halfcheetah-medium-replay-v2 result, T eval too long or too short can both lead to performance drops. More ablations are available in Appendix G.  
related work: 
 Online Finetuning of Decision Transformers. While there are many works on generalizing decision transformers (e.g., predicting waypoints [5] , goal, or encoded future information instead of return-togo [22, 5, 57, 36] ), improving the architecture [37, 16, 53, 65] or addressing the overly-optimistic [46] or trajectory stitching issue [63] ), there is surprisingly little work beyond online decision transformers that deals with online finetuning of decision transformers. There is some loosely related literature: MADT [31] proposes to finetune pretrained decision transformers with PPO. PDT [64] also studies online finetuning with the same training paradigm as ODT [74] . QDT [66] uses an offline RL algorithm to re-label returns-to-go for offline datasets. AFDT [76] and STG [75] use decision transformers offline to generate an auxiliary reward and aid the training of online RL algorithms. A few works study in-context learning [33, 34] and meta-learning [60, 30] of decision transformers, where improvements with evaluations on new tasks are made possible. However, none of the papers above focuses on addressing the general online finetuning issue of the decision transformer.Transformers as Backbone for RL. Having witnessed the impressive success of transformers in Computer Vision (CV) [17] and Natural Language Processing (NLP) [11] , numerous works also studied the impact of transformers in RL either as a model for the agent [45, 38] or as a world model [39, 50] . However, a large portion of state-of-the-art work in RL is still based on simple Multi-Layer Perceptrons (MLPs) [35, 28] . This is largely because transformers are significantly harder to train and require extra effort [45] , making their ability to better memorize long trajectories [42] harder to realize compared to MLPs. Further, there are works on using transformers as feature extractors for a trajectory [37, 45] and works that leverage the common sense of transformer-based Large Language Model's for RL priors [10, 9, 70] . In contrast, our work focuses on improving the new "RL via Supervised learning" (RvS) [7, 18] paradigm, aiming to merge this paradigm with the benefits of classic RL training.Offline-to-Online RL. Offline-to-online RL bridges the gap between offline RL, which heavily depends on the quality of existing data while struggling with out-of-distribution policies, and online RL, which requires many interactions and is of low data efficiency. Mainstream offline-to-online RL methods include teacher-student [51, 6, 59, 72] and out-of-distribution handling (regularization [21, 29, 62] , avoidance [28, 23] , ensembles [2, 15, 24] ). There are also works on pessimistic Q-value initialization [69] , confidence bounds [26] , and a mixture of offline and online training [56, 73] . However, all the aforementioned works are based on Q-learning and don't consider decision transformers. 
conclusion: 
 In this paper, we point out an under-explored problem in the Decision Transformer (DT) community, i.e., online finetuning. To address online finetuning with a decision transformer, we examine the current state-of-the-art, online decision transformer, and point out an issue with low-reward, sub-optimal pretraining. To address the issue, we propose to mix TD3 gradients with decision transformer training. This combination permits to achieve better results in multiple testbeds. Our work is a complement to the current DT literature, and calls out a new aspect of improving decision transformers.Limitations and Future Works. While our work theoretically analyzes an ODT issue, the conclusion relies on several assumptions which we expect to remove in future work. Empirically, in this work we propose a simple solution orthogonal to existing efforts like architecture improvements and predicting future information rather than return-to-go. To explore other ideas that could further improve online finetuning of decision transformers, next steps include the study of other environments and other ways to incorporate RL gradients into decision transformers. Other possible avenues for future research include testing our solution on image-based environments, and decreasing the additional computational cost compared to ODT (an analysis for the current time cost is provided in Appendix H). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: See Appendix I.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: The Appendix is organized as follows. In Sec. A, we discuss the potential positive and negative social impact of the paper. Then, we summarize the performance shown in the main paper in Sec. B. After this, we will explain our choice of RL gradients in the paper in Sec. C, and why our critic serves as an average of policies generated by different context lengths in Sec. D. We then provide rigorous statements for the theroetical analysis appearing in the paper in Sec. E, and list the environment details and hyperparameters in Sec. F. We then present more experiment and ablation results in Sec. G. Finally, we list our computational resource usage and licenses of related assets in Sec. H and Sec. I respectively.Our work generally helps automation of decision-making by improving the use of online interaction data of a pretrained decision transformer agent. While this effort improves the efficiency of decisionmakers and has the potential to boost a variety of real-life applications such as robotics and resource allocation, it may also cause several negative social impacts, such as potential job losses, human de-skilling (making humans less capable of making decisions without AI), and misuse of technology (e.g., military).In this section, we summarize the average reward achieve by each method on different environments and datasets, where the result for Adroit is shown in Tab. 2, and the result for Antmaze is shown in Tab. 3. As the summary table for MuJoCo is already presented in Sec. 4, we show the reward curves in Fig. 6 . For a more rigorous evaluation, we also report other metrics including the median, InterQuartile Mean (IQM) and optimality gap using the rliable [3] library. See Fig. 7 for details. Breakdown analysis for each environment can be downloaded by browsing to https://kaiyan289. github.io/assets/breakdown_rliable.rar. C Why Do We Choose TD3 to Provide RL Gradients?In this section, we provide an ablation analysis on which RL gradient fits the decision transformer architecture best. Fig. 8 illustrates the result of using a pure RL gradient for online finetuning of a pretrained decision transformer (for those RL algorithms with stochastic policy, we adopt the same Table 3 : Average reward for each method in Antmaze Environments before and after online finetuning. The best result is marked in bold font and all results > 90% fo the best performance are underlined.To save space, the name of the environments and datasets are abbreviated as follows: U=Umaze, UD=Umaze-Diverse, MP=Medium-Play, MD=Medium-Diverse, LP=Large-Play and LD=Large-Diverse. U+M=Umaze and Medium maze. Our method performs the best on umaze and medium maze, while IQL performs the best on large maze. Both methods are much better than the rest on average. TD3+BC diverges on antmaze in our experiments. Figure 6 : Results on MuJoCo [58] Environments. The TD3 gradient significantly improves the overall performance of the decision transformer; autoregressive algorithms, such as ODT and PDT, fails to improve policy in most cases (especially on random dataset), while TD3+BC and IQL's improvement during finetuning is generally limited.architecture as ODT which outputs a squashed Gaussian distribution with trainable mean and standard deviation). It is apparent that TD3 [21] and SAC [25] are the RL algorithms that suit the decision transformer best. Fig. 9 further shows the performance comparison between a decision transformer with SAC+ODT mixed gradient and TD3+ODT mixed gradient (both with coefficient 0.1). The result shows that TD3 is the better choice when paired with supervised learning.Note, While PPO is generally closely related with transformers (e.g., Reinforcement Learning from Human Feedback (RLHF) [44] ), and was used in some prior work for online finetuning of decision transformers with a small, discrete action space [38] , in our experiments, we find PPO generally does not work with the decision transformer architecture. The main reason for this is the importance sampling issue: PPO has the following objective for an actor π θ parameterized by θ:max θ min E (s,a)∼π θ old π θ (a|s) π θold (a|s) A π θ old (s, a), clip π θ (a|s) π θold (a|s) , 1 -ϵ, 1 + ϵ A π θ old (s, a) . (7)Here, π θold is the policy at the beginning of the training for the current epoch. Normally, the denominator of the red part, π θold (a|s), would be reasonably large, as the data is sampled from that distribution. However, because of the offline nature caused by different RTGs and context lengths at rollout and training time, the denominator for the red part in Eq. ( 7) could be very small in training, which will lead to a very small loss if A π θold (s, a) > 0. This introduces significant instability during the training process. Fig. 10 illustrates the instability and degrading performance of a PPO-finetuned decision transformer. Pr(reward ≤ x)dx). Our method indeed outperforms all baselines on Adroit, MuJoCo and antmaze (umaze and medium).In contrast, RLHF, does not exhibit such a problem: it does not use different return-to-go and context length in evaluation and training. Thus RLHF does not encounter the problem described above.Besides the RL gradients mentioned above, as IQL works well on large Antmazes, we also explore the possibility of using IQL as the RL gradient for decision transformer instead of TD3. We found that IQL gradients, when applied to the decision transformer, indeed lead to much better results on antmaze-large. However, IQL fails to improve the policy when the offline dataset subsumes very low reward trajectories, which does not conform with our motivation. This is probably because IQL, as an offline RL algorithm, aims to address out-of-distribution evaluation issue, which is a much more important source of improvement in exploration in the online case. Thus, we choose TD3 as the RL gradient applied to decision transformer finetuning in this work. Fig. 11 shows the result of adding TD3 gradient vs. adding IQL gradient on Antmaze-large-play-v2 and hopper-random-v2.Context Lengths?As we mentioned in Sec. 2, When updating a deterministic DT policy, the following loss is minimized:EQUATIONwhere T train is the training context length and RTG real is the real return-to-go. , SAC [25] , AWAC [40] , AWR [47] and PPO [52] . We also plot ODT's performance as a reference. The result shows that generally, TD3 and SAC work the best, while PPO does not work at all. However, if we consider a particular action a t in some trajectory τ of the dataset, during training (both offline pretraining and online finetuning), the policy generated by the decision transformer fitting a t will beEQUATIONT is actually sampled from a distribution U ′ (1, T train ) over integers between 1 and T train inclusive; this distribution U ′ is introduced by the randomized starting step of the sampled trajectory segments, and is almost a uniform distribution on integers, except that a small asymmetry is created because the context length will be capped at the beginning of each trajectory. See Fig. 12 for an illustration.Therefore, online decision transformers (and plain decision transformers) are actually trained to predict with every context length between 1 and T train . During the training process, the context length is randomly sampled according to U ′ , and a critic is trained to predict an "average value" for the policy generated with context length sampled from U ′ . In this section, we will state the theoretical analysis summarized in Sec. 3.3 more rigorously. We will first provide an explanation on how the decision transformer improves its policy during online finetuning, linking it to an existing RL method in Sec. E.1 and Sec. E.2. We will then bound its performance in Sec. E.3.Advantage-Weighted Actor Critic (AWAC) [40] is an offline-to-online RL algorithm, where the replay buffer is filled with offline data during offline pretraining and then supplemented with online experience during online finetuning. AWAC uses standard Q-learning to train the critic Q : |S| × |A| → R, and update the actor using weighted behavior cloning, where the weight is exponentiated advantage (i.e., exp A(s,a) λ where λ > 0 is some constant).We denote β as the underlying policy of the dataset, and P β as the distribution over states, actions or returns induced by β. Note such P β can be either discrete or continuous. By prior work [7] , for decision transformer policy π DT , we have the following formula holds for any return-to-go RTG ∈ R of the future trajectory: Based on Eq. ( 10), we have the following lemma:Lemma 2. For state-action pair (s, a) in an MDP, RTG ∈ R, assume the distributions of return-to-go (RTG) P β (RTG|s, a) and P β (RTG|s) are Laplace distributions with scale σ, then for any RTG large Step 𝑗𝑗 ≥ 𝑅𝑅 trainStep 𝑖𝑖 < 𝑅𝑅 train 𝑅𝑅 train (b) Distribution U ′ of T2 shows the distribution U ′ of T 2 ; while T ′ 2 for step j ≥ T train is uniformly sampled between 1 and T train because the start of the segment is uniformly sampled, T 2 for step i < T train will be capped at the start of the trajectory. Thus U ′ is not exactly uniform. Though the assumption is strong to form the exact same formula, it shows the basic idea of how we link P β (RTG) to its distance to V β (s) and Q β (s, a) in this paper.enough (more rigorously, RTG ≥ max{Q β (s, a), V β (s)}), π(a|s, RTG) is updated in the same way as AWAC. 4Proof. If return-to-go are Laplace distributions, then by the symmetric property of such distributions, the mean of RTG given s or (s, a) would be the expected future return, which by definition are value functions for β, i.e., V β (s) for P β (RTG|s) and Q β (s, a) for P β (RTG|s, a). See Fig. 13 as an illustration. As RTG eval ≥ max{Q β (s, a), V β (s)}, we haveEQUATIONAnd thus we haveP β (RTG|s,a) P β (RTG|s) = exp Q β (s,a)-V β (s) σ = exp( A β (s,a) σ ), where A β is the advantage function.While the Laplace distribution assumption in this lemma is strict and impractical for real-life applications, it gives us three crucial insights for a decision transformer:• For any state s or state-action pair (s, a), we haveE β [RTG|s, a] = Q β (s, a), E β [RTG|s] = V β (s), which is an important property of P β on RTG;• For decision transformer, the ability to improve the policy by collecting rollouts with high RTG, similar to AWAC, is closely related to advantage. In the above lemma, for example, if the two Laplace distributions have different scales σ V , σ Q , we will have the ratioP β (RTG|s,a) P β (RTG|s) being exp Q β (s,a) σ Q -V β (s) σ V; if the distributions are Gaussian, we will get similar results but with quadratic terms of Q β and V β .• Different from AWAC, the "policy improvement" of online decision transformers heavily relies on the global property of the return-to-go as RTG eval moves further away from Q β and V β . If the return-to-go is far away from the support of the data, we will have almost no data to evaluate P β , and its estimation can be very uncertain (let alone ratios). In this case, it is very unlikely for the decision transformer to collect rollouts with high RTG true and get further improvement. This is partly supported by the corollary of Brandfonbrener et al. [7] ,where the optimal conditioning return they found on evaluation satisfies RTG eval = V β (s 1 ) at the initial state s 1 . This is also supported by our single-state MDP experiment discussed in Sec. 3.1 and illustrated in Fig. 2 .Those important insights lead to the intuition that decision transformers, when finetuned online, lack the ability to improve "locally" from low RTG true data, and encourages to study the scenario where P β (RTG|s) and P β (RTG|s, a) are small.As mentioned in Sec. E.2, we study the performance of decision transformers in online finetuning when P β (RTG|s) and P β (RTG|s, a) is small. Specially, in this section, r(s, a) is not a reward function, but a reward distribution conditioned on (s, a); r i ∼ r(s i , a i ) is the reward obtained on i-th step. Such notation takes the noise of reward into consideration and forms a more general framework. Also, as the discrete or continuous property of β is important in this section, we will use Pr β to represent probability mass (for discrete distribution or cumulative distribution for continuous distribution) and p β to represent probability density (for probability density function for continuous distribution).By prior work [7] , we have the following performance bound tight up to a constant factor for decision transformer for every iteration of updates: Theorem E.1. For any MDP with transition function p(•|s, a) and reward random variable r(s, a), and any condition function f , assume the following holds:• Return coverage: P β (g = f (s 1 )|s 1 ) ≥ α f for any initial state s 1 ;• Near determinism: for any state-action pair (s, a), ∃ s ′ such that Pr(s ′ |s, a) ≥ 1 -ϵ, and ∃ r 0 (s, a) such that Pr(r(s, a) = r 0 (s, a)) ≥ 1 -ϵ;• Consistency of f : f (s) = f (s ′ ) + r for all s when transiting to next state s ′ .Then we haveEQUATION)where α f > 0, ϵ > 0 are constants, p ini is the initial state distribution, and H is the horizon of the MDP. π DT is the learned policy by Eq. (10).Proof. See Brandfonbrener et al. [7] .In our case, we define f (s) as follows: Definition E.2. f (s 1 ) = RTG eval for all initial states s 1 , f (s i+1 ) = f (s i ) -r i for the (i + 1)-th step following i-th step (i ∈ {1, 2, . . . , T -1}).Further, we enforce the third assumption in Thm. E.1 by including the cumulative reward so far in the state space (as described in the paper of Brandforbrener et al. [7] ). Under such definition, we have a tight bound on the regret between our target RTG eval and the true return-to-go RTG true = H i=1 r i by our learned policy at optimal, based on current replay buffer in online finetuning.We will now prove that under certain assumptions, 1 α f grows superlinearly with respect to RTG; as the bound is tight, the expected cumulative return term E τ =(s1,a1,...,s H ,a H )∼β H i=1 E ri∼r(si,ai) r i will be decreasing to meet the bounds.To do this, we start with the following assumptions: Assumption E.3. We assume the following statements to be true:• (Bounded reward) We assume the reward is bounded in [0, R max ] for any state-action pairs.• (High evaluation RTG) RTG eval ≥ RTG βmax , where RTG βmax is the largest RTG true in the dataset of n trajectories generated by β.• (Beta prior) We assign the prior distribution of RTG generated by policy β to be a Beta distribution Beta(1, 1) for the binomial likelihood of RTG falling on [0,RTG βmax ] or [RTG βmax , T R max ].Remark E.4. The Beta distribution can be changed to any reasonable distribution; we use Beta distribution only for a convenient derivation. Considering the fact that by common sense, trajectories with high return are very hard to obtain, we can further strengthen the conclusion by changing the prior distribution.We then prove the following lemma:Lemma 3. Under the Assumption E.3, given underlying policy β of the dataset, for any state s with value function V β (s) and any state-action pair (s, a) with Q-function Q β (s, a), any c ≥ 0 and RTG eval ∈ R, with probability 1 -δ , we haveEQUATION)where δ = 1 -CDF Beta(n+1,1) (ϵ), and the CDF is the cumulative distribution function.Proof. With the beta prior assumption in Assumption E.3, we know that with n samples where RTG ≤ RTG βmax , we have the posterior distribution to be Beta(n + 1, 1), i.e., with probability 1 -CDF Beta(n+1,1) (ϵ), we have Pr β (RTG ≥ RTG βmax ) ≤ ϵ for ϵ > 0.Thus, by Chebyshev inequality, we knowEQUATION)and a similar conclusion holds for Pr β RTG -Q β (s, a) ≥ c|s, a . Thus, the probability decays superlinearly with respect to RTG.Given this lemma, it remains to connect the bound of Pr β (RTG ≥ c 0 ) to P β (RTG = c 0 ) on RTG, c 0 ∈ R. For discrete distribution, the connection is straightforward: Pr β (RTG ≥ c 0 ) ≥ Pr β (RTG = c 0 ) = P β (RTG = c 0 ) for any condition s or (s, a).Thus, we immediately get the following corollary:Corollary 2. Assume the reward is bounded in [0, R max ] for any state and action, and the number of possible different return-to-go one can get is finite or countably infinite. Then for the f condition function defined in Def. E.2, with probability of at least 1-δ, we haveα f ≤ (1-ϵ)RTG βmax +ϵR 2 max T 2 -[V β (s)] 2 (RTG-V β (s)) 2 , i.e., 1α f grows in the order of Ω(RTG 2 eval ). Remark E.5. While the limitation on return-to-go seems strong theoretically, it is very easy to satisfy such assumption in practice because it has no requirement on the discreteness of state and action space. Such corollary can be applied on reward discretization with arbitrary precision (including implicit ones by float precision).For continuous distribution, to bound p β (RTG = c 0 ) with Pr β (RTG ≥ c 0 ) on RTG, we would need to assume that "peak" does not exist (see Fig. 14 for illustration), i.e. there does not exist cases where p β is large but Pr β is small. Thus, we made the following assumption: Assumption E.6. (Lipschitzness on uncovered RTG distribution) p β (RTG|s) is K V -Lipschitz where RTG is larger than RTG βmax . Remark E.7. The assumption is reasonable because when RTG is larger than any of the RTG true in the dataset, we have no data coverage for the performance of the underlying policy β under such RTG, and thus we can choose any inductive bias for β. Remark E.8. Note the Lipschitzness of p β does not rely on the Lipschitzness of the reward function. For example, consider a single-state, single-step MDP where we have a uniformly random policy a ∼ U (0, 1) with reward r(a) = 2 -1 a . The reward is clearly not Lipschitz on a ∈ (0, 1), but the distribution of r is p r (r 0 ) = p a (r -1 (r 0 ))• ∂r -1 (r0) ∂r = 1 (r-2) 2 , which is Lipschitz on (-∞, 1) as a ∈ (0, 1).With such assumption, we have the following corollary: Corollary 3. Under assumption E.6, for the f condition function defined in Def. E.2, we haveα f ≤ √ 2K V Ω RTG -1.5 eval , i.e., 1 α f grows in the order of Ω(RTG 1.5 eval ) with probability of at least 1 -δ.Under assumption E.6, by Lipschitzness, for any RTG where p β (RTG|s) > p 0 , we havep β (RTG + c|s) > p 0 -K V • c for any c ∈ [0, p0 K V ]. Thus, we know that if ∃ RTG 0 > RTG βmax such that p β (RTG 0 |s) > p 0 , then we have Pr β (RTG ≥ RTG 0 |s) > p 2 0 2K V (See Fig. 14 for an illustration). By the contra-positive statement of the above conclusion, we know thatEQUATIONand RTG eval is applicable to the inequality above by the high evaluation RTG assumption in Assumption E.3. We then apply the proof lemma 3, but apply the inequalityP (x -E[x] ≥ c) ≤ E(x-E[x]) 3 c 3instead of Chebyshev inequality which leads to the conclusion. Datasets. The dataset has a size of 128, with 100 actions uniformly sampled in (-1, 0.95), and the remaining 28 actions uniformly sampled in (0.5, 1). The dataset is designed to conceal the reward peak in the middle. DDPG and ODT+DDPG successfully recognized the reward peak but ODT failed. Environments. Adroit is a set of more difficult benchmark than Mujoco in D4RL, and is becoming increasingly popular in recent offline and offline-to-online RL works [28, 23] . We test four environments in adroit in our experiments: pen, hammer, door and relocate. Fig. 15 shows an illustration of the four environments. 1. Pen. Pen is a locomotion environment where the agent needs to control a robotic hand to manipulate a pen, such that its orientation matches the target. It has a 24-dimensional action space, each of which controls a joint on the wrist or fingers. The state space is 45-dimensional, which contains the pose of the palm, the angular position of the joints, and the pose of the target and current pen. 2. Hammer. Hammer is an environment where the agent needs to control a robotic hand to pick up a hammer and use it to drive a nail into a board. The action space is 26-dimensional, each of which corresponds to a joint on the hand. The state space is 46-dimensional, which describes the angular position of the fingers, the pose of the palm, and the status of hammer and nail. 3. Door. In the door environment, the agent needs to use a robotic hand to open a door by undoing the latch and swinging it. The environment has a 28-dimensional action space, which are the absolute angular positions of the hand joints. It also has 39-dimensional observation space which describes each joint, the pose of the palm, and the door with its latch. 4. Relocate. In the relocate environment, the agent needs to control a robotic hand to move a ball from its initial location towards a goal, both of which are randomized in the environment.The environment has a 30-dimensional action space which describes the angular position of the joints on the hand, and a 39-dimensional space which describes the hand as well as the ball and target. 4 : The size and the average and standard deviation of the normalized reward of the Adroit datasets from D4RL [19] used in our experiments.Datasets. For each of the four environments, we test our method across three different qualities of datasets: expert, cloned and human, all of which provided by the DAPG [49] repository. The expert dataset is generated by a fine-tuned RL policy; the cloned dataset is collected from an imitation policy on the demonstrations from the other two datasets; and the human dataset is collected from human demonstrations. Tab. 4 shows the size and average reward of each dataset.Environments. Antmaze is a more difficult version of Maze2D, where the agent controls a robotic ant instead of a point mass through the maze. It has a 27 dimensional-state space and a 8-dimensional action space. We test our method on six variants of antmaze: Umaze, Umaze-Diverse, Medium-Play, Medium-Diverse, Large-Play and Large-Diverse, where "Umaze", "Medium" and "Large" describes the size of the maze (see Fig. 16 for an illustration), and the "Diverse" and "Play" describes the type of the dataset. More specifically, "Diverse" means that in the offline dataset, the starting point and the goal of the agent are randomly generated, while "Play" means that the goal is generated by a handcraft design. "Umaze" without suffix is the simplest environment where both the starting point and the goal are fixed. Datasets. Similar to Adroit and MuJoCo, we test our method on datasets provided by D4RL. Tab. 5 shows the size and normalized reward of each dataset. Note, following IQL [28] and CQL [29] , we conduct reward shaping: subtracting from all rewards in the dataset and environment the value 1 during training of both our method and baselines to provide denser reward signal for all antmaze environments. However, we still count original sparse reward when comparing the performance.Environments. We test our method on four widely used environments: Hopper, Halfcheetah, Walker2d and Ant. Fig. 17 shows an illustration of the four environments. 5 : The size and the average and standard deviation of the normalized reward of the Antmaze datasets from D4RL [19] used in our experiments. 1. Hopper. Hopper is a locomotion task on a 2D vertical plane, where the agent manipulates a single-legged robot to hop forward. Its state is 11-dimensional, which describes the angle and velocity for the robot's joints. Its action is 3-dimensional, which corresponds to the torques applied on the three joints for the current time step respectively.Halfcheetah. Halfcheetah is also a 2D environment which requires the agent to control a cheetah-like robot to run forward. The states are 17-dimensional, containing the coordinate and velocity of the joints The actions are 6-dimensional, which control the torques on the joints of the robot.Ant. In Ant, the agent controls a four-legged 8-DoF robotic ant to walk in a 3D environment and tries to move forward. It has a 111-dimensional state space describing the coordinates and velocities of the joints.Walker2d. Walker2d is a 2D environment in which the agent needs to manipulate a 8-DoF two-legged robot to walk forward under the agent's control. Its state space is 27-dimensional.Datasets. We test our method across three different qualities of datasets: medium, medium-replay and random. The medium dataset contains trajectories collected by an agent trained with RL, but early-stopped at medium-level performance. The medium-replay dataset is the collection of trajectories sampled in the training process of the agent mentioned above. The random dataset contains trajectories collected by an agent with random policy. Tab. 6 shows the size and normalized reward of each dataset.Environments. Maze2D is another set of D4RL environment, where the agent needs to control a point mass to navigate through a 2D maze and arrive at a fixed goal. It has a 4-dimensional state space describing its coordinate and velocity, and a 2-dimensional action describing its acceleration. The reward is determined by its current distance to the goal. We test our method on four variants of maze: Open, Umaze, Medium and Large with increasing difficulty. The map of each maze is illustrated in Fig. 16 . Maze2D environment is tested in Sec. G.2.Datasets. We again test our method on datasets provided by D4RL. Tab. 7 shows the size and normalizeed reward of each dataset. 7 : The size and the average and standard deviation of the normalized reward of the Maze2D datasets from D4RL [19] used in our experiments.For all networks, we use a simple MDP with two hidden layers of width 128, and ReLU [1] as activation function. We add a Tanh activation function to limit the output for ODT and the actor of DDPG to [-1, 1]. For both methods, we use Adam [27] as the optimizer, and the learning rate is set to 10 -3 . We pretrain 5 epochs on offline data (20 gradient steps) and 16 epochs for online finetuning, with a batch size of 32 for gradient update and collect 64 new rollout states for each epoch (thus we train 2n + 4 steps for the n-th online finetuning epoch). RTG eval is set at 1, which serves as the (constant) input for ODT rollout and DDPG actor. Both DDPG and ODT uses deterministic actor with an exploration noise uniform in [-0.01, 0.01] during online rollouts.Tab. 8 summarizes hyperparameters that are common across all environments, and Tab. 9 summarizes hyperparameters that are different across environments. For environments that exist in ODT [74] , we follow the hyperparameters from ODT medium environments. We did not use positional embedding as suggested by ODT [74] . Specially, for antmaze, we remove most (all but 10) 1-step trajectories, because the size of the replay buffer for decision transformers is controlled by the number of trajectories, and antmaze dataset contains a large number of 1-step trajectories due to its data generation mechanism (immediately terminate an episode when the agent is close to the goal, but do not reset the agent location). Also, we add Layernorm [4] after each hidden layer of the critic for Adroit, Maze and Antmaze environments, according to Yue et al. [71] 's advice. We found that such practice stabilizes the training process (see Sec. G.5 for ablation).For ODT and TD3 baseline, we use the same code as our TD3+ODT, while setting coefficients for RL and supervised gradients accordingly. For PDT baseline, we use the default hyperparameter in PDT paper, and pretrain PDT for 40K steps for all experiments. For TD3+BC and IQL, we use the default hyperparameter in their codebase, and pretrain them for 1M steps for all experiments (remaining the same as that in the codebase). Though we have tested MuJoCo environments in Sec. 4, it is worth noting that many offline RL algorithms have addressed the MuJoCo benchmark quite well [28, 23] . Thus, we also tested settings where RL struggles to obtain good performance to further analyze the performance of using RL gradients for decision transformers.Environment and Experimental Setup. In this experiment, we use the same experiment and dataset as in Sec. 4 except for one major difference: the rewards are not given immediately after each step. Instead, the cumulative reward during a short period of M steps is given only at the end of the period, while the rewards observed by the agents within a period are all zero. We adopt such a setting from prior influential work [43] , which creates a sparse-reward setting where RL algorithms struggle. We test M = 5 in this experiment.Results. We use the same baselines as that in Sec. 4; Tab. 10 summarizes the performance of each method. Generally, DT with TD3 gradient still works very well, much better than ODT. While TD3+BC works well in several scenarios, it struggles on random environments. See Fig. 18 for reward curves.We test on navigation tasks in D4RL [19] where the agents need to control a pointmass through four different mazes: Open, Umaze, Medium and Large with different dataset respectively. Fig. 19 lists the performance of each method on Maze2D before and after online finetuning, and Tab. 11 summarizes the performance before and after online finetuning. The result shows that our method again significantly outperforms autoregressive-based algorithms such as ODT and PDT, which validates our motivation in Sec. Fig. 20 shows the result of using different context lengths on hammer-cloned-v1 environment (in this experiment, we use T eval = 1 to demonstrate the effect of more different T train ). It is shown from the experiment that the selection of T train needs to be balanced between more information taken into account and training stability; while longer T train brings faster convergence when growing from 1 to 5, the reward curves with T train ∈ {10, 20} oscillates more than that with T train = 5.In some environments, such as hopper-random-v2, walker2d-random-v2 and ant-random-v2, our proposed method still seems to be improving after 500K online samples. In Fig. 21 , we show the finetuning result of our proposed method with more online transitions, which effectively shows that our method has greater potential in online finetuning when finetuned for more gradient steps.As we have mentioned in Sec. F.2, as suggested by Yue et al. [71] , we apply Layernorm [4] to critic networks for environment other than MuJoCo for better stability in training. In our experiment, we found that it greatly stabilizes the critic on complicated environments such as Adroit, but makes online policy improvement less efficient on easier MuJoCo environments. Fig. 22 shows the performance and critic MSE loss comparison on some environments with and without layernorm; it is clearly shown that layernorm helps stabilizes online finetuning in some cases such as pen-cloned-v1, but hinders performance increase on other environments such as halfcheetah-medium-v2.As mentioned in Sec. 3.2, we use reflexive critics (i.e., critics that only take the current state nand action) to add RL gradients to decision transformer, and this creates an average effect among policies generated by different context lengths (see Sec. D in the appendix for detail). In this section, we explore recurrent critic by substituting the MLP critic using a LSTM, such that for a trajectory segment, the evaluation for the i-th action a i is based on all state-action pairs (s 1 , a 1 ), . . . , (s i-1 , a i-1 ) and current state s i . As shown in Fig. 23 , we found that recurrent critics are much less stable than The reward curves of our method when finetuned for more steps (we only report curves until the black line in Fig. 6 ). It is clearly shown that our method has greater potential for improvement when finetuned for more steps. In the Adroit environment results discussed in Sec. 4, we found that the baseline of ODT finetuned using pure TD3 gradients struggles due to catastrophic forgetting. Inspired by Wołczyk et al. [61] , we test whether adding a KL regularizer can fix the forgetting problem. Though our policy is deterministic, we can approximately interpret the policy as Gaussian with a very small variance. Thus, a KL regularizer can be simply added using c 0 • ∥a -a old ∥ 2 , where a is the current action and a old is the action predicted by the pretrained policy. We set c 0 = 0.05 and test this method on the Adroit cloned and expert dataset. We illustrate the result in Fig. 24 . We find that the KL regularizer effectively addresses the issue on expert environments for both TD3 and TD3+ODT. But it can sometimes hinder the policy improvement of TD3+ODT with low return during online finetuning. Figure 24 : The result of ODT with better exploration (only in cloned dataset) and TD3/TD3+ODT forgetting mitigation. The result shows that 1) ODT with curriculum RTG does not work; 2) even with exploration supported by an oracle, ODT can still fail on some environments such as hammer; 3) JSRL with the pretrained policy does not work for forgetting mitigation; 4) KL regularizer effectively addresses the issue on expert environments for both TD3 and TD3+ODT, but it can hinder the improvement of TD3+ODT with low return.In Sec. 3.1, we state that ODT cannot explore the high-RTG region when pretrained with low-quality offline data, and we ran a simple experiment to verify this (Fig. 2 ). In this section, we test two potential alternatives for addressing the exploration problem: JSRL [59] and curriculum learning.For JSRL, an expert policy is used for the first n steps in an episode, before ODT takes over. We set n = 100 (100% max episode length for adroit pen, 50% max episode length for other adroit environments) initially, and apply an exponential decay rate of 0.99 for every episode. We test two settings of JSRL: the expert policy being the offline pretrained policy, and the expert policy being oracle, i.e., an IQL policy trained on the Adroit expert dataset.For curriculum learning, we use ODT with a gradually increasing target RTG with the current RTG for rollouts being RTG eval -0.99 N (RTG eval -RTG data ). Here, N is the number of episodes sampled in online stage, and RTG data is the average RTG of the offline dataset.Results are summarized in Fig. 24 . We found that curriculum RTG does not work, probably because the task is too hard and cannot be improved by random exploration without gradient guidance. Further, even with oracle exploration, ODT is not guaranteed to succeed: it fails on the hammer environment where TD3+ODT succeeds, probably because of insufficient expert-level data and an inability to improve with random exploration.In this section, we further examine the source of the performance gain of our method compared to TD3+BC. There are two key differences as stated in Sec. 3.2: The architecture and RL via Supervised (RvS) learning [18] . We can hence assess two more baselines: TD3+BC with our transformer architecture and TD3+RvS using TD3+BC's architecture. We present the ablation result on the Adroit cloned environment in Fig. 25 . The result shows that only TD3+BC with our architecture works (albeit still worse than our method). We hypothesize that this is because a simple MLP is hard to model the complicated policy which takes both RTG and state into account.To further assess if simply adding more layers to the MLP works, we conduct an ablation on the number of layers for TD3+RvS. The result is illustrated in Fig. 26 . It shows that simply adding a few layers to the MLP does not aid performance. We speculate that it is probably the transformer architecture that helps modeling the state-and-RTG-conditioned policy. We conduct all experiments with a single NVIDIA RTX 2080Ti GPU on an Ubuntu 18.04 server equipped with 72 Intel Xeon Gold 6254 CPUs @ 3.10GHz. Mujoco experiments takes about 6 -8 hours, and the bottleneck is the gradient update; about 50% time is spent on backpropagation and update of parameters. Our critic appended to ODT only takes up about 20% time to train, in which 90% of the critic training time is spent on decision transformer inference to get action for "next state". For the actor, the training overhead of our method is negligible since it only contains an MLP critic inference to get the Q-value. Therefore, overall our method only uses 20% extra time compared to ODT for training, but attains much better results.Our code is developed upon multiple algorithm repositories and environment testbeds.Algorithm Repositories. We implement our method on the basis of online decision transformer repository, which has a CC BY-NC 4.0 license. We also refer to IQL [28] , PDT [64] and TD3+BC [20] repository when running baselines, all of which have MIT licenses.Environment Testbeds. We utilize OpenAI gym [8] , MuJoCo [58] , and D4RL [19] as testbed, which have an MIT license, an Apache-2.0 license, and an Apache-2.0 license respectively.NeurIPS Paper Checklist38th Conference on Neural Information Processing Systems (NeurIPS 2024).ODT uses different RTGs for evaluation and online rollouts, but we refer to both as RTGeval as they are both expert-level expected returns.Eq. (4) is very informal. See Appendix E for a more rigorous description.Note we use "max" instead of "βmax" as this is a property of the environment and not the dataset.Decision transformers have a sequence of past RTGs as input, but the past sequence can be augmented into the state space to fit into such form. 


Decision: This paper has been unanimously recommended for acceptance. It tackles an interesting problem—the offline-to-online problem in reinforcement learning—applied to the decision transformer and provides some theoretical and empirical results. In one way or another, the rebuttal addressed most of the authors' concerns, and I'm recommending the acceptance of this paper.

Reviews: review: paper_summary: paper_summary: This paper addresses the challenge of online finetuning of online decision transformers (ODT). Theoretical results are provided to show the target return-to-go can hamper finetune. The authors propose to have TD3 gradients added to ODT finetune and improve its performance especially when pretrained with low-reward data. Extensive empirical results are provided to show the proposed method can achieve stronger or competitive results across a large number of environments.
strength_weakness: strength_weakness: - Maybe I missed something but I feel it might be good to have ODT baselines that try to tackle online finetuning with alternative (and more naive) methods. For example, authors argue that a main difficulty when finetuning DT is that when pretrained with low-reward data, the target RTG at finetuning stage is simply hard to obtain. But if we provide a realistic target RTG initially (based on pretrain data), and gradually increase it, will that help performance? Another thing is if we have a better exploration policy, will that help ODT finetune to the same extent? 
- Adding the TD3 component can make the training more complex and slower.
questions: questions: - Will alternative strategies that improve exploration at finetune phase help ODT to the same extent? 
- The random datasets are not commonly tested in previous papers, when the baselines are tested on these datasets, are the hyperparameters tuned for these datasets?
limitations: limitations: Some limitations are discussed in section 6.
paper_summary: paper_summary: The authors introduce a novel framework for improving the performance of Online Decision Transformer through adding TD3 gradients to the fine-tuning ODT objective. This is motivated by a theoretical analysis of ODT, that highlights an issue with low-reward, sub-optimal pretraining. The authors also provide an extensive empirical investigation.
strength_weakness: strength_weakness: * Overclaim of the contribution of “We propose a simple yet effective method to boost the performance of online finetuning of decision transformers.” The empirical results of relocate-expert-v1 show the opposite, where ODT outperforms TD3+ODT. Perhaps the authors can discuss this result and/or refine the claim to environments with particular properties where the expectation is that TD3+ODT outperforms ODT.
* Minor: Section 3.1 could highlight the conceptual figure of the plot of Figure 1, c, to help the reader understand the graph at the beginning of section 3.1.
questions: questions: * Can the authors comment or perform an ablation with ODT+DDPG for all the main experiments as well?
* What is the impact of context length on your approach?
* For the main experiments how many random seed runs are each experimental result over? E.g. in Figure 2.
limitations: limitations: Limitations are adequately discussed in section 6.
paper_summary: paper_summary: This paper presents a method for fine-tuning decision transformers (DT) online. The proposal is to integrate TD3 with DT such that the gradients from TD3 can help the online policy to explore highly rewarding trajectories, hence further improving the agent performance. It can be seen as a variant of TD3+BC with the MLP being replaced by a transformer. The experimental results show the proposed method outperforms several baselines, and is effective even when the offline dataset is of low quality, a setting which traditional online DT cannot do well.
strength_weakness: strength_weakness: - The TD3 gradients for policy update are not clear. For example, in equation 2, it seems that $Q$ is a constant if you optimize $\mu^{RL}$? Or is $a_t$ predicted by $\mu^{RL}$?
- Some notations are a bit confusing. For example, there are RTG, RTG_real, RTG_eval. Better to clearly define them first.
questions: questions: - Could you please explain more about the point (2) in line 182: how does DT prioritize trajectories with higher return?
limitations: limitations: - Compared to TD3+BC, the proposed method uses transformer instead of MLP. It costs much more computation to gain better performance. Such limitation is not mentioned in the paper.
paper_summary: paper_summary: The authors consider the problem of online fine-tuning decision transformers. Current approaches for this problem do not work well when the offline data is low-quality. The authors analyze the online DT algorithm theoretically and show why DT-induced policy update would not work if the RTG used for conditioning is much higher than RTG seen in the dataset. They then propose combining the ODT update with RL gradients, in particular, the ones obtained from the TD3 model. This approach was then tested through Adroit, Antmaze, and Mujoco experiments, where the method obtained strong results. Finally, the authors perform additional analyses and make improvements to their proposed approach.
strength_weakness: strength_weakness: - The empirical results seem to be noisy, and a more careful approach to statistical significance is needed. Looking at, e.g., Figure 5, I'm not sure if TD3+ODT is better than TD3 as the plots overlap quite a lot, but the corresponding Table 1 makes it look like TD3+ODT is a clear winner. 
    - This includes bolding "the best solution" in some of the tables. If one of the solutions achieves a score of 101 and the other of 100, in the RL regime, it is usually not possible to tell which one is statistically better and the fair thing to do is to bold both. 
    - I didn't find the information on how many seeds are used for each experiment, and this is very important in the RL context.
    - In summary, I would suggest the authors adopt the recommended practices of [1] to have a clearer picture of the results.  
- I think the baselines considered here should be made stronger
    - The TD3 baseline collapses right after fine-tuning starts in many cases. It looks like it might be caused by forgetting which was shown to be a problem in RL fine-tuning [2]. I wonder if combining TD3 with some forgetting mitigation technique would help.
    - The authors point out that their method improves on TD3+BC in two ways: architecture (MLP -> DT) and objective (BC -> RvS). I think these two axes of improvement should be investigated separately. That is, introducing two additional baselines would show us where the improvements come from: TD3+BC+Transformer and TD3+RvS (with MLP rather than DT). 

[1] Agarwal, Rishabh, et al. "Deep reinforcement learning at the edge of the statistical precipice." Advances in neural information processing systems 34 (2021): 29304-29320. \
[2] Wolczyk, Maciej, et al. "Fine-tuning Reinforcement Learning Models is Secretly a Forgetting Mitigation Problem." Forty-first International Conference on Machine Learning.
questions: questions: Currently, I would be inclined to accept the paper if not for the problems with the evaluation. As such, I would like to ask the authors:
- Please discuss and address the questions about the evaluation setting, as raised in the Weaknesses section above. If this is sufficiently resolved, I'd be happy to increase the score.
- Please consider introducing stronger baselines. This is not as crucial, but it would make the paper stronger.

Less important but interesting to discuss:
- The authors say: "training a transformer-based value function estimator is quite hard." I agree with this opinion based on my own experiences, but I don't think there has been a thorough discussion of this in the literature. The papers referenced by the authors do not really address this issue well in my opinion. So, it's not a weakness of the paper, but I would like to ask the authors where they think the problems of value learning in transformers come from (or maybe even their online training in general).
limitations: limitations: Although the authors do discuss the limitations of their work, I think there are at least two more points that should be discussed:
* The method has not been tested on image-based datasets and environments.
* There is an increased computational cost as compared to a regular ODT, since we have to train the value functions.

Comments: comment: Thanks for your appreciation of our work and response! We have started the experiments and will surely have those results ready in our revised version.
I thank the authors for their detailed rebuttal response, and I particularly appreciate the new experimental ablations in the global response. Thank you for re-phrasing your claim. I look forward to seeing the results of ODT+DDPG on all additional environments in the revised version. I am still keeping my score the same.
Thanks a lot for appreciating our work and rebuttal! We will revise our paper as discussed in the reviews for the next version.
Thanks for your rebuttal. Given the idea is interesting and the experimental results are thorough, I will adjust my score to 6.
Thanks a lot for your timely reply and for appreciating our rebuttal. Below are our follow-up responses:

**Q1. About the results in Fig. 3.**

Yes, those results are averaged over all the settings due to the 1-page limit of the response pdf. For example, for adroit, we average over 12 different settings ({expert, cloned, human}x{pen, hammer, relocate, door}). We will update figures on each setting in the revised paper. To begin, here, we provide some numbers for IQM (which is more robust than median as suggested by the paper mentioned in the review) and optimality gap on the adroit pen environment, as the mean and standard deviation are already provided in the main paper. The numbers in parentheses are the 95% confidence interval. 

**IQM**

Method | pen-expert-v1 | pen-cloned-v1 | pen-human-v1 
---|---|---|---| 
TD3+BC | 40.15 (24.00~48.17)  | -2.39 (-2.82~-1.55) | 18.63 (8.49~34.17)
IQL | **149.44 (146.54~151.13)** | 76.06 (72.17~78.46) | 98.13 (91.18~101.85)
ODT | 129.31 (125.10~133.52) | 26.24 (19.74~32.73) | 32.28 (25.23~39.33)
PDT | 27.91 (14.50~41.32) | 14.05 (5.14~30.09) | 2.67 (1.75~3.58)
TD3 | 69.48 (61.52~77.44) | 70.52 (67.25~73.79) | 32.28 (19.25~45.31)
TD3+ODT (ours) | 118.30 (102.80~135.14) | **130.51 (124.43~135.65)** | **116.71 (110.05~120.61)** 

**Optimality Gap**

Method | pen-expert-v1 | pen-cloned-v1 | pen-human-v1 
---|---|---|---| 
TD3+BC | 59.85 (48.17~76.00) | 102.39 (101.55~102.83) | 81.36 (65.83~91.51)
IQL | **0.00 (0.00~0.00)** | 23.94 (21.54~27.83) | 2.93 (0.00~8.81)
ODT | **0.00 (0.00~0.00)** | 73.75 (67.26~80.25) | 67.61 (60.66~74.76)
PDT | 72.08 (58.68~85.50) | 85.95 (69.90~94.85) | 97.33 (96.42~98.24)
TD3 | 30.51 (22.56~38.48) | 29.48 (26.21~32.75) | 67.72 (54.69~80.74)
TD3+ODT (ours) | **0.00 (0.00~0.00)** | **0.00 (0.00~0.00)** | **0.00 (0.00~0.00)**

**Q2. More seeds are needed for experiments.**

We completely agree, thorough evaluation is important. We started experiments with two additional seeds, and we will include those in the revised paper.

**Q3. Selective forgetting mitigation on cloned datasets.**

1. “Forgetting all the behaviors that are not useful” might be suboptimal. While expert trajectories are certainly more useful, it is hard to say which behaviors are not useful at all in the offline dataset. For example, all trajectories may provide information on environment dynamics and possibly undesirable states. We speculate that it would be better if such data could be memorized in some way not reflected in the evaluation policy, e.g., dynamic models in model-based RL, areas of state-action space with low Q-values, or encoded in the policy but “hidden” by conditioning on high RTG. Our method is a combination of the latter two.

2. We think the ODT gradient in TD3+ODT during the online phase could be interpreted as a kind of selective forgetting mitigation. Since TD3+ODT (as well as our TD3 / original ODT) keeps the offline data in the replay buffer for ODT itself during the online phase, the supervised learning gradient on those offline data can be regarded as a regularization towards the RTG-conditioned empirical policy, which, ideally, should be equal to the pretrained policy if the algorithm converges during the offline phase. 

3. Given the performance increase due to the KL regularizer used with TD3+ODT on adroit expert datasets, we agree that an RTG-weighted regularizer could be interesting to test and could potentially further improve results. We have started some experiments, and will add results on this in the revised paper.

**Q4. Is the transformer crucial, or bigger MLPs would suffice?**

Good question! While ODT is the main baseline of our paper as we mentioned before, we feel ablations on using bigger MLPs / RNNs for TD3+RvS are interesting. We started some experiments for our revised paper.
I appreciate the thorough response from the authors. After reading it as well as the other reviews, I decided to increase my score. 

Here's my detailed response:
* I appreciate that the authors ran the rliable library to get more stable results. As for the results in Fig. 3 -- are these averaged over all the settings (e.g., cloned and expert in Adroit case)? It would be nice to also see this analysis on each setting.
* I understand that having more seeds is expensive, but I think we need them to be sure about the results. Given some overlaps in Figure 3 presented by the authors I would recommend adding at least 2 more seeds (5 in total)
* I really appreciate the forgetting mitigation experiments. It makes sense that BC works well on expert datasets but not on cloned ones. I wonder if one could apply some sort of selective forgetting mitigation on the cloned datasets to remember what is useful and forget all of the behaviors that are not useful.
* It's quite interesting that the transformer architecture seems to be much more important than the conditioning (i.e., BC vs RvS). I wonder if the transformer is crucial or just having bigger MLPs or RNNs/SSMs would suffice.