Q-learning Decision Transformer: Leveraging Dynamic Programming for Conditional Sequence Modelling in Offline RL
Authors: Taku Yamagata
Ahmed Khalil
Raul Santos-Rodriguez
Venue: Submitted to ICLR 2023 (2023)
Keywords: offline reinforcement learning
reward conditional model
dynamic programming
decision transformer

Abstract: Recent works have shown that tackling offline reinforcement learning (RL) with a conditional policy produces promising results. The Decision Transformer (DT) combines the conditional policy approach and a transformer architecture, showing competitive performance against several benchmarks. However, DT lacks stitching ability -- one of the critical abilities for offline RL to learn the optimal policy from sub-optimal trajectories. This issue becomes particularly significant when the offline dataset only contains sub-optimal trajectories.
On the other hand, the conventional RL approaches based on Dynamic Programming (such as Q-learning) do not have the same limitation; however, they suffer from unstable learning behaviours, especially when they rely on function approximation in an off-policy learning setting. In this paper, we propose the Q-learning Decision Transformer (QDT) to address the shortcomings of DT by leveraging the benefits of Dynamic Programming (Q-learning). It utilises the Dynamic Programming results to relabel the return-to-go in the training data to then train the DT with the relabelled data. Our approach efficiently exploits the benefits of these two approaches and compensates for each other's shortcomings to achieve better performance. We empirically show these in both simple toy environments and the more complex D4RL benchmark, showing competitive performance gains.


Under review as a conference paper at ICLR 2023
Q-LEARNING DECISION TRANSFORMER: LEVERAG- ING DYNAMIC PROGRAMMING FOR CONDITIONAL SE- QUENCE MODELLING IN OFFLINE RL
Anonymous authors Paper under doublc-blind review
ABSTRACT
Recent works have shown that tackling offline reinforcement learning (RL.) with a conditional policy produces promising results. The Decision Transformer (DT) combines the conditional policy approach and a transformer architecture, showing competitive performance against several benchmarks. However, DT lacks stitch- ing ability one of the critical abilities for offline RL to learn the optimal policy from sub-optimal trajectories. This issue becomes particularly significant when the offline dataset only contains sub-optimal trajectories. On the other hand, the conventional RL approaches based on Dynamic Programming (such as Q- learning) do have the same limitation; however, they suffer from unstable learning behaviours, especially when they rely on function approximation in an off-policy learning setting. In this paper, we propose the Q-learning Decision Transformer (QDT) to address the shortcomings of DT by leveraging the bene- fits of Dynamic Programming (Q-learning). It utilises the Dynamic Programming results to relabel the return-to-go in the training data to then train the DT with the relabelled data. Our approach efficiently exploits the benefits of these two approaches and compensates for each other's 's shortcomings to achieve better per- formance. We empirically show these in both simple toy environments and the more complex D4RI. benchmark, showing competitive performance gains.
INTRODUCTION
The transformer architecture employs a self-attention mechanism to extract relevant information from high-dimensional data. It achieves state-of-the-art performance a variety of applications, in- cluding natural language processing (NLP) (Vaswani et al. 2017 Radford et al. 2018 Devlin al.) 2018) or computer vision (Ramesh et al., 2021). Its translation to the RL domain, the Decision transformer (DT) (Chen al.] 2021). successfully applies the transformer architeeture to offline reinforcement learning tasks with good performance when shifting their focus on the sequential modelling. employs goal conditioned policy which converts offline RL into supervised learn- ing task, and it avoids the stability issues related to bootstrapping for the long term credit assign- ment (Srivastava et 2019 Kumar et al. 2019b Ghosh et al.) 2019). More specifically, DT con- siders sum of the Future rewards return- to-go (RTG), as the goal and learns a policy conditioned on the RTG and the state. It is categorised reward conditioning approach.
Although DT shows very competitive performance in the offline reinforcement learning (RL) tasks, fails to achieve one of the desired properties of offline RL agents, stitching. This property is an ability to combine parts of sub-optimal trajectories and produce an optimal one (Fu et al. 2020) Below, we show a simple example of how DT (reward conditioning approaches) would fail to find the optimal path.
To demonstrate the limitation of the reward conditioning approaches (DT), consider a task to find the shortest path from the left-most state to the rightmost state without going down to the fail state in Fig. [] We set the reward as -1 at every time step and -10 for the action going down to the fail state. The training data covers the optimal path, but none of the training data trajectories has the entire optimal path. The agent needs to combine these two trajectories and come up with the optimal path. The reward conditioning approach essentially finds trajectory from the training data

---

Under review as a conference paper at ICLR 2023
training data
goal
1n
start
fall
trajectory
trajectory
DT agent
Q-Learning agent
Figure I: A simple example demonstrates the decision transformer approach's issue (lack of stitch- ing ability) - fails to find the shortest path to the goal. In contrast, Q-learning finds the shortest path. The numbers the arrows rewards the path and the numbers the states are RTGs.
simpple
mezeed
MujoCo deleyed reward
100.0
COL
DT
00.0
OCT
total reware
40.0
20.0
0.0
20.0
open-646
umaze dense
medium dense
large- brge-dense
hopper- medium
haitcheetah medium
walker medium
Figure 2: Evaluation results for conservative Q-learning (CQL), Decision Transformer (DT) and Q- learning Decision Transformer (QDT). The left two plots (simple and maze2d environments) show that the DT does not perform as it fails to stitch trajectories, and the right plot shows that CQL fails learn from sparse reward scenario (delayed reward). In contrast, QDT achieves consistently good results across all the environments.
that gives the ideal reward and takes the same action as the trajectory. In this simple example, trajectory has a meagre reward. Hence, it always follows the path of trajectory despite trajectory 2 giving the optimal path for the first action.
In contrast to the reward conditioning approaches (DT), Q-learning does not suffer from the issue and finds the optimal path quickly in this simple example. Q-learning takes each time step separately and propagates the best future rewards backwards. Hence can learn from the first optimal action from trajectory 2. However, Q-learning has some issues on long time horizon and sparse reward scenarios. It attempts propagating the value function backwards to its initial state, often struggling to learn across long time horizons and sparse reward tasks. This is especially true when Q-learning uses function approximation in an off-policy setting as discussed in Section 11.3 in (Sutton & Barto, 1998).
Here, we devise a method to address the issues above by leveraging Q-learning to improve DT. Our approach differs from other offline RL algorithms that often propose a new single architecture of the agent and achieves better performance. We propose a framework that improves the quality of the offline dataset and obtains better performance from the existing offline RL algorithms. Our approach exploits the Q-learning estimates to relabel the RTG in the training data for the DT agent. The motivation for this comes from the fact that Q-learning learns RTG value for the optimal policy. This suggests that relabelling the RTG in the training data with the learned RTG should resolve the DT stitching issue. However, Q-learning also struggles situations where the states require large time step backward propagation. In these cases, argue that DT will help as it estimates the sequence of states and actions without backward propagation. Our proposal (QDT) exploits the strengths of each of the two different approaches to compensate for other's 's weaknesses and achieve a more robust performance. Our main evaluation results are summarised in Fig. 2 The left two
In this paper, will use the Q-learning and Dynamic Programming interchangeably to indicate any RI. algorithm relying on the Bellman-backup operation.
2

---

Under review as a conference paper at ICLR 2023
plots (simple and maze2d environments) show that DT does not perform well as it fails to stitch trajectories, while the right plot illustrates that CQL (Q-leaming algorithm for offline reinforcement learning) fails to learn in sparse reward scenario (delayed reward). These results indicate that neither of these approaches works well for all environments, and we might have abysmal results by selecting the wrong type of algorithms. In contrast, QDT performs consistently well across all environments and shows robustness against different environments. Through our evaluations, we also find that some of the evaluation results in the prior works may be directly comparable, and it causes some contradicting conclusions. We touch on the issue in Section 6
PRELIMINARIES
Offline Reinforcement Learning. The goal of RL is to learn policy that maximises the expected sum of rewards in a Markov decision process (MDP), which four-tuple (S, A, p,r)) where S is set of states, A is a set of actions, p is the state transition probabilities, and is a reward function.
In the online or on-policy RL settings, an agent has access to the target environment and collects new set of trajectories every time it updates its policy. The trajectory consists of {st, ar, Fr re)/ where St, ag and re are the state, action and reward at time respectively, and T is the episode time horizon.
In off-po RL case, the agent also has access to the environment to collect trajectories, but it can update its policy with the trajectories collected with other policies. Hence, improves its sample efficiency as it can still make use of past trajectories.
Offline RL goes step further than off oof-po RL. It learns its policy purely from static dataset that is previously collected with an unknown behaviour policy (or policies). This paradigm can be precious in case of the interaction with the environment being expensive or high risk (e.g., safety critical applications).
Decision Transformers. DT architecture (Chnn al. 2021 casts the RL problem conditional sequence modelling. Unlike the majority of prior RL approaches that estimates value functions or compute policy gradients, DT outputs desired future actions from the target sum of future rewards RTGs, past states and actions.
T (Ri-K+1, S1- QI-K+1 R 81-1 Q1-1, R,, St).
(1)
Equation| shows the input of a DT, where K the context length, RTGs (Re = ΣT Te), states and is actions. Then DT outputs the next action (a,).
DT employs Transformer architecture (Vaswani et al., (0177), which consists of stacked self-attention layers with residual connections. It has been shown that the Transformer architecture successfully relates scattered information in long input sequences and produces accurate outputs (Vaswani et al.
2017 Radford et al.) 2018 Devlin al. 2018 Ramesh et al. 2021].
Conservative Q learning. In this work, we use the conservative Q learning (CQL) framework (uu- mar et al.) 2020) for the Q-learning algorithm. CQL is offline RL framework that learns Q- functions that are lower-bounds of the true values. It augments the standard Bellman error objective with regulariser which reduces the value function for the out-of-distribution state-action pair while maintaining ones for state-action pairs in the distribution of the training dataset. In practice, it uses the following iterative update equation learn the Q-function under a learning policy p(als).
gi+1 +- arg min (BenD (Q Be.and(((sss))
Q
(1(8,0)+ Q( + (2)
+ a, DD 2 d' u(a'ls")
where D is the training dataset and y is a discount factor. Kumar et al. 2020) showed that while the resulting function,, Q" lim Q* may not be point-wiee wise lower- bound, it is lower bound of (s), i.e. ( (Q, a)] < (s).
3

---

Under review as a conference paper at ICLR 2023
3 METHOD
We propose a method that leverages Dynamic Programming approach (Q-leaming) to compensate for the shortcomings of the reward conditioning approach (DT) and build robust algorithm for the offline RL setting, Our proposal consists of three steps. First, the value function is learned with Q-learning- Second, the offline RL dataset is refined by relabelling the RTG values with the result of Q-learning. Finally, the DT is trained with the relabelled dataset. The first and third steps do not require any modifications of the existing algorithms.
The reward conditioning approach (DT) takes an entire trajectory sequence and conditions on it using the sum of the rewards for that given sequence. Such an approach struggles on tasks requiring stitching (Fu et al.] 2020) the ability to learn an optimal policy from sub-optimal trajectories by combining them. contrast, the Q-learning alternative propagates the value function backwards for each time step separately with the Bellman backup, and pools the information for each state across trajectories. It therefore does not have the same issue. Our proposal tackles the stitching issue of the reward conditioning approach by relabelling the RTG values with the learned Q- Q-functions. With the relabelled dataset, the reward conditioning approach (DT) can then utilize optimal sub-trajectories from their respective sub-optimal trajectories.
We now discuss how to relabel the RTGs values with the learned Q-functions. Replacing all of the RTGs values with Q-functions is not adequate because not all the learned Q-functions are accurate, especially in a long time horizon and sparse reward case. Ideally, we would like to replace the RTGs values where the learned Q-functions are accurate. In this work, we employ the CQL framework for the offline Q-learning algorithm, which learns the lower bound of the value function. We replace the RTGs values when the RTG in the trajectory is lower than the lower bound. With this approach, our method substitutes the RTGs values where the learned value function is indeed accurate (or closer to the true values). We also replace all RTG values prior to the replaced RTG along with the trajectory by using reward recursion (R,-1 = Tt-1 Rt). This propagates the replaced RTG values to all the time steps prior to the replaced point. To apply this idea, we initialise the last state RTG to zero (Rr = 0), and then start the following process from the end of the trajectory to the initial state backwards in time. First, the state value is computed for the current state with the learned value function (st) = (aawstales)) [Q( a)), where the л is the learned policy. Next, the value function is compared (s:)) against the RTG value for the current state (Ra). If the value function is greater than that of the RTG, the RTG for the previous time step is set from (Rz-1) to r (sz), otherwise it is set to r + Rr We repeat this process until the initial state is reached. This process summarised in Algorithm []
The above relabelling process might introduce inconsistencies between the reward and RTG within the DT input sequence (Eq. [.. The RTG value is sum of the future rewards, hence must always be R. + Re+1. However, the relabelling process might break this relationship. To maintain this consistency within the input sequence of DT, we regenerate the RTG for the input sequence
(f R R ) by copying the last RTG (it = R₂) and then repeatedly apply RG = r + R backwards until = t - K + 1. We repeat this for each the input sequences to maintain the consistency of the rewards and RTGs. This process is summarised in Algorithm
Algorithm Relabelling return-to-ge go
Algorithm 2 Generating return to-go for DT
Input
Input
TETT ((s)
rewards
M--+1:: rewards
learned value function
Re
return to go for time
time horizon (trajectory length)
context length
Output
Output RiT
RIT
relabelled return go
relabelled return to go for DT
RT +0
Rg + R₁
+-T T
T - t t-1
while - > do R,-1 + -r + max(R, (s+))
while > t K do Rr + rr + R
767-1
1-
end while
end while
4

---

Under review as a conference paper at ICLR 2023
Theoretical considerations of QDT. Q-learning Decision Transformer (QDT) relies on DT as the agent algorithm, which can be seen as a reward conditioning model. A reward conditioning model takes the states and RTG as inputs and outputs actions. If we assume the model is trained with the state 8t and the optimal state-action value function (8t, (t)), then we can guarantee that the model will output the optimal action (arg max, Q" (st,a) for long it is given 8, and max, (st, a) as inputs (Srivastava et al., 2019). In practice, we do not know the optimal value function Q" a), hence DT (and similarly other reward conditioning approaches) uses RTG in- stead. RTG is collected through the behaviour policy (or policies) and often is not optimal with the majority of values being much lower than the corresponding optimal value function (Q" a)). As QDT uses CQL to learn the optimal conservative value function, Th. 3.2 in Kumar et al. (2020) shows that the conservative value function is a lower bound of the true value function. Hence the QDT relabelling process moves the RTG in the training dataset closer to the optimal value function (see Appendix D).
4 RELATED WORK
Offiine reinforcement learning. The offine RL learns its policy purely from static dataset that was previously collected with an unknown behaviour policy (or policies). As the learned policy might differ from the behaviour policy, the offline algorithms must mitigate the effect of the dis- tributional shift (Agarwal et al. 2020 Prudencio et al.) 2022). One of the most straightforward approaches to address the issue is by constraining the learned policy to the behaviour policy (Fu-) jimoto et al., 2019; Kumar et al. 2019a; Wu et al. 2019). Other methods constrain the learned policy by making conservative estimates of future rewards (Kumar et al.] 2020 Yu et al., 2021). Some model-basee methods estimate the model's 's uncertainty and penalize the actions whose conse- quences are highly uncertain (Janner et al. 2019 Kidambi et al., 2020). Some approaches address the distributional shift without restricting the learned policy. One such approach group is weighted imitation learning (Wang et al. 2018 Peng et al. 2019; Wang et al., 2020 Nair et al. 2020 Chen
let al. 2020 (Sieel et al. [2000] Brandfonbrener et al. (021), which carries out imitation learning by putting higher weights on the good state-action pairs. It usually uses an estimated advantage function as the weight. As this approach imitates the selected parts of the behaviour policy, and naturally restricts the learned policy within the behaviour policy. The other group of the approaches without restricting the learning policy conditional sequence modelling, which learns policy con- ditioned with particular metric for the future trajectories. Some examples of the metrics are sum of the future rewards (Srivastava et al. 2019 Chen et al., 202T), a certain state (sub goal) (Codev- illa et al., 2018) Ghosh et al. 2019 Lynch et al.) 2020) and even learned features from the future trajectory (Furuta et al. 2021).
Our approach does not belong to any of these groups but is related to the approach of learning pessimistic value function, the conditional sequence modelling and weighted imitation learning ap- proaches. Essentially. our method is conditional sequence modelling approach as it learns the following action conditioned on the current state and the sum of the future rewards, but the training data is augmented by the result of the learned pessimistic value function. Also, the overall high-level structure is somewhat similar to the weighted imitation learning, which learns the value function and uses it to weight the training data in the following imitation learning stage. However, each compo- nent is very different from ours, and it uses the value function to weight the training data, whereas our approach relabels the RTG values by tracing back the trajectory with the learned value function as well as the trajectory itself where the learned value function is not reliable. Also, in our approach, the policy learned with conditional sequence modelling, whereas they use non-conditional non- sequential models. We believe we can apply our relabelling approach to the weighted imitation learning algorithms, and is an exciting avenue for future work.
Data centric approach. Andrew Ng recently spoke about the importance of the training data to achieve good performance from a machine learning model and suggests we should spend more of our effort on data than on the model (Data-centric Approach) (Press) 1021). He said, "In the Data- centric Approach, the consistency of the data is paramount and using tools to improve the data quality that will allow multiple existing models to do well." Our method can be seen as Data-centric Approach for offline RL, as we focus on improving the training data and using the existing models. Our method provides tool to improve data quality.
5

---

Under review as a conference paper at ICLR 2023
5 EVALUATION
We investigate the performance of QDT relative to the offline RL algorithm with the Dynamic Pro- gramming based approach as well as the reward conditioning approach. As QDT utilises the result of CQL and it is considered as the sfate-of-art offline RL method, we pick CQL as the benchmark for the Dynamic Programming based approach and DT for the reward conditioning approach for the same reason. From the evaluations in this section, we would like to demonstrate the benefits and weaknesses of the Dynamic Programming approach and the reward conditioning approach and how our proposed approach (QDT) helps their weaknesses.
We start our investigation with a simple environment with sub-optimal trajectories. As it is a simple environment, Dynamic Programming approach (CQL) should work well, and as it uses sub-optimal trajectories, the reward conditioning approach (DT) will struggle. It is interesting to see how much QDT helps in the cireumstance. We also evaluate them on Maze2D environments designed to test the stitching ability with different levels of complexity. We expect that DT struggle whereas CQL and QDT performs well on them. Then, we evaluate the algorithms on complex control tasks- Open AT Gym MuJoCo environments with delayed (sparse) reward as per Chen et al. (202T). They have zero rewards all the non-terminal states and put the total reward the terminal state. should make the Dynamic Programming approach (CQL) learning harder as it requires propagating the reward from the terminal state all way to the initial state. Finally, we show the evaluation results for Open AI Gym MuloCo environments with the original dense reward setting for the reference.
Simple environment. To highlight the benefit of QDT, we evaluate our method in a simple envi- ronment, which has 6-by-6 discrete states and eight discrete actions. The goal of the task is to find the shortest path from the start to the goal state. We prepare an offline RI. dataset with a hundred episodes from uniformly random policy and then remove episode that achieves close to the optimal total reward to make sure it only contains sub-optimal trajectories. Refer to Appendix for further details of the environment and the dataset.
Table T show the summary of the evaluation results. We also evaluate the performance of CQL, which is used for relabeling. It shows vanilla DT fails badly, which indicates DT struggles to learn from sub-optimal trajectories, whereas CQL performs well as it employs a Dynamic Programming approach, which can pool information across trajectories and successfully figure out the near-optimal policy. It shows QDT performs similar to CQL,, which indicates that although QDT uses the con- ditional policy approach, overcomes its limitation and learns the near-optimal policy from the sub-optimal data. Further details and results are available in Appendix B
trajeetoon
Table 1:
Simple Environment Evaluation Results. Average and standard deviation scores are re-
ported over ten seeds.
CQL 40.0 + 0.0 15.9 = 4.4
DT
QDT
Total Reward
42.2 + 6.3
Maze2D environments. Maze2D domain is navigation task requiring an agent to reach fixed goal location. The tasks are designed to provide tests of the ability of offline RL algorithms be able to stitch together parts of different trajectories (Fu et al.) (020). It has four kinds of environments open, umaze, medium and large, and they are getting more complex mazes the order (Fig. 3 Also, it has two kinds of reward functions normal and dense. The normal gives positive reward only when the agent reaches the goal, whereas the dense gives the rewards at every step exponentially proportional to the negative distance between the agent and the goal. For the model, we use the DT source code provided by the authors P and d3ripy library for CQL, then build QDT by adding small code (replacing the return- go) to the DT source code before its training.
44] (Imai & Seno 2021) - offline RL.
Table 2 shows the summary of the results. All of the numbers in the table are the normalised total reward (score) such that 100 represents an expert policy (Fu et al. 2020). CQL works well, espe-

---

Under review as a conference paper at ICLR 2023
Figure 3: Four Maze2D environment layouts (from left to right: open, umaze, medium and large).
+
Table 2: Maze2D Evaluation Results. Average and standard deviation scores are reported over 5 seeds. The result for each seed is obtained by evaluating the last learned model on the target environment. The best average values are marked in bold. Dataset
CQL
DT
QDT
maze2d-open--o
216.7 1 80.7
196.4 1 39.6
190.1 37.8
Sparse Reward
maze2d-umaze-vl
94.7 23.1
31.0 21.3
57.3 8.2
mazz2d-medium-vi
41.8 13.6
4.2
13.3 1.3
maze2d-lagee-vl
49.6
2.9
31.0 19.8
mazze2d-open-dense-v0
307.6 43.5
346.2
325.7
Dense
Reward
mazz2d-umazz-dennse-//
72.7 10.1
10.9
58.6
maze2ddmmedium-dnsse-w
70.9
3.75
42.3 - 7.1
maze2d-large-denss.
90.9 + 19.4
4
62.2 + 9.9
cially with the dense rewards. DT struggles in many cases due to the lack of stitching ability. (These environments are designed to test the stitching ability.) QDT clearly improves DT performance, especially where CQL performs well. It indicates that QDT brings the stitching capability to DT approach. We discuss the performance gap between CQL and QDT in Section
Open AI Gym MuJoCo environments with delayed (sparse) reward. We also evaluate our ap- proach (QDT) on complex control tasks - Open AI gym MuJoCo environments with the D4RL offline RL datasets (Fu al. 2020). The Open Al gym MuJoCo environments consist of three tasks Hopper, HalfCheetah and Walker2d. We test on medium and medium-repla, v2 datasets. To demonstrate the shortcoming of the Dynamic Programing approach (CQL), we follow [Cen et al. (222T) and evaluate the algorithms with delayed (sparse) reward scenario in which the agent does not receive any reward along the trajectory and receives the sum of the rewards the final time step. Again
the DT and CQL models from the existing source code for the MuJoCo Gym environments without any modifications and add extra code for the relabelling of the RTG values.
Table 3 shows the simulation results (scores) for the delayed reward case. We also copy the simula- tion results from from/Chen et al. (2021) for DT and CQL for the reference. All of the numbers in the table the normalised total reward (score) such that 100 represents an expert policy (Fu et al.) 2020)- As expected, CQL struggles to learn good policy, whereas the DT shows good performance. Also, QDT performs similar DT though they are using the results of CQL that performs badly. indicates that QDT successfully use the information from CQL where it is useful. One exception is the medium--repla--we replay-walker22 result. QDT performs worse than DT here. Through some investiga- tions, we found that the CQL algorithm overestimates the value function in the majority of the states in the medium-replay wewpau---ppaaaallelll2 walker2d dataset. We touch the issue in the following discussion section.
Open AI Gym MuJoCo environments, We also evaluate our approach (QDT) on Open AI gym MuloCo environments with the original dense reward for the reference. As they have dense rewards and contain reasonably good trajectories, both CQL and DT would work well.
Table shows the summary of our simulation results for CQL, DT and QDT. We also copy the simulation results from Chen et al.) (202T) for DT and Emmons et al.] (2021) for CQL for the refer- ence. Firstly, we can see that our simulation results are aligned with the references except for the medium-replay-hhopper result. Because it has relatively high variance, it is probably due to the small number of samples (five random seeds). Secondly, CQL performs equal or better than DT and QDT in this evaluation. It is understandable they have dense rewards (they do not require propagating value function in the trajectory). Finally, from the comparison between DT and QDT, QDT performs the same DT.

---

Under review as a conference paper at ICLR 2023
Table 3: Open Al Gym MuloCo with Delayed Reward Evaluation Results. Average and standard deviation scores are reported with 5 seeds. Our simulation results are in the Results columns, best average boldfaced. Ref. +2 are the results copied from Chen et al. (2021). We are not sure which
version of dataset the authors used for Ref. +2 and only Hopper results are available in the paper.
CQL
DT
QDT
Dataset
Results
Ref. +2
Results
Ref.*² +2
Results
Medium Medium
Hopper-v2
23.3 1.0
5.2
57.3 + 2.4
60.7 + 4.5
50.7 5.0
HalfCheetah--2
1.0 1.0
42.2 + 0.2
42.4 0.5
Walker2d-v2
0.0
69.9 = 2.0
63.7 + 6.4
Replay
Hopper-v2
7.7
2.0
50.8 - 14.3
78.5 + 3.7
26.7
HalfCheetah-v2
6.8
33.0 = 4.8
32.8 + 7.3
Walker2d-v2
3.2
51.6 24.6
29. 15.5
Table 4: Open AI Gym MuJoCo Evaluation Results. Average and standard deviation scores reported over seeds. Our simulation results are Results columns. The best average values are marked in bold. Ref." is the results copied from Emmons al. 2021) Ref. is the results copied
from Chen et al. 2021).
CQL
DT
QDT Results
Dataset
Results
Ref." +1
Results
Ref.
Medium
Hopper-v2
69.4 + 13.1
64.6
60.3 + 5.5
67.6 + 1.0
66.5 + 6.3
HalfCheetah-v2
49.2 + 0.5
49.1
42.1 + 0.5
42.1 0.1
42.3 + 0.4
Walker2d-v2
83.0 + 0.6
82.9
73.3 + 2.5
74.0 1.4
67.1 + 3.2
Medium
Hopper-v2
96.2 + 7.9
97.8
12.7
7.0
52.1
Replay
HalfCheetah-v2
49.8 + 0.5
47.3
14.1111
36.6
35.6 1 0.5
Walker2d-v2
76.5
86.1
13.9
66.6
58.2 2 5.1
DISCUSSION
Stitching ability. To demonstrate the stitching ability, we evaluate the performance each algo- rithm with varying degrees of the sub-optimal dataset. We pick the medium-replay dataset for the MuloCo Gym environment as contains trajectories generated by various agent levels removes the best X% of the trajectories. As X is increased, more good trajectories are removed from the dataset. Thereby moving further away from the optimal setup. Fig. Aeg.Assswws the CQL, DT and QDT results well the best trajectory return in the dataset. It shows that CQL offers better results than the best trajectory within the dataset except X = O, where the trajectory contains the best score; hence can be better than that, In contrast, DT fails to exceed the best trajectory, which indicates DT fails to stitch the sub-optimal trajectories. QDT performs better than DT and becomes close to the CQL results X 40 and 50 (in the regime with 60 50% bottom trajectories).
hopper-mediim-repplayyy
Figure 4: Evaluation results (scores) for CQL, DT and QDT with the hopper- medium-replay-v2 dataset removed top X% trajectories. The shaded area shows one standard deviation range of the re- sults. It also has the maximum score in the dataset as a reference. CQL results are generally better than the maximum score, which indicates CQL successfully stitches sub-optimal trajectories, whereas DT fails so. QDT improves DT through re- labelling, being better than the maximum score on the right-hand side of the plot.
CQL
DT
QDT
80.0
max score
score
60.0
40.0
top X% removed from the dataset
8

---

Under review as a conference paper at ICLR 2023
Performance gap between QDT and CQL. Although QDT improves DT on the sub-optimal dataset scenario (Fig. 4 QDT does not perform as well as CQL for the range of small X. The results from Emmons et al. 202I) indicate that DT can perform as well CQL when plenty of good trajectories are available (medium-expert dataset). It implies that there is still room for im- provements for DT and QDT approaches with datasets that contain far from optimal trajectories. To address this, we are considering using a Q-learning algorithm specific to QDT approach, but this is left for the future work.
Conservative weight. CQL has a hyperparameter called conservative weight, denoted by a in 2 It weights the regulariser term, where the higher value. the more conservative are the value function estimations. Ideally, we would like to set it as small as possible so that the estimated value function becomes tighter lower bound; however, too small conservative weight might break the lower bound guarantee, and the learned value function might give a higher value than the true value (Kumar et al.) 202T). Empirically, we discovered that this is exactly what happens in our delayed reward experiment (Table 3) for the seedduu-rrelly-wakkete
dataset example. The value function learned by CQL in the dataset has higher values than the corresponding true value in many states, and it causes the wrong relabelling of RTG and, subsequently. a worse QDT performance. We evaluated it with higher a values- - increased from 5.0 to 100. Though this improves the QDT result from 30.3 + 16.2 46.9 + 13.8, it is still worse than DT. This is left for future work for further investigation. In this paper, we assume we have access to the environment in order to optimise the hyperparameters. However, this should be done purely offline for proper offline RL setting. Although there are some proposals (Paine et al. 2020 Fu et al. 2021 Emmons et al. 2021) this is still an active research area.
Reproducing results for benchmarking. There have been many attempts to establish a benchmark for the offline RL approaches by building datasets (Fu et al., 22020; Agarwal et al. 2020), sharing their source code, as well as producing a library focusing on offline RL (Imai & Seno [002T). How- ever, we still found some conflicting results between papers. The leading cause of the issue the requiring a vast amount of effort and computational power to reproduce the other researcher's re- sults. As a result, most authors have no choice but to re-use the original results from state-of-th the--rt papers in the literature to establish comparison. However, this leads to conflicting results due to the difficulties of reproducing all the details involved in these very diverse experimental setups. For example, many offline RL papers use D4RL MuJoCo datasets to evaluate their algorithms and com- pare them against other approaches. In this case, the datasets have three versions- namely, v0, vl and v2. While not always clearly stated, most papers use version vO. However, some use version v2, which causes some of the conflicting results. For example, Chen et al. (2021) appears to evaluate their model with the v2 dataset while referencing other papers' results that use vO. A second issue with benchmarking the results in this manner is the usual insufficient number of simulations. As the simulations require large processing power, it is not feasible to run large number of simulations. Most authors (including us) evaluate only three random seeds, which is often insufficient to compare the results. In this paper, we emphasise and analyse carefully the results from the simple environ- ment, as they helps demonstrate the characteristics of the algorithm. The more complex and realistic environments are still helpful; however, the estimated variance suggest that several cases should be handled with care when extracting conclusions.
7 CONCLUSIONS
We proposed Q-learning Decision Transformers, bringing the benefits of Dynamic Programming (Q-learning) approaches to reward conditioning sequence modelling methods to address some of their well-known weaknesses. Our approach provides a novel framework for improving offline reinforcement learning algorithms. In this paper, to illustrate the approach, we use existing state-of- the-art algorithms for both Dynamic Programming (CQL) and reward conditioning modelling (DT). Our evaluation shows the benefits of our approach over existing offline algorithms in line with the expected behaviour. Although the results are encouraging, there is room for improvement. For example, the QDT results for Maze2D (Table are better than DT but still not good as CQL. On the other hand, the QDT results for Gym MuJoCo delayed reward (Table 3) are significantly better than CQL but not as good DT in the walker2d environment. These need further investigation. We are also interested in trying different Dynamic Programming and reward conditioning algorithms in the proposed framework.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. pp. 104-114, 2020.
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Mare Bellemare. Deep reinforcement learning at the edge of the statistical precipice. Advances in neural informa- fion processing systems, 01-29304-29322 2021.
David Brandfonbrener, William F Whitney, Rajesh Ranganath, and Joan Bruna. Quantile filtered imitation learning. arXiv preprint 1xXiv:2112.00950 2021.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Picter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34, 2021.
Xinyue Chen, Zijian Zhou, Zheng Wang, Che Wang, Yanqiu Wu, and Keith Ross. Bail: Best- action imitation learning for batch deep reinforcement learning. Advances in Neural Information Processing Systems, 33:18353-18363, 2020.
Felipe Codevilla, Matthias Müller, Antonio López, Vladlen Koltun, and Alexey Dosovitskiy, End- to-end driving via conditional imitation learning. pp. 4693-4700, 2018.
Jacob Devlin, Ming- Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep
bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 2018.
Scott Emmons, Benjamin Eysenbach, Ilya Kostrikov, and Sergey Levine. Rvs: What is essential for offline via supervised learning? arXiv preprint arXiv:21 10751, 2021.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning, 2020.
Justin Fu, Mohammad Norouzi, Ofir Nachum, George Tucker, Ziyu Wang, Alexander Novikov, Mengjiao Yang, Michael R Zhang, Yutian Chen, Aviral Kumar, et al. Benchmarks for deep off- policy evaluation. arXiv preprint 103.:2003.16596 2021.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference Machine Learning, pp. 0062-2062. PMLR, 2019.
Hiroki Furuta, Yutaka Matsuo, and Shixiang Shane Gu. Generalized decision transformer for offline hindsight information matching. arXiv preprint arXi::2111.10364 2021.
Dibya Ghosh, Abhishek Gupta, Ashwin Reddy, Justin Fu, Coline Devin, Benjamin Eysenbach, and Sergey Levine. Learning to reach goals via iterated supervised learning. arXiv preprint arXiv:1912.06088 2019.
Hado Hasselt. Double q-learning. Advances in neural information processing systems, 23, 2010.
Michita Imai and Takuma Seno. d3rlpy: An offline deep reinforcement library. 12 2021.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model- based policy optimization. Advances in Neural Information Processing Systems, 32, 2019.
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model- based offline reinforcement learning. Advances in neural information processing systems, 33: 21810-21823, 2020.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off- off- aff-policy q- learning via bootstrapping enror reduction. Advances in Neural Information Processing Systems, 32, 2019a.
Aviral Kumar, Xue Bin Peng, and Sergey Levine. Reward-conditioned policies. arXiv preprint arXiv: 13465, 2019b.
10

---

Under review as a conference paper at ICLR 2023
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1179-1191, 2020.
Aviral Kumar, Anikait Singh, Stephen Tian, Chelsea Finn, and Sergey Levine. A workflow for offline model-free robotic reinforcement learning. arXiv preprint urXiv:2109.108/1 2021.
Aviral Kumar, Joey Hong, Anikait Singh, and Sergey Levine. When should we prefer offline rein- forcement learning over behavioral cloning? arXiv preprint arXiv:2204. 05618, 2022.
Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, and Pierre Sermanet. Learning latent plans from play. pp. 1113-1132, 2020.
Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine. Accelerating online reinforcement learning with offline datasets. arXiv preprint (ixxx:20006.93559, 2020.
Tom Le Paine, Cosmin Paduraru, Andrea Michi, Caglar Gulcehre, Konrad Zolna, Alexander Novikov, Ziyu Wang, and Nando de Freitas. Hyperparameter selection for offline reinforcement learning. arXiv preprint arXiv:2007.00055. 2020.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. weightt regression: Simple and scalable off- off-policy reinforcement learning. arXiv preprint
2019.
Gil Press.
Andrew ng launches htwp:////nnuneennnnnaanneeeooroo
campaign for data-centric ai.
http://wwwwfooere..commtterrrres centricaai//hhh-2296117777-7-77 June 2021.
Rafael Figueiredo Prudencio, Marcos
Maximo, and Esther Luna Colombini. A sur- vey on offline reinforcement learning: Taxonomy, review, and open problems. arXiv preprint arXiv:2203.01 01387, 2022.
Alec Radford, Karthik Narasimhan, Tim Salimans, and llya Sutskever. Improving language under- standing by generative pre-training. 2018.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pp. 8821-8831 PMLR, 2021.
Noah Y Siegel, Jost Tobias Springenberg, Felix Berkenkamp. Abbas Abdolmaleki, Michael Ne- unert, Thomas Lampe, Roland Hafner, Nicolas Heess, and Martin Riedmiller. Keep doing what worked: Behavioral modelling priors for offline reinforcement learning. arXiv preprint arXiv:2002.08336. 2020.
Rupesh Kumar Srivastava, Pranav Shyam, Filipe Mutz, Wojciech Jaskowski, and Jürgen Schmidhu- ber. Training agents using upside-down reinforcement learning. arXiv preprint 191iv:1912.02888 2019.
Richard Sutton and Andrew G Barto. Reinforcement Learning. The MIT Press, 1998.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- fion processing systems, 30, 2017.
Qing Wang, Jiechao Xiong, Lei Han, Han Liu, Tong Zhang, et al. Exponentially weighted imitation learning for batched historical data. Advances in Neural Information Processing Systems, 31, 2018.
Ziyu Wang, Alexander Novikov, Konrad Zolna, Josh S Merel, Jost Tobias Springenberg, Scott Reed, Bobak Shahriari, Noah Siegel, Caglar Gulcehre, Nicolas Heess, et al. Critic regularized regression. Advances in Neural Information Processing Systems, 33:7768-7778. 2020.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning.
arXiv preprint arXiv: 1911.191166666 2019.
Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offline model-based policy optimization. Advances in Neural Information Processing Systems, 34, 2021.
11

---

Under review as a conference paper at ICLR 2023
Appendices
A SIMPLE ENVIRONMENT EXAMPLE TRAJECTORY DATA AND ITS COMPUTATION
This section describes the trajectory data and some computation details for the simple example shown in Fig. We bring the figure here and added the state IDs in the circle (Fig. 5). The two
training data
goal
2
fail
trajectory
trajectory 2
DT agent
O-Leaming agent
Figure 5: simple example demonstrates the decision transformer approach's issue (lack stitch- ing ability) fails to find the shortest path to the goal. In contrast, Q-learning finds the shortest path.
trajectories of training data are:
trajectoryl
81=1, 0 down, 11
82 0: ri-igh,,, 12=
Sy= N5,43/NAA, 03=
(3)
trajeclory2
Ss un - ro= 11,
81 01 down, 1 10,
82= 02F,0,2-AA,, r2=
We compute the returntoo-o (RTG) from the reward ri Eq.
T
R₂
TT
where R, is RTG at time step and T is the episode length. The trajectories with the RTGs becomes follows:
trajectoryl
-
1o=1, R1 -
82 3==2,;:=right, Ro=
8,=G,o-NAA,;=00, R3=0)
(5)
trajectory2
| Ro 1
81 -1 01 -1-2, R1 --110 10,
R, R2
DT (the reward-conditioned approach) is trained to predict actions from the state and RTG, so takes [st, Rı] as the input and outputs 04- (Here, we assume the context length K 1 for DT 0 case, the DT agent is trained to predict = up from [s=S, R (trajectory 1) and right from [s=S, R- 11] (trajectory 2). For the evaluation, we set the RTG the best value (-2 in this case) at t = 0, and then the agent predicts the action from [s=S, - 2]. Because the input [s=S, R= 2] is closer to [s=S, R=-3] (trajectory 1) than [s=S, R= (trajectory 2), the agent predict a = up (trajectory 1) despite the optimal action is right (trajectory 2).
for simplicity.) For example, in
12

---

Under review as a conference paper at ICLR 2023
B SIMPLE ENVIRONMENT EVALUATION DETAILS
B.I ENVIRONMENT
The environment has 6-by-6 discrete states and eight discrete actions as shown in Fig. 6 The goal of the task find the shortest path from the start to the goal state. Each time step gives 10 reward and +100 reward at the goal. The optimal policy gives +50 total reward (= 100 10 * 5). We also remap the action so that the same action index not always optimal. The mapping differs for each state but is fixed across the episodes.
-10 reward at each time
+100 reward for the
The oplimal trajecolry (+50 total reward)
Eight discrete
Figure 6: A simple 2D maze environment, which has (-by-6 grid world and eight actions for moving eight directions. -10 reward each time step and +100 reward for the goal. The optimal trajectory keeps moving up-right the goal, which has total reward +50 (= 100 10 * 5). The action is remapped so that the same action index is not always the optimal action. The mapping differs for each state, but fixed across the episodes.
B.2 DATASET
We prepare offline RL dataset with hundred episodes from a uniformly random policy and then remove an episode that achieves positive total reward to make sure only contains sub-optimal trajectories. As result, the dataset used in this evaluation has one hundred episodes and 4,454 time steps. The maximum return of the hundred episodes is -10.0, the minimum return -490 as we terminate the episode at 50 time step, and the average return is -415.5.
B.3 CQL MODEL DETAILS
We build the CQL model for the simple environment based on Double Q-learning (Hasselt, 2010) and employ embedding lookup table module to convert the discrete state continuous high dimensional embedding space. The detailed model parameters are Table
Table 5: Simple Enviormment CQL Model Parameters
Parameter
Value
State embedding dimension
32
DQN type
fully connected
DQN number layers
2
DQN number of units
32
Optimizer
Adam
Optimizer betas
0.9, 0.999
Optimizer learning rate
5.0e-4
Target network update rate
1.0e-2
Batch size
128
Number of training steps
1000 updates
Conservative weight (a)
0.5
13

---

Under review as a conference paper at ICLR 2023
B.4 DT AND QDT MODEL DETAILS
Our DT and QDT model for the simple environment is constructed based on minGPT open-source code The detailed model parameters are in 6
Table 6: Simple Environment DT/QDT Model Parameters
Parameter
Value
Number of layers
Number of attention heads
4
Embedding dimension
64
Nonlinearity function
ReLU
Batch size
64
Context length
2
retur-too-o conditioning
50
Dropout
0.1
Learning rate
4.0c-4
B.5 FURTHER EVALUATION RESULTS FOR SIMPLE ENVIRONMENT
The following tables have the simple environment results for all ten seeds. Table shows the reward for the highest value during the training period. Table & shows the reward with the model at the end of training. DT and QDT have more significant differences between these two tables than the CQL results, which indicates that DT and QDT have overfitting issues and unstable learning behaviour.
Table Simple Environment Full Results (Best). The results from the best performing model during the training.
CQL
DT
QDT
40.0
18.2
43.6
40.0
20.4
42.0
40.0
11.2
49.2
results for ten random seeds
40.0
13.8
42.6
40.0
12.6
39.2
40.0
8.4
27.8
40.0
19.6
47.2
40.0
21.2
47.4
40.0
14.4
37.4
40.0
18.8
46.0
40.0
15.9
42.2
std.
0.0
4.4
6.3
C OPEN AI GYM MUJoCo AND MAZE2D EVALUATION DETAILS
C.1 COL MODEL DETAILS
For MuJoCo Gym CQL evaluation, we use d3rlpy library (Imai & Seno 2021). It provides script to run the evaluation (cbtly//mprouuc/offffnnccccgppp and it uses the same hyperparameters as
hhtps///hthhb.ccm/aaaattttmnn
14

---

Under review as a conference paper at ICLR 2023
Table 8: Simple Environment Full Results (Last).
The results from the model the end of the
training.
CQL
DT
QDT
40.0
-39.2
13.8
40.0
8.6
35.8
40.0
-25.4
46.6
results for ten random seeds
40.0
-20.8
16.6
30.0
-50.2
29.2
40.0
-26.0
19.6
40.0
9.4
44.0
30.0
-3.00
47.4
40.0
-10.2
23.2
40.0
7.8
35.0
mean
38.0
-18.1
31.1
std.
4.2
21.3
12.5
Kumar et al. 2020). For Mazed2d simulations, we re-use the same d3rlpy script with the same hyperparameter settings.
C.2 DT AND QDT MODEL DETAILS
For DT simulations, we use the code provided by the original paper authro for both MuJoCo Gym and Maze2D environments. For QDT simulations, added extra code to relabelling the return-to- go to the DT script (decision-trane hrtpiisnnrrrrssmommmomg///cmeame Algorithm and
The relabelling code is described in
C.3
EVALUATION PROCESS
CQL We train the CQL model with five random seeds for 500,000 updates with 256 batch size, then evaluate the model at the end of the training with 10 episode roll-outs. We inherit these CQL. settings from d3ripy offline RL library (Imai & Seno 2021)
DT We train the DT model with five random seeds for 100,000 updates with 64 batch size, then evaluate the model at the end the training with 100 episode roll-outs. We inherit these DT settings from the source code provided by the DT paper authors") (Chnn et al. 2021).
QDT We train the QDT model with five random seeds, each of them employing its own trained CQL model to relabel the dataset. QDT model is trained for 100,000 updates for MuJoCo Gym and 150,000 updates for maze2d with 64 batch size, then evaluate the model at the end of the training with 100 episode roll-outs same as DT.
C.4 HYPER PARAMETER SEARCH
We use the same hyper- pyper-parameter settings as the original papers (Kumar et al. 2020 Chen et al, 2021]. However, did some hyper-paramete searches for the conservative weight (a). It is because the optimal conservative weight value could be different for CQL and QDT.
For MuloCo Gym environments, medium-replay dataset. We take these values from the CQL paper. Then, reduce these values to if the performance of CQL and QDT varies. Table and Table shows CQL and QDT results
start with
10.0 for medium dataset and 5.0 for
15

---

Under review as a conference paper at ICLR 2023
respectively. These results show that 10.0 for medium dataset and 5.0 for medium- dataset perform well for QDT and do not degrade performance sigmificantly for CQL. Also, they are the same values the original paper, decide to keep them the same the paper.
Table 9: CQL results for Open Al Gym MuloCo with conservative weight parameter (a) sweep. Average and standard deviation scores reported over three seeds.
CQL
Dataset
a 10.0
a 5.0
a 2.5
a 1.25
Replay Medium
Hopper-v2
68.7 + 16.4
72.5 + 9.5
83.6 3.8
HalfCheetah-v2
48.9 2.4
51.8 + 2.4
57.0 1.1
Walker2d-v2
83.3 0.5
86.2 + 0.5
43.5 + 43.6
Medium
Hopper-v2
95.4 11.6
87.5 + 24.7
90.7 + 14.5
HalfCheetah-v2
49.9 - 2.9
51.8 = 2.7
54.3 = 0.2
Walker2d-v2
88.9 + 3.7
50.6 + 36.3
16.8 = 14.2
Table 10: QDT results for Open AI Gym MuJoCo with conservative weight parameter (a) sweep. Average and standard deviation scores are reported over three seeds.
QDT
Dataset
10.0
5.0
a 2.5
1.25
Medium Medium
Hopper-v2
68.6 + 7.5
65.3 + 1.3
57.5 + 6.6
HalfCheetah-v2
42.2 = 0.5
42.2 + 0.05
42.1 = 0.4
Walker2d-v2
65.9 + 3.6
70.1 + 2.4
68.8 + 6.9
Replay
Hopper-v2
55.3 28.0
4
64.0 22.9
HalfCheetah-v2
35.7 + 0.6
35.5 = 0.4
33.0 + 0.5
Walker2d-v2
59.1 + 2.8
64.3 - 5.9
45.2 = 39.5
For maze2d environment, we start with 10.0 which is the value used the CQL paper for MuJoCo Gym environments medium datasets. Then, reducing these values to see if the performance of CQL varies. Table II shows the simulation results. We pick a 1.0 as performs the best. is lower values might perform better. We see QDT shows good improvement
possible that DT with a = 1.0, so we use the value for this paper. We would like to try further optimisation the future.
Table II: CQL results for Maze2D with conservative weight parameter (a) sweep. Average and standard deviation scores are reported over three seeds.
CQL
Dataset
a 10.0
a = 2.0 66.1 + 9.8
a= 1.0
maze2d-umaze-v maze2d-medium--i
27.3 12.2
96.0 + 32.2
-3.5 + 1.3
36.6 = 3.7
35.9 + 15.3
maze2d-large-vl
-2. 0.0
40.8 + 6.0
53.2 + 7.0
D JUSTIFICATION OF REPLACING RTG WITH THE LEARNED VALUE FUNCTION
Define the optimal state value function as V* (*+)), the learned lower bound of the value function as (st) and the corresponding return-to-go value R2 We show that when (((s) RG, the error in V (st) is smaller than the error Rg. We start from the condition,
((s)) > R
(6)
(*(*)) - V(sr) < V (st) Rr-
16

---

Under review as a conference paper at ICLR 2023
As (st) is the lower bound of (s:),, (s:)) 2 (sz). Hence both sides of the above equation are non-negative. We can take the absolute of both terms, and we get,
[V' (s:) < IV (st) Ral.
This indicates that the error in (st) smaller than the error in R..
E FURTHER DISCUSSIONS
E.1 WHY CQL OUTPERFORMS DT/QDT ON MAZE2D, BUT FAILS ON MUJoCo GYM DELAYED REWARD?
It is because maze2d are simpler environments and have shorter episodes than the MuJoCo control tasks. Table 12 shows that the action dimension, the state (observation) dimension and the episode length averaged over the top 5% returns in the dataset. It can be seen that MuloCo tasks have higher action/state dimensions and longer episode lengths than Maze2d. Also, the evaluation results for the Sparse maze2d-medium and -large show some notable performance loss against the Dense counterparts, which is aligned with the fact that their episode lengths are longer than the maze2d- open and -umaee.
Table 12: MuJoCo Gym and Maze2D environments comparison. The table shows that the action di- mension, the state (observation) dimension and the episode length averaged over the top 5% returns in the dataset.
Environment
Action
State Dimension
Good Episode
Dimension
Average Length
hopper
3
11
708.2
halfcheetah
6
17
1000.0
walker2d
6
17
996.7
maze2d-open
2
49.8
maze2d-umanze
2
128.6
maze2d-medium
2
224.1
maze2d-large
2
314.6
E.2 WHY QDT OUTPERFORMS DT ON MAZE2D WHEREAS IT DOES NOT ON GYM DESPITE BOTH HAVING DENSE REWARDS?
is due the difference in the training data. maze2d dataset is designed test the stitching ability; hence it only has sub-optimal trajectories, whereas the MuJoCo Gym dataset has some optimal trajectories. If the dataset has some optimal trajectories, DT will perform well. On the other hand, if the dataset has only suboptimal trajectories, DT will struggle, and QDT improves such cases by utilising the information in CQL.
As maze2d only has suboptimal trajectories, DT struggles with them, and QDT can perform better than DT. For MuJoCo Gym cases, the dataset has some optimal trajectories; hence DT performs well, and as QDT.
Strictly speaking, there are some exceptions. MuJoCo halfcheetah-mmedium and halfcheetah- medium-replay dataset does not have optimal trajectory, still QDT performs similarly to DT. because even CQL struggles to achieve good performance on these datasets. (CQL only per- forms similarly to DT even though CQL can stitch the suboptimal trajectories.) As CQL struggled, QDT could not get much help from CQL.
The other exception is maze2d-open and maze2d-open-denae These datasets have good trajectories. actually aligned with our evaluation results. The results for maze2d-open and maze2d-open: dense show good performance with DT.
Table 13 shows the maximum, 95 percentile and 90 percentile values of the normalised returns (score) in the dataset. As discussed above, Maze2d has suboptimal trajectories (except open
17

---

Under review as a conference paper at ICLR 2023
and open-dense), and MuJoCo Gym has (near) optimal trajectories close to 100 (except halfcheetah).
Table 13: Scores in MuJoCo Gym and Maze2D datasets. This table shows that maximum score, 95 percentile score and 90 percentile score values for each dataset. Dataset
max. score
95 pet. score
pet. score
maze2d-open-vo
232.4
130.7
116.2
mazz22-open-denss--0
188.9
128.4
117.4
maze2d-umaze-vl
21.1
13.2
10.3
maze2o-umazz-denns--ll
-1.4
-11.7
-18.3
maze2d-mediun-vi
12.8
6.8
4.9
maze20-medium-ddnss--
8.9
4.0
0.3
maze2d-large-vl
16.9
6.5
-2.5
maze2ddlarge-densse-v
14.6
7.9
-2.4
hopper-mediumn-2
99.5
63.2
57.0
hopper-neediun-replay-22
98.6
46.4
31.5
halfchhettah-mediiun-22
45.0
43.0
42.5
halfchhetul-mmediun--rlllaaaa
42.4
39.9
39.2
walker2d-medium-
92.0
83.4
82.4
walkerrdd-medium-eppaaa---
89.9
66.6
42.5
E.3
WHYY QDT PERFORMS CLOSE TO DT, NOT CQL IN FIG. (GYM HOPPER)?
The main reason is that QDT employs DT as its agent algorithm. The difference lays in its training data. If the environment/dataset has specific characteristics that work against DT approach, those also work against QDT. Some of these properties, such as dataset sub-optimality, are fixed/mitigated by QDT. However, there may be other elements that are against DTT and QDT, e.g., the environment having a few critical states (Kumar et al. 2002). If this is also behind the gap between CQL and DT, then possible QDT performs close/the same DT.
Kumar et al. (002)) studied the Dynamic Programming approach and the imitation learning ap- proach and compared the upper bounds of their sub-optimality (the difference between the return from the optimal policy and the learned policy). They show that the Dynamic Programming ap- proach preferred over imitation learning when the environment has few critical states the return of the episode mostly depends upon the actions in these states. The results in Kumar et al. (2022) are based on theoretical analysis (sub-optimality upper bounds). Hence, it is possible that the imitation learning approach (DT and QDT) can perform as well as better than Dynamic Pro- gramming approaches (such as CQL) in practice. Kumar et al. (2022) empirically shows that the goal-conditioned approach remains competitive by selecting the right level of model capacity and the goal. There are still many open and ongoing discussions regarding the comparison.
E.4 EXTRA RESULTS FOR REMOVING TOP X%
We the same experiment the Stitching ability subsection in Section|oonn the other two MuJoCo Gym environments. The results (Fig. 7 do not show a clear benefit of QDT over DT. We think is because the cause the gap between CQL and DT is not just the sub-optimality in the dataset (still the ootimitiial can be the cause of the difference, but it is not the only cause in these cases.)
E.5 CONSISTENCY RELABELLING ABLATION EXPERIMENT
have tried the ablation experiment for the consistency relabelling (Algorithm 2) on a subset of environments. The results summarised it in Table [4 We run ten random seeds for Simple Environment and three for others. The Simple Environment result shows some benefits of using Algorithm 2 in its average value, although it is not significant. For the other more complex environ- ments, we do not see clear benefits of Algorithm2. We think this is because the changes applied by Algorithm2 relatively minor compared to the original RTG variations. We think is better to
18

---

Under review as a conference paper at ICLR 2023
bahlfchectah medium replay v2
walker2d medium replay v2
60.0
80.0
40.0
Score
Score
30.0
40.0
COL
COL
10.0
QDT
20.0
QDT
max Score
max Score
top X% removed from the dataset
top X% removed from the dataset
Figure 7: Evaluation results (scores) for CQL, DT and QDT with the malscheetah-mediumee and wwwkered-mmdium-reppe -erpuuuyurrrp-2222222- dataset removed top X% trajectories. The shaded area shows one standard deviation range of the results. It also has the maximum score in the dataset reference.
keep Algorithm 2, at least for now because the training data could have non-realistic (inconsistent) samples without the algorithm.
Table 14: Scores in Simple Environment, MuloCo Gym and Maze2D datasets. This table shows QDT results and QDT without the consistency relabelling (Algorithm 2).
Dataset
QDT
QDT w/o
Alg.2
Simple Environment
42.2 : 6.3
29.7 + 13.8
hopper-medium-
65.3 2.0
65.7 + 3.9
halfcheetah meeiutta-a-mmium--222-
2.3
42.4 0.1
walker2d meeiuum2-mm maze2d-large-vl
71.3
80.2 + 10.8
35.0 + 24.2
23.0 5.0
E.6 AGGREGATED EVALUATION RESULTS
We compute the aggregated evaluation results for each group of environments (maze2d, MuJoCo Gym delayed reward and MuloCo Gym) with three different metrics median, Interquantile mean (IQM) and mean (Fig. 8). It uses 95% stratified bootstrapped confidence interval (Agarwal al., 2021
The results support our conclusions 1) DT struggles in maze2d, but QDT improves DT performance by getting help from CQL. 2) CQL fails in MuJoCo Gym delayed reward. 3) DT and QDT perform similarly in MuJoCo Gym. Note that QDT has never failed in any of these groups of environments.
19

---

Under review as a conference paper at ICLR 2023
Median
IQM
Mean
CQL
DT
QDT
25
50
75
100
0
25
50
75
100
0
25
50
75
100
Normalized Score (maze2d)
Median
IQM
Mean
CQL
DT
QDT
25
50
75
100
0
25
50
75
100
0
25
50
75
100
Normalized Score (MujoCo Gym delayed reward)
Median
IQM
Mean
CQL
DT
QDT
0
25
50
75
100
0
25
50
75
100
0
25
50
75
100
Normalized Score (MujoCo Gym)
Figure 8: Aggregated evaluation results (scores) for each group of environments (maze2d, MuJoCo Gym delayed reward and MuloCo Gym) with three different metrics median, Interquantile mean (IQM) and mean. The results support our conclusions 1) DT struggles in maze2d, but QDT improves DT by getting help from CQL. 2) CQL fails in MuJoCo Gym delayed reward. 3) DT and QDT perform similarly in MuJoCo Gym.
20


Reviews:
[{"review_id": "vhRu32Lf-g", "reviewer": "Reviewer_J4fs", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on the problems with two popular offline RL algorithms: DT (which struggles with inferring optimal policies from partial solutions, referred to as the stitching problem) and CQL (which struggles to propagate rewards with a long time horizon). To address the shortcomings of both methods, the authors propose a method (QDT) that modifies returns in training trajectories with value functions (learned via CQL) when returns are lower than learned value functions, then uses this data to train DT. \n\nOn environments selected to test either stitching ability (Maze2D) or learning with sparse rewards (Mujoco with delayed reward), the authors demonstrate that, while QDT rarely outperforms both DT and CQL, it can learn when either of the two methods fails.   ", "summary_of_the_review": "To improve the ability of DT to learn from partial solutions, the authors propose a method (QDT) that modifies returns in training trajectories with value functions (learned via CQL). On environments selected to test either stitching ability or learning with sparse rewards, the authors demonstrate that, while QDT rarely outperforms both DT and CQL, it can learn when either of the two methods fails.   \n\nThe paper would be stronger if the proposed method had: (a) stronger theoretical justification, (b) better empirical results (regularly matching the best method in either setting), (c) the authors did a more thorough investigation into the performance difference between CQL and QDT (partly mentioned as left for future work in Section 6: conservative weight).", "strength_and_weaknesses": "The paper does a good job of defining and demonstrating the problems through both examples and the choice of environments. The experiments are fairly extensive and I appreciated the thorough discussion on the limitations of the method. The results are mostly consistent with the hypothesis that DT struggles with the stitching problem and CQL struggles with the delayed rewards.  \n\nGiven the problems with DT are most acute when the training dataset does not contain (near) optimal trajectories, I found the experiment with an increased percentage of removed top trajectories (Fig 4) particularly interesting, and would appreciate seeing the same experiment run on multiple environments. It should be noted QDT starts outperforming DT only in the regime with 60-50% bottom trajectories. Overall QDT seems much closer to DT than CQL in performance. Why is that the case?  \n\nSince the main contribution is the method by which the returns are relabelled, Section 3 should be rewritten to be clearer and move the theory and pseudo-algorithm from the Appendix to Section 3. The lack of strong theoretical justification for the given procedure is one of the main weaknesses of the paper.  \n\nCQL outperforms DT/QDT on Maze2D even with sparse reward, which seems inconsistent with the initial hypothesis. Why is this the case? Separating dense and sparse environments in Table 2 would highlight this better. \n\nI'd be also curious to see the results with more random seeds if the authors ran more experiments since the submission. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "da5ph_dczeF", "reviewer": "Reviewer_C8yF", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors have observed that DT (for offline RL) lacks the ability to stitch suboptimal sub-trajectories, and propose the following two-step procedure:\n\n1) learn a Q function using the CQL critic\n2) relabel the RTG of the offline dataset using the learned critic, then train a DT on the relabelled dataset\n", "summary_of_the_review": "A paper with interesting idea but lacks ablation study. The experiment results show some inconsistent patterns and it is unclear the proposed method can perform consistently better than state-of-the-art.", "strength_and_weaknesses": "This manuscript studies an interesting question for DT and offline RL. The idea of RTG relabelling is simple, the resulting algorithm is easy to implement. It does improve DT's performance on several antmaze tasks.\n\nOverall, I see this is an interesting idea, yet I think further explorations can be made in a few directions to make the current manuscript a strong submission. See discussions below.\n\n*  What the authors want to show in Figure 1 is that trajectory 1 and 2 have very different and even polarized RTGs, when conditioning on a reasonable RTG (which can be out of distribution), DT might predict the suboptimal move at the initial state. \n\nA minor point I want to mention --  the example trajectory 2 in Fig 1, actually suggests the MDP the authors consider is not episodic. For continuing tasks/MDPS, people tend to use discounted rewards, and the RTG would be finite (assuming bounded reward).  Assuming episodic MDP (a finite horizon or a terminating state), trajectory 2 wouldn't contain any infinite RTG either. The original paper of DT actually implicitly assumed episodic MDP. I believe the above idea can be illustrated with examples with finite RTGs as well.\n\n* Given this example, the central question to answer becomes how to choose the right move if the optimal subtrajectory is associated with suboptimal RTG. I can see the problem can be generalized to if the return distribution of the offline dataset contains different modes, how to stitch suboptimal trajectories associated with trajectories in different modes. The authors propose a DP approach to relabel the RTGs. I suggest the authors present the algorithms in the main paper for clarity, also use the concise math notations like :  R_{t-1} = max(R_t, Vhat(s_t)) + r_{t_1}, where Vhat is the estimated value function.\n\nOne particular design I noticed is that for each input sequence, the RTG is further relabelled as in Algorithm 2. The authors argue this is to keep the consistency between the reward and RTG within an input sequence. I'd suggest the authors run an ablation experiment to verify this is needed, and give some explanations. Will the performance of DT crash with such inconsistency? \n\nBesides, this means, for the same state action tuple (s_t, a_t), the associated RTG will be different when it is sampled in different input sequences. I wonder if this is related to data augmentation / perturbation -- you smooth out the input data distribution by those \"noisy\" RTGs, and I wonder how much performance gain will you obtain by introducing this extra labelling step. \n\n* It's better to give more details for the experiments, e.g. - how many evaluation rollouts per seed are taken? Besides, each algorithm only takes 3 seeds. Preferably, the authors can run experiments with 5 to 10 seeds. (The authors can also use the stratified bootstrap to compute statistics rather than mean: https://arxiv.org/pdf/2108.13264.pdf )\n\n* For the experiments, the proposed QDT improves upon DT on some antmaze tasks, but the performance is roughly the same as DT on the mujoco tasks (both the delayed reward mode and the normal mode). It suggests the improvement is marginal. For the ablation experimment shown in Fig 4,  DT and QDT perform all most the same.  I'm also curious why CQL works for antmaze with sparse rewards, but not Gym tasks with delayed reward?\n\nThe results are also somewhat inconsistent. QDT improves upon DT for antmaze tasks with dense rewards, but the performance are roughly the same for Gym tasks whose rewards are also dense. I'd suggests the authors try to answer why such inconsistency happens.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "4Z-EzYXflj", "reviewer": "Reviewer_QqTj", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper tries to unify Q-learning and Decision Transformer (imitation learning), which hopes to allow DT to obtain some level of trajectory stitching capabilities.", "summary_of_the_review": "If the Decision Transformer were not a hot topic at this very moment (2022 Fall), this paper should be rejected because although the idea is novel, it's a bit simplistic and Important weaknesses haven't been fully addressed. However, because this field is very hot and fast-moving, we should reward fast workers -- people who can try out ideas and offer interesting insights (even if their idea \"fails\"). In this case, I would consider QDT a failed attempt but a very interesting failed attempt. Although this is the direction we all want to go, QDT isn't the right approach. In the spirit of 1). Rewarding negative results (and the honesty of the authors); 2). Encouraging more work in this fast-moving area; 3). Reward how well-written this paper is and how insightful the discussions/experiments are; I recommend this paper be accepted.", "strength_and_weaknesses": "Strength:\n- This is a very timely paper. Augmenting Decision Transformer is a very trendy area at this moment -- using $V(s)$ to relabel RTG is a very straightforward idea, but this is the first paper to do it. Not only did they implement the change, but they also showed a comprehensive set of experiments AND motivated the change well. I find it impressive to pull this off in such a short time.\n- This paper is not afraid of revealing QDT's weakness. There are quite a few experiments and discussions that show that QDT does not quite work (yet). This might be a problem if this isn't such a fast-moving area of research -- since this area is very hot, any kind of new insight is valuable. The authors clearly know this and aren't afraid of sharing these negative results on their own algorithm. This will allow fast iteration of ideas and future research to build on these observations. \n- CQL overestimates Q values (despite conservatism). This problem has been pointed out in [1], and I'm also glad the author dedicated a whole paragraph to discuss this and offered a small ablation experiment.\n\nWeakness:\n- I wish more thought could be put into the algorithm design and address the weaknesses/issues discussed. But I also understand there isn't enough time to get these done.\n- QDT essentially only works (outperforms DT) in 1 environment (aka Maze2D). So, even though the idea is cool and well-motivated, there's not enough evidence that this is the **right** algorithm. But then again, research is iterative, and future papers will probably get it right.\n\n[1] A Workflow for Offline Model-Free Robotic Reinforcement Learning", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject