When should we prefer Decision Transformers for Offline Reinforcement Learning?
Authors: Prajjwal Bhargava
Rohan Chitnis
Alborz Geramifard
Shagun Sodhani
Amy Zhang
Venue: ICLR 2024 poster (2024)
Keywords: offline reinforcement learning
sequence modeling
reinforcement learning
Primary area: reinforcement learning

Abstract: Offline reinforcement learning (RL) allows agents to learn effective, return-maximizing policies from a static dataset. Three popular algorithms for offline RL are Conservative Q-Learning (CQL), Behavior Cloning (BC), and Decision Transformer (DT), from the class of Q-Learning, Imitation Learning, and Sequence Modeling respectively. A key open question is: which algorithm is preferred under what conditions? We study this question empirically by exploring the performance of these algorithms across the commonly used D4RL and Robomimic benchmarks. We design targeted experiments to understand their behavior concerning data suboptimality, task complexity, and stochasticity. Our key findings are: (1) DT requires more data than CQL to learn competitive policies but is more robust; (2) DT is a substantially better choice than both CQL and BC in sparse-reward and low-quality data settings; (3) DT and BC are preferable as task horizon increases, or when data is obtained from human demonstrators; and (4) CQL excels in situations characterized by the combination of high stochasticity and low data quality. We also investigate architectural choices and scaling trends for DT on \textsc{atari} and D4RL and make design/scaling recommendations. We find that scaling the amount of data for DT by 5x gives a 2.5x average score improvement on Atari.

Published as a conference paper at ICLR 2024
WHEN SHOULD WE PREFER DECISION TRANSFORMERS FOR OFFLINE REINFORCEMENT LEARNING?
Prajjwal Bhargava', Rohan Chitnis', Alborz Geramifard', Shagun Sodhani', Amy Zhang!
Meta Al University of Texas, Austin
(pradd ronucnit, alborzg, sodhani, amyzhang) @met a. com
ABSTRACT
Offline reinforcement learning (RL) allows agents to learn effective, return- maximizing policies from a static dataset. Three popular algorithms for offline RL are Conservative Q-Learning (CQL), Behavior Cloning (BC), and Decision Transformer (DT), from the class of Q-Learning, Imitation Learning, and Sequence Modeling respectively. A key open question is: which algorithm preferred under what conditions? We study this question empirically by exploring the performance of these algorithms across the commonly used D4RL and ROBOMIMIC benchmarks. We design targeted experiments to understand their behavior concerning data sub- optimality, task complexity, and stochasticity. Our key findings are: (1) DT requires more data than CQL to learn competitive policies but is more robust; (2) DT is substantially better choice than both CQL and BC in sparse-reward and low-quality data settings; (3) DT and BC are preferable task horizon increases, when data is obtained from human demonstrators; and (4) CQL excels in situations characterized by the combination of high stochasticity and low data quality. We also investigate architectural choices and scaling trends for DT on ATARI and D4RL and make design/scaling recommendations. We find that scaling the amount of data for DT by 5x gives a 2.5x average score improvement on ATARI.
INTRODUCTION
Offline reinforcement learning (RI.) (Levine et al. 2020 Lange et al., 2012: Frnst al., 2005) aims to leverage existing datasets of agent behavior in an environment to produce effective policies. key open question is: which learning method is preferred for offline RL? In this paper, we empirically investigate this question. Among many offline RL algorithms, we focus three that have been studied extensively and are thus relatively easy to interpret and compare: Conservative Q-Learnig (CQL) (Kumar et al. 2020), Behavior Cloning (BC) (Bain and Sammut, 1995). and Decision Transformer (DT) (Chen et al. 2021). CQL, from the class of Q-Learning (Sutton and Barto 2018), uses temporal difference (TD) updates learn value function through bootstrapping. In theory, it can learn effective policies even from highly suboptimal trajectories in stochastic environments, but practice, suffers from instability and sensitivity to hyperparameters in the offline setting (Brandfonbrener et al.) 2022). BC, from the family of Imitation Learning (Hussein al. 2017), mimics the behavior policy of the data; however, it relies on the data being high-quality. DT, from the class of Sequence Modeling, is recently popularized paradigm that aims to transfer the success of Transformers (Vaswani et al. 2017) into offline RL (Chen et al. 2021
[ Janner et al..
202 yet they have shown to struggle with stochastic dynamics (Brandfonbrener et al. 2022).
We design targeted experiments to understand how these three algorithms perform as we vary properties of the data, task, and environment. Our experiments are conducted across the commonly used D4RL, ROBOMIMIC, and ATARI benchmarks. Table [ shows a high-level summary of our key findings. In Section 4.1 begin by establishing baseline results in benchmark tasks for CQL, BC, and DT, for both dense-reward and sparse-reward settings. Then, we perform experiments to answer several key questions, which form the core contributions of this paper:
(Sections
How are agents affected by the presence of suboptimal data? As the notion of suboptimality take many meanings in offline RI, we consider three definitions:

---

Published as a conference paper at ICLR 2024
Property
Algorithm
SPARSE
BEST X%
WORST X
LENGTH
NOISE
HORIZON
SPACE
STOCHASTICITY
DT
X
V
X
CQL
X
V
X
X
BC
X
X
X
X V
X
X
Table I: This paper compares the robustness of three algorithms for offline RL various properties of the data, task, and environment: sparse rewards (SPArSE), training on the best X% of data (BEST X)), training on the worst X% of data (WORST X)). trajectory lengths in the dataset (LENGTH), presence of highly suboptimal actions in the dataset (NOISE), long task horizon (HORIZON), large task state space (SPACE) and Stochasticity (STOCHASTICITY). A check mark indicates relative robustness, while a cross indicates notable deterioration in performance. We see that each algorithm preferable different situations.
(Section 4.2) Our first setting involves varying the amount of data the agents are trained on. More specifically, agents to either the best X% or the worst X% of the data for varying values of X. This enables to study sample efficiency when learning from high-quality and low-quality data. (Section 4.3) In our second experiment, we study the impact of suboptimality arising from increased trajectory lengths in the dataset. In longer trajectories, rewarding states are typically further away from early states, which can impact training dynamics.
sort trajectories in the dataset by their returns and expose the
(Section 4.4) Finally, our third experiment, we examine the impact of adding noise to data, in the form of random actions. This setting be seen as simulating common practical situation where the offline dataset was accompanied by a lot of exploration.
(Section 4.5) How do agents perform when the task complexity is increased? To understand this, we study the impact of both state space dimensionality and task horizon on agent performance. [Section[46) How do agents perform in stochastic environments? To investigate this, we evaluate the performance agents as we vary the degree stochasticity and data quality.
(Sections (.77) How can one effectively use DT in practice? Based on the overall strength of DT our findings, we provide guidance on architecture (Appendix (A) and hyperparameters for DT and perform detailed analysis of scaling the model size and amount of data in ATARI
Our key findings are: (1) DT is more robust than CQL but also requires more data; (2) DT is the best choice in reware and low-quality data settings; (3) DT and BC are preferable as task horizon increases, or when data is obtained from suboptimal human demonstrators; (4) CQL excels in situations characterized by the combination of high stochasticity and low data quality; (5) larger DT models require less training, and scaling the amount of data improves scores on ATARI.
Our work aligns with recent research trend that examines the trade-offs among various algorithms for offline RL. |Brandfonbrener et al. (2022) outlined theoretical conditions, such as nearly deterministic dynamics and prior knowledge the conditioning function, under which Sequence Modeling (called "RCSL" " in their work) is preferable. Our paper extends their research by asking carefully designed questions targeted to provide novel empirical insights. Kumar al. (2023) investigated when Q-Learning might be favored over Imitation Learning. Our research expands on this inquiry by incorporating the recently popularized DT as part of the Sequence Modeling paradigm, thereby providing insight into the training dynamics and learned policies of each algorithm.
We hope this paper helps researchers determine which offline RI. algorithm to use their applica- tion. Throughout the paper, provide practical guidance which algorithm is preferred, given characteristics of the application. Code and data: httpss//gtthb..cmm/accookrrsrrrr paradg
RELATED WORK
Our work addresses the question of which learning algorithm most suitable for offline RL [Levine al.) 2020 Fu et al. 2020 Prudencio et al. 22033) by examining three prominent algorithms from the following paradigms in the field: Q-Learning Imitation Learning, and Sequence Modeling (Brandfon brener al. 2022). We chose CQL and DT as representative algorithms for Q-Learnin and Sequence Modeling, respectively, based on their popularity in the literature and their strong performance across
standard benchmarks (Kumar al. 2020 2023 Chen et al. 2021
Lee al. 2022 Kumar al.
2

---

Published as a conference paper at ICLR 2024
2023 Other options are possible well such as BCQ (Fujimoto et al. 2019), BEAR (Kumar et al.
2019a), and IQL (Kostrikoy et al. (021). The Trajectory Transformer is another algorithm within the Sequence Modeling paradigm (Janner et al.. 2021) (model-basedl Similar to Sequence Modeling, other studies have explored approaches that aim to learn policies conditioned on state and reward (or return) predict actions (Schmidhuber 2019 Srivastava et al.) [019 [Brandfonbrener al.)
2022; Kumar et al., 2019b; Emmons et al. (022), in both online and offline settings. Finally, despite its simplicity, Behavior Cloning (BC) remains widely used baseline among Imitation Learning algorithms (Kumar et al. [023)] Chen et al. 2021] Ho and Ermon (016, and Gul (2211a) and thus was chosen as the representative algorithm from the Imitation Learning paradigm. Alternative options, such as TD3-BC (Fujimoto and Gu 2021b), could also be considered.
While our research focuses on these three paradigms, it also worth mentioning model-based RI. approaches, which have begun to increase in popularity recently. These approaches have delivered promising results in various settings (Janner et al.) 2022 Yu et al. [020aa Kidambi et al. 2020a] Argenson and Dulac-Arnold (02T). but we do not examine them in our work, instead choosing to
focusing on the most prominent paradigms in offline RL Tarasov et al. 2022).
In light of the recent interest in scaling foundational models (Hoffmann et al. 2022). both Kumar et al. (2023) and Lee et al. (2022) have demonstrated that DT scales with parameter size. Moreover, Kumar ct al. (2023) indicate that CQL performs better suboptimal dense data in the Atari domain. Our findings concur with these studies but offer a more comprehensive perspective, as we also explore sample efficiency, as well the scaling of parameters and data together.
3 PRELIMINARIES
Here, we discuss brief background (see Appendix C for more details) and our experimental setup.
3.1 BACKGROUND
In reinforcement learning (RL), an agent interacts with Markov decision process (MDP) (Puterman 1990), taking actions that provide a reward and transition the state according to an unknown dynamics model. The agent's s objective to learn policy that maximizes its return, the sum of expected rewards. In offline RL (Levine et al. 2020). agents cannot interact with the MDP, but instead learn from a fixed dataset of transitions D (s, a, r, s')}, generated from an unknown behavior policy.
Q-Learning (Sutton and Barto 2018) uses temporal difference (TD) updates to estimate the value of actions via bootstrapping. We focus on Conservative Q-Learning (CQL) (Kumar et al. 22220. which addresses overestimation by constraining Q-values so that they lower-bound the true value function. Behavior Cloning (BC) (Bain and Sammut 1995) is a simple algorithm that mimics the behavior policy via supervised learning D. Sequence Modeling is recently popularized class of offline RL algorithms that trains autoregressive models to map the trajectory history to the next action. We focus on the Decision Transformer (DT) (Chen et al. 2021. in which the learned policy produces action distributions conditioned on trajectory history and desired retursstoogo. Rs = Σ - Tt- DT is trained using supervised learning on the actions. Conditioning on returns-to enables DT to learn from suboptimal data and produce a wide range of behaviors during inference.
3.2 EXPERIMENTAL SETUP
Data: We consider tasks from two benchmarks: D4RL and ROBOMIMIC, chosen due to their popular- ity (Nie al., 2022, Goo and Niekum| (022). We also explore the HUMANOID GYM environment, which is not part of D4RL:; in the process, we generate D4RL-style datasets for HUMANONI (details in Appendix D). We additionally conduct experiments
ATARI, which allows us to study the scaling propertics of DT with image observations. All tasks are deterministic and fully observed (see Laidlaw ct al. (2023) for a description of why deterministic MDPs are still challenging in RL) and have continuous state and action spaces, except for ATARI, which has discrete state and action spaces.
In D4RL (Fu et al., 2020), we focus on the HALFCHEETAH, HOPPER and WALKER tasks. For each task, three data splits are available: medium, medium replay, and medium- expert. These splits differ in size and quality. The medium split is obtained by early stopping the training of a SAC agent (Haarnoja al. 2018) and collecting 1M samples from the partially trained behavior policy.
3

---

Published as a conference paper at ICLR 2024
The medium replay split contains ~100-200K samples, obtained by recording all interactions of the agent until it reaches a medium level of performance. The mediummexxper split contains 2M samples obtained by concatenating expert demonstrations with medium data. In ROBOMIMIC (Mandlekar iet al. 2021), we consider four tasks: Lift, Can, Square, and Transport. Each task requires a robot to manipulate ate objects into desired configurations; see Mandlekar al. (202T) for details. For each task, three data splits are provided: proficient human (PH), multi-human (MH), and machine-generated (MG). PH data has 200 demonstrations collected by single experienced teleoperator. while MH data has 300 demonstrations collected by teleoperators at varying proficiency levels. MG data has 300 trajectories, obtained by rolling out various checkpoints along the training of a SAC agent, and has mixture of expert and suboptimal data. Appendix F gives details on ATARI AAarwal et al. 2020).
On D4RT., we evaluate agents on normalized average returns, following Fu et al. (2020). On ROBOMIMIC, we measure success rates following Mandlekar et al. (202T). ATART scores were normalized following Hafner al. (202T). Scores are averaged over 100 evaluation episodes for D4RL and ATARI, and 50 for ROBOMIMIC. All experiments report the mean score and standard deviation over five independent seeds of training and evaluation.
Evaluation Metrics:
Details: For DT, we used a context length of 20 for D4RL and 1 for ROBOMIMIC; see Section A for experiments and discussion on how context length impacts DT. All agents have fewer than 2. parameters (for instance, in D4RL, we have the following parameter counts: BC 77.4k, CQL 1.1M, 730k). BC and CQL use MIP architectures. For more details, see Appendix
4 EXPERIMENTS
4.1
ESTABLISHING BASELINE RESULTS
We start by analyzing the baseline performance of the three agents on each dataset since we will make alterations to the data in subsequent experiments. We studied both the sparse-reward and the dense-reward regimes in D4RL and ROBOMIMIC. Because D4RL datasets have dense rewards only, we created the sparse-reward versions by exposing the sum of rewards in each trajectory only on the last timestep. For ROBOMIMIC experiments, we used the MG dataset (Section 3.2). which contains both sparse-reward and dense-reward splits. See Table and Table for results.
We noticed three key trends. (1) DT was generally better than or comparable to CQL in the dense- reward regime. For example, on ROBOMIMIC, DT outperformed CQL and BC by 154% and 51, respectively. However, DT performed about 8% worse than CQL on D4RL. (2) DT was quite robust to reward sparsification, outperforming CQL and BC by 88% and 98% respectively and 62% respectively on ROBOMIMIC. These results are especially interesting because the dense and sparse settings have the same stafes and actions in the dataset; simply redistributing reward the last step of each trajectory caused CQL's performance to reduce by half in the case of D4RL. while DT's performance remained unchanged. A likely reason that sparser rewards mean CQL must propagate TD errors over more timesteps to learn effectively, while DT conditions on the returns-to-go go-go at each state, and so is less affected by reward redistribution. (3) BC was never competitive with the best agent due to the suboptimality of the data collection policy.
D4RL, and 194%
We note that our methodology behind making D4RL tasks sparse is the same as (Chen et al. 202 and might break Markovian dynamics. We provide an additional data point on the Maze2D environment, which provides sparse and dense rewards following Markovian dynamics, in Table 6 in Appendix G
DT
CQL
BC
Dataset
Sparse
Dense
Sparse
Dense
medium
62.56 + 1.16
63.66 + 0.55
43.94 + 4.7
67.11 +:2 0.24
53.91 + 5,93
medium replay
64.08 1 1.25
65.22 1 1.57
49.04 1 13.79
78.41 1 0.45
14.6
1 8,32
medium expert
103.15 0.77
103.64 0.12
29.36 5.14
105.39
0.84
47.72 5.5
Average
76.6 +
77.51 + 1.12
40.78 + 7.88
83,64 10.51
38.74 +: 6.58
Table 2: Results on D4rL in both sparse and dense reward settings, averaged over the HALFCHEETAH, HOPPER and WALKER tasks. BC is reward-agnostic. DT performed best with sparse rewards, while CQL performed best with dense rewards. For full results, refer to Table 12 in Appendix G
4

---

Published as a conference paper at ICLR 2024
DT
CQL
BC
Dataset
Sparse
Dense
Sparse
Dense
Lifi
93.2 3.2
96 +
60 13.2
68.4 6.2
59.2 6.19
Can
83.2
83.2 =
0
2 1.2
55.2 -: 5.8
Average
88.2 1: 1.6
89.6 1: 1.4
30 1 6.6
35.2 1 3.7
57.2 6
Table 3: Results on the MG dataset for two tasks (Lift and Can) in ROBOMIMIC. We see that DT performed best in both sparse and dense reward settings. Results on the Square and Transport task can be found in Table 13 and Figure (TTak Horizon) respectively.
Although CQL excels in certain dense-reward settings, its performance is subject to volatility in other environments. CQL is less preferable for use in rpware settings. Meanwhile, DT is competitive and risk-averse option that performs well in dense- reward settings and stands out as the perfffrffring agent in sparse-reward settings.
Practical Takeaway:
4.2 How DOES THE AMOUNT AND QUALITY OF DATA AFFECT EACH AGENT'S PERFORMANCE?
In this section, we aim to understand how agent performance varies given various amounts of high- return and low-return dense-reward data. To do this, we sorted the trajectories based on their returns and trained the agents on the "best" and "worst" X% of the data for various values of X. Analyzing each agent's s performance on the best X% enables us to understand sample efficiency: how quickly do agents learn from high-return trajectories? Analyzing the performance on the worst X% enables us to understand how well the agents learn from low-quality data. See Figure [ for results D4RL.
The " "-best" curves show that both CQL and DT improved as they observed more high-return data, but CQL was more sample-efficient, reaching its highest score at ~5% of the dataset, while DT required 20%. We hypothesize that CQL best in the low-data regime because in this case, the behavior policy is closer to the optimal one. However, going down between 20% and 80%, adding lower-return data could sometimes harm CQL, perhaps because (1) the difference between the optimal policy and the behavior policy grew larger and (2) TD updates had less opportunity to propagate values for high-return states. Meanwhile, DT was more stable and never worsened with more data, likely because conditioning on returns-to-go allowed to differentiate trajectories of varying quality. BC's performance was best with small amount of high-return data and then deteriorated greatly, which is expected because BC requires expert data.
evidenced by the "CQL-best" line in Figure Ia
The "-worst" curves show that DT learned from bad data 33% faster on average than CQL in medium replay (Figure Tap. but that they performed similarly in medium expert (Figure Ib]. This is sensible because the low-return trajectories in medium replay are far worse than those in medium expert, and we have already seen that DT is more stable than CQL when the behavior policy is more suboptimal.
In Figure 10 in Appendix G.2 we show the same graphs as Figure I but for the sparse-reward D4RL dataset. This experiment revealed two new takeaways: (1) DTT-best far more sample-efficien and performant than CQL-best when rewards are sparse; (2) suboptimal data plays more critical role for CQL in the sparse-reward setting than the dense-reward setting.
efficient agent given a small amount of high-quality data; 2) While DT requires more data than CQL, it scales more robustly with additional suboptimal data due DT's reliance on returns-to-go being more robust to variance in rewards; 3) DT can be slightly better than CQL with very low-quality data; 4) DT and CQL are preferable over BC, especially in the presence of suboptimal data.
Practical Takeaways: 1) CQL is the most
4.3 How ARE AGENTS AFFECTED WHEN TRAJECTORY LENGTHS THE DATASET INCREASE?
In this section, study how performance varies as function of trajectory lengths in the dataset; this is an important question because, in practice, data suboptimality often manifests as longer demonstrations, To study this question, turned to human demonstrations, which are qualitatively different from synthetic data (Orsini et al., (02I]): human behavior is multi-modal and may be non-Markovian, so humans exhibit greater diversity in demonstration lengths when solving task.
5

---

Published as a conference paper at ICLR 2024
DARL Medium Replay, Dense Rewards
Medium Expert,
ST-owet
Cql-bbet
Average Nornal aard Retum
00
Avergne Normlized Iteturm
1:00
Tieett
Cal-beat
pCoeett
80
40
50 test
DTivors
BD 21
20
Fersentage of dataset
training
Percentage cataset the training
(a)
(b)
Figure 1: Normalized D4RL returns obtained by training DT, CQL, and BC on various amounts of highest-return ("best") or lowest-return ("worst") data. The left plot (a) is on medium replay data, while the right plot (b) is medium expert data; both plots average HOPPER and WALKER tasks. Notice that the Y-axis limits are set lower on the left plot. We observe that CQL was the most sample-efficient agent when only small amount of high-quality data was available, but it could degrade with lower-quality data. Meanwhile, the performance of DT never worsened with more data. Also, BC performed best with 20-40% of high-quality data, highlighting the importance of expert data for BC. For full results, refer to Figure 9 Appendix G
the HALFCHEETAH,
We used the ROBOMIMIC benchmark, which contains PH (proficient human) and MH (multi-human sparse-reward datasets (Section 3.2). Because the rewards are fixed and given at the end of trajectories, the lengths of the trajectories are proxy for optimality. highlighted by Mandlekar al. (2021). The MH datasets are further broken into "Better," "Okay," and "Worse" splits based on the proficiency of the demonstrator. We leveraged this to do granular experimentation. See Table 4 for results.
Dataset Type
Training Trajectory Length
DT
CQL.
BC
PII
105 1 13
83.1 1 0.8
45.6 1 5.0
91.3 1 0.9
MH-Better
133 = 33
53.5 +
36.5
4.7
80.2
+ 2.3
MH- Better-Okay
156 + 50
$5.3+1
3.0
82.2 + 2.3
MH-Okay
180 + 51
54.2
29.8 29.8499
65.1 + 2.6
MII- Belter-Okay- Okay-Worse
194 1 93
72.2 1.2
26.5
79.6
1 3.6
MH-B -etteer---ooee
201 + 107
60.0 0.8
32.4+ 10.7
74.2
+ 3,3
-kay-kay-Wir Worse
224
52.9 1 0.5
2..7423
67.4
3,4
MH- Worse
113
52.4 1
5.8
59.5
Table 4: Average training trajectory length and corresponding success rates for DT, CQL, and BC on human- numan-generated datasets in ROBOMIMIC. Each row aggregates over the Lift, Can, and Square tasks.
We that BC outperformed all other agents. All methods performed best using shortest trajectories and deteriorated similarly as trained on longer trajectories. For full results, refer to Table [3] in Appendix G The finding that BC performed best especially interesting in light of Table 3 where BC performed far worse than DT. The difference is that Table 3 was on MG (machine-generated)) data, while Table 4 studies human genaaated data. Orsini et al. (202T) have found the source of data to play prominent role in determining how well an agent performs on a task. This finding is consistent with many prior works (Brandfonbrener et al. 2022 Mandlekar et al. [2021] Bahl et al..
2022 Gandhi et al. 2022 Wang et al., [2023] Lu et al. 2022) that have found Imitation Learning to work better than Q L-Leanning when the policy is trained on human-generated data. We corroborate
this finding the Adroit Pen-human-v -uu task, also Table 5
As trajectory lengths increase, bootstrapping methods are susceptible to worse performance because values must be propagated over longer time horizons. This is especially a challenge in sparse-reward settings and may explain why we observed CQL performing far worse than BC and DT.
Given that used context length of for DT in ROBOMIMIC (Section 3.2). the key differences between BC and DT are 1) conditioning on returns-to-go and 2) the MLP versus Transformer architecture. We hypothesize that BC performs better than DT because the PH and MH datasets

---

Published as a conference paper at ICLR 2024
100
% Deterforation
40
20
Robomime PHi Spanse
Rabomimic MG Dense
Robomimic MG Sparae
D4RL Sparse
D4RL Dunse
Figure 2: The impact of adding random data to the offline dataset for DT, CQL, and BC. The Y-axis shows the relative deterioration in the evaluation score. Results averaged over tasks and over two strategies for creating random data. The ROBOMIMIC PH dataset does not have a dense-reward split.
are high-quality enough for Imitation Learning to be effective while being too small for Sequence Modeling. Refer to Appendix | for detailed study that attempts to disentangle these differences.
Practical Takeaway: Agents are affected similarly as trajectory lengths are increased, but when the data was generated by humans, BC is preferable.
4.4 How ARE AGENTS AFFECTED WHEN RANDOM DATA ADDED THE DATASET?
This section explores the impact of adding an equal amount of data collected from random policy to our training dataset. We considered two strategies to ensure our results were not skewed based on particular strategy for collecting random data. In "Strategy 1," we rolled out a uniformly random policy from sampled initial states. In "Strategy 2," we rolled out a pre-trained DT agent for several steps and executed single uniformly random action. The number of steps in each rollout was chosen uniformly at random to be within standard deviation of the average trajectory length in the offline dataset. We can see that Strategy 1 adds random transitions around initial states, while Strategy 2 adds random transitions along the entire trajectory leading to goal states. Because results did not differ 2 significantly between the two strategies (see Appendix G.4). Figure Serure shows results averaged across
both. Akin to Section we consider dense- and sparse-reward settings in D4RL and ROBOMIMIC.
Compared BC, both CQL, and DT demonstrated enhanced robustness injected data, experiencing less than 10% deterioration. However, the resilience of these agents manifests differently. CQL's resilience is more volatile than DT's, as evidenced by the larger standard deviations of blue bars compared to orange ones. In Appendix G.4 we present per-task breakdown of the results in Figure 2 which shows several intriguing trends. CQL's performance varies greatly across tasks: it improves in some, remains stable in others, and deteriorates in the rest. Interestingly, when CQL's performance on the original dataset is poor, adding random data can occasionally improve it, as corroborated by Figure CQL's volatility is particularly apparent in ROBOMIMIC, with Figure 15 showing that CQL deteriorates nearly 100% on the Lift PH dataset improves on the Can PH dataset by nearly 2x.
The low deterioration of BC on ROBOMIMIC MG is likely just because the MG data was generated from several checkpoints of training SAC agent, and therefore already has significantly lower data quality than either ROBOMIMIC PH or D4RL. In Section 4.3 we found BC to be superior when the behavior policy was driven by humans. However, when suboptimal data is mixed with high-quality human data, DT becomes preferable over BC.
Practical Takeaway: BC is likely to suffer the most from the presence of noisy data, while DT and CQL are more robust. However, DT is more reliably performant than CQL in this setting.
4.5 How ARE AGENTS AFFECTED BY THE COMPLEXITY OF THE TASK?
now focus on understanding how increasing the task complexity impacts the performance of our agents. Two major factors contributing to task complexity are the dimensionality of the state space and the horizon the task MDP. To understand the impact state space dimensionality, we utilized the HUMANOID environment, which has 376-dimensional states, along with other D4RL tasks, which have substantially smaller state spaces. To understand the impact of task horizon, we utilized the ROBOMIMIC PH datasets for Lift, Can, Square, and Transport tasks (listed in order of increasing task
7

---

Published as a conference paper at ICLR 2024
Effect increasing State Gimendionllty
Effect Increasing Task
02L
100
190
COLPN 0CPM
Azor Roar
Average Suociss
State Space 3imens on
Approcimate Tack Horizon
Figure 3: Impact of increasing state-space dimensionality on D4RL medium expert data (left) and task horizon on ROBOMIMIC with both PH data and PH + equal amount of random data (right).
Stalinng DT Data and Data-Pammeeers on Atani
Scaling DTT Paramoters on Atarl
Aestage Sooes
140
Avctage Normalzed Grrner Soore
100
120
2.
100
60
1.54
KM
300H
TM
1.3M 20 2.59
3V
3.5M
1.5
27
3.5
43
51 67
12
of (17000001))
Pasematers (Milloon))
Figure 4: ATARI results for DT while scaling data only (left, blue), parameters only (right), and both simultaneously (left, orange), averaged over BREAKOUT, QBERT, SEAQUEST, and PONG. Numbers on top of the orange curve show model parameter sizes. We found that scaling data was more impactful than scaling parameters, but scaling both together gave minor gains over scaling data only.
horizon). Although trajectory lengths in the dataset are an artifact the behavior policy while task horizon is inherent property of the task, found average trajectory lengths in the dataset to be a useful proxy for quantifying task horizon, because exactly computing task horizon is non-trivial. Akin to the previous section, experimented with both the PH datasets and the same datasets with an equal amount of random data added in. Figure 3 shows the results averaged on tasks with identical dimensions (left) and task horizon (right) for DT, CQL, and BC.
The performance of all agents degraded roughly equally as the state space dimensionality increased (11 111-177 376). Regarding task horizon, with high-quality data (PH), all three agents started near a 100% success rate, but BC deteriorated least quickly, followed by DT and then CQL. In the presence of the suboptimal random data -PH-suboptinal) DT performed the best while BC performed poorly, consistent with our observations in Section 4.4 Additionally, CQL benefited from the addition of suboptimal data, as seen by comparing the solid and dotted blue lines. This suggests that adding such data can improve the performance of CQL in long-horizon tasks, consistent with Figure |
Practical Takeaway: All agents experience similar deterioration when the dimensionality of the state space is increased. When the task horizon is increased, DT remains a robust choice, but BC may be preferable when data is known to be high-quality.
4.6 HOw DO AGENTS BEHAVE IN STOCHASTIC ENVIRONMENTS! ?
Understanding the behavior of these agents in stochastic environment is pivotal for practical appli- cations. We evaluated our previously trained agents under the influence of Gaussian noise added to the predicted action during evaluation, with a probability of p on each timestep as follows: action action + (0, 1) * + 41), where and μ are stochasticity parameters. This modification (Fujimoto et al. 2018) is intended to simulate the effect of stochastic environment dynamics. Our find- ings, presented in Figure 5) reveal that all three agents experience similar degradation in performance when trained on medium expert data with moderate stochasticity (p- = 0.25, o = 0.25). However, in the case of sub-optimal (medium replay) data and higher stochasticity setting (p 0.5, 0.5),
8

---

Published as a conference paper at ICLR 2024
Impact of Stochasticity
60
+1
50
BC
Delerforalion
40
47
WS
Medium Repply p=...25, 020.25
Medium Repay
Medium Expert peo 29, gm0.25
Medium Experi prO.5, am0
p-0 an0
Dataset and Stochasticity parameters
Figure 5: Results on dense-reward D4RL with added stochasticity during evaluation, averaged over the HALFCHEETAI,, IIOPPER and WALKER tasks. deterioration (Y-axis) represents the performance drop relative to running evaluations without any stochasticity. Numbers within boxes above each bar represent the average normalized scores obtained on that dataset with the stochasticity parameters.
CQL demonstrated superior resilience compared to DT and BC. Building the observation made by Paster et al. (2022). who looked at DT's limitations in stochastic environments, our study provides evidence indicating that DT's performance may hold up reasonably well with CQL in continuous action spaces when stochasticity is moderate and data high-quality.
Practical Takeaway: While DT exhibits a comparable decline to CQL when trained on high-quality data in continuous action spaces, CQL is expected to be relatively more robust as data quality declines or stochasticity increases.
4.7 SCALING PROPERTIES OF DECISION TRANSFORMERS ON ATARI
Based on the relative robustness of DT in many of the previous sections, studied the scaling properties of DT along three dimensions: the amount training data, the number of model parameters, and both jointly. We focused the ATART benchmark because of its high-dimensional observations, for which we expect scaling to be most helpful. We scaled the number of model parameters by increasing the number layers in the DT architecture {6, 8, 12, 16, 18, 20, 24, 32, 64). The results are shown in Figure 4 The performance of DT reliably increased with more data up until 1.5M timesteps in the dataset (blue, left), but was insensitive or perhaps even hurt by increasing the number of parameters (green, right). Upon scaling DT to certain size (3.5M+ parameters), we observed that larger model outperformed its smaller counterpart given same amount of data. Additionally, we discuss architectural properties of DT in Appendix A
Practical Takeaway: When scaling up DT for complex environments, prioritize scaling the data before the model size. Scaling both simultancously may improve performance further.
5 LIMITATIONS AND FUTURE WORK
This work addressed the question which learning method amongst CQL, BC and DT should be preferred for offline reinforcement learning. One of the limitations of our work is that we could broaden our study to include more representative algorithms from each paradigm, such as Implicit Q-Learning (Kostrikov et al., 2021 and Trajectory Transformers (Janner et al.) 2021),
well as
paradigms we did not explore here, such as model-based offline RL (Kidambi et al. [2020b] Yu et al. 2020b) and diffusion models (Ajay al. 2022). However, note that resource limitations make this challenging: each datapoint in every plot our experiments required around 1,000+ GPU-hours, considering the aggregations over domains and random seeds. Each agent we add exponentially increases the computational demand, exceeding our budget. We also hope to evaluate on a larger set of benchmarks that includes compositional tasks, like those in embodied AI (Duan al. 2022).

---

Published as a conference paper at ICLR 2024
REFERENCES
Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. In International Conference Machine Learning, 2020.
Anurag Ajay, Yilun Du, Abhi Gupta, Joshua Tenenbaum, Tommi Jaakkola, and Pulkit Agrawal. Is con- ditional generative modeling all you need for decision-making? arXiv preprint arXiv:2211.1563 2022.
Arthur Argenson and Gabriel Dulac-Amold. Model-based offline planning. In International Confer- ence on Learning Representations, 2021. URL /ttDs:://peereevee...eek wttp://wwwwweteeetmeemmma
OKNB1G5xzd4
Shikhar Bahl, Abhinav Gupta, and Deepak Pathak. Human-to-robot imitation in the wild. In RSS, 2022.
Michael Bain and Claude Sammut. A framework for behavioural cloning. In Machine Intelligence 15, pages 103-129, 1995.
Marc G. Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforcement learning. Proceedings of the 34th International Conference on Machine Learning- Volume 70, ICML'17, page 449-458. JMLR.org. 2017.
David Brandfonbrener, Alberto Bietti, Jacob Buckman, Romain Laroche, and Joan Bruna. When does return-condinn conditionit supervised learning work for offline reinforcement learning? In Alice Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL http::ttts///ett http::://openreviice wwwpinrreiieieeeetfeeeeeeeeeeeee net forum wwt.://eeetoommmddeexxeeeoomo Xig-XXy4koXX
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:1 10415540415097, 2021.
Jiafei Duan, Samson Yu, Hui Li Tan, Hongyuan Zhu, and Cheston Tan. survey of embodied ai:
From simulators to research tasks. IEEE Transactions on Emerging Topics in Computational Intelligence, 6(2):230-22) 2022.
Scott Emmons, Benjamin Eysenbach, Ilya Kostrikov, and Sergey Levine. Rvs: What is essential for offline RL via supervised learning? In International Conference on Learning Representations, 2022. /
net/ fott:/ffooomnnnmm87aaaaa
Damien Ernst, Pierre Geurts, and Louis Wehenkel. Tree-based batch mode reinforcement learning. Journal of Machine Learning Research, 6, 2005.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.072113 2020.
Scott Fujimoto and Shixiang (Shane) Gu. minimalist approach offline reinforcement learning. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 20132-20145 Curran Associates, Inc., 202 URL 1tt0002//f1eeee/ http://eleeeeaaaaaacccaaaaaa77a-
nwwrips cttccccccppp/cttacc paper files/
Scott Fujimoto and Shixiang Shane Gu. minimalist approach offline reinforcement learning. Advances in neural information processing systems, 34:20132-20144 2021b.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International conference on machine learning, pages 1587-1596. PMLR, 2018.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference Machine Learning, pages 2052-2062. 2019.
Kanishk Gandhi, Siddharth Karamcheti, Madeline Liao, and Dorsa Sadigh. Eliciting compatible demonstrations for humat imitation learning. In 6th Annual Conference on Robot Learning, 2022. URL http::t////// /ttpss:/opperrev/ ew. net. ftte//orumm??m d= RabxvK3
10

---

Published as a conference paper at ICLR 2024
Wonjoon Goo and Scott Nickum. Know your boundaries: The necessity of explicit behavioral cloning in offline rl. arXiv preprint TrXiv:2206.0069) 2022.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforeement learning with a stochastic actor. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Pruceedings of Machine Leurning Reseurch, pages 1861-1870. PMLR, 10-15 Jul 2018. URL
https::/pproce
ngs m press v watpass/vv///aaaaaaa ha 8..htm.
Danijar Hafner, Timothy P Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. In International Conference on Learning Representations, 2021. URL
htttp:://openni
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016. URL
https: https://proceeiinggs hewpeediiinn..nuuiinee cttpcp/cpppppaeeeeee Flles//appx:2215/111 022-288785500222221443338888-8 Pappees.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, al.
Training compute-optimat large language models. arXiv preprint arXiv:2203, 2022.
Ahmed Hussein, Mohamed Medhat Gaber, Eyad Elyan, and Chrisina Jayne. Imitation learning: A survey of learning methods. ACM Computing Surveys (CSUR), 50(2):133,,2))) 2017.
Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. Advances in neural information processing systems, 34: 12:3-22-1 2021.
Michael Janner, Yilun Du, Joshua Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. In International Conference Machine Learming, 2022.
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel:
Model-based offline reinforcement learning, In H. Larochelle, M. Ranzato, R. Had- sell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 21810-21823. Curran Associates, Inc., 2020a. URL
https///prroceeiinnss.tt neurips CC/ ww/paper.pppp....ppp..e..ee http::/aw...........a/.aaa/e///a flees/pape:/220//1ee
E Teeaaeff6leeebb8dd3222774010 Paper. .ddf
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model- based offline reinforcement learning. Advances in neural information processing systems, 33: 21810-21823, 2020b.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, abs/1412.6980, 2014.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. arXiv preprint 10..0221 2021.
Aviral Kumar, Justin Fu. Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. Advances Neural Information Processing Systems, 32, 2019a.
Aviral Kumar, Xue Bin Peng. and Sergey Levine. abs/1912. 13465, 2019b.
Reward-coonditonee policies. ArXiv,
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances Neural Information Processing Systems. 33:1179-1199 2020.
Aviral Kumar, Rishabh Agarwal, Xinyang Geng, George Tucker, and Sergey Levine. Offline q-learning diverse multi-task data both scales and generalizes. In International Confer- ence on Learning Representations, 2023. URL https:///////...ttt https:://ppnreevie. net/ hwtp:netee..aaiieeeeem forum?id
4 1-k7kJavA

---

Published as a conference paper at ICLR 2024
Cassidy Laidlaw, Stuart Russell, and Anca Dragan. Bridging rl theory and practice with the effective horizon, 2023.
Sascha Lange, Thomas Gabel, and Martin A. Riedmiller. Batch reinforcement learning. In Reinforce- ment Learning, 2012.
Kuang-Huei Lee, Ofir Nachum, Sherry Yang. Lisa Lee, C. Daniel Freeman, Sergio Guadarrama, Ian game decision
Fischer, Winnie Xu, Eric Jang, Henryk Michalewski, and Igor Mordatch. Multi-game transformers. Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors. Advances in Neural Information Processing Systems, 2022. URL / net/forum? id=
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint (rXiv:2005.01643 2020.
Hao Liu and Picter Abbeel. Behavior from the void: Unsupervised active pre-training. In
M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Ad- vances in Neural Information Processing Systems, volume 34, pages 18459-18473. Curran Asso- ciates, Inc., 2021. URL/htttp::///rroeeddnn neuri wttpcc/ppppppee..e files/paper Paper.
2021/ htt.///ee//000....30066666666600
Yiren Lu, Justin Fu, George Tucker, Xinlei Pan, Eli Bronstein, Becca Roelofs, Benjamin Sapp, Brandyn White, Aleksandra Faust, Shimon Whiteson, Dragomir Anguelov, and Sergey Levine. Imitation is not enough: Robustifying imitation with reinforcement learning for challenging driving 11491
scenarios, 2022. URL hwt.:htttt//
11419
Ajay Mandlekar, Danfei Xu, Josiah Wong, Soroush Nasiriany, Chen Wang, Rohun Kulkarni, Li Fei- Fei, Silvio Savarese, Yuke Zhu, and Roberto Martin-Martn. What matters in learning from offline human demonstrations for robot manipulation. In arXiv preprint arXiv:2088.03298. 2021.
Paul Michel, Omer Levy, and Graham Neubig. Are sixteen heads really better than one? In H. Wallach, H. Larochelle, A. Beygelzimer, F. s'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Asso- ciates, Inc., 2019. URLLttpss//ppoeeeddng... neurips............e..... wttpsuurrpscccpppppeeeeeeeeeeeee cc/paper..........p.pe files/paper 220///Ele22200aeb222f66222222222
Paper.
Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Andrei A. Rusu, Joel Veness, Mare G. Belle- mare, Alex Graves, Martin A. Riedmiller, Andreas Kirkeby Fidjeland, Georg Ostrovski, Stig Petersen, Charlie Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518:529-532 533,, 2015.
Allen Nie, Yannis Flet-Berliac, Deon Jordan, William Steenbergen, and Emma Brunskill, Data- efficient pipeline for offline reinforcement learning with limited data. Advances in Neural Informa- tion Processing Systems, 35:14810-1448 2022.
Manu Orsini, Anton Raichuk, Leonard Hussenot, Damien Vincent, Robert Dadashi, Sertan Girgin, Matthieu Geist, Olivier Bachem, Olivier Pietquin, and Marcin Andrychowicz. What matters for adversarial imitation learning? In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021. URL https //openreview net. htt://ormmmmmm d=-Or=a33bbG11
Keiran Paster, Sheila MclIraith, and Jimmy Ba. You can't count on luck: Why de- cision transformers and rvs fail in stochastic environments. In S. Koyejo, S. Mo- hamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural In- formation Processing Systems, volume 35, pages 38966-38979, Curran Associates, Inc., 2022. URL httpps./pproceedings http://wwwweeiiiooeeeeeeeeeeeeee http:///pppppeeeeeeeeeeeeeeeeeee File/ 110.1111121000233111111111010100 fitees/ppppeee/222///1
Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International Conference Machine Learning (ICMI.), 2017.
12

---

Published as a conference paper at ICLR 2024
Rafael Figueiredo Prudencio, Marcos ROA Maximo, and Esther Luna Colombini. A survey offline reinforcement learning: Taxonomy, review, and open problems. IEEE Transactions on Neural Networks and Learning Systems, 2023.
Martin L Puterman. Markov decision processes. Handbooks in operations research and management science, 2:331-434, 1990.
Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable-baselines, Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22(268): 2021. URL http::ttt///tttt /
20 1364.htm.
Jürgen Schmidhuber. Reinforcement learning upside down: predict rewards just map them to actions. CoRR, abs/1912.02877. 2019. URL httt::://aax.i..or//a/////////// 02875
Rupesh Kumar Srivastava, Pranav Shyam, Filipe Wall Mutz, Wojciech Jaskowski, and Jürgen Schmidhuber. Training agents using upside-down reinforcement learning. ArXiv, abs/1912.02877, 2019.
Richard Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Denis Tarasov, Alexander Nikulin, Dmitry Akimov, Vladislav Kurenkov, and Sergey Kolesnikov. CORL: Research-oriented deep offline reinforcement learning library. In 3rd Offline RL Work- shop: Offline RI. as a "Launchpad", 2022. URL https:://ppereeeeeeeee ew. net. forum? id=
SyAS49bBcV
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Chen Wang, Linxi Fan, Jiankai Sun, Ruohan Zhang, Li Fei-Fei, Danfei Xu, Yuke Zhu, and Anima Anandkumar. Mimicplay: Long-horizon horizon imitation learning by watching human play. arXiv preprint arXiv:2302.12222. 2023.
Denis Yarats, David Brandfonbrener, Hao Liu, Michael Laskin, Pieter Abbeel, Alessandro Lazaric, and Lerrel Pinto. Don't change the algorithm, change the data: Exploratory data for offline reinforcement learning. arXiv preprint 2rXiv:2201.13425 2022.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neu- ral Information Processing Systems, volume 33, pages 14122. Curran Associates, Inc., neurips wttpuuiipp..pppppp....ppppppppee 2112//222222200077222444777788
2020a. URL htttp::/ppocceedinnneee
Files/paper:2020/ Paciidaa-pperpppp..
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization, Advances in Neural Information 2020b.
Processing Systems.
13

---

Published as a conference paper at ICLR 2024
Effect increasing Context
Effect on Attention Heads
Alat 048L
40
80
Conghn
of Amention Heads
Figure 6: Increasing the context length (left) and number of attention heads (right) of DT on both ATARI and D4RL. ATARI curves average over BREAKOUT, QBERT, SEAQUEST, and PONG. D4RL curves average over all tasks and data splits. We see that scaling context length and attention heads benefits performance on ATARI, but not D4RI., though the observed benefit of increasing the number attention heads is not statistically significant.
A ARCHITECTURAL PROPERTIES OF DECISION TRANSFORMERS
Here, we study the impact of architectural properties of DT, namely the context length, # attention heads, of layers, and embedding size. Full experimental results are in Appendix
Context length: The use of context window makes DT dependent the history of states, actions, and rewards. unlike CQL and BC. Figure O (left) demonstrates the role of context length for DT. Having context length larger than (i.e., no history) did not benefit DT on D4RL, while helped on ATARI. where performance maximized at a context length of 64. This finding indicates that some tasks may benefit from more extensive knowledge of history than others. The deterioration in performance, context length increases, is likely due to DT overfitting to certain trajectories.
Attention Heads: While the importance of the number of Transformer attention heads has heen noted in NLP (Michel al. 2019). is an open question how the trends carry over to offline RL. ( shows the impact of this hyperparameter on DT. We observed monotonic improvement on ATARI, but no improvement on D4RL. One of the primary reasons for this discrepancy is that there is more room for an agent to extract higher rewards ATARI than D4RL. For instance, DT achieved expert-normalized score of over 320 the BREAKOUT ATARI game, but never gets over 120 in D4RL. This suggests there is more room for scaling the data/parameters to improve results on ATARI.
Number of layers: This was discussed in Section 4.7 with results in Figure 4
Embedding size: Increasing the embedding size of DT beyond 256 did not result in any improvement in ATART; see Table in Appendix for results.
B ADDITIONAL RESULTS
this section, we provide additional data points on three agents on different environments of D4RL.
Pen-Human-vD from Adroit has been generated from human demonstrations and is higher dimensional compared to other tasks. We observed that DTT and BC outperform CQL, while (Brandfonhrener ct al.) 2022) showed that DT outperformed IQL.
Trajectory stitching:
Maze euvironments such as Antmaze and Maze2D require agents perform trajectory stitching. As (Brandionbrrener et al. (002)) noted, DT based methods require trajectory level information prohibiting DT to make use of information across trajectories. It is due to these reasons that Q-Learning based methods might be considered preferable in the case data requires trajectory stitching to be performed.
Dataset
DT
CQL.
BC
Pen-HHmman-vO 78.36 1 4.09
72.52 1 2.31
85.68 6.77
Table 5: Results of all the agents on Pen-Human-vO.
14

---

Published as a conference paper at ICLR 2024
Dataset
DT
CQL
BC
untmazz-umaze-vl
66.66 +: 5.31
71.33 + 2.05
62.6 + 6.8
Maze 2D Sparse
6.7 5.89
19.86 0.95
15.39 6.57
Maze 2D Dense
27.9 = 6.85
49.62 7.79
5.23
Table 6: Results of all the agents on Maze environments. CQL outperforms DT on these tasks due to the ability of Q learning based methods to perform trajectory stitching.
C EXTENDED BACKGROUND
In the reinforcement learning (RL) problem, an agent interacts with Markov decision process (MDP) (Puterman 1990). defined as tuple AS R. H). where S and A denote the state and action spaces. T('', (s',, s) Pr(s' (s, a) is the transition model, [R(s, a) IR is the reward function, and H is the (finite) horizon. At each timestep, the agent takes an action a E A, the environment state transitions according to T. and the agent receives a reward according to R. The agent does not know T or R. Its objective is to obtain a policy π : S + A such that acting under the policy maximizes its return, the sum of expected rewards: B8| B R(st, [(+))]] In offline RL (Levine et al. (020), agents cannot interact with the MDP but instead learn from a fixed dataset of transitions D {8,, r, }. generated from an unknown behavior policy.
O-Learning and CQL. One of the most widely studied learning paradigms is Q-Lcarning (Sutton and Barto, 2018), which uses temporal difference (TD) updates to estimate the value of taking actions from states via bootstrapping. Although Q-Learning promises to provide general-purpose decision-makin
framework, in the offline setting algorithms such as Conservative Q-Learning (CQL) (Kumar el al.] (020)) and Implicit Q-Learning (IQL) (Kostrikov et al. 2021) are often unstable and highly sensitive to the choice of hyperparameters (Brand- Tonbrener et al. 2022). In this paper, we will focus on Conservative Q-Learning (CQL). CQL (Kumar et al. 2020) proposes modification to the standard Q-learning algorithm to address the overestimation problem by constraining the Q-values so that they do not exceed lower bound. This constraint is highly effective in the offfine setting because it mitigates the distributional shift between the behavior policy and the learned policy. More concretely, CQL. adds a regularization term to the standard Bellman update:
min E.-D log Σ exp(Qo(s.
- -Ea.a-D
(Qo(s.a)) + TDError(0; DD),
(I)
where a is the weight given to the first term, and the second term is the distributional TDError D) under the dataset, from C51 (Bellemare et al. (0017).
Imitation I Learning and BC. When the dataset comes from expert demonstrations or is otherwise near-optimal, researchers often turn to imitation learning algorithms such Behavior Cloning (BC) or TD3+BC (Fujimoto and Gul 2021b) to train a policy. BC simple imitation learning algorithm that performs supervised learning to map states s in the dataset to their associated actions a. Due to the -wwar--ggoostic nature of BC, it typically requires expert demonstrations to Ieam an effective policy. The clear downside of BC is that the imitator cannot be expected to attain higher performance than attained in the datasct.
Sequence Modeling and DT. Sequence modeling is a recently popularized class of offline RL algorithms that includes the Decision Transformer (Chen el al. [02T) and the Trajectory Transformer (Janner et all] (02T). These algorithms train autoregressive sequence models that map the history of the trajectory to the next action. In the Decision Transformer (DT) model, the learned policy produces action distributions conditioned on trajectory history and desired returns-to-ge R2 = Σi= Ft. This leads to the following trajectory representation, used for both training and inference: T = (R1, 81, 01, R2, 82, .2,..., Ri, 8r, ar). Conditioning on returns-to-go enables DT to learn effective policies from suboptimal data and produce range of behaviors during inference.
D ADDITIONAL DATASET DETAILS
Humanoid Data In this section, we present the details of the HUMANOID offline Reinforcement Learning (RL) dataset created for our experiments. We trained a Soft Actor-Critic (SAC) agent (Haarnoja et al. (2018) for
3 million steps and selected the best-performing agent, which achieved a score of 5.5k, to generate the expert split. To create the medium split, utilized agent that performed at one-third of the expert's 's performance. We then generated the medium-expert split by concatenating the medium and expert splits. Our implementation is based on (Raffin et al. (2021) and employs the default hyperparameters for the SAC agent (behavior policy). Table 7 displays the performance of all agents across all splits of the HUMANOID task. Furthermore, Table g provides statistical information the HUMANOID dataset.
Robomimic:
We visualize the return distribution of ROBOMIMIC tasks, employing discount factor of 0.99, illustrated in Figure 7 and Figure 8 It hecomes evident that the discount factor has a significant impact on the
15

---

Published as a conference paper at ICLR 2024
Task
BC
CQL
Humanoid Medium
13.22 + 4.25
23.67
2.48
49.51 0.77
Humanoid Medium Expert
13.67 + 5.21
52.63
1.92
54.1 = 1.36
Humanoid Expert
24.22 + 5.37
53.41 + 1.32
63.69 + 0.22
Average
17.03
43.23
55.76
Table Performance agents on high dimensional state space task. CQL seems better suited to tasks involving higher state space dimensions.
Task
# of trajectories
of samples
Humanoid Medium
4000
1.76M
Humanoid Expert
4000
3.72M
Humanoid Medium Expert
8000
5.48M
Table Dataset statistics for HUMANOID. The number of samples refers to the number of environ- ment transitions recorded in the dataset.
data's optimality characteristics. PH features shorter trajectories, resulting in a higher proportion of high-retum data.
Distribution of Returns across trajectories
Gamma=0.99 (Robomimjc Can)
64
60
Proficient Human
Multi Human
50
45
Frequency
40
40
37
35
30
26
32
32
30
28.
14
11
10
0-0.2
0.2.4
0.40666
0.6.888
0.8-1
1-1.2
1.2414
1.4..6
1 1.6.6 8
1-22
2.2.2
2222
0
Returns
Figure 7: Return Distribution in Proficient and Multi Human splits of Robomimic Can environment shown with gamma set 0.99. Proficient Human trajectories are shorter in length compared Multi Human trajectories. Even though PH and MH datasets contain the same number of trajectories, the PH dataset contains more near-expert data.
E ADDITIONAL EVALUATION DETAILS
We specify our sampling procedure used evaluate the ATARI benchmark, used in Section 4.7 and Scction A The ATARI offline datasct (Agarwal et al. 2020) contains the interaction of DQN agent (Mnih et al. 2015) as it is trained progressively in 50 buffers. Each observation in the dataset contains the last 4 frames of the game, stacked together. Buffers 1 and 50 would contain interactions of the DQN agent when is naive and expert respectively. Our results averaged of 500k timestep
four different experiments. Each experiment performed sampling the buffers numbered 1) 1-50 2) 40-50 (DQN competitive), 3) 45-50, and 4) 49-50 (DQN is expert). We study the architectural and scaling properties of DT this dataset, where we consider four games: BREAKOUT, QBERT, SEAQUEST, and PONG. We follow the protocol of Lee et al. (2022) and Kumar et al. (2023) by training the Atari DQN Replay dataset, which used sticky actions, but then evaluating with sticky actions disabled.
16

---

Published as a conference paper at ICLR 2024
Distribution of Returns across trajectories Machine Generated (Robomimic Can) dense; gamma=0.99
Distribution of Returns across trajectories Machine Generated (Robomimic Can)
Sparse: ; gamma=0.99
30
1750
25
1500
Frequency
1250
Frequency
20
1000
750
10
0-5
5:10
10-15
15:20
20-25
25:30
30:35
35-40
40-45
45-50
50 55
55.60
05
6:10
10-15
15-20
20-25
25:30
30-35
35-40
40-45
45:50
50-55
55-60
60-45
Returns
Returns
Figure 8: Reward Distribution in Machine Generated splits of Robomimic Can environment shown
with gamma set to 0.99. The sparse variant has more high return trajectories.
F DISENTANGLING DT AND BC
In this section, experimented with additional baseline, "BC Transformer." modified version of DT that does not perform conditioning on retums-to-go vector and has a context length of previously mentioned, we set the context length to in the case of DT for running experiments on ROBOMIMIC as well. This section aims to investigate the discrepancy between the performance of DT and BC, specifically, we wanted understand how much of the discrepancy differences between DT and BC which is typically implemented using a stack of MLPs. By having the BC Transformer baseline, only distinguishing component between and BC BC is often better-perree better-performing agent than BC Transformer on PH Table 9 MG Table 10 and MH tasks Table IT] Additionally, it can also be observed that RTG conditioning only plays critical role when the distribution of reward shows variation. Unlike expert data such PH and MH where the RTG vector remained the same, see DT outperforming BC Transformer on MG significantly.
be attributed to RTG conditioning compared architectural
architecture. We observed that
Layers
BC TRANSFORMER
DT
BC
CQL
Lift-PH
0
100 =
100 0
92.7
Can-PH
94.6 2.4
96 10
95.3 95.3
38 7.5
Square-PH
52 4 5.8
53.3 4 2.4
78.7 1 1.9
5.3 1 2.5
Transport-P11
0
1.3
29.3 0.9
010
Table 9: Results of agents ROBOMIMIC PH tasks
Layers
BC TRANSFORMER
DT
BC
CQL
Lift-MG-sparse
77.9 7.11
96 - 2.1
65.3 - 2.5
64 2.8
Can-MG-sparss
48.66 1.8
92.8 1 4.1
64.7 3.4
1.3 0.9
Lift-MG-dent
68 3.2
93.99 - 2.5
60 2
63.3 5.2
Can-MG-dense
43.99 + 3.2
82.4
64
4.3
0
Table 10: Results of agents on ROBOMIMIC MG tasks
17

---

Published as a conference paper at ICLR 2024
Layers
BC TRANSFORMER
DT
BC
COL
Lift-MH
100 0
100 0
100 0
56.7 + 40.3
Can-MH
93.3 4.1
95.33
86 4.3
5.7
Square-MH
18.66 - 4.1
21.33 + 3.7
5.67
0.7 = 0.9
Transport-MIf
0
0
11.3 2.5
0
Table 11: Results of agents ROBOMIMIC MH tasks
18

---

Published as a conference paper at ICLR 2024
G ADDITIONAL RESULTS ON D4RL AND ROBOMIMIC
This section contains results obtained individual tasks of D4RL. and ROBOMIMIC benchmarks. We the averaged-oul results obtained on all the tasks from respective benchmark our analysis.
G.1 ESTABLISHINGG BASELINES
This section presents baseline results individual tasks both sparse and dense settings D4RL. The average outcomes are detailed in Table 2 outperforms CQL and BC in nearly all tasks within the sparse setting of the D4RL benchmark. Although CQL achieves marginally higher average medium-replay splits), also exhibits significantly higher volatility, evidenced by the standard deviations. contrast, DT remains competitive robust. the sparse reward setting, CQL surpasses BC by 5.2%. As highlighted in Scction subsection 4.1 CQL is most effective in the dense reward setting of the D4RL benchmark.
4.1). Our observations indicate that DT consistently
the Hopper task (3.4% ahead of DT for medium and
DT
CQL
BC
Dataset
Sparse
Dense
Sparse
Dense
Medium
Half Cheetah
42.54 + 0.12
42.35 + 0.02
38.63 = 0.81
47.03 + 0.075
42.76 0.17
Hopper
69.75 + 2.31
73.03 + 0.74
73.89
10.12
70.54 0.41
64.35 5.6
Walker
1.07 75.42
75.42 + 0.9
19.31
3.17
83.77 0.25
54.62 12.04
Average
62.56 1.16
63.66 0.55
43.94 +
67.11 0.24
$3.91 5.93
Medium Replay
Half Cheetah
38.76 0.26
37.09 +
35 2.56
46.12 0.06
9.81 9.2
Hopper
82.24 + 1.58
89.15 + 1.19
83.1 + 19.21
101.27 + 0.46
16.19 + 10.8
Walker
71.24 +: 1.93
69.44 + 3.14
29.02 19.63
87.85 + 0.83
17.82 1: 4.96
Average
64.08 + 1.25
65.22 + 1.57
49.04 = 13.79
78.41 + 0.45
14.6 + 8.32
Medium Expert
Half Cheetah
90.55 1: 1.53
91.67 1 0.21
24.35 1 2.38
94.95 1 0.25
42.95 1: 0.14
Hopper
110.29 1: 0.58
110.76 + 0.06
42.44
12.52
109.67 1: 2.12
62.21 1 6.5
Walker
108.63 1:
108.49
21.3 0.55
111.58 1 0.17
38 9.86
Average
103.15 = 0.77
103.64 0.12
29.36 5.14
105.39 + 0.84
47.72 + 5.5
Total Average
76.6 1 77.51 +: 1.12
40.78
7.88
83.64 1.0.51
38.74 1: 6.58
Table 12: Results on D4RL in both sparse and dense reward settings, averaged over the HALFCHEETAH, HOPPER, and WALKER tasks. The presented results here are an extension of Table
19

---

Published as a conference paper at ICLR 2024
G.2 How DOES THE AMOUNT AND QUALITY OF DATA AFFECT BACH AGENT'S PERFORMANCE?
Figure 9 illustrates the performance of agents on individual tasks within the D4RL benchmark as data quality and quantity varied. DT improved plateaued (upon reaching maximum performance) as additional data was provided. contrast, CQL exhibits volatility, displaying significant performance declines HOPPER and WALKER2D medium-replay tasks. The performance of BC tends deteriorate when trained on low-return data.
Half Cheetah Medium Replay, Dense Rewards
Half Cheetah Medium Expert, Dense Rewards
100
Average Normalized Return
CQL-best
Average Normalized Return
DT-best
CQL-wrrt
80
BC-best
OT-worst
DT-best
60
D DT-worst
20
BC best
COL-wrrs
40
BCwerar
BC-worst
0
20
40
60
80
100
40
60
80
100
Percentage of dataset given to the agent for training
Percentage dataset given to the agent for training
Hopper Medium Replay, Dense Rewards
Hopper Medium Expert, Dense Rewards
DT-best
COL-best
Average Fizellizede Return
80
Average ize Return
DT
BC
80
COL-bet
ized
80
DT worst
DT-warst
CQL
60
Bo
wor-werst
BC-warst
20
20
40
100
0
20
40
60
80
100
Percentage dataset given agent for training
Percentage dataset go agent for training
(d)
Walker Medium Replay, Dense Rewards
Walker Medium Expert, Dense Rewards
CQL best
Average Normalized Return
80
Average Normalizeed Return
DT-best
DT-best
CQL-worst
worst
BC-best CQl-best
worst
BC-t
DTwworst
CQL-worst
20
BC-worst
Percentage of dataset given to the agent for training
Percentage of dataset given the agent for training
(c)
(f)
Figure 9: Results demonstrating the behavior of agents as the amount of data and quality is varied on medium-replay and medium- splits of D4RL benchmark in dense reward setting. The results presented here are an extension of Figure
20

---

Published as a conference paper at ICLR 2024
Figure 10 illustrates the performance behavior of agents the quantity and quality of data adjusted in sparse setting on the D4RL dataset. A more detailed exploration of this behavior across individual tasks presented TigureFiigure 11
Two key observations can be made from these results. 1) In the sparse reward setting, DT becomes a markedly more sample-efficient choice, with its performance either improving remaining steady as the quantity of data increases. In contrast, CQL displays grealer variability and fails to exceed BC in scenarios involving expert data (medium-ed (medium-exppert).. 2) The sub-optimal data plays significantly more important role CQL sparse settings as compared to dense settings. Our hypothesis is that the sparsity of feedback makes learning about course correction more critical than learning from expert demonstrations. Notably, we discovered that the worst 10% of data features trajectories with substantially higher return coverage, which contributes to greater diversity within the data. This, in turn, enhances CQL's ability learn superior Q-values (course correction) when compared the best. 10% data in medium- -xppert data setting.
D4RL Medium Reploy, Sparse Rewards
Medium Expart, Sparse Rewards
80
D1-bwal
Average Norma aomaaaoddan Retum
60
Avergee Normlized Itetur
Cil-ward
20
SC-teet
40
BC 10
Diwwo
C2B Avee
20
best
Cqu-beet
80 word
Persentage of dataset for training
Percentage cataset the for training
(a)
(b)
Figure 10: Normalized D4RL returns obtained by training DT, CQL, and BC various amounts of highest-return ("best") or lowest-return ("worst") data in sparse reward setting. The left plot (a) is on medium replay data, while the right plot (b) is medium expert data; both plots average HALFCHEETAH, HOPPER and WALKER tasks. Notice that the Y-axis limits are set lower on the left plot. We observed that DT was substantially more sample-efficient agent than CQL. Sub-optimal data was noticeably crucial for CQL in sparse reward setting.
the
21

---

Published as a conference paper at ICLR 2024
Half Cheetah Medium Replay, Sparse Rewards
Half Cheetah Medium Expert, Sparse Rewards
DT-best
DT-best
Average Nonmalzzed Return
Average Normalized Return
80
BC-besh
30
6.0
20
CQL-worst
BC-beet
40
woL-w
CQL-best
20
EQU QQt-best
Boworrs
20
Percentage of dataset given to the agent for training
Percentage of dataset given to the agent for training
(a)
(b)
Hopper Medium Replay, Sparse Rewards
Hopper Medium Expert, Sparse Rewards
COL-wrrt
100
DT-best
Average Normalized Return
Average Normalzzed Return
BC-best
DT-best
80
coL-w
Cos-best
60
DTwwrrst
Bo-worst
DT-worst
40
20
20
CQL-best
worst
Percentage of dataset given to the agent for training
Percentage of dataset goven the agent for training
(c)
(d)
Walker Medium Replay, Sparse Rewards
Walker Medium Expert, Sparse Rewards
70
DT-best
DT
Average Normalized Retur
Average Normal izd Return
100
DT worst
50
DT TTwworst
80
BC-worst
woraw
60
PC best
80
CQL-best
EQL bast
20
C0L Wont
10
BC-worst
20
40
60
80
100
20
40
60
80
100
Percentage dataset given agent for training
Percentage dataset given agent for training
(f)
Figure 11: Results demonstrating the behavior of agents as the amount of data and quality is varied on medium-replay and medium-expert splits of D4RL benchmark sparse reward setting.
22

---

Published as a conference paper at ICLR 2024
G.3 How ARE AGENTS AFFECTED WHEN TRAJECTORY LENGTHS IN THE DATASET INCREASE?
Table showcases the performance all agents across individual ROBOMIMIC tasks, encompassing both synthetic -uman-generated datasets. surpasses both agents all synthetic tasks within the ROBOMIMIC benchmark, for both sparse and dense settings. Interestingly, BC demonstrates a strong aptitude for numerous tasks with human-generated data, particularly excelling the SQUARE task.
Dataset Type
Average Trajectory Length
DT
CQL
BC
Lift MG Dense
0
+ 1.2
68.4 6.2
59.2+6619
Lift MG Sparse
150 + 0
93.2 + 3.2
60.2
59.2:1.199
Can MG Dense
150 1: 0
83.2 0 0
2 1.2
49.6.12.4
Can MG Sparse
150 + 0
83.2 = 1.59
0
49.6:t2.4
Lift-PH
48 1 6
100
92.7
100+00
Lif-MMH-Better
+
94.6 +: 1.8
5.9
98.7 +
Lift-MH--kay--Berr
83 + 29
100 +
6.5
99.3 1 0.9
Lift-MH-OK.e
94 + 30
97.33 +
67.3 10.5
96 = 1.6
Lift-MH
104 + 44
100 + 0
56.7=
100 100+00 0
Lifi-MH- Lif-MMH-Worse-Bettee
109 49
100
75.3 25.6
100 1 0
Can-PH
116 14
1
3..7
95.3=0.9
Lift-MH- Worse-Okay
119 - 44
100 =
6..7
98.7 + 1.9
Can-MH-Better
143 + 29
52.66 + 0
7.4
83.3 + 2.5
Lift-MHI-Worse
145 + 40
94.6 1
13.3 9
100 + 0
Square-PH
151 20
53.3 + 2.4
5.3
78.7 + 1.9
Can-MH-Okay --kty-Better
162 44
75.33 3.3
90.7 + 3.7
Can-MH-Okay
47
52.66 + 0
2.
72 2.8
Square-MH-Betttr
185 46
13.3
0.7
58.7 2.5
Can-MH
209 + 114
95.33 0
22
86 4.3
Can-MH- Worse-Better
224 + 134
73.99 111
2.7
7.3
Manare-MH-Okayy
225 76
20.66 +
1.3
56.7 + 4.1
Can-MH--Worse-Oka
Worse-Okay
242 + 126
54 1.6
18.8
5.7
Square-MH-Oka
265 + 78
12.6 10
040
27.3 1 3.4
Square-MH
269 + 123
21.33
0.7
52.7 + 6.6
Square-MH- Worse-Better
271 + 140
6+0
1.3
46.7 5.7
Can-MH-Worse
304 + 148
51.33
4133
56.7 2.5
Square-MH- Worse-Okay
Mouare-MH-Woore-Okaaa
+ 128
4.6
2.7
2.5
Square-Mat MH---HH-W-W-ooo
357 + 150
11.30
0#0
22 4.3
Table 13: Success rates for different synthetic and human-generated datasets in ROBOMIMIC, Results presented here are an extension of Table 4 (from (ubsecttonn 4.3) and have been sorted as per the trajectory length of the task. DT outperformed both agents on synthetic data as seen by performance on MG tasks. BC seemed well suited to numerous tasks for which data was numan-generann (PH and MH). Bold numbers represent the berf-perf agent on the corresponding task.
23

---

Published as a conference paper at ICLR 2024
G.4 How ARE AGENTS AFFECTED WHEN SUBOPTIMAL DATA IS ADDED TO THE DATASET?
Figure 12 depicts the behavior agents when random data introduced according "Strategy the dense reward data regime the D4RL benchmark. As previously described, "Strategy I" involves rolling out uniformly random policy from sampled initial states to generate random data. Our observations indicate that both CQL and DT maintain stable performance, while BC exhibits instabilities, occasionally failing observed the HALF CHEETAH task.
Half Cheetah Medium Replay
Half Cheetah Medium Export
100
Nome Lerd futumo
-3
Co.
20
Satathat
of modom
Hopper Medium Replay
Hopper Medium expert
100
100
60
catather
of
Walker Medium Realoy
Walker Medium Export
23
110
Nomal aad INtAATE
DOL
D7
-8- CO.
DT
DC
of catathrr
of
Figure 12: Impact of adding suboptimal data on all agents in dense reward data regime on D4RL.
tasks using Strategy 1. The results presented here are an extension to Figure 2 (from subsection 4.4).
24

---

Published as a conference paper at ICLR 2024
Figure 13 displays the behavior of agents random data is incorporated according Strategy 2 in the dense reward data regime of the D4RL benchmark. "Strategy 2," we roll out a pre-trained agent for a certain number of steps, exccute a single uniformly random action, and then repeat the process. While Strategy produces random transitions primarily clustered around initial states, Strategy 2 generates random transitions across the entire manifold of states, spanning from initial states to high-reward goal states.
Half Medium
Half Cheetah
Medium
Norakedd hraums
60
13
50
6.
0:1
cata
of
Hopper Medium
Hopper
Medium
60
00
53
22
41
74
data
dadd mindom
Medium
Medium
Noraleedd Pecumn
+- C0
Figure 13: Impact of adding suboptimal data in dense reward data regime on D4RL tasks using Strategy 2. The results presented here are an extension to Figure 2 (from ssubsection 4.4). A noteworthy observation from this experiment is that the performance of CQL improved on HOPpeR medium-expert task with the addition of random data. This suggests that the random data might be aiding in the learning of more accurate Q-values for various states.
25

---

Published as a conference paper at ICLR 2024
Figure 14 illustrates the behavior of agents when random data introduced according to Strategy in the sparse reward data regime the D4RL benchmark. We observed that the performance of CQL drastically declined on the HOPPER-MEDIUS -MEDIUM- REMLLAY task, while its performance stayed the same on other tasks.
Half Cheetoh Meeium Replay
Halt Cheetah Medium Expert
cata that
ofidate tratin mindom
Hopper Medium Repley
Hopper Medium Expert
103
100
COL
80
23
cata that
of rindom
Walker Medium Roplay
Walker Medium expert
Nemolaed Rntumno
5
60
CO.
Sata that
of
Figure 14: Impact of adding suboptimal data in sparse reward data regime on D4RL tasks using Strategy 2. The results presented here are extension to Figure 2 (from subsection 4.4). The performance of CQL. sharply decreases in the HOPPER medium-replay task. Unlike its performance in the dense reward data regime, CQL does not exhibit the same level of resilience in the sparse reward setting and demonstrates considerably higher volatility.
26

---

Published as a conference paper at ICLR 2024
Figure 13
depicts the behavior of agents when random data is incorporated following Strategy in the sparse reward data regime (human- generated) of ROBOMIMIC benchmark. Notably, we observed drastically different performance trends with CQL. Its performance plummeted by 80% when the proportion of random data reached 51%. In contrast, its performance nearly doubled on the CAN MG task when the proportion of random data was increased to 30%.
(human-generated)
Succes Rate
Success
COL
20
random
that
Figure 15: Impact of adding suboptimal data in sparse reward data regime in ROBOMIMIC using Strategy 1. The results presented here extension Figure 2 (from subsection 4.4). CQL exhibited markedly different behaviors across these tasks. Its performance declined sharply on the LIFT PH task. In contrast, its performance significantly improved, nearly doubling, with the addition of random data on CAN PII task.
Figure 16 illustrates the behavior of agents when random data is introduced according to Strategy 2 in the sparse reward data regime (euunan-eeeeraed) of the ROBOMIMIC benchmark.
100
Rate State
Sundnes Rate
Reee
Su000s
60
20
21
61
72 768
24 30
99
45
15 data that is random
3 data that random
Figure 16: Impact of adding suboptimal data in sparse reward data regime in ROBOMIMIC using Strategy 2. The results presented here are an extension to Figuue Subsection 4.4), Interestingly. when following Strategy for random data generation, CQL maintains stable performance in the LIFT PH task. We observed a similar behavior to that seen with Strategy on the CAN PH task.
27

---

Published as a conference paper at ICLR 2024
Figure 17 and Figure 18 presents the behavior of agents random data added following Strategy 2 in sparse dense reward data regime (synthetic) respectively on ROBOMIMIC benchmark. all four scenarios, DT maintained its peak performance reasonably well, indicating its resilience in very noisy data settings. Conservative Q-Learning (CQL), however, showed signs of deterioration on both dense and sparse variants of LIFT tasks. Notably, CQL failed to perform on the CAN task. As mentioned earlier, BC didn't exhibit deterioration, possibly because the Mixed-Goal (MG) data already contains substantial amount of highly sub-optimal data.
Lift MG Dense
Can Dense
100
Succoss Rate
CDI
37
Sucooes Rete
COL
80
20
01234
1900 120
17
21
25
25
47
012 435
10
29
34
37
45
% data that la random
of data that is rancom
Figure 17: Impact of adding suboptimal data in dense reward data regime in ROBOMIMIC using Strategy 2. The results presented here are an extension to Figure 2 (from subsection 4.4).
Lift Sparse
MG Sparse.
100
Eluceess Rate
Surznes
68
02
CD)
D2
01234
17
456
random
random
Figure 18: Impact of adding suboptimal data in sparse reward data regime in ROBOMIMIC using Strategy 2. The results presented here are an extension to Figure 2 (from subsection 4.4
H ADDITIONAL EXPERIMENTAL DETAILS
We used the original author implementation as reference for our experiments wherever applicable. In settings where implementation required, we referenced the implementation which has been known to provide competitive/state-ollll results on D4RL. We provide details on compute and hyperparameters below.
Compute All experiments were run on an A100 GPU. Most experiments with DT typically require 10-15 hours of training. Experiments with CQL and BC require 5-10 hours of training. We used Pytorch 1.12 for our implementation.
Hyperparameters We mentioned all the hyperparameters used across various algorithms below. Our implementations are based on original author-provided implementations, without any modifications the hyperparameters. learn more about the selection hyperparameters, we recommend viewing the associated papers. Due to the stable training objective of DT and BC, both of these agents do not require substantial hyperparameter sweep experiments. DT was trained using Adam optimizer (Kingma and Ba 2014) with Multi-Step Learning Rate Scheduler. Each experiment
five times to account for seed variance.
28

---

Published as a conference paper at ICLR 2024
Hyperparameter RETURNS-TO-GO
Robomimic PH
Robomimic MH
Robomimic MG
120
Table 14: Returns-to-go values used with DT on D4RL and ROBOMIMIC.
Hyperparameter
VALUE
Reference implementation
https//tthhllooooo//oooooooooeee (MIT License)
Number of attention heads
Number of layers
6
Embedding dimension
128
Context Length (Breakout)
Context Length (Qbert)
30
Context Length (Seaquest)
30
Context Length (Pong)
50
Number of buffers Batch size
50
Learning Rate (LR)
6c 4
Number of Steps
500000
Return-to-go conditioning
90 Breakout (R: 1 max in dataset)
2500 Qhert (+z 5x max in dataset)
20 Pong (+s x max in dataset)
1450 Seaquest (+s 5x max in dataset) ReLU, encoder
Nonlinearity
GeLU, otherwise 32, 64, 64
Encoder channels
Encoder filter sizes
8 x 8, 4 x 4, 3 3
Encoder strides
4, 2,
Max epochs
Dropout
0.1
Learning rate
6 10-4
Adam betas
(0.9, 0.95) 1.0
Grad norm clip
Weight decay
Learning rate decay
Linear warmup and cosine decay (sce code for details)
Warmup tokens
512 20
Final tokens
2 500000 K
Table 15: Hyperparameters used with DT on ATARL
29

---

Published as a conference paper at ICLR 2024
Hyperparameter
VALUE
Reference implementation
(MIT License)
Number layers
Number of attention heads
Embedding dimension
128
Nonlincarity function
ReLUU
Batch size
512
Return-to-go conditioning (N/A to BC)
6000 HalfCheetah
3600 Hopper
5000 Walker
Reacher
6000 Humanoid
Dropout
0.1
Learning
10
Grad norm clip
Weight decay
10
Learning rate decay
Linear warmup for first 102 training steps
Context Length (N/A to BC)
Maximum Episode length
1000
Learning Rate
6c 4
Number of workers
16
Number evaluation episodes
100
Number iterations
10
Steps iteration
5000
Table 16: Hyperparameters used with DT and BC on D4RL.
Hyperparameter
VALUE
Reference implementation
(Apache 2.0 License)
Batch Size
2048
Steps per Iteration
1250
Number of Iterations
100
Discount
0.99
Alpha multiplier
Policy Learning Rate
3e-4
QF Learning Rate
3e-4
Soft target update rate
Se-3
BC steps
100k
Target update period
CQL actions
10
CQL importance sample
Truc
CQL lagrange
Falsc
CQL target action gap
-1
CQL temperature
CQL min weight
5
Table 17: Hyperparameters used with CQL on D4RL. We used identical hyperparameters as the reference implementation.
30

---

Published as a conference paper at ICLR 2024
Hyperparameter
VALUE
Reference implementation
https//ggthubb.cm///diiiiaaaaaae (MIT License)
BC hidden dim
I024
BC hatch size
1024
BC LR
1c-4
CQL
1c-4
CQL Critic Target tau
0.01
CQL Critic Lagrange
False
CQL target penalty
5
CQL Batch size
1024
Table 18: Hyperparameters used with CQL on EXORL. We used identical hyperparameters as the reference implementation.
Hyperparameter
VALUE
Reference implementation
http://ittbbbcmmmAAttSssiiuttutn (MIT License)
BC LR
le-4
BC Learning Decay Factor
BC encoder layer dimensions
[300, 400]
BC decoder layer dimensions
[300, 400]
CQL. discount
0.99
CQL Q-network LRR
1c-3
CQI. Policy I.R
3c-4
CQL Actor MLP dimensions
[300-400]
CQL Lagrange threshold
Table 19: Hyperparameters used with BC and CQL on ROBOMIMIC. We used identical hyperparame- ters as the reference implementation.
31

---

Published as a conference paper at ICLR 2024
I ABLATION STUDY TO DETERMINE THE IMPORTANCE OF ARCHITECTURAL COMPONENTS OF DT
In this section, we present the results of our ablation study, which was conducted to assess the significance of various architectural components of the DT. To isolate the impact of individual hyperparameters, we altered one time while keeping all others constant. Our findings indicate that the ATARI benchmark is better suited for examining scaling trends compared the D4RL benchmark. This likely due the bounded rewards present D4RL tasks, which may limit the ability to identify meaningful trends. We did not observe any significant patterns in the D4RL context. A key insight from this investigation is that the performance of DT, when averaged across Atari games. improved as increased the number of attention heads. However, we did not notice similar trend when scaling the numher of layers (Figure 4j. It is also important mention that the original DT study featured two distinct implementations of the architecture. The DT variant used for reporting results on ATARI benchmark had heads and layers, while the one employed for D4RL featured a single head and layers.
Number of Heads
BREAKOUT
QBERT
SEAQUEST
PONG
267.5 + 97.5
15.4 11.4
2.5 0.4
106 + 8.1 90.2 = 11.33
8
155.01 + 49.86
32.05 + 14.23
2.08
16
220.67 51.02
23.74 10.75
1.94
54.32 44
246.2 62.8
34.27
1.97 1 0.38
72.91 + 41.36
64
249.54 + 54.13
24.531111.33
2.11 + 0.56
69.76 + 34.49
Table 20:
Ablation study to determine the importance of the number of heads the performance of DT on the ATARI benchmark. The experiment was conducted with fully mixed Data (50 buffers) with 10 evaluation rollouts (number of layers=6).
Number of Layers
BREAKOUT
QBERT
SEAQUEST
PONG
229.58 + 49.86
24
2.08 0.44
90.2 11.33
8
160.8 + 44.1
32.34 14.07
1.9 0.36
73.35 + 40.4
12
226.25 1 79.47
1
3.75
94.23 1 22.44
16
218.55 1 58.92
3.401
0.28
55.95 1 48.55
159.02 1 49.35
40.68 = 13.99
1.3
95.75 1 8.36
32
239.31 + 71.21
8.12
2.49
86 = 29.72
64
223.71 + 59.65
31.63 = 12.56
0.61
73.6 + 39.68
Table 21: Ablation study to determine the importance of the number of layers on the performance of DT on the ATARI benchmark. The experiment was conducted with fully mixed Data (50 buffers) with 10 evaluation rollouts (number of heads=8).
Embedding dimension
BREAKOUT
QBERT
SEAQUEST
PONG
128
229.58 49.86
24 9.28
2.08 1 0.44
90.2 1 11.33
256
289 1 76.79
32.34 1 14.07
1.91 1 0.34
39.25 1 48.09
512
216.17 49.13
25.5919..1
2.08 1 0.59
1024
59.7 = 78.78
8.44
0.11 + 0.06
Table 22: Ablation study to determine the importance of embedding dimension on the performance of DT the ATARI benchmark. The experiment was conducted with fully mixed Data (50 buffers) with 10 evaluation rollouts (number of heads=8).
32

---

Published as a conference paper at ICLR 2024
Context Length
BREAKOUT
QBERT
SEAQUEST
PONG
229.58 49.86
24 =
+ 0.44
61.04 38.47
32
227.51 68.3
2.09
2.16 0.32
36
213.19 66.82
23.43 + 15.31
0.49
42
219.98 = 53.5
25.58 9.86
1.99 = 0.35
48
263.29 1 49.68
22.91 15.13
1.96 + 0.37
Table 23: Ablation study to determine the importance context length on the performance of DT on the ATARI benchmark. The experiment was conducted with fully mixed Data (50 buffers) with 10 evaluation rollouts (number of heads=8, num of layers=6).
Heads
BREAKOUT
QBERT
SEAQUEST
PONG
4
194.15 + 36.66
20.88 10.05
1.94 + 0..7
86.96 + 16.67
212.58 38.55
29.24 10.83
2.08 0.31
52.95 42.54
236.68 31.73
8.37
1.94
85.57
14.29
32
221.17 23.81
23.34 6.34
1.73 0.24
90.75 + 10.99
64
35.55 + 10.86
2.01 + 0.34
63.36 + 36.23
Table 24: Ablation study determine the importance of heads DT performance the ATARI benchmark. The experiment was conducted with fully mixed Data (50 buffers) with 100 evaluation rollouts (number of layers=6).
Layers
BREAKOUT
QBERT
SEAQUEST
PONG
212.58 +: 38.55
29.21 10.83
2.08 0.31
70.91
34.63
8
220.35 + 45
32.12 +
2.12 0.59
70.1 36.28
12
206.83 46.2
7.44
2.06 0.4
90.77+12.99
16
212.67 36.98
29.04
2.35
37.44
24
173.43 56.4
21.41
0.48
87.5
32
206.07 1 45.69
32.15
2.18
66.4 40.82
64
177.64 + 102.81
27.74 5.97
2.04 1
35.98 + 43.27
Table 25: Ablation study determine the importance of layers the performance of DT the ATARI benchmark. The experiment was conducted with fully mixed Data (50 buffers) with 100 evaluation rollouts (number of heads=8).
Heads
BREAKOUT
QBERT
SEAQUEST
PONG
53.85
30.93 4:
3.5 0.81
85.74 4: 25.64
4
155.01 43.58
4 11.12
4.37
4.37 1.25
85.01 16.68
8
188.38 + 67.43
33.91
3.53
0.73
65.9
16
176.58 = 69.9
2.77
3.58
75.35
32
225.85 68.21
6.49
0.73
33.7
64
246.62 = 120.95
37.25
17.25:2000 20.03
3.23
72.14
Table 26: Ablation study to determine the importance of heads the performance of DT on the ATARI benchmark. The experiment was conducted with the last buffers (out of 50) with 100 evaluation rollouts (number of layers=6).
33

---

Published as a conference paper at ICLR 2024
Layers
BREAKOUT
QBERT
SEAQUEST
PONG
151.71 1 37.17
21.49 8.54
3.65 1 1.12
79.2 26.77
4
186.49 1 59.76
13.01-16.33
3.73
73.55 1 37.41
6
188.38 1 67.43
13.94
3.5310.7 0.73
38.08
8
160.8 + 44.1
22.72
3.11
74.11
12
225.85 + 68.21
6..9
0.73
18.7-37.21
16
246.62 + 120.95
37.25 + 20.03
3.23 0.87
72.14 37.78
24
159.02 + 49.35
40.68 + 13.99
+ 1.3
70.4 + 35.52
Table Ablation study determine the importance of layers on the performance of DT on the ATARI benchmark. The experiment was conducted with the last 10 buffers (out of 50) with 100 evaluation rollouts (number of heads=8).
Heads
HALF CHEETAH
HOPPER
WALKER2D
42.74 (0.2)
77.08 (7)
74.64 (1.6)
42.62 (0.2)
(3.3
74.08 (0.8)
16
42.79 (0.2)
74.07 (3.4)
73.59 (1.3)
32
42.53 (0.2)
72.76
76.17 (1.4)
42.51 (0.1)
71.77
74.51 (2.4)
Table 28: Ablation study to determine the importance of heads on the performance of DT the D4RL.
medium tasks. The results are averaged across 100 evaluation rollouts (number of layers=6).
Heads
HALF CHEETAH
HOPPER
WALKER2D
37.87 (0.3)
89.06 (0.4)
67.93 (2.7)
38.07 (0.3)
(3.6)
72.7 (2.7)
16
38,46 (0.4)
88.88 (1.3)
(1)
32
37.88 (0.3)
88.15 (2.9)
72.65 (2.8)
64
38.17 (0.5)
90.29 (3.5)
71.99 (5)
Table 29: Ablation study determine the importance of heads on the performance of DT on the D4RI.
medium-replay tasks. The results are averaged across 100 evaluation rollouts (number of layers=6).
Heads
HALF CHEETAH
HOPPER
WALKER2D
4
91.52 (0.7)
110.58 (0.1)
108.57 (0.3)
8
90.55 (0.9)
110.18 (0.4)
108.69 (0.3)
16
90.61 (0.7)
110.79 (0.3)
108.28 (0)
32
92.06 (0.2)
110.84 (0.3)
108.16 (0)
64
91.35 (0.3)
110.87 (0.4)
108.45 (0.1)
Table 30: Ablation study determine the importance of heads on the performance of DT the D4RL medium-expert tasks. The results are averaged across 100 evaluation rollouts (number of layers=6).
Layers
HALF CHEETAH
HOPPER
WALKER2D
42.51 (0)
72.05 (1.8)
74.68 (0.6)
6
42.62 (0.2)
73.37 (4.4)
74.08 (0.8)
8
42.48 (0)
73.25
75.04 (1.9)
12
42.55 (0.2)
70.37 (2.6)
74.49 (0.89)
16
42.45 (0)
63.3 (1.7)
74,53 (1.9)
24
42,4 (0.1)
69.07 (4.35)
75.64 (1.4)
Table 31: Ablation study determine the importance of layers the performance of DT the D4RL medium tasks. The results are averaged across 100 evaluation rollouts (number of heads=8).
34

---

Published as a conference paper at ICLR 2024
Layers
HALF CHEETAH
HOPPER
WALKER2D
38.8 (0.7)
92.09 (3)
69.33 (4.2)
6
38.07 (0.3)
91.32
72.2 (2.7)
8
37.14 (1.3)
90.71 (0.64)
70.14
12
37.29 (1)
89.11 (2.1)
69.36 (2.3)
16
37.15 (0.4)
83.3)
71.4 (1.8)
24
37.46 (0.7)
83.17 (4.6)
73.66 (2.4)
32
38.39 (0.9)
87.94 (1.6)
69.43 (0.9)
64
38.09 (0.7)
81.95 (0.46)
68.01 (2.28)
Table 32: Ablation study to determine the importance of layers on the performance of DT on the D4RL medium-replay tasks. The results are averaged across 100 evaluation rollouts (number of heads=8).
Layers
HALF CHEETAH
HOPPER
WALKER2D
92.21 (0.2)
110.26 (0.6)
108.33 (0.6)
8
90.55 (0.9)
110.18 (0.4)
108.37 (0)
16
90.52 (0.8)
110.49 (0.2)
108.79 (0.1)
90.06 (0.6)
110.8 (0.2)
108.61 (0.4)
64
90.59 (0.6)
110.46 (0.4)
108.82 (0)
Table 33: Ablation study determine the importance of layers on the performance of DT on the D4RL medium-expert tasks. The results are averaged across 100 evaluation rollouts (number of
heads=8).
Context Length
HALF CHEETAH
HOPPER
WALKER2D
10
42.68 (0)
(3.3)
(1.31)
20
42.71 (0.1)
(5.4)
75.25 (0.9)
30
42.72 (0.1)
81.55 (10.2)
75.8 (0.2)
40
42.67 (0.2)
73.88 (0.9)
74.73 (1.8)
50
42.66 (0)
83.14
72.52 (0.7)
60
42.49 (0)
85.54 (6.9)
74.18 (0.3)
70
42.54 (0.1)
83.75 (0.03)
74.04 (2.7)
Table 34: Ablation study to determine the importance of context length on the performance of DT on the D4RL medium tasks. The results are averaged across 100 evaluation rollouts (number of layers=6. number of heads=8).
Context Length
HALF CHEETAH
HOPPER
WALKER2D
10
36.98 (0.2)
80.07 (1.1)
69.73 (2.2)
20
37.7 (0.1)
84.28 (2.6)
67.62 (2.6)
35.71 (0.1)
91.37 (3.9)
68.71 (1.4)
40
35.34 (0.6)
87.08 (1.5)
69.33 (2.4)
50
35.36 (0.5)
87.99 (2.3)
64.61 (5.2)
60
37.19 (0.5)
86.82 (3.5)
64.55 (1.2)
36.62 (1.4)
90.27 (0.8)
63.48 (2.5)
Table 35: Ablation study to determine the importance context length on the performance of DT on the D4RL. medium-replan tasks. The results averaged across 100 evaluation rollouts (number of layers=6, number of heads=8).
35

---

Published as a conference paper at ICLR 2024
Context Length
CHEETAH
WALKER2D
10
87.61 (2.64)
109.99 (0.48)
108.97 (0.92)
86.21 (3.75)
109.54 (0.72)
107.97 (0.82)
30
88.24 (2.17)
110.16 (0.48)
108.24 (0.26)
40
87.38 (1.15)
110.71 (0.4)
108.6 (0.04)
50
88.22 (0.96)
110.74 (0.41)
107.89 (0.56)
Table 36: Ablation study to determine the importance context length on the performance of DT on the D4RL medium-expert tasks. The results are averaged across 100 evaluation rollouts (number of layers=6, number of heads=8).
Context Length
HALF CHEETAH
HOPPER
WALKER2D
32
86.27 (0.3)
108.92 (0.5)
108.18 (0.3)
64
86.62 (2)
(0.7)
107.85 (0.5)
128
89.15 (1.2)
109.47 (0.3)
108.53 (0)
256
90.98 (0.9)
109.47 (1.1)
107.73 (0)
512
91.3 (0.3)
109.61 (0.7)
107.72 (0)
1024
91.38 (0.4)
109.04 (0.8)
108.42 (0.2)
Table 37: Ablation study to determine the importance of context length on the performance of DT on the D4RI. medium-replay tasks. The results are averaged across 100 evaluation rollouts (number of layers=6, number of heads=8).
36

---

Published as a conference paper at ICLR 2024
J DT ON EXORL
We additionally conduct smallcr-scale experiments in EXORI, which allows study the performance of DT on reward-free play data. Typical offline datasets are collected from a behavior policy that aims optimize some (unknown) reward. Contrary to this practice, the EXORL benchmark (Yarats et al. (022) was obtained from reward- free exploration. After the acquisition of an (s, a, s') dataset, reward function is chosen and used include rewards in the data. This same reward function is used during evaluation. We consider the WALKER WALK, WALKER RUN, WALKER STAND environments (APT). All scores are averaged over 10 evaluation episodes.
In the following section, we present the results obtained using DT on three distinct environments from the EXORL framework. Returns-to-ge in the tables presented below represent returns-to-valus provided DT at the time of inference. Upon comparing the metries in the EXORL study, noticed that DT's performance falls short compared CQL, which may be attributed to the data being collected reward free setting. Although investigating the behavior of these agents in reward-free settings presents an avenue for future research, we propose the following hypothesis. Typically, the exploration of new states in reward-free environment is conducted through heuristies such curiosity (ICM) (Pathak et al. 2017) entropy maximization (APT) (Liu and Abbeel (02T). The reward functions defined by these heuristics differ from those used for relaheling data when training oftline agents. Consequently, bootstrapping-bar based methods might better equipped to learn mapping between the reward function determined by the heuristic and the one employed for data relabeling.
Returns-to-G
LR
300
400
500
600
800
6e-4
123.28
(1.9)
120.18 (9.3)
126.46 (2.1)
131.82 (7.4)
128.47 (4.1)
130.95 (9.1)
125.61 (7.4)
le-3
119.25 (1.2)
121.59 (5.1)
128.85 (5)
132.35 (6.5)
136.59 (10.7)
124.58 (7.7)
121.68 (5.8)
3e-3
123.68 (5.3)
123.43 (5.3)
128.23 (0.9)
122.54 (8.9)
127.24
116.33 (1.9)
122.34 (6.3)
5e-3
110.35 (7.3)
120.65 (8.1)
123,82 (11.4)
117.38 (6.7)
120.45 (2.8)
112.27 (11.5)
104.63 (8.1)
7e-3
120.6 (7.8)
125.98 (2.8)
117.76 (11)
123.76 (14.9)
121.7 (6.4)
110.95 (6.7)
120.18
9c-3
125.04 (2.5)
114.34 (5.62)
116.57 (5.8)
115.17 (1.6)
123.71 (15.8)
124.65 (6.8)
123.9 (5.2)
1-e2
126.93 (8.2)
125.18 (3.9)
130 (4.2)
118.92 (10.8)
107.87 (10.4)
126.28 (8.7)
119.57 (13)
Table 38: Performance of DT WALKER WALK from EXORL returns-to-go value is varied. The experiment was run with the following parameters: batch size=7200, warmup steps=10, 200k timestep data, 150 gradient updates, and context length: 20.
Returns-to to-Go
L.
200
300
400
500
600
700
800
6e-4
133.68
(10.9)
139.51 (5.1)
137.5 (12.3)
131.2 (3.8)
130.14 (6.5)
129.8 (1.4)
129.79 (3.9)
le-3
131.62 (8.8)
126.34 (5.6)
125.71 (4.4)
129.58 (9.2)
(11.44))1))
127.94 (8.4)
(2..6
3e-3
126.71 (3.3)
129.85 (1.6)
119.36 (3.7)
(2..56
128 (12.4)
125.86 (4.4)
118.13 (5)
5e-3
123.05 (4)
119.7 (6.5)
124.79 (7.4)
124.48 (8.3)
(12.3)
122.25 (8.1)
129.83 (4.2)
7e-3
108.4 (2.5)
116.85 (6.9)
120.8 (10.5)
112.59 (9.1)
116.12 (3.2)
113.55 (3.5)
117.96 (1.8)
9c-3
107.79 (7.4)
98.57 (18)
110.52
(9.6)
101.21 (20.4)
102,91 (20.1)
90.19 (21.9)
1-e2
112.69 (10.5)
100.42 (13.7)
114 (24.3)
118.13
105.55 (9.9)
108.74 (14.9)
126.21 (16.2)
Table 39: Performance of DT WALKER WALK from EXORL. The experiment was with the following parameters: batch size=7200, warmup steps=30, 200k timestep data, 150 gradient updates, and context length: 20.
37

---

Published as a conference paper at ICLR 2024
Returns-to-Go
LR
Warmup
200
300
400
500
600
700
800
6c-4
30
133.68 (10.9)
139.51 (5.1)
137.5 (12.3)
131.2 (3.8)
130.14 (6.5)
129.8 (1.4)
129.79 (3.9)
4c-4
30
136.16 (4.7)
131.23 (6.5)
131.67 (5.2)
121.06 (6.2)
121.95 (3.5)
129.54 (0.1)
137.26 (0.9)
le-4
30
117.7 (4.3)
124.08 (9.7)
114.02 (1.9)
118.9 (3.8)
119.09 (0.7)
123.83 (9.7)
120.86 (7.0)
6e-4
50
130.14 (11.2)
129.53 (8.2)
136.92 (7.4)
123.96 (0.4)
132.93 (1)
(5.51
136.91 (9.6)
6e-4
80
125.44 (1)
131.98 (2.3)
129.48 (7.8)
123.6 (6.8)
127.62 (2.9)
127.64 (6.7)
123.53 (3.1)
6e-4
124.18 (4.5)
128.26
122.23 (6.8)
127.9 (0.8)
125.78 (3.7)
131.39 (5.9)
131.83 (5.1)
Table 40: Performance of DT on WALKER WALK from EXORL. The experiment was run with the following parameters: batch size=7200, 200k timestep data, 150 gradient updates, and a context length of 20.
Returns-to-
LR
Warmup
200
300
400
500
600
700
800
6e-4
50
123.53 (13)
127.81 (5.3)
121.44 (6.7)
112.79 (5.6)
123.33 (2.7)
120.44 (1.9)
124.58 (7)
6e-4
117.02 (4.3)
137.5 (6.5)
126.24 (8.1)
124.64 (6.8)
125.43 (9.9)
115.1 (2.0)
125.67 (7.8)
7e-4
120,91 (9.1)
128.69 (3.4)
121.03 (4.4)
112.11 (6)
120.67 (4.5)
127.57 (3.8)
120.05 (5.9)
le-3
30
123.64 (1.3)
118.84 (11.7)
129.66 (7.5)
126.77 (5.5)
122.59 (7.6)
127.9 (5.6)
127.19 (12.3)
Table 41: Performance of DT on WALKER WALK from EXORL. The experiment was run with the following parameters: batch size=7200, 200k timestep data, 300 gradient updates, and a context length of 20.
Returns-to--G
LR
Warmup
200
300
400
500
600
700
800
6c-4
50
57.56 (0.8)
57.49 (1.4)
56.25 (1.3)
56.38 (1.1)
57.4 (1)
57.87 (1.6)
56,23 (2)
6e-4
100
56.99 (0.6)
58.64 (0.9)
$7.88 (1.3)
57.15 (1.3)
60.11 (2.6)
$7.25
(0.8)
Se-4
40
59.84
57.98 (2.4)
57.82 (1.4)
56.9 (2.2)
(2.2)
56.48 (2.4)
60.36
(2.1)
6e-4
200
56.47 (2.5)
59.69 (2.8)
56.6
59.36 (1.0)
(2.0)
59.18
58.36 (0.1)
7c-4
100
60.3 (0.1)
57.12 (0.7)
57.03 (2.0)
56.32 (0.7)
59.55 (1.2)
(1.3
58.6 (2.2)
lc-3
30
62.48 (3.3)
59.23 (1.8)
56.62
58.69 (1.7)
58.94
56.25 (1.5)
57.55 (1.7)
Table 42: Performance of DT on WALKER RUN from EXORL. The experiment was run with the following parameters: batch size=7200, 200k timestep data, 300 gradient updates, context length: 20.
Returssto-Go
LR
Warmup
200
300
400
500
600
700
6c-4
100
263.71 (6.8)
292.08 (6.3)
278.26 (14.7)
271.19 (3.8)
283.36 (11.1)
284.05 (15.2)
279.87 (8.8)
Sc-4
270.83 (1.7)
271.04 (14.6)
275.7 (5.5)
273.92 (13.9)
269.5 (0.7)
290.78 (14.7)
286.77 (6.8)
6c-4
200
266.37 (14.8)
271.16
278.27 (8.2)
291.59 (9.4)
287.37 (4.4)
274.2 (6.9)
272.6 (5.7)
Table 43: Performance DT' on WALKER STAND from EXORI.. The experiment was run with the following parameters: batch size=7200, 200k timestep data, 300 gradient updates, context length: 20.
38

---

Published as a conference paper at ICLR 2024
Returns to to-Go
IR
200
300
400
500
600
700
800
le-4
128.52 (3)
125.83 (3.2)
116.24 (2.5)
130.23 (12.7)
137.82 (3.7)
126.32 (3.4)
125 (5.5)
6e-4
114.56 (11.9)
107.41 (5.5)
127.49 (10.4)
115.08 (8.4)
99.1 (4.0)
117.1 (7.3)
97.73 (1.3)
9
106.24 (7.5)
102.84 (16.0)
102.74 (14.7)
95.55 (5.5)
110.13 (10.4)
105.28 (13.3)
100.03 (7.0)
le-3
92.71 (3.1)
104.86 (15.3)
99.35 (1.5)
96.8 (8.0)
86.95 (10.0)
98.01 (14.7)
92.7 (1.6)
4c-3
45.65
42.86
39.6
41.93
39.71
39.88
43.05
8c-3
56.3
39.3
44.63
48.93
39.35
44.12
52.88
Table 44: Performance of DT on WALKER WALK from EXORI. The experiment was run with the
following parameters: batch size=7200,, warmup steps= 100, 200k timestep data, 900 gradient updates.
Returns-to-Go
LR
200
300
400
500
600
700
800
le-4
113.53 (2.5)
123.63 (2.7)
115.69 (2.0)
116.66 (1.7)
114.54 (1.1)
115.48 (2.1)
122.86 (6.4)
6e-4
103.02 (5.5)
102.17 (1.3)
94.05 (0.6)
98.45 (0.0)
95.22 (1.5)
101.86 (8.9)
114.71 (3.8)
9c-4
92.71 (8.7)
92.18 (0.8)
102.49 (5.0)
88.61 (11.8)
92.95 (3.2)
94.97 (2.2)
97.31 (3.6)
le-3
84.65
93
97.18
93.57
95.27
92.04
88.7
4e-3
57.96 (1.0)
66.73 (2.9)
61.93 (14.4)
46.71 (3.4)
58.77 (9.0)
51.75 (0.2)
47.44 (5.1)
Se-3
39.59 (2.4)
(4.66
44.71 (12.2)
41.27 (2.9)
(5.8)
48.63 (4.7)
45.2 (7.2)
Table 45; Performance of DT on WALKER WALK from EXORL. The experiment was run with the following parameters: batch size=72000, 200k timestep data, warmup steps=100, 900 gradient updates, context length=10.
Returns-to-Go
LR
200
300
400
500
600
700
800
lc-4
113.53 (2.5)
123.63 (2.7)
115.69 (2.0)
116.66 (1.7)
114.54 (1.1)
115.48 (2.1)
122.86 (6.4)
6c-4
103.02 (5.5)
102.17 (1.3)
94.05 (0.6)
98.45 (0.0)
95.22 (1.5)
101.86 (8.9)
114.71 (3.8)
9c-4
92.71 (8.7)
92.18 (0.8)
102.49 (5.0)
88.61 (11.8)
92.95 (3.2)
94.97 (2.2)
97.31 (3.6)
le-3
84.65
97.18
93.57
95.27
92.04
88.7
4e-3
57.96 (1.0)
66.73 (2.9)
61.93 (14.4)
46.71 (3.4)
58.77 (9.0)
51.75 (0.2)
47.44 (5.1)
8e-3
39.59 (2.4)
38.9 (4.66
44.71 (12.2)
41.27 (2.9)
40.34 (5.8)
48.63 (4.7)
45.2 (7.2)
Table 46: Performance of DT on WALKER WALK from EXORL. The experiment was run with the following parameters: batch size=72000, 200k timestep data, warmup steps=100, 900 gradient updates, context 1ength=10.
Returns-to-Go
LR
200
300
400
500
600
700
800
le-4
116.61
(7.8)
111.97 (6.3)
114.26 (4.4)
116.88 (4.9)
123.97 (3.4)
111.21 (3.7)
108.62 (2.5)
6e-4
112.06 (6.3)
113.17 (2.5)
104.78 (3.3)
112.57 (6.7)
107.5 (3.4)
112.69 (4.6)
106.85 (2.8)
9
108.42 (5.7)
102.44 (2.3)
99.46
105.42 (1.7)
106.62 (9.3)
19.22
102.55 (8.4)
le-3
103.74 (6.5)
106.42 (1.8)
98.7 (6.2)
103.34 (4.2)
116.92 (3.4)
107.22 (7.0)
105.08 (11.6)
4o-3
87.2 (6.4)
83.95 (4.2)
89.53 (8.1)
77.35 (12.4)
(2.6)
89.65 (8)
93.22 (7.8)
8e-3
57.73 (10.6)
59.11 (9.3)
56.23 (4.2)
58.82 (12.9)
52.2 (3.6)
62.79 (2.4)
54,29 (9.0)
Table 47: Performance of DT on WALKER WALK (Proto) from EXORL. The experiment was run with the following parameters: batch size=7200. 200k timestep data, 900 gradient updates, context length=10.
39

---

Published as a conference paper at ICLR 2024
Returns-to-Go
LR
200
300
400
500
600
700
800
le-4
116.61 (7.8)
111.97 (6.3)
114.26 (4.4)
116.88 (4.9)
123.97 (3.4)
111.21 (3.7)
108.62 (2.5)
6e-4
112.06 (6.36)
113.17 (2.5)
104.78 (3.3)
112.57 (6.7)
107.5 (3.4)
112.69 (4.6)
106.85 (2.8)
9c-4
108.42 (5.7)
102.44 (2.3)
(8.16
105.42 (1.7)
106.62 (9.3)
(6.2)
102.55 (8.4)
1e-3
103.74 (6.5)
106.42 (1.8)
98.7
103.34 (4.2)
116.92 (3.4)
107.22 (7.0)
105.08 (11.6)
4e-3
87.2 (6.4)
83.95 (4.2)
89.53 (8.1)
77.35 (12.4)
90.34 (2.6)
89.65 (8.0)
93,22 (7.8)
8e-3
57.73 (10.6)
59.11 (9.3)
56.23 (4.2)
58.82 (12.9)
52.2 (3.6)
62.79 (2.4)
54.29 (9.0)
Table 48: Performance of DT on WALKER WALK (Proto) from EXORL. The experiment was run with the following parameters: batch size=7200, 200k timestep data, 900 gradient updates, context length=10.
40


Reviews:
[{"review_id": "ALDa0umpWr", "reviewer": "Reviewer_pkaE", "summary": "This paper contributes to the field of offline Reinforcement Learning (RL) by conducting a comprehensive research study comparing three popular algorithms: CQL, BC, and DT. It investigates their performance across benchmark datasets like D4RL and ROBOMIMIC, analyzing their behavior under varying data quality, task complexity, and stochasticity.\n\nThe key findings reveal that DT requires more data than CQL to achieve competitive performance, but exhibits higher robustness to suboptimal data. Notably, DT outperforms both CQL and BC in sparse reward and low-quality data settings. Interestingly, DT and BC demonstrate superior performance for tasks with longer horizons or data collected from human demonstrations. Additionally, CQL shines in situations characterized by both high stochasticity and lower data quality.\n\nBeyond comparative analysis, the paper delves deeper into DT, exploring optimal architectural choices and scaling trends on ATARI and D4RL datasets. Notably, it demonstrates that increasing the data volume by fivefold results in a 2.5x average score improvement for DT on ATARI.\n\nOverall, this research offers valuable insights for selecting the most appropriate offline RL algorithm based on the specific characteristics of the task and data conditions.", "questions": "1. Please offer theoretical insights in each of the empirical findings where DT is more robust.\n2. Please perform experiments in more complex tasks such as Androit, Antmaze and vision-based tasks, e.g. vision-based drawer manipulation tasks in [1] and Atari games.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This paper conducts a thorough empirical analysis on offline RL algorithms, focusing on DT and the comparison to other offline RL algorithms, in particular, CQL, and BC. The paper presents a number of valuable practical insights, which offer great value to the RL community regarding selecting offline RL algorithms in different scenarios.\n\nThe paper is well-organized, especially listing the main practical insights at the end of each section, which makes readers easy to follow and understand.", "weaknesses": "1. While the paper offers great empirical analysis, it would be still great to provide some technical/theoretical justifications regarding each empirical result shown in the paper, similar to [1], which will further strengthen the claim in the paper.\n\n2. The empirical experiments are done in Robomimic and D4RL, but the D4RL tasks are only mujoco locomotion tasks. The authors should include experiments on more complex tasks such as Androit, Antmaze and vision-based tasks, e.g. vision-based drawer manipulation tasks in [1] and Atari games. \n\n[1] Kumar, Aviral, Joey Hong, Anikait Singh, and Sergey Levine. \"Should i run offline reinforcement learning or behavioral cloning?.\" In International Conference on Learning Representations. 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CpDrmWKmK2", "reviewer": "Reviewer_2vu5", "summary": "The paper empirically investigates the performance of three representative offline RL algorithms – Conservative Q-learning (CQL), Decision Transformer (DT), and Behavior Cloning (BC) – as several dimensions of the dataset, task, or model are varied, with the aim of determining which algorithm is most appropriate, depending on the setting. The authors provide concrete conclusions and recommendations based on their findings.", "questions": "* Am I misunderstanding something in my complaints about the D4RL sparsification or the stochasticity? Please correct me if so.\n* What is a “high-quality continuous action space”? (near top of pg. 9) Does this just refer to the degree of determinism?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "* The experiments investigate a number of significant practical considerations in offline RL, such as reward structure, inclusion of suboptimal data, and task complexity.\n* The authors provide concrete conclusions and recommendations based on their findings. A practitioner trying to decide which algorithm to use for their task/dataset would likely benefit from reading this paper.\n* The paper is clear, readable, and presented well.\n* The D4RL-style humanoid dataset may be of independent interest to researchers, and I hope that the authors will release it.", "weaknesses": "* I am skeptical of the sparsified version of D4RL because it violates the Markov property of the reward function: the reward now depends on the whole trajectory rather than just the current state/action. And two trajectories may end at the same state but have significantly different total rewards, causing disagreement. I can see why this transformation would break CQL but not DT, and indeed CQL deterioriates much more on sparse D4RL than sparse Robomimic. So I think the results here do not convincingly show that CQL is worse than DT for sparse environments in general; on D4RL, this artificial transformation violates an assumption underlying the CQL algorithm, and on Robomimic, CQL was already worse than DT on Robomimic even when given dense rewards.\n* The experiment on stochasticity does not seem to perfectly answer the question “how good are these algorithms in stochastic environments?” because the data you are training on was collected with deterministic dynamics. So really what you are testing is “how well does a policy trained by these algorithms in a deterministic MDP transfer to a similar stochastic MDP?”. This may also be an interesting question to study, but I think that if you intend to answer the first question, the experiment is not entirely appropriate, and it would be best to repeat it with the training data also collected in a stochastic MDP.\n* Some of the conclusions regarding BC are obvious (e.g. BC suffers more than offline RL when there is suboptimal/noisy data) or already widely known (e.g. BC works better than offline RL on human-generated demonstrations). It could provide more insight to include a hybrid method such as TD3-BC and show how the degree of regularization affects the behavior.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UDty89GC3E", "reviewer": "Reviewer_q2Sb", "summary": "The paper investigates three offline RL algorithms DT, CQL and BC under different settings, to see the effective use case for each algorithm. The experiments involve the settings with sparse-dense rewards, different data qualities, various trajectory lengths, random data injection, different complexity of tasks, stochastic environments and scaling with data and model parameters. The experiments are quite thorough with observed facts summarized, as well as conjectural reasons. It is an interesting paper providing some insights for the offline RL community.", "questions": "In Fig. 1(a), it shows when the dataset contains more sub-optimal samples, CQL degrades its performance more than DT, but when it uses 100% data the performance increases back to as good as DT. Is there any explanation for this?\n\nIt is interesting to see that DT is more reliable than CQL when adding random data (as Sec. 4.4), while less robust when injecting stochasticity in the environment (as Sec. 4.6). What’s the potential reason for DT to perform less robustly in the latter case?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well-written with small formatting issues. The experiments are quite thorough.\n\nIn comparing DT, CQL and BC, the paper involves at least seven different aspects. The observed facts are quite informative. Apart from summarizing the observed facts, the authors also provide reasonable explanations for most of the results.", "weaknesses": "Although explanations are provided for most of the observed facts, most of them are speculative without further evidence. For example, in Sec. 4.1, ``a likely reason … mean CQL must propagate TD errors …’’. If these conjectures are dived deeper with further evidence and justification, it will make the paper more solid.\n\nAnother major critique is that the algorithms CQL and BC are no longer the SOTA results for these offline RL baselines, and more effective algorithms are proposed with improvement. Examples like TD3BC [1], percentage BC, Diffusion Q-learning [2] for BC type methods,  and implicit Q-learning, implicit diffusion Q-learning [3], for Q-learning type methods, etc. Comparing DT with CQL, BC is a good starting point, but it could also be more interesting to see those more advanced algorithms in these comparisons. For example, it is known that BC does not perform well as long as the dataset contains not only high-return samples, but how does percentage BC perform in settings other than Sec. 4.2? Would it be a robust and reliable method as well in the offline setting?\n\nMinor issues: \n\nPosition of Tab. 2 is problematic.\n\nReference:\n\n[1] Fujimoto, Scott, and Shixiang Shane Gu. \"A minimalist approach to offline reinforcement learning.\" Advances in neural information processing systems 34 (2021): 20132-20145.\n\n[2] Wang, Zhendong, Jonathan J. Hunt, and Mingyuan Zhou. \"Diffusion policies as an expressive policy class for offline reinforcement learning.\" arXiv preprint arXiv:2208.06193 (2022).\n\n[3] Hansen-Estruch, Philippe, et al. \"Idql: Implicit q-learning as an actor-critic method with diffusion policies.\" arXiv preprint arXiv:2304.10573 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CPUnm3Ytcy", "reviewer": "Reviewer_wytN", "summary": "The authors compare three approaches to Offline-RL: CQL (representing Q-learning approaches), BC (IL approaches), and Decision Transformer (Sequence Modeling approaches) under a range of environments and data regimes (e.g. varying the amount of data, quality of data, or whether it was generated by a human or agent).\nBased on the results, they distill concise, actionable advice for practitioners.", "questions": "Please let me know if I mischaracterized or overlooked some of your contributions.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "I think this paper is a useful read for every practitioner in Offline-RL as it concisely evaluates and tests a range of relevant environmental factors and how those should influence the choice of algorithm. \nFairly comparing and evaluating different algorithms against each other is a seemingly simple, but actually not that straightforward thing to do as a lot of choices have to be made well: For example, which algorithms to compare, which environments to use and how to distill the results into concise statements that readers will actually remember.\nAnd I think the authors did a great job here in this regard. \nOf course, there are various additional things that they could have looked at (e.g. continuous vs. discrete actions) or a more modern approach than BC - but I do understand that lines have to be drawn somewhere.", "weaknesses": "I think the paper has no important weaknesses w.r.t what it set out to do, namely comparing important algorithmic approaches on a range of relevant environments.\nIn terms of publication, one could argue that there is limited novelty - not just in terms of algorithmic novelty, but also insights generated: None of the insights were very surprising and some were quite obvious (e.g. CQL performs worse when the back-propagation of the reward signal is impeded through sub-optimal or longer trajectories). \n\nI do believe this is a valid concern, but nevertheless regard the paper as very useful for the community, hence my initial rating of 6 (with a tendency towards 8).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)