Skill Decision Transformer
Authors: Shyam Sudhakaran
Sebastian Risi
Venue: Submitted to ICLR 2023 (2023)
Keywords: Transformer
Offline Reinforcement Learning

Abstract: Recent work has shown that Large Language Models (LLMs) can be incredibly effective for offline reinforcement learning (RL) by representing the traditional RL problem as a sequence modelling problem. However many of these methods only optimize for high returns, and may not extract much information from a diverse dataset of trajectories. Generalized Decision Transformers (GDTs)  have shown that by utilizing future trajectory information, in the form of information statistics, can help extract more information from offline trajectory data. Building upon this, we propose Skill Decision Transformer (Skill DT). Skill DT draws inspiration from hindsight relabelling and skill discovery methods to discover a diverse set of \emph{primitive behaviors}, or skills. We show that Skill DT can not only perform offline state-marginal matching (SMM), but can discovery descriptive behaviors that can be easily sampled. Furthermore, we show that through purely reward-free optimization, Skill DT is still competitive with supervised offline RL approaches on the D4RL benchmark.

Under review as a conference paper at ICLR 2023
SKILL DECISION TRANSFORMER
Anonymous authors
Paper under double-blind review
ABSTRACT
Recent work has shown that Large Language Models (LLMs) can be incredibly effective for offline reinforcement learning (RL) by representing the traditional RL problem sequence modelling problem (Chen et al., 2021; Janner et al., 2021). However many of these methods only optimize for high returns, and may not extract much information from a diverse dataset of trajectories. Generalized Decision Transformers (GDTs) (Furuta et al., 2021) have shown that by utilizing future trajectory information, in the form of information statistics, can help extract more information from offline trajectory data. Building upon this, we propose Skill Decision Transformer (Skill DT). Skill DT draws inspiration from hindsight relabelling (Andrychowicz et al., 2017) and skill discovery methods to discover a diverse set of primitive behaviors, or skills. We show that Skill DT can not only perform offline state-marginal matching (SMM), but can discovery descriptive be- haviors that can be easily sampled. Furthermore, we show that through purely reward-fiee free optimization, Skill DT is still competitive with supervised offline RL approaches on the D4RL benchmark.
INTRODUCTION
Reinforcement Learning (RL) has been incredibly effective in variety of online scenarios such as games and continuous control environments (Li, 2017). However, they generally suffer from sample inefficiency, where millions of interactions with an environment are required. In addition, efficient exploration is needed avoid local minimas (Pathak al., 2017; Campos et al., 2020). Because of these limitations, there is interest in methods that can learn diverse and useful primitives without supervision, enabling better exploration and re-usability learned skills (Eysenbach al., 2018; Strouse et al., 2021; Campos et al., 2020). However, these online skill discovery methods still require interactions with an environment, where access may be limited.
This requirement has sparked interest in Offline RL, where dataset of trajectories is provided. Some of these datasets (Fu et al., 2020) are composed of large and diverse trajectories of varying performance, making it non trivial to actually make proper use of these datasets; simply applying behavioral cloning (BC) leads to sub-optimal performance. Recently, approaches such as the Deci- sion Transformer (DT) (Chen et al., 2021) and the Trajectory Transformer (TT) (Janner al., 2021), I), utilize Transformer architectures (Vaswani et al., 2017) to achieve high performance on Offline RL. benchmarks. Furuta et al. (2021) showed that these methods are effectively doing hindsight infor- mation matching (HIM), where the policies are trained to estimate a trajectory that matches given target statisties of future information. The work also generalizes DT as information-statistic con- ditioned policy, Generalized Decision Transformer (GDT), This results in policies with different capabilities, such as supervised learning and State Marginal Matching (SMM) (Lee et al., 2019), just by simply varying different information statistics.
In the work presented here, we take inspiration from the previously mentioned skill discovery meth- ods and introduce Skill Decision Transformers (Skill DT), special case of GDT, where we wish to condition action predictions on skill embeddings and also future skill distributions. We show that Skill DT is not only able to discovery a number of discrete behaviors, but it is also able to ef- fectively match target trajectory distributions. Furthermore, we empirically show that through pure unsupervised skill discovery, Skill DT is actually able to discover high performing behaviors that match or achieve higher performance on D4RL benchmarks (Fu et al., 2020) compared to other state-of ofttte-rrttttt offline RI. approaches.

---

Under review as a conference paper at ICLR 2023
VQVAE
nin
Causal Transforner
Encuder
0
one (B.)
Figure 1: Skill Decision Transformer. States are encoded and clustered via VQ-VAE VAE codebook embeddings. A Causal Transformer, similar to the original DT architecture, takes in sequence of states, a latent skill distribution, represented as the normalized summed future counts of VQVAE encoding indices (details can be found in the "generate histtorram" function in A.5), and the corre- sponding skill encoding of the state at timestep t.
Our method predicts actions, conditioned by previous states, skills, and distributions of future skills. Empirically, we show that Skill DT can not only perform SMM on target trajectories, but can also match or achieve higher performance on D4RL. benchmarks (Fu et al., 2020) compared to other state-of-the-art offline RL approaches. Skill DT also has the added benefit of using discrete skills, which are useful for easily sampling diverse behaviors.
2
RELATED WORK
2.1 SKILL DISCOVERY
Many skill methods attempt to learn a latent skill conditioned policy <alss, z), where state ~ p(s) and skill Z. that maximizes mutual information between S and Z (Gregor et al., 2016; Sharma et al., 2019; Eysenbach et al., 2018). Another way of learning meaningful skills through variational inference, where is learned via reconstruction loss (Campos al., 2020). Explore, Discover and Learn (EDL.) (Campos et al., 2020) an approach, which discovers a discrete set of skills by encoding states via VQ-VAE: p(=[s), and reconstructing them: p(s[2). We use similar approach, but instead of reconstructing states, we utilize offline trajectories and optimize action reconstruction directly (p(a)s, =)). Since our policy is autoregressive, our skill encoding actually takes into account temporal information, leading to more descriptive skill embeddings. Offline Primitive Discovery for Accelerating Offline Reinforcement Learning (OPAL) (Ajay et al., 2020), also discovers offline skills temporally, but instead uses a continuous distribution of skills. These continuous skills are then sample by a hierarchical policy that optimized by task rewards. Because our approach is completely supervised, we wish to easily sample skills. To simplify this, we opt to use a discrete distribution of skills. This makes it trivial to query the highest performing behaviors, accomplished by just iterating through the discrete skills.
2.2 STATE MARGINAL MATCHING
State marginal matching (SMM) (Lee et al., 2019) involves finding policies that minimize the dis- tance between the marginal state distribution that the policy represents p" (s), and a target distribu- tion (s)) These objectives have an advantage over traditional RL objectives in that they do not require any rewards and are guided towards exploration (Campos et al., 2020). CDT has shown impressive SMM capabilities by utilizing binned target state distributions to condition actions in order to match the given target state distributions. However, using CDT in a real environment is difficult because target distributions must be provided, while Skill DT learns discrete skills that can be sampled casily. Also, CDT requires a low dimensional state space, while Skill DT in theory can work any type of input long as it can be encoded effectively into vector.
2

---

interested in SMM through skills, where we extract as Under review as a conference paper at ICLR 2023
3 PRELIMINARIES
In this work, we consider learning in environments modelled as Markov decision processes (MDPs), which can be described using varibles (S, A, R),, where S represents the state space, represents the action space, and P(s++1 1 ax) represents state transition dynamics of the environment.
3.1 GENERALIZED DECISION TRANSFORMER
The Decision Transformer (DT) (Chen et al., 2021) represents RI. as a sequence modelling problem and uses GPT architecture Alec Radford & Sutskever (2018) predict actions autoregressively, Specifically, DT takes in sequence of RTGs, states, and actions, where R. Ee Tt, and trajectory r (Ro, 80: 0o.-.. Rpr 81 ([r)) DT uses previous tokens to predict a with a deterministic policy which is optimized by a mean squared error loss between target and predicted actions. For evaluation, target return Rearget provided and DT attempts achieve the targeted return in the actual environment. Furuta et al. (2021) introduced a generalized version of DT, Generalized Decision Transformer (GDT). GDT provides a simple interface for representing variety different objectives, configurable by different information statistics (for consistency, we represent variations of GDT with nod();
Te St, Oes rts ... ST, OT, IT, 1 information statistics function
Generalized Decision Transformer (GDT):
gd (agl 8o, ap.... (T))) 8 ((-1,
Decision Transformer (DT):
do,..., Wit (ael (To), So, ao .... Tus (Ts), 8t-1, 01-1) where 1 (T)) = El 7 * discount factor
Categorical Decision Transformer (CDT):
Tode (azl (o (TO), 80. a0 odt (Tt), ; Sc: az), where Tht (T,) = histogram($s: 8T)
CDT is the most similar to Skill CDT captures future trajectory information using future state distributions, represented histograms for each state dimension, essentially binning and counting the bin ids for each state dimension. Skill DT instead utilizes learned skill embeddings to generate future skill distributions, represented histograms of full embeddings. In addition, Skill DT also makes use of the representation learnt by the skill embedding by also using it in tandem with the skill distributions.
SKILL DECISION TRANSFORMER
4.1 FORMULATION
Our Skill DT architecture is very similar the original Decision Transformer presented in Chen et al. (2021). While the classic DT uses summed future returns to condition trajectories, we instead make of learned skill embeddings and future skill distributions, represented as a histogram of skill embedding indices, similar to the way Categorical Decision Transformer (CDT) (Furuta et al., 2021) utilizes future state counts. One notable difference Skill DT has to the original Decision Transformer (Chen et al., 2021) and the GDT (Furuta et al., 2021) variant is that we omit actions in predictions. This is because much information from states.
Formally, Skill DT represents a policy:
T Zt- K, 2t- K, ... 1,
where is the context length, and are the learnable parameters the model. States are encoded skill embeddings é, which then quantized using learned codebook of embeddings
3

---

Under review as a conference paper at ICLR 2023
argmin,ll2 2n 122 The future skill distributions represented as the normalized histogram of summed future one hot encoded skill indices: Z. Ei one hot())) Connecting this to GDT, our policy can be viewed as:
Thes (ael (To), So. 1 La (Te): st), where 1 (T) = (histogram(2:.... 2T), 21).
(Tr)
4.1.1 HINDSIGHIT SKILL RE-LABELLING
Hindsight experience replay (HER) a method that has been effective in improving sample- efficiency of goal-oriented agents (Andrychowicz et al., 2017; Rauber et al., 2017). The core concept revolves around goal relabelling, where trajectory goals are replaced by achieved goals vs. inteded goals. This concept of re-labelling information has been utilized in a number of works (Ghosh et al., 2019; Zheng et al., 2022; Faccio et al., 2022), to iteratively learn an condition predictions on target statistics. Bi-Directional Decision Transformer (BIT) (Furuta et al., 2021), utilizes an anti- causal transformer to encode trajectory information, and passes it into a causal transformer action predictor. every training iteration, BDT re-labels trajectory information with the anti-causal trans- former. Similarly, Skill DT re-labels future skill distributions at every training iteration. Because the skill encoder is being updated consistently and skill representations change during training, the re-labelling of skill distributions is required to ensure stability in action predictions.
4.2 ARCHITECTURE
Many previous works have represented discrete skills as categorical vari- ables, sampled from categorical distribution prior (Strouse al., 2021; Eysenbach et al., 2018). VQ-VAEs (van den Oord et al., 2017) have shown impressive capabilities with discrete variational inference in the space of computer vision (Razavi al., 2019; Esser al., 2020), planning (Ozair et al., 2021), and online skill discovery (Campos et al., 2020). Because of this, we use VQ- VAE to quantize encoded states into set of continuous skill embeddings. We encode states into vectors 2, and quantize to nearest skill embeddings 2. To ensure stability, we minimize the loss:
VQ-VAE Skill Encoder.
VQLOSS(2, 2) MSE(2, 2)
(1)
Where 2 is the output of the MLP encoder and z is the nearest embedding in the VQ-VAE codebook.
Optimizing this loss minimizes the distance of skill encodings with their corresponding nearest VQ-VAE embeddings. This is analagous to clustering, where we are trying to minimize the distance between datapoints and their actual cluster centers. In practice, we optimize this loss using an exponential moving average, as detailed in Lai et al. (2022).
Causal Transformer. The Causal Transformer portion of Skill DT shares similar architecture to that of the original DT (Chen et al., 2021), utilizing a GPT (Alec Radford & Sutskever, 2018) model. It takes in input the last K states 8t-Ki, skill encodings 2t-Ktt+ and future skill embedding distributions Z.- K:t- As mentioned above, the future skill embedding distributions are calculated by generating a histogram of skill indices from timestep 1 T, and normalizing them so that they use learned linear layers to create token
add up to 1. For states and skill embedding distributions, embeddings. To capture temporal information, we also learn a timestep embedding that added to each token. Note that we don't 't tokenize our skill embeddings because want to ensure that don't lose important skill embedding information. It's important to note that even though we don't 't add timestep embeddings the skill embeddings, they still capture temporal behavior because the attention mechanism (Vaswani et al., 2017) of the causal transformer attends the embeddings to tem- porally conditioned states and skill embedding distributions. The VQ-VAE and Causal Transformer components are shown visually in Fig. 1.
4.3 TRAINING PROCEDURE
Training Skill DT is very similar to how other variants of GDT are trained (CDT, BDT, DT, etc.).
First, before every training iteration re-label skill distributions for every trajectory using our VQ- VAE encoder. Afterwards, we sample minibatches of sequence length K, where timesteps are sam- pled uniformly. Specifically, at every training iteration, we sample 7 (St, St. K, at 11....444 k)),,
4

---

Under review as a conference paper at ICLR 2023
Viol
Rt....
quarzze
7i.... 2
histgram
VQUE Excoder
Causal Transformer f.
Mass dey
Training
(11
Data
(as.... (11x)
MSE((A,, A.18) (0....)))xx)))))
VQLOSS So.... Soo S.....
Figure 2: Training procedure for Skill Decision Transformer. sub-trajectories of states of length & are sampled from the dataset and encoded into in latents and discretized. All three variables are passed into the causal transformer output actions. The VQVAE parameters and Causal Transformer parameters are backpropagated directly using an MSE loss and VQ-VAE regularization loss, shown
where is sampled uniformly for each trajectory in the batch. The sampled states, (8t, (81, K).. are encoded into skill embeddings using the VQVAE encoder. We then in the states, encoded skills, and skill distributions into the causal transformer to output actions. Like the original DT (Chen et al., 2021), also did not find it useful to predict states or skill distributions, but it could be useful for actively predicting skill distributions without having to actually provide states to encode. This topic hope to explore more in the future. The VQVAE encoder and causal transformer updated by backpropagation through an MSE loss between target actions and predicted actions and the VQVAE regularization loss referenced in 1. The simplified training procedure is shown in Al- gorithm 1.
Algorithm Offline Skill Discovery with Skill Decision Transformer
Initialize offline dataset D, Causal Transformer So. VQVAE Encoder eg, context length K, num updates per iteration J for training iterations 1...N do
Sample timesteps uniformly: t e ml..mnxlleen
Label dataset trajectories with skill distributions Zr, = = one hot(2+) for all t//l Sample batch of trajectory states: T = (St. (Stt 02, (k),kk)) for I...J do
& &
Encode skills
ZreetK quantize(2ree4) Quantize skills with VQVAE
0reesx fol ZTc 81, Zre X ZT
L9.s 1 x (at - āz)² VQLOSS@(2..... (Zran Ry &max)
backprop Log.s w.r.t 0
end for
end for
5 EXPERIMENTS
5.1 TASKS AND DATASETS
For evaluating the performance of Skill DT, we use tasks and datasets from the D4RL benchmark (Fu et al., 2020). D4RL. has been used as standard for evaluating many offline RI. methods (Kumar

---

Under review as a conference paper at ICLR 2023
al., 2020; Chen al., 2021; Kostrikov et al., 2021; Zheng et al., 2022). We evaluate methods on mujoco gym continuous control tasks, as well as two antmaze tasks. Images of some these environments can be seen in A.4
5.2 EVALUATING SUPERVISED RETURN
Can Skill DT achieve near or competitive performance, using only trajectory information, compared to supervised offline RL approaches?
Mujoco Mean Results
Env Name
DT
CQL
IQL
OPAL
KMeans
Skill
num
Dataset
DT
DT
skills
Max
(best
(Skill
Reward
skill)
DT)
walker2d-medie
74
79
78.3
76
82
10
92
malfcheetah-medine
43
44
47
10
45
ant-medium
94
101
100
106
10
107
hopper-meding
68
58
66
66
76
32
100
balfchectah-mediun replay
37
46
44
39
32
42
hopper-mediin-rrplayy
63
95
95
71
81
32
99
antmaze-umaze
59
75
88
73
100
32
100
autmaze-umazze-divesse
53
84
62
100
32
100
autmaze-mediin--iiees
0
61
71
13
64
100
untmaze-medium-ppe play
0
54
70
81
0
64
100
Average normalized returns on Gym and AntMaze tasks. We obtain some results as reported other works (Chen et al., 2021; Kumar et al., 2020; Kostrikov et al., 2021; Zheng al., 2022), and calculate Skill DT's returns as an average over seeds (for gym) and (for antmaze). Skill DT outperforms the baselines most tasks, but fails beat them on replay tasks and antmaze- medium. However, Skill DT can consistently solve the
Table 1:
umaze tasks.
Other offline skill discovery algorithms optimize hierarchical policies via supervised RI,, utilizing the learned primitives to maximize rewards of downstream tasks (Ajay et al., 2020). However, because we are interested in evaluating Skill DT without rewards, we have rely on learning enough skills such that high performing trajectories are represented. To evaluate this in practice, rollouts for each unique skill and take the maximum reward achieved. Detailed python sudocode for this is provided in A.5. For a close basle comparison to Skill DT, we use a K- Means augmented Decision Transformer (K-Means DT). K-Means DT differs from Skill DT in that instead of learning skill embeddings, instead we cluster states via K-Means and utilize the cluster centers the skill embeddings.
Surprisingly, through just pure unsupervised skill discovery, we are able to achieve competitive re- sults on Mujoco continuous control environments compared state-of-the-ar offline reinforcement learning algorithms (Kumar et al., 2020; Kostrikov et al., 2021; Chen et al., 2021). As we can see in our results in Table Skill DT outperforms other baselines on most of the tasks and DT all of the tasks. However, it performs worse than the other baselines on the cilmaze-mediune / -replay tasks. We hypothesize that Skill DT performs worse in these tasks because they contain multimodal and diverse behaviors. We think that with additional return context online play, Skill DT may be able to perform better in these environments, and we hope to explore this as future work. Skill DT. like the original Decision Transformer (Chen et al., 2021), also struggles on harder exploration problems like the antmaze- medium environments. Methods that perform well on these tasks usually utilize dynamic programming like Trajectory Transformer(Jamnn forcement learning like OPAL (Ajay et al., 2020). Even though Skill DT performs marginally better than DT, there is still lot of room for improvement in future work.
al.. 2021) or hierarchical rein-

---

Under review as a conference paper at ICLR 2023
DISCUSSION
6.1
ABLATION STUDY
What is the effect of the number of skills?
Ablation Results
Env Name
5 skills
10 skills
16 skills
32 skills
walker2d-mediug
80
82
balfcheetab-mmedum
44
44
44
44
ant-medium
100
106
106
106
hopper-mediug
65
70
76
hopper--ediumm-rphee
28
31
46
81
34
39
41
41
ant-umaze
80
100
100
100
diteere
66
100
100
100
Table 2: Best reward obtained from skills for a varying number of skills
Because Skill DT a completely unsupervised algorithm, evaluating supervised return requires evaluating every learnt skill and taking the one that achieves the maximum reward. This means we are relying entirely on Skill DT's ability to capture behaviors from high performing trajecto- ries from the offline dataset. We found that increasing the number of skills has less of effect performance, environments that have a large number of successful trajectories (-medium environ- ments). We hypothesize that these datasets have unimodal behaviors, and Skill DT does need many skills to capture descriptive information from the dataset. However, for multimodal datasets (such the rrepaay environments), Skill DT's performance improves with increasing number of skills. In general, using a larger number of skills can help performance, but the tradeoff is increased computation time because each skill needs to be evaluated in the environment. These results are reported in Table 2. Images of skills learnt can be seen in A.4
6.2 SMM WITH LEARNED SKILLS
How well can Skill DT reconstruct target trajectories and perform SMM in a zero shot man- ner?
Ideally, if an algorithm is effective at SMM, it should be able to reconstruct a target trajectory in an actual environment. That is, given target trajectory, the algorithm should be able to rollout a sim- ilar trajectory. The original DT can actually perform SMM well, on offline trajectories. However, when actually attempting this an actual environment, tory because it is unable to be conditioned on accurate future state trajectory information. Skill DT, similar CDT, is able to perform SMM in an actual environment because encodes future state information into skill embedding histograms. The practical process for this is fairly simple and de- tailed Algorithm 2. addition to state trajectories, learned skill distributions of the reconstructed trajectory and the target trajectory should also be close. We investigate this by looking into target trajectories from antmaze- umazzz-z2. antmaze antmaze-umaze-de umaze-diverse-v v2. For a more challenging example, handpicked trajectory from antmaze- umaze-di diverse that is unique in that it has a loop. Even though the trajectory is unique, Skill DT still able to roughly recreate it in a zero shot manner (Fig. 3), with rollouts also including loop in the trajectory. Additional results can be found in A.3.
unable reconstruct a target trajec-

---

Under review as a conference paper at ICLR 2023
012
0.0
10
15
(a) Target antmaze-umaze trajectory
(b) Target antmare--mmaze-diverss skill dist.
(c) reconstructed trajectories
134
419
018
444
1.9
002
(d) reconstructed skill dist.
Figure 3: From the antmaze-umaze-diveer Environment: The target trajecty is complex, with loop and with noisy movement. Reconstructed rollouts also contain a loop
6.3 SKILL DIVERSITY AND DESCRIPTIVENSS
How diverse and descriptive are the skills that Skill DT discovers?.
In order to evaluate Skill DT as a skill discovery method, we must show that behaviors are not only diverse but are descriptive, or more intuitively, distinguishable. We are able to visualize the diversity of learned behaviors by plotting each trajectory generated by skill on both antmaze-umant and ant environments, shown below. To visualize Skill DT's ability to describe states, we show the the projected skill embeddings and quantized skill embedding clusters (Fig. 5). For diversity metric, we utilize a Wasserstein Distance metric between skill distributions (normalized between [0, 1J), similar to the method proposed in (Furuta al., 2021). We report this metric in Table 3.
Wasserstein Distances
Env Name walker2d-men memmiium
min
max
avg
0.007
0.015
0.010
ant-medium halfihhettah-medilu--eppppp
0.007
0.012
0.008
0.009
0.036
0.027
0.011
0.033
0.019
ant-umaze-diverse
0.008
0.026
0.011
Table 3: Wasserstein distance metric (computed between each skill and all others). In tasks with unimodal behaviors (-medium), Skill DT discovers skills that result in trajectories that are more similar eachother than complex tasks (-replay and antmaze)
8

---

Under review as a conference paper at ICLR 2023
-0
Figure 5: t-SNE projections of Ant-v2 states. Left: States encoded into unquantized skill em- beddings and projected via TSNE. Right: States are encoded into quantized skill embeddings and projected via TSNE
Figure 4: Left: ant-umaze-v2. v2, Right: ant-umaze -inaze-divess--22 Trajectories made using 32 skills for both ant-umaze variants. The diverse variant contains lots of noisy trajectories, but Skill DT is still learn diverse and distinguishable skills
6.4 LIMITATIONS AND FUTURE WORK
Our approach is powerful because it is unsupervised, but it is also limited because of it. Because we do not have access rewards, we rely on pure offline diversity ensure that high performing tra- jectories are learned and encoded into skills that can be sampled. However, this is not very effective longer sequence prediction. Skill DT could bene-
for tasks that require dynamic programming fit from borrowing concepts from hierarchical skill discovery (Ajay et al., 2020) to re-use learned skills on downstream tasks by using an additional return-conditioned model. In addition, it would be interesting to explore online component to the training procedure, similar to the work in Zheng et al. (2022).
7 CONCLUSION
We proposed Skill DT, a variant of Generalized DT, to explore the capabilities of offline skill dis- covery with sequence modelling. We showed that a combination of LLMs, hindsight-relabelling can be incredibly useful for extracting information from diverse offline trajectories.. On standard offline RL environments, we showed that Skill DT is capable of learning a rich set of behaviors and perform zero-shot SMM through state-encoded skill embeddings. Skill DT can further be im- proved by adding an online component, hierarchical component that utilizes returns, and improved exploration.

---

Under review as a conference paper at ICLR 2023
ACKNOWLEDGEMENTS
Removed for blind review.
REPRODUCIBILITY STATEMENT
The code to to reproduce every experiment in this paper is available at [removed].
REFERENCES
Anurag Ajay, Aviral Kumar, Pulkit Agrawal, Sergey Levine, and Ofir Nachum. OPAL: offline primitive discovery for accelerating offline reinforcement learning. CoRR, abs/2010.13611, 2020. URL https://azxiv..oomm http://wwww.....aa////.aaaaaa./e h2tp://bb///200 1361...1111.
13611.
Tim Salimans Alec Radford, Karthik Narasimhan and Ila Sutskever. Improving language under- /
standing by generative pre-training. 2018. URL /ttp:::////uussest-222 amaaaaazaaaa..
assets coveeeeeseerccccooooeeeooooeeeee language wrr.gaage__odderrtttannnnapprprr otspre/eeooooo.a...g.uuuuusrrreo unsupervised/
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. CoRR, abs/1707.01195, 2017. URL /tttp://azee
org/ abs abs/1707 01495.
Víctor Campos, Alexander Trott, Caiming Xiong, Richard Socher, Xavier Giró-i-Nieto, and Jordi Torres. Explore, discover and learn: Unsupervised discovery of state-covering skills. CoRR, abs/2002.03647,, 2020. URL httppp////aaaa...rr////////00 wttpg///ab////aa/ae
03647.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. arXiv preprint urXiv:2106.03455 2021.
Patrick Esser, Robin Rombach, and Björn Ommer. Taming transformers for high-resolution image synthesis. CoRR, abs/2012.09841, 2020. URL /ttp::://axxvvooo org/ abs 2012. 09841.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without reward function. CoRR, abs/1802.06070, 2018. URL http:
//aaxii..rr/aba/1110... 06070.
Francesco Faccio, Vincent Herrmann, Aditya Ramesh, Louis Kirsch, and Jürgen Schmidhuber. Goal-conditioned generators of deep policies, 2022. URL 2207.01517
orriii.orrraaae abs/
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. 2020.
Hiroki Furuta, Yutaka Matsuo, and Shixiang Shane Gu. Generalized decision transformer for offline hindsight information matching. CoRR, abs/2111. 2021. URL hRt.httss:/axxxiv.oom
abs/ 2211.1111111 10364.
Dibya Ghosh, Abhishek Gupta, Justin Fu, Ashwin Reddy, Coline Devin, Benjamin Eysenbach, and Sergey Levine. Learning to reach goals without reinforcement learning. CoRR, abs/1912.06088, 2019. URL /attp://aate
org/ abs / /1912. 06088.
Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra. Variational intrinsic control. CoRR, /e//1611.07500 2016. URL http: / //wpvvaaavvvv//////////a org/ abs 07507.
Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. Advances in Neural Information Processing Systems, 2021.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine, Offline reinforcement learning with implicit q- learning. CoRR, abs/21 10.06169, 2021. URL WtRthttss:///axiit wwtpsxiwivorrr.oooooeooe wtgg/bb//// abs/2110 06169.
10

---

Under review as a conference paper at ICLR 2023
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline /bs/
reinforcement learning. CoRR, abs/2006.04779, 2020. URL
0006.04777
Chieh-Hsin Lai, Dongmian Zou, and Gilad Lerman. Robust vector quantized-variationaal autoen- coder. CoRR, abs/2202.01988 2022. URL https: htt///axxvv./// http://xxwvv.../.ee.ee/.. wttg////////////////////eeeeee// 2202. 01987.
Lisa Lee, Benjamin Eysenbach, Emilio Parisotto, Eric P. Xing, Sergey Levine, and Ruslan Salakhut- dinov. Efficient exploration via state marginal matching. CoRR, abs/1906.05274, 2019. URL
http:///rxtai org/ httgg/abs/1999 05274.
Yuxi Li. Deep reinforcement learning: An overview. arXiv preprint arXiv: 171.07777,, 2017.
Sherjil Ozair, Yazhe Li, Ali Razavi, loannis Antonoglou, Aäron van den Oord, and Oriol Vinyals. Vector quantized models for planning, CoRR, abs/2106.04615, 2021. URL https: //arivv. org/ arg/abs// 20....0000002 04615.
Deepak Pathak, Pulkit Agrawal, Alexei Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. CoRR, abs/1705.0536.3 2017. URL http:ht/////xii..rrgggggoe/
7705.0536..
Paulo E. Rauber, Filipe Mutz, and Jürgen Schmidhuber. Hindsight policy gradients. CoRR, abs/1711.06006, 2017. URL http: wawpvv.vvvv..////v http://ww..rg//b/////aa// 06006.
Ali Razavi, Aäron den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with VQ-VAE-2. CoRR, 1bs/1996.00446, 2019. URL
abs 11999.96.00000 00446.
Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman, Dynamics-aware unsupervised discovery of skills. CoRR, abs/1907.01657, 2019. URL http:://axxi..orttt abs/1907. 01657.
DJ Strouse, Kate Baumli, David Warde-Farley, Vlad Mnih, and Steven Hansen. Learning more skills through optimistic exploration. CoRR, 0tns/2107.14226, 2021. URL org/abs/ 2107. 14226.
Aäron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation leamn ing. CoRR, abs/1711.00933 2017. URL
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is you need. CORR, abs/1706.03762, 2017. URL http://tp/// /tt:://axiiv.. org/ abs/1706 03762.
Qinqing Zheng, Amy Zhang, and Aditya Grover. Online decision transformer, 2022. URL https: / ///aa/ii.rrrgabb////////e/ 0567.

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.1
DATASET STATISTICS
Dataset Stats
Env Name
state
act dim
num trajec- tories
avg
max
min
avg
max
min
dim
dataset reward
dataset
dataset reward
dataset
dataset
dataset
reward
d4rl
d4rl
d4rl
walker2d
17
1190
2852
4227
62
92
0
medium
halfcheetah
17
1000
4770
5309
-310
45
0
-medium ant
111
1202
3051
4187
-530
80
107
-5
medium
hopper
11
3
2186
1422
3222
316
44
100
10
-medium
halfcheetah
17
202
3093
4985
-638
27
42
-3
-medium
rrppaa
hopper-
11
1801
529
3193
-0.5
17
99
medium
-replay
antmaze-
29
2815
0.5
0
52
100
antmaze-
29
1011
0.012
0
1.2
100
0
umaze
-diverse
antmaze-
29
1137
0.125
0
12.5
100
0
medium
-diverse
antmaze-
29
1204
0.2
0
20.0
100
0
medium
-play
Table Dataset statistics
A.2
HYPERPARAMETERS
Common Hyper Parameters for Causal Transformer
hyperparameter
value
Number of layers
Number of attention heads
4
Embedding dimension
256
Context Length
20
Dropout
0.0
Batch Size
256
Updates between rollouts
50
Ir
le-4
gradient norm
0.25
Table 5
A.3 FEW SHOT TARGET TRAJECTORY RECONSTRUCTION
Skill DT, like other skill discovery methods, towards a particular goal. In this case, we are interested in recreating target trajectories as close as possible. The detailed algorithm for few shot skill reconstruction:
its state-to-skill encoder guide its actions
12

---

Under review as a conference paper at ICLR 2023
0200
0200
0175
0115
0130
0150
0125
0100
007
0050
0000
0005
0005
0100
0000
(a) Target Ant Skill Distribution
(b) Reconstructed Ant Skill Distribution
Figure 7: From the Ant-v2 Environment: Skill distributions of target trajectory and the recon- structed trajectory from rolling out in the environment. Because Ant-v2 is simpler environment, we can see that the reconstructed skill distributions are very close to the target.
(a) Target antmazz-umaze x-y trajectory
(b) Reconstructed antmaze-umaze x-y trajectory
0.200
0.20
0325
0.00
0.0
0.890
0.05
0.005
10
10
15 20
(c) Target antmaze-umaze skill dist.
(d) Reconstructed antmaze-umaa skill dist.
Figure 6: From the Antmaze-Umazt Envidiverseromment: One of the longer and highest performing trajectories in the dataset reconstructed by Skill DT. The trajectory is not quite identical to the target, but it follows similar path, where it hugs the edges of the maze just like the target.
Algorithm Reconstructing target trajectories
Initialize target trajectory т, skill encoder Eg, Skill DT transformer π
1. (su ..... ST
Jarget
target) + T,
> Extract states from target trajectory target
2. (Zo, ... 2T) (zindexo, ... zinderr) = Es taagge ...., ST
target 3. (Zo, ..., Zr) = bastogram(zindezz,, (zindezo, zindert). ) b Create skill distributions by creating a
encode states
histogram of skill encoding indices
4. ~ (a Zo. 20, so....,
> rollout real environment, see A.5 for details
13

---

Under review as a conference paper at ICLR 2023
A.4 TRAJECTORY SKILL VISUALIZATIONS
Figure 8: Skills learned in the ant-medium-22 environment. Each row corresponds to skill
Figure 9: Skills learned the halfchectah halichectah-meiiu-rrpllaea medium-replay-v2 environment. Each row corresponds to skill
Figure 10: Skills learned the hupper-medium-replayy environment. Each row corresponds to skill
14

---

Under review as a conference paper at ICLR 2023
Figure 11: Skills learned in the
v2 environment. Each row corresponds to skill
Figure 12: Skills learned in the ant-umaze-diverse-v2 environment. Each row corresponds to skill
15

---

Under review as a conference paper at ICLR 2023
A.5 EVALUATING SKILL DT's PERFORMANCE
Because Skill DT a purely unsupervised algorithm, evaluate the performance in an actual en- vironment, we perform rollouts for each skill and evaluate each to determine which is the best. To do this, we first populate buffer of skills (here we denote with 2) and skill histograms Z. When we rollout in the actual environment, the causal transformer utilizes this buffer to actually make predictions. However, updates the skill encodings that actually sees the environment each timestep. This is because even though the policy is completely conditioned to follow a single skill, it may end up reaching states that are classified under another. Python sudocode shown below:
def generate histogram (one hot__kiill skill ids):
trajectory_Lente histogram torch.tenser (copy (cne hot skill ids))
1en (one_hot,. htt_shiotlsll_____ idds
reverse order
for i in range (traaeecttry__eegghhh -1, -1]:
if != trajectory histogram[i] histcgrammil
(ittogrami+1]]
return histogram / Aistogram.ssu(/ # normalize zange [o, 11
def Solluatteskiill dt (skill dt, env, steps, cen) num_skills 3kill_dt.num__ee
rewards -
for skill_id in range (num (rtumsskilss)
skill_ids repeat (skill id, max steps)
create on hot skill ids
ex: one hot [[2, 5) fro, 1, 0, o, 01, 10, 1, 0, 0, 011 one_hot_skilllide one_hct (skill ids, num num_skills)
initialize_staa state reset
t 0
total_reward =
state buffer state_buffer butter zeros (max ste__stepa)
(max_ mma__steps)
while t < max_steps:
is vquae embedding
skill id is the index of the vgvae embedding in codebook
2, skill
encoee skill paacodd_ssilllsttaaaaee
= one hot one_hot (skilliid)
z = skill_dt. oot.e/ettttccitttoroooooooooooe histograms
$ create
state_buffer state
z [_buffer(]]
if t context_ien:
curr_states bufffrr [t:t:context_len)
b len]
curr 2 2it:t#context__.
actions = dt.lidd.ccausa transtormerrreeeeeeeeo (curr_ 2, curr
curr_states)
action actions[t]
else:
curr_states = state_buffer
curr_2 btt.fffelttcconett__eee
curr 2 2 2lt-conntet__en--1ttt
actions 3kll_cct.ccaallttaasffffmoe
(curr_3 curr_z,
curr_states)
action actions(-1)
state, reward, done = env, step (action)
tetal_reward reward
16

---

Under review as a conference paper at ICLR 2023
if done:
break
rewards. ( (total_reward) return max (rewards)
17


Reviews:
[{"review_id": "dgR_17OTfY0", "reviewer": "Reviewer_Tnhd", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces the Skill Discovery Decision Transformer, an extension of Decision Transformer [1] in which that does not use the return-to-go for policy conditioning. Instead, individual states are clustered and conditioning is performed with a \"skill-distribution-to-go\". The general motivation is a relation to (unsupervised) skill discovery algorithms which often do state-space clustering in order to learn goal-conditioned low-level policies (e.g., implicitly as DIAYN [2] or explicitly as EDL [3], OPAL [4]). However, in this paper, the policy is modeled with a transformer rather than an MLP.\n\n[1] http://arxiv.org/abs/2106.01345\n[2] http://arxiv.org/abs/1802.06070\n[3] http://arxiv.org/abs/2002.03647\n[4] https://openreview.net/pdf?id=V69LGwJ0lIN", "summary_of_the_review": "While the overall idea of the work at hand is interesting, I don't see it being fit for publication in the current form. The overall algorithm is not fully described, and several important analyses and ablations are missing. Pages 7 and 8 are populated with Figures that could well be placed in the Appendix, making room for more relevant additions.", "strength_and_weaknesses": "Strengths:\n- Decision Transformers are a recently introduced paradigm, and further developments on RL and sequence modeling are highly relevant to the community, as are skill discovery approaches.\n- The paper is generally easy to follow.\n- The proposed model can recover high-performing trajectories on benchmark offline RL datasets, although several questions remain open (see below)\n\nWeaknesses:\n- The method is not fully described. How is the VQVAE learned that s used to quantise states in Algorithm 1? Why is another encoder learned? Is the algorithm working with a random code-book? Why not use k-means instead then?\n- I think the evaluation here is somewhat unfair. On the one hand, your method can act in a zero-shot manner, but on the other hand you have to try every single skill and then pick the top-performing one. This should be made clear in the table, e.g., by relabeling the \"Skill DT\" column as \"Best Skill DT\". What's the average and worst performance of the skills? \n- The experiments leave several questions unanswered which would be very helpful to better understand the method:\n  - What do the resulting skills look like? Figure 3(a) only shows rollouts with a fixed skill, right? Given that this is a maze and that X/Y coordinates are part of the state, is there no correlation between skill partition and X/Y position at all? If it's not X/Y, what do the skills represent then? Or, put differently, what do a state at the start and end of a trajectory have in common so that they end up in the same cluster? It's hard for me to see how single-state clustering can be used to describe full trajectories consistently. \n  - Figure 5 and 6 are hard to interpret. The conclusion seems to be that a clustering is learned, but it's hard to judge the quality here. The VQVAE (for which it's unclear how it's learned) is described to perform better than k-Means. Why not try k-Means in the full pipeline instead to get a more grounded comparison?\n  - What is the quantitative benefit of providing codebook vectors rather than one-hot vectors for the active skill in a specific state?\n  - What is the relation between the number of skills and final performance or behaviors learned?\n- In Figure 4, results indicate that the policy is able to replay a reference trajectory. Would that not also work with the vanilla Decision Transformer, or another GDT variant that takes future statistics (e.g., simply the next 5 states) into account? You say \"[SDT] is able to follow the general path in a zero shot manner\", but is there any trajectory in the dataset that does *not* go from the start to the target?\n- No videos of acting policies provided.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "TqjoihO1CoP", "reviewer": "Reviewer_L7Nj", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes Skill Decision Transformer (Skill DT), a novel offline RL method with transformer-based sequential modeling, which can be interpreted as an extension of Generalized Decision Transformer, proposed by Furuta et al.\nSkill DT first encodes each state into a discrete latent variable $z_t$ with VQ-VAE codebook, converts it into a one-hot vector, and then makes a histogram over the future time steps. After that, causal transformer takes the sequence of such future-aggregated skill distribution, skill, and state as an input, and auto-regressively predicts the actions at each time step.\nThe experiments show that Skill DT achieves superior or comparable performance to other existing offline RL methods (DT, CQL, IQL) with several D4RL MuJoCo-locomotion, and antmaze datasets.", "summary_of_the_review": "Combining VQ-VAE with transformer-based offline RL (DT, GDT) might be an important direction to leverage diverse and unstructured behavioral datasets. However, this paper doesn't have enough experimental evaluations and has many unclear points that should be revised. Considering those aspects, I vote for rejection.", "strength_and_weaknesses": "### Strength\n- While the quantization of offline behavioral data for sequential modeling seems to be shown in GDT paper, the learned skill embedding with VQ-VAE may be a novel approach for transformer-based offline RL.\n- The experimental results presented in Table 1 show a certain amount of improvement among competitive offline RL methods (DT, CQL, IQL).\n\n### Weaknesses\n - The domain where Skill DT is effective seems limited. For instance, while Skill DT shows notable performance in antmaze-umaze or mujoco-medium settings that have unimodal behaviors, it doesn't show improvement in antmaze-medium or mujoco-medium-replay that have multimodal behaviors. Since skill discovery methods usually learn diverse and multimodal behaviors, these trends are not intuitive.\n\n- There is no skill-based baseline. GDT-variants (CDT/BDT), k-means clustering-conditioned DT, or some offline skill-based method (OPAL, LiSP [1]) can be relevant baselines.\n\n[1] https://arxiv.org/abs/2012.03548", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "MJHEPc0Tou", "reviewer": "Reviewer_2Y2V", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper extends Decision Transformers and Generalized Decision Transformers for skill discovery. They learn primitive skills based on unsupervised learning (i.e., without rewards information). Technically, they first use a VQ-VAE to encode discrete skills, then they utilize a causal transformer just like the original decision transformer. For training, they condition the encoded skill,  the distribution of skills, etc, to predict the actions. In experiments, they use D4RL benchmark, where they show the skill decision transformer could leverage the unsupervised-learned skills to improve the performance. ", "summary_of_the_review": "The paper combines VQ-VAE and decision transformers to tackle the important problem of skill discovery. However, the experiments could be improved to demonstrate the strength of such a neat combination. ", "strength_and_weaknesses": "Strength:\n+ The idea is clearly stated, and the implementation is relatively simple based on the well-established components such as VQ-VAE and Decision Transformers;\n+ The proposed method achieves improved performance across a large number of domains in D4RL benchmarks;\n\nWeaknesses\n- The proposed idea is rather incremental based on the decision transformers and generalized decision transformers, where I don't see many specific technical contributions other than combining VQ-VAE to have a discrete skill set;\n- The number of skills set in the proposed method seems rather arbitrary from Table 1. How should the users specify how many skills the proposed method should employ for better performance? \n- The illustrations of Figure 5 and Figure 6 do not seem to add much information for understanding the method;\n- I would expect an experiment, where sampling and conditioning on different skills the trajectory could generate different styles/behaviors;\n- The proposed method should be more effective in tasks requiring hierarchical RL in complex control domains, where low-level skills are discovered in an unsupervised fashion. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "I6VuT6EP6r", "reviewer": "Reviewer_tKoW", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a transformer-based architecture to discover diverse behaviors from the offline dataset.", "summary_of_the_review": "This paper proposed a transformer-based architecture to discover diverse behaviors from the offline dataset. The paper is somewhat incremental work and with unsatisfied experimental evaluation.", "strength_and_weaknesses": "Strength:\n1. This paper is well-writing.\n2. The discovery of diverse behaviors is an interesting problem.\n\nWeakness:\n1. The experiment results are not good enough. The proposed method is much worse than baselines in some tasks (such as antmaze-medium and antmaze-medium\u0002diverse).\n2. Lack of diverse behavior analysis. (1) The author should give some visualization of diverse behaviors their method finds on the D4RL dataset. (2) The author should give a diversity metric to measure the diversity of behaviors on different tasks.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject