Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposed a transformer-based architecture to discover diverse behaviors from the offline dataset.
strength_and_weaknesses: Strength:
1. This paper is well-writing.
2. The discovery of diverse behaviors is an interesting problem.

Weakness:
1. The experiment results are not good enough. The proposed method is much worse than baselines in some tasks (such as antmaze-medium and antmaze-mediumdiverse).
2. Lack of diverse behavior analysis. (1) The author should give some visualization of diverse behaviors their method finds on the D4RL dataset. (2) The author should give a diversity metric to measure the diversity of behaviors on different tasks.

clarity,_quality,_novelty_and_reproducibility: - Clarity: Good.
- Quality: Fair.
- Novelty: Poor. This work is just built on previous works, such as Decision Transformer and  Skill Discovery
- Reproducibility: Poor. The authors provide no code, no appendix, and no hyper-parameter description. 
summary_of_the_review: This paper proposed a transformer-based architecture to discover diverse behaviors from the offline dataset. The paper is somewhat incremental work and with unsatisfied experimental evaluation.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper extends Decision Transformers and Generalized Decision Transformers for skill discovery. They learn primitive skills based on unsupervised learning (i.e., without rewards information). Technically, they first use a VQ-VAE to encode discrete skills, then they utilize a causal transformer just like the original decision transformer. For training, they condition the encoded skill,  the distribution of skills, etc, to predict the actions. In experiments, they use D4RL benchmark, where they show the skill decision transformer could leverage the unsupervised-learned skills to improve the performance. 
strength_and_weaknesses: Strength:
+ The idea is clearly stated, and the implementation is relatively simple based on the well-established components such as VQ-VAE and Decision Transformers;
+ The proposed method achieves improved performance across a large number of domains in D4RL benchmarks;

Weaknesses
- The proposed idea is rather incremental based on the decision transformers and generalized decision transformers, where I don't see many specific technical contributions other than combining VQ-VAE to have a discrete skill set;
- The number of skills set in the proposed method seems rather arbitrary from Table 1. How should the users specify how many skills the proposed method should employ for better performance? 
- The illustrations of Figure 5 and Figure 6 do not seem to add much information for understanding the method;
- I would expect an experiment, where sampling and conditioning on different skills the trajectory could generate different styles/behaviors;
- The proposed method should be more effective in tasks requiring hierarchical RL in complex control domains, where low-level skills are discovered in an unsupervised fashion. 

clarity,_quality,_novelty_and_reproducibility: The paper is in good clarity, of decent quality, and relatively borderline novelty. The reproducibility of the paper should be good.
summary_of_the_review: The paper combines VQ-VAE and decision transformers to tackle the important problem of skill discovery. However, the experiments could be improved to demonstrate the strength of such a neat combination.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper introduces the Skill Discovery Decision Transformer, an extension of Decision Transformer [1] in which that does not use the return-to-go for policy conditioning. Instead, individual states are clustered and conditioning is performed with a "skill-distribution-to-go". The general motivation is a relation to (unsupervised) skill discovery algorithms which often do state-space clustering in order to learn goal-conditioned low-level policies (e.g., implicitly as DIAYN [2] or explicitly as EDL [3], OPAL [4]). However, in this paper, the policy is modeled with a transformer rather than an MLP.

[1] http://arxiv.org/abs/2106.01345
[2] http://arxiv.org/abs/1802.06070
[3] http://arxiv.org/abs/2002.03647
[4] https://openreview.net/pdf?id=V69LGwJ0lIN
strength_and_weaknesses: Strengths:
- Decision Transformers are a recently introduced paradigm, and further developments on RL and sequence modeling are highly relevant to the community, as are skill discovery approaches.
- The paper is generally easy to follow.
- The proposed model can recover high-performing trajectories on benchmark offline RL datasets, although several questions remain open (see below)

Weaknesses:
- The method is not fully described. How is the VQVAE learned that s used to quantise states in Algorithm 1? Why is another encoder learned? Is the algorithm working with a random code-book? Why not use k-means instead then?
- I think the evaluation here is somewhat unfair. On the one hand, your method can act in a zero-shot manner, but on the other hand you have to try every single skill and then pick the top-performing one. This should be made clear in the table, e.g., by relabeling the "Skill DT" column as "Best Skill DT". What's the average and worst performance of the skills? 
- The experiments leave several questions unanswered which would be very helpful to better understand the method:
  - What do the resulting skills look like? Figure 3(a) only shows rollouts with a fixed skill, right? Given that this is a maze and that X/Y coordinates are part of the state, is there no correlation between skill partition and X/Y position at all? If it's not X/Y, what do the skills represent then? Or, put differently, what do a state at the start and end of a trajectory have in common so that they end up in the same cluster? It's hard for me to see how single-state clustering can be used to describe full trajectories consistently. 
  - Figure 5 and 6 are hard to interpret. The conclusion seems to be that a clustering is learned, but it's hard to judge the quality here. The VQVAE (for which it's unclear how it's learned) is described to perform better than k-Means. Why not try k-Means in the full pipeline instead to get a more grounded comparison?
  - What is the quantitative benefit of providing codebook vectors rather than one-hot vectors for the active skill in a specific state?
  - What is the relation between the number of skills and final performance or behaviors learned?
- In Figure 4, results indicate that the policy is able to replay a reference trajectory. Would that not also work with the vanilla Decision Transformer, or another GDT variant that takes future statistics (e.g., simply the next 5 states) into account? You say "[SDT] is able to follow the general path in a zero shot manner", but is there any trajectory in the dataset that does *not* go from the start to the target?
- No videos of acting policies provided.
clarity,_quality,_novelty_and_reproducibility: Clarity: Overall the presentation is quite clear, however there is no description of the VQ-VAE training procedure.
A further (minor) question: the formulas for computing the skill distribution do not include normalization, but the diagrams in Figure 4 suggest that the resulting histogram is normalized?

Quality: Given the missing experiments mentioned above, I don't think the paper fully supports the proposed architecture.

Novelty: The building blocks for the architecture stem from prior work: Decision Transformers and state partitioning for skill discovery. The combination is novel but not sufficiently evaluated.

Reproducibility: Details regarding the VQ-VAE training procedure are missing and source code is not provided; as such, I don't think that work could be reproduced as is.
summary_of_the_review: While the overall idea of the work at hand is interesting, I don't see it being fit for publication in the current form. The overall algorithm is not fully described, and several important analyses and ablations are missing. Pages 7 and 8 are populated with Figures that could well be placed in the Appendix, making room for more relevant additions.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes Skill Decision Transformer (Skill DT), a novel offline RL method with transformer-based sequential modeling, which can be interpreted as an extension of Generalized Decision Transformer, proposed by Furuta et al.
Skill DT first encodes each state into a discrete latent variable $z_t$ with VQ-VAE codebook, converts it into a one-hot vector, and then makes a histogram over the future time steps. After that, causal transformer takes the sequence of such future-aggregated skill distribution, skill, and state as an input, and auto-regressively predicts the actions at each time step.
The experiments show that Skill DT achieves superior or comparable performance to other existing offline RL methods (DT, CQL, IQL) with several D4RL MuJoCo-locomotion, and antmaze datasets.
strength_and_weaknesses: ### Strength
- While the quantization of offline behavioral data for sequential modeling seems to be shown in GDT paper, the learned skill embedding with VQ-VAE may be a novel approach for transformer-based offline RL.
- The experimental results presented in Table 1 show a certain amount of improvement among competitive offline RL methods (DT, CQL, IQL).

### Weaknesses
 - The domain where Skill DT is effective seems limited. For instance, while Skill DT shows notable performance in antmaze-umaze or mujoco-medium settings that have unimodal behaviors, it doesn't show improvement in antmaze-medium or mujoco-medium-replay that have multimodal behaviors. Since skill discovery methods usually learn diverse and multimodal behaviors, these trends are not intuitive.

- There is no skill-based baseline. GDT-variants (CDT/BDT), k-means clustering-conditioned DT, or some offline skill-based method (OPAL, LiSP [1]) can be relevant baselines.

[1] https://arxiv.org/abs/2012.03548
clarity,_quality,_novelty_and_reproducibility: ### Quality
There are some points that should be improved in writing.
- The title in the paper should be fixed (Skill Discovery Decision Transformer --> Skill Decision Transformer).  
- In Section 3, the definition of $\tau_t$ is missing.
- The explanation of GDT in Section 3.1 seems to miss $a_t$.
- In Section 3.1, the information statistics of DT should be $\Sigma \gamma^t r_t$ rather than $\Sigma \gamma * r$ (\gamma is also not defined).
- Citation of "a GPT architecture ..." seems broken.
- In Section 5.1, "timeplapse" --> "timelapse".
- In Algorithm 1, $e_{\phi}(s_t), .. e_{\phi}(s_t)$ --> $e_{\phi}(s_t), .. e_{\phi}(s_{t+K})$.
- The GDT-like formalization of Skill DT in Section 4.1 might be wrong: $I = (histogram, z_t)$ --> $I = histogram$?
- The sentence in Section 7 has duplicate periods. 

### Clarity
There are some unclear points in the paper.
- Compared to DT or GDT, Skill DT doesn't have actions as inputs. Is there any justification/explanation?  
- How does skill DT prepare the future skill distribution in the evaluation time?
- The suffix of Figure 1 should reflect context length: i.e. $t - K, ..., t$ or $t, .., t+K$, instead of $t, ..., T$.
- Continuous skill-based methods often assume gaussian distribution as a skill distribution, which is easy to sample latent skills. It should be clarified how "easily sampling diverse behaviors" in Skill DT. Since Skill DT seems to sample the discrete skill variable at every timestep, it seems difficult to sample desired / consistent behaviors.
- Figure 3 said "Skill DT is still learn diverse and distinguishable skills", but I don't think those trajectories are distinguishable. They are very cluttered.
- The visualization in Figure 6 might not be so different from each other compared to Figure 5. Some quantitative metrics would be helpful.
- The number of skills in Figure 5 seems different between VQ-VAE and k-means. It should be aligned.
- The ablation of num_skills presented in Table 1 could be important. While the medium dataset has unimodal behaviors, it requires 10 skill variables, which doesn't seem intuitive.



### Originality
The quantization of offline behavioral data for sequential modeling seems to be shown in GDT paper. The originality of this paper is the replacement of the state-discretization / anti-causal transformer encoder with VQ-VAE encoder.


summary_of_the_review: Combining VQ-VAE with transformer-based offline RL (DT, GDT) might be an important direction to leverage diverse and unstructured behavioral datasets. However, this paper doesn't have enough experimental evaluations and has many unclear points that should be revised. Considering those aspects, I vote for rejection.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes an extension of Decision Transformers to condition on skills. The original model [Chen et al.] conditions on returns, whereas another extension Generalized DT conditions on arbitrary statistics. The current works extends this to conditioning on skills. While the reviewers appreciated the clarity of the work, there were uniform concerns among the reviewers on the limited novelty of the work (changing the conditioning to skills and use of VQ-VAE encoder) and the sparse details of the empirics. Following up one of the reviewer suggestions, I'd encourage the authors to investigate hierarchical RL environments for further investigation and analysis of the proposed approach.
justification_for_why_not_higher_score: Very low novelty given prior work.
Decision: Reject