Venue: NeurIPS 2022 Workshop Offline_RL

Reviews:

final_score: rating: 7: Good paper, accept
review_content: Summary

This paper proposed an offline skill-learning algorithm leveraging a world model. The method first pretrain a world model that learns latent space dynamics. Then they train the skill encoder and decoder which encodes/decodes between the skill space and the model’s latent space. The author also proposed a code resampling technique that can ensure the skill space (the codebook) is well covered. Finally, the skill learning is performed on the imagined trajectory generated by the world model, maximizing a mutual information-based intrinsic reward. In the experiment, they finetuned a high-level controller for solving downstream tasks and compared it with other unsupervised RL methods, where it outperformed the other existing methods.

Strength:
1. The paper is well written, where the 4 steps: learning world model, skill discovery, skill learning, and skill adaptation are clear.
2. The idea of learning skills in the codebook combined with the code resampling technique is compelling. 
3. The derivation for the calculation of the mutual-information objective is reasonable.
4. Overall, the idea of leveraging a world model to learn skills in an offline manner is interesting, and the proposed method has shown a promising approach to make it work.


Weakness:
1. The main issue is the lack of baseline comparison. Since this work’s main direction is to extract useful skills from the offline unlabeled datasets, they might include the comparison to the works listed below for example:
https://arxiv.org/abs/2010.11944
https://arxiv.org/abs/2010.13611

2. Since the baseline scores are mostly cited from the URLB paper, I believe the skill-based baseline like DIAYN is naively finetuned rather than training a high-level planner. This is unfair because the proposed method is training a high-level planner (they called meta-controller), while the baseline is not.
review_title: The method is interesting, but the baseline comparison is insufficient

final_score: rating: 7: Good paper, accept
review_content: Summary:

The paper proposes a new method to learn exploration strategies/skills from unlabeled offline data. The backbone of the model is a standard Dreamer latent variable model. Two additional policies are proposed: 
1) A skills-based policy, which is conditioned on a skill variable. The policy is trained using the standard mutual information justification, but uses a new VQ-VAE approach to estimating the quantity, which tries to simultaneously aim for a particular codeblock, while maximizing codeblock entropy.
2) A meta-controller, which at test time selects skills to use to maximize task rewards. 
The authors show good performance on a variety of unsupervised RL learning benchmarks.

Strength And Weaknesses:

Strengths:
-The proposed model outperforms baselines across tasks in unsupervised RL benchmark. 

Weaknesses:
-The paper could benefit from some additional model-based baselines.
-Improvement seems small on most domains.
-It seems quite strange that random actions is the best baseline on robot manipulation tasks.

Discussion:
I believe the paper proposes a novel approach and is well written, with extensive comparison and analysis. 

1. However, other works have consider World-model based skills learning before i.e. Xie. et. al. Latent Skill Planning for Exploration and Transfer, which uses a similar objective.  
2. It is not immediately clear what component of the proposed model contributes to performance, especially given the complexity of individual modules. Is it model-based learning vs model-free learning over the baselines? In particular, performance seems pretty close to the LBS approach, which is also a model-based method. 
3. The visualized skills do not necessarily seems to perform consistent meaningful actions.






review_title: New model-based method to learn exploration from unsupervised data

Decision: Accept