Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This submission introduces a general-purpose prior for option discovery from demonstrations: minimum description length, i.e., a compression objective. The authors extend the VTA model from Kim et al., 2019, to include a compression objective and a policy decoder (which can constitutes the option policies). The experiments demonstrate that the algorithm is effective in discrete action settings with both 2D and 3D observations.
strengths_and_weaknesses: The paper is well-written, and the motivation and presentation is good, and I think that finding good, general priors for option discovery is of high interest. The experiments are described clearly and, although they clearly expose a very suitable structure for learning discrete options, showcase the efficacy of the method sufficiently well.

As the authors note, MDL for option discovery was recently proposed by Zhang et al., 2021 -- it would have been interesting to include this as another baseline for comparison.

In terms of ablations, I'd encourage the authors to further analyze the effective numbers of skills that are used (explaining the effect of filtering unused skills as described in section 5). In a similar vein, I was wondering how robust the proposed algorithm is to the number of options that should be learned. If I understand correctly, in the experiments the number of objects to navigate to is known beforehand and the number of options is specified accordingly. Suppose that this information is hard to obtain from a given set of demonstrations -- how much would it hurt to specify too many or too few options? To add to that, in Fig. 8 (Appendix), 7 out of 10 skills seem to result in the same actions. Why is that?
questions: See above
limitations: The conclusion in the main body mentions possible future work but fails to raise awareness on limitations, or at least on open (and as for now) unaddressed open questions. For example, would the proposed method potentially work on the maze navigation tasks in D4RL (https://sites.google.com/view/d4rl/home)? I think that corresponding evaluations can be regarded as outside the scope of this work, but if the authors see possible issues in applying the method to other popular offline RL tasks, this would be an interest addition to the paper's conclusion.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents a method from offline demonstration learning to HRL. The paper builds upon the generative model of VTA and add an information cost to enforce a better segmentation, which ends up aligning with the ground truth boundaries, The recovered options can be used in the HRL with augmented action space which shows good performance,
strengths_and_weaknesses: Strengths:
The motivation is clear, and the paper is well written and easy to follow. I find the proposed method to be novel and clean. The experiment results also suggest the method is effective at recovering the ground truth boundaries. The connection to MDL also seems convincing. The experiment on HRL and learning new tasks is also impressive.


Weakness:
I would like to see discussion and comparison with CompILE [1], which I think is quite similar to this work, since they are both based on generative models.


[1] https://arxiv.org/abs/1812.01483.
questions: Can you provide the comparison between your method and compILE? 
Can you also provide some experiments on robotics task with continuous state/action space like jaco pinpad in [1] or reacher?

[1] https://arxiv.org/pdf/2103.10972.pdf
limitations: Not that I can see.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a framework to discover options from the offline data. The framework is build upon the Variational Temporal Abstraction(VTA) method. To discover semantically meaningful skills, they involve an optimization object of compression. They carry out experiments on 2D and 3D multi-task domains. The experiment results show that the proposed framework is able to discover meaningful options and reuses them for new tasks.
strengths_and_weaknesses: Strengths:
1. The proposed method is reasonable for option discovery.
2. The analysis and introduction of the method is very clear and make it easy to follow.

Weaknesses:
1. Although I'm not an expert in the field of option discovery from offline data, the idea of penalizing option switching or reward option continuation is common is option discovery from online data., e.g. option-critic[1] and [2]. Thus, the novelty may be not enough.
2. The choice of comparison method. There are a lot of methods for offline option discovery proposed rencently. Can you explain the reason of choosing DDO?
[1] The Option-Critic Architecture
[2] A Compression-Inspired Framework for Macro Discovery
questions: 1. There are a lot of methods for offline option discovery proposed rencently. Can you explain the reason of choosing DDO?
limitations: The authors have adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents a regularization method for learning skills from data. The authors are inspired by the minimum description length and combine the maximum likelihood objective with the compressing term to improve the effectiveness of learned skills. The resulting approach LOVE demonstrates the faster-converging speed and semantically important results compared to prior work. Furthermore, LOVE can scale to tasks with image observations. 
strengths_and_weaknesses: ## After reading other reviewers' comments which are more positive than I thought and reading other rebuttals, I decided to increase my score from five to six and then to seven.
# Strength
1. Learning skills from demonstrations is an important task in hierarchical reinforcement learning.
2. The proposed compression metric should be effective to improve the learned skills in HRL.
3. The proposed visualizations in Figure3 and Table1 demonstrate the effectiveness of LOVE in various perspectives. 
4. The code is provided.

# Weakness
1. The authors state in many places, "prior works do not directly learn skills from high-dimensional pixel observations". This is false. In fact, many work tries to learn skills on Atari games (real-image games), such as the option critic architecture [4] and RAPL [83].
2. The proposed compression term is merely a trick beyond VTA. And the compression metric is already used in RAPL [83] and the unsupervised skill discovery literature. So the novelty is limited. 
3.  The title is a bit misleading. The title states "learning options via compression". However, the compression is merely a regression term in the learning scheme, and the major body of this approach is still VTA.
questions: 1. Why don't the authors try to experiment on Atari to show the learned skills?
2. Why can LOVE discover better boundaries in Figure 3?
3. How to balance the maximum likelihood target and the length constraint target? If there are many skills of the same length, would the target become ineffective?
4. Can the proposed compresion scheme generalize to other option learning frameworks?
limitations: The authors shortly describe the limitations in the conclusion part. However, I don't quite understand that part and I think the important limitations of this paper are not discussed (refer to the above limitation part).
review_title: null

Meta-review: metareview: This paper studies the problem of learning options in multi-task reinforcement learning. The authors note that previous works optimize an underspecified objective and they propose adding an extra term to the objective function that relates to the description lengths of skills. The authors study their approach and show empirically that it scales to high-dimensional problems and performs well compared to previous approaches. The discovered skills can also be used to solve new tasks using fewer samples than previous approaches.

The initial reviews were overall very positive for this paper. During the author-reviewer discussion, the authors provide additional results and provided satisfying answers to most reviewer comments. As a result, the reviewers are unanimous that this work should be accepted and the discussion period did not reveal any other elements to report here. 

I am pleased to recommend acceptance, congratulations! It seems like the current version of your manuscript already addresses most if not all of the points raised by reviewers. In addition, please do not forget to discuss the limitations of your current work including e.g. different domains that it might (not) work in (see the comment from reviewer CrHt).

confidence: Certain
Decision: Accept