Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary
-------

To best leverage diverse datasets of logged experience, the authors
propose to extract a space of primitive behaviors in a continuous space,
and to use these for downstream learning. The primitive behaviors are
learned through a VAE loss, and CQL is applied to learn a policy over
the primitives. The authors claim that this approach to offline RL
avoids known distribution shift and allows for temporal abstraction. The
method is also applied to few-shot imitation learning, exploration and
transfer to online RL.

Decision
--------

Despite this paper being an interesting read, I feel that my concerns
about the experiments lead me to not be confident in the proposed
approach. As such, my preliminary rating for the paper is "Okay but not
good enough". The baselines chosen for the experiments do not seem
representative of the problem being addressed. This also leads into the
motivation, where OPAL is motivated from the offline RL perspective but
does not explicitly mitigate the issues in offline RL. Only the
theoretical results investigate the effect of a fixed replay-buffer, but
even these claims are framed in terms of what $\mathcal{D}$ should be to
ensure downstream RL.

Originality
-----------

The proposed approach seems limited in its novelty, combining approaches
from skill-discovery, hierarchical RL and somewhat from offline RL. The
theory section is interesting, however the technical arguments seem
similar to that of Nachum et al. (2018).

Quality and Clarity
-------------------

The overall quality of the paper is very high. The writing is clear, and
the theoretical arguments are put into an easily understandable context.

Strengths
---------

-   OPAL leverages techniques from many areas of machine learning:
    unsupervised learning, hierarchical RL and offline RL. This is an
    interesting combination, and deserves to be investigated. It seems
    like the workhorse of OPAL is the unsupervised learning component
    however, and perhaps this should be emphasized and investigated
    independently.
-   The theoretical section is very clear and well contextualized. The
    claims in the paper seem correct, and they do provide much needed
    validation for hierarchical RL. The assumption of an optimal
    high-level controller seems strong, but the analysis is nonetheless
    interesting.

Weaknesses
----------

-   Despite Figure 2, OPAL is difficult to disentangle with many moving
    parts. It doesn't help that there is no discussion of how
    hyperparameters were chosen for each component, or an ablation study
    to investigate different hierarchical RL approaches, fine-tuning,
    offline RL algorithms, or VAE losses.
-   The proposed method is not specifically designed to leverage
    anything particular in offline RL. As you show, it can be applied to
    online RL. However, why is this motivated from the offline
    perspective? OPAL itself does not seem to mitigate distributional
    shift.
-   It would be helpful to directly compare to other skill-discovery
    methods that have been applied to online RL but can be adapted to
    offline RL. For example, what prevents the work by Campos et
    al. (2020) and the baselines therein to be applied in conjunction
    with an offline RL algorithm? The baselines for the experiments do
    not seem representative of the problem you are addressing. You use
    standard offline RL algorithms, yet claim that the temporal
    abstraction of skill discovery is crucial to the results. As such,
    you should compare OPAL to other skill-discovery algorithms that can
    be combined with offline RL.

Detailed Comments
-----------------

-   Section 4.1: " Prior: ρω(z|s0) tries to predict the encoded
    distribution of the sub-trajectory.." Prior doesn't seem like the
    right word here, since it is being learned.
-   Section 4.2: In what way is the task-specific policy $\pi_\psi$
    different from the learned prior in Section 4.1? Both are
    distributions over latent $z$ conditioned on a state. While the
    prior is designed to only be conditioned on the initial state, this
    is still quite similar to the high-level policy $\pi_\psi$.
-   Section 4.2: "\[to\] ensure that the c-step transitions remain
    consistent … with the labelled latent action $z_i$" Why is this
    necessary, should CQL ensure consistency at the action-level, while
    the latent actions are consistent by design?
-   Corollary 4.1.1: How would $\epsilon$ actually approach
    $\mathcal{H}_c$ in the algorithm? The condition is never explicitly
    enforced because you formulate it as a constrained optimization
    problem and you are never able to change $\mathcal{H}_c$ since it is
    a constant.
-   Section 5.1, baselines: shouldn't baselines be compared to different
    unsupervised skill discovery algorithms, paired with offline RL
    algorithms? BEAR and EMAQ are offline RL algorithms without any
    temporal abstraction. And as you say in the results section with an
    ablation study, this temporal abstraction is crucial.
-   Section 5.1, results: An ablation study is discussed but I cannot
    find the corresponding results table/figure in the paper.
-   Section 5.2, Table 3: With so many 0.0's, this leads me to believe
    that the baselines are quite weak, or the problem is too hard for
    the baselines. How were the baselines chosen. For example, why was
    DDCO paired with DDQN instead of SAC?

Minor Comments
--------------

-   Section 5.2 mis-capitalized We "in this setting We use the Antmaze
    environ-"
-   Section 5.3, Table 4: why are there no standard errors for SAC?

Post Rebuttal
--------------
After discussion with the authors, I have decided to increase my score to a 6. The authors have addressed many of my concerns with respect to motivation, theoretical analysis and empirical evidence. As it stands, I still think OPAL is hampered by the many "moving parts" involved. The theoretical analysis and empirical evidence suggests a very effective approach however, and should lead to further work combining variational sequence encoding techniques for primitive extraction in RL. My decision would be to "accept the paper if there is room".

References
--------------
Campos, Víctor, Alexander Trott, Caiming Xiong, Richard Socher, Xavier Giro-i-Nieto, and Jordi Torres. 2020. “Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skills.” *arXiv:2002.03647*.
<http://arxiv.org/abs/2002.03647v4>.

Nachum, Ofir, Shixiang Gu, Honglak Lee, and Sergey Levine. 2018. “Near-Optimal Representation Learning for Hierarchical Reinforcement Learning.” *arXiv:1810.01257*. <http://arxiv.org/abs/1810.01257v2>.


review_title: While interesting,  baselines are weak and OPAL does not directly address problems in offline RL

final_score: rating: 7: Good paper, accept
review_content: ## OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning
### Summary
The authors present OPAL, an offline reinforcement learning approach that distills useful common behaviors from offline transition data. They leverage a variational GRU to encode trajectories into a latent space of primitive policies. These learned primitives can later on be transferred to other tasks by learning a high-level controller over the extracted primitive latent space. Their experimental results show that OPAL compares favorably against other SOTA methods in offline, online and few-shot reinforcement learning settings.

Overall, the paper is clearly written, and the approach is potentially practical for real-world applications, due to its good performance on both offline and few-shot adaptation settings. 

### Strength
- The proposed method can leverage large unlabeled data without pre-defined reward functions.
- The property of OPAL is mathematically analyzed.
- OPAL can potentially benefit from the advancement of offline RL algorithms by simply replacing CQL.
- Can be adapted to a new task through few-shot imitation learning.

### Weakness
- Training details are missing for the OPAL encoder.
- Not exactly a weakness, but it would be nice to see if the method works in real-world settings (real-world on/offline data). 
review_title: Good paper with (potentially) practical values for real-world robotic tasks

final_score: rating: 6: Marginally above acceptance threshold
review_content: In the RL setting, this paper tackles the case where an agent may have access to large amounts of offline experience data. The objective of the work is to find an effective way to leverage this data for finding temporally extended primitive behaviors. The paper provides results that show how performing offline primitive learning can be leveraged for improving few-shot imitation learning as well as exploration and transfer on a variety of benchmark domains.

The paper tackles an important question in reinforcement learning: learning temporally extended primitive behaviors from off-policy data is a very relevant question. However, I found the motivation for the approach quite vague as well as different elements that require clarification (see below) and because of this, I can't recommend acceptance.

Motivation for the approach: 
- if the downstream policy has to stay close to the offline data distribution, it seems to me that if the offline data distribution is obtained with a bad policy, this can not lead to interesting decision-making.
- can you explain why equation 2 "motivates better generalization"?

Other concerns:
- Why can gamma not have a value of 0 (first paragraph preliminaries)
- What does it mean "To capture multi-modality in a dataset" ? (second paragraph preliminaries)
- Figure 1:  What does the (81.1%) and (70.3%) refer to?
- Table 4: why is SAC given without standard deviation?

Some text improvements:
- "we focus on focus on"

Other comment:
- The number of seeds (4) should ideally be increased.
review_title: Interesting research question but motivation of the method unclear

Meta-review: This paper presents an interesting mix of new theoretical and empirical results showing how learning temporally extended primitive behaviors can help improve offline (batch) RL.

Although 2/3 reviewers initially raised concerns regarding the motivation of the approach and some of the choices that were made, the authors did an excellent job at addressing these concerns in detail, and there is now a consensus towards acceptance.

I consider that this work is a meaningful contribution towards better offline RL, which is definitely a very important use case in practice. The authors have given convincing explanations to motivate their approach, and made several improvements to the paper. As a result, I am recommending it for acceptance, as a poster.
Decision: Accept (Poster)