DSR: Reinforcement Learning with Dynamical Skill Refinement

Authors: Dongxiang Chen, Ying Wen
Primary area: reinforcement learning
Keywords: Reinforcement Learning, Reinforcement Learning with Skills, Reinforcement Learning with Demonstrations

Abstract:
Reinforcement learning with skills (RL with skills) is an efficient paradigm for solving sparse-reward tasks by extracting skills from demonstration datasets and learning high-level policy which selects skills. Because each selected skill by high-level policy is executed for multiple consecutive timesteps, the high-level policy is essentially learned in a temporally abstract Markov decision process (TA-MDP) built on the skills, which shortens the task horizon and reduces the exploration cost. However, these skills are usually sub-optimal because of the potential low quality and low coverage of the datasets, which causes the sub-optimal performance in the downstream task. Refining skills is intuitive, but the change of skills will in turn lead to the non-stationarity of the transition dynamics of TA-MDP which we name temporal abstraction shift. To address the dilemma of sub-optimal skills and temporal abstraction shift, we unify the optimization objectives of the entire hierarchical policy consisting of the high-level policy and the low-level policy whose latent space embeds the skills. We theoretically prove that the unified optimization objective guarantees the performance improvement in TA-MDP, and that optimizing the performance in TA-MDP is equivalent to optimizing a lower bound of the performance of the entire hierarchical policy in original MDP. Furthermore, in order to overcome the phenomenon of skill space collapse, we propose the dynamical skill refinement (DSR) mechanism which names our method. The experiment results empirically validate the effectiveness of our method, and show the advantages over the state-of-the-art (SOTA) methods.

Decision: reject
Reviewer scores: 5, 6, 5

Reviews and discussion:
<thread>
creation date:
1729935056537

thread_type:
official_review

<thread_object>
creator: reviewer_tpgy

<content>
summary:
This paper addresses the problem of the sub-optimal dataset in the skill-based reinforcement learning settings. The authors propose a unified optimization objective to train a hierarchical skill structure, providing a mathematical proof within the TA-MDP. Additionally, the authors introduce a dynamic skill refinement approach to mitigate potential collapse in the skill space.

strengths:
- Learning skills from sub-optimal datasets and overcoming skill space collapse are crucial challenges in skill-based reinforcement learning scenarios.

weaknesses:
- I am not fully convinced how the unified optimization objective addresses the issue of sub-optimality in the datasets. DSR learns the low-level policy, making it more likely to optimize the action sequence for the downstream policy. If this is the case, the authors should emphasize more on mitigating the collapse in the skill space. 
- The contribution on dynamic skill refinement (Section 5.2) seems minimal, and its benefits are not fully demonstrated in Ablation 6.2, as DirectRefinement also shows comparable performance. Otherwise, the authors should provide more ablation studies on the learning of TA-MDP.
- In Section 6.1, the authors mention that DSR improves performance by avoiding skill space collapse. However, in Ablation 6.2, skill space collapse appears to occur only in the PyramidStack task. If the authors want to emphasize their contributions regarding skill space collapse, they should create more scenarios similar to PyramidStack and elaborate on why these tasks cause the collapse, and why it does not occur in other tasks.
- The presentation could be further improved. Figure 4 is smaller compared to Figure 2, and Figure 3 could be replaced with an equation for better clarity. For clarity, it would be better to include the pseudocode in the main manuscript rather than in Appendix C.

questions:
- As the paper addresses the problem arising from the sub-optimality (low quality and low coverage) of the datasets, the authors should elaborate more on their dataset collection regarding to its sub-optimality.
-  In Figure 6, the y-axis denotes the episodic return. How is the episodic return computed? Additionally, can the authors elaborate more on their sparse reward setting? Is a reward given only upon task completion?
- In Ablation 6.2, what do the authors mean by "the extracted skills are not well initialized"? In Appendix D.2, the authors mention using datasets collected from TableCleanup, SlipperyPush, and PyramidStack, but why does only the PyramidStack task show this performance gap in Ablation 6.2?
- According to the original ReSkill paper, ReSkill achieves an average return of over 15 in the PyramidStack task. However, in Figure 6, ReSkill achieves only an average return of 10. The authors mentioned that the experimental settings are equivalent to those in ReSkill as described in Appendix D.2. Why is there a performance gap?
- It would be benificial, if the authors provide more ablation studies on their learning mechanism of TA-MDP. (Section 4)
- Can the authors provide additional experiments on other robotic manipulation environments, including long-horizon scenarios such as MetaWorld or Franka Kitchen?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730581131125

thread_type:
official_review

<thread_object>
creator: reviewer_uuww

<content>
summary:
This paper aims to address two issues in RL with skills: **(1)** sub-optimal pretrained skills due to low quality and low coverage of the datasets, and **(2)** temporabl abstraction shift due to refining skills. The authors propose to unify the optimization objectives of both high-level and low-level policies as the future return and use random network distillation to dynamically adjust the weight of action refinement. The paper demonstrates the effectiveness of the proposed method by providing theoretical analysis and empirical results.

strengths:
* The proposed method is well-motivated, theoretically grounded, and demonstrates better performance against several state-of-the-art RL-with-skills methods.
* The author uses multiple figures to improve the clarity of the paper.

weaknesses:
While well-motivated, I have some questions about this work:

1. In the introduction, the authors mention that previous works (e.g., SPiRL [1]) assumes near-optimal dataset to pretrain the skills. However, even if the skill contains some sub-optimal behavors, could the high-level policy still learns to avoid choosing the bad skills as it aims to maximize the reward?
2. From the theoretical perspective, how is DSR different from the classic online hierarchical RL methods? For instance, [2] updates both high-level and low-level policies with RL objective with representation learning, and also provides the optimization objective bound. A similar bound is provided in [3] for offline settings. These two bounds actually quantify the exact form of sub-optimality, hence I think they are stronger than Theorem 3 in this work. I would appreciate it if the authors could discuss these methods in the paper.
3. How well does DSR can transfer skills to new tasks? I think one promise of skill-based RL is that it can conbine the learned skills to novel tasks, as mentioned in [1]. I am wondering if the authors could show some results in their experimental setting, such as training on data from one task and evaluating on other tasks.

Overall, while this paper presents some interesting ideas, I am unable to recommend acceptance at this stage given the questions mentioned above. However, I would consider raising my score if the authors could address my concerns.

[1] Pertsch, Karl, Youngwoon Lee, and Joseph Lim. "Accelerating reinforcement learning with learned skill priors." Conference on robot learning. PMLR, 2021.

[2] Nachum, Ofir, et al. "Near-optimal representation learning for hierarchical reinforcement learning." *arXiv preprint arXiv:1810.01257* (2018).

[3] Yang, Yiqin, et al. "Flow to control: Offline reinforcement learning with lossless primitive discovery." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 37. No. 9. 2023.

questions:
There are some questions and concerns, which I have outlined in the previous section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730713338936

thread_type:
official_review

<thread_object>
creator: reviewer_qk82

<content>
summary:
The paper introduces Dynamical Skill Refinement (DSR), an on-policy reinforcement learning method designed to optimize hierarchical policies in environments with sparse rewards. The paper refines the skills in an on-policy manner and proves that the temporal abstraction shift is circumvented by simultaneously updating the high-level policy and skills. The method prevents skill space collapse, which can lead to performance issues, by incorporating a dynamical mechanism that refines skills without disrupting the latent space. Empirical results demonstrate that DSR outperforms state-of-the-art methods in complex sparse-reward robotic manipulation tasks.

strengths:
1- The paper is well-written and easy to read.
2- The proofs for theorems are solid.
3- Good comparisons with SOTA methods.

weaknesses:
1- The main difference between your approach and ReSKILL is that yours handles the temporal abstraction shift but it is not shown in your theorems.
2- This method needs lots of extra parts such as an extra residual policy to avoid skill collapse. Also, using RND makes sure that a skill is refined enough but still you cannot make sure that modifying the skill does not impact the state of other skills in the same state. Moreover, RND is an approach that needs lots of tuning such as how you control the learning rate of the variable network and the threshold for being accurate enough for the refinement. This would impact the results and we expected to see an analysis of these parameters.

3- Usually we have complex navigation robotic environments in the experiments with skills as the macro-action to reach the subgoals or landmarks. We do not see any navigation environment here.

questions:
1- Can you explain again how skills are extracted using VAE?
2- ReSKILL has a similar on-policy approach as your method. You said that ReSKILL cannot theoretically guarantee performance improvement, but you did not mention proof. Can you elaborate more on this?
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes a method for learning skills from a sub-optimal dataset, which are used in a hierarchical policy to solve sparse-reward tasks. A the key component of the method is a mechanism to prevent skill space collapse. The paper includes both theoretical analysis and empirical results demonstrating improved performance relative to state-of-the-art methods.

Reviewers appreciated the importance of the problem setting, the proposed method's strong empirical performance, the correctness of the theoretical results, and clear writing and motivation in the text.

Reviewers had concerns about the similarities with prior work (ReSKILL, SPiRL, classical HRL methods), including the novelty of Theorem 3 (which may have appeared in prior work). They also had concerns about the complexity of the proposed approach. Reviewers suggested additional baselines (ReSKILL) and evaluation tasks (complex navigation robotic environments, transfer to new settings, clarifying the results from Ablation 6.2), and minor presentation recommendations.

Overall, I think this has the makings of the strong paper (combining theoretical analysis with a new method), but there remain some unaddressed concerns about the relationship with prior work. I therefore am recommending that the paper be rejected. I would encourage the authors to resubmit after addressing the the feedback mentioned in the reviews.