Interplay Between Task Learning and Skill Discovery for Agile Locomotion

Authors: Seungeun Rho, Kartik Garg, Harshil Vagadia, Sehoon Ha
Primary area: applications to robotics, autonomy, planning
Keywords: Unsupervised Skill Discovery, Reinforcement Learning, Robot Learning, Locomotion

Abstract:
Agile locomotion of legged robots, characterized by high momentum and frequent contact changes, is a challenging task that demands precise motor control. Therefore, the training process for such skills often relies on additional techniques, such as reward engineering, expert demonstrations, and curriculum learning. However, these requirements hinder the generalizability of methods because we may lack sufficient prior knowledge or demonstration datasets for some tasks. In this work, we consider the problem of automated learning agile motions using its intrinsic motivation, which can greatly reduce the effort of a human engineer. Inspired by unsupervised skill discovery, our learning framework encourages the agent to explore various skills to maximize the given task reward. Finally, we train a parameter to balance the two distinct rewards through a bi-level optimization process. We demonstrate that our method can train quadrupeds to perform highly agile motions, ranging from crawling, jumping, and leaping to complex maneuvers such as jumping off a perpendicular wall.

Decision: reject
Reviewer scores: 3, 3, 5, 6

Reviews and discussion:
<thread>
creation date:
1730227448585

thread_type:
official_review

<thread_object>
creator: reviewer_vxjr

<content>
summary:
This paper presents a novel RL framework for learning agile and diverse locomotion skills. This is achieved using the combination of maximising a task specific objective summed with a diversity one waited using a learnt parameter. The authors present a two stage training algorithm to optimise both the task and diversity objectives: The agent policy is updated using gradients from both objectives, whilst the learnt parameter is updated using only the task rewards. The algorithm in this paper is compared with a task-only, diversity-only (based on prior work METRA) and RND, which maximises reward exploration. Training with task and diversity objectives together outperform the base-lines and produces impressive results for three agile manoeuvres. These manoeuvres include leap, climb, crawl and a wall jump.

strengths:
The formulation using a learnt weighting to balance the task and diversity objective is a strong contribution and is original. This is used to learn agile manoeuvres for quadruped locomotion in interesting environments and facilitates skill discovery. The quality of this work is generally good. The work is presented clearly and it is easy to follow. The work builds on METRA resulting in a contribution of merit to the field.

weaknesses:
Despite the impressive results and soundness of the paper, there are a few weaknesses. These weaknesses are all readily addressable and addressing these would strengthen the paper. The most significant weaknesses are the comparisons to the baselines. The reason for using RND (Burda et al., 2018) as a baseline is not provided in the paper. RND should be included in the literature review and a few sentences explaining why this is a fair comparison should be provided Section 4.1. The METRA (div-only) baseline fails to learn a usable policy. It would be of benefit to the reader to explore why this fails and if the skill latent space learns anything useful. A potential weakness is that METRA and LSD permits the composition of skills albeit in a straightforward setting, is this possible with the task and diversity algorithm? The training curves in Fig. 7 e) and f) would be improved by showing the standard deviation across multiple repeats similarly to Fig. 4. Finally, a minor weakness is that Fig. 2 is of a slightly lower quality compared to the other figures.

questions:
I have a few questions which mostly address some of the weaknesses of the paper.

Please could you point me to where you state how many repeats are used in Figure 4. 

Please could you explain why there is such a broad range of obstacles successfully traversed in Figure 4. 

As mentioned in weaknesses, why is RND used as a comparison baseline method? Why did the Div-only objective fail to solve the tasks? Did the latent representations of METRA fail to learn any reasonable skills? 

Does the skill collapse mean that there is no smooth transition between skills? 

Can you learn the skills needed for the wall jump individually and tie them together?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730647644609

thread_type:
official_review

<thread_object>
creator: reviewer_dmla

<content>
summary:
The paper proposes using a combination of skill discovery with goal-conditioned reinforcement learning, where the parameter balancing the two objectives is also optimized rather than held constant. The skill discovery is achieved by aligning latent space transitions with the sampled skills, subject to a temporal distance constraint. The approach allows learning agile behaviors for robot parkour with only a few simple task rewards, and the skill discovery module which encourages exploration. The method is verified on the quadruped robot A1 in simulation on a variety of tasks, including crawling, jumping, and wall-kicking. It outperforms the shown baselines and other exploration methods (RND) by a significant margin.

strengths:
- Interesting application of skill discovery as a method of exploration in goal-conditioned policies, and applied to highly dynamic quadruped skills.

- The robot learns very diverse skills and the approach helps it converge quicker (or at all) to a solution, where using purely task rewards fails.|

- Simple task rewards result in impressive behaviors.

weaknesses:
- Somewhat limited novelty - the approach uses an established skill discovery approach and balances its weight with task rewards, which prior work has already shown.

- No comparison with a few relevant prior works that also combine skill discovery with task policies.

- Lack of hardware validation, which makes it difficult to evaluate how useful this method can be in robotics.

- Some concerns with respect to the theoretical foundation of the method, as elaborated in the comments/questions.

questions:
## Main questions/notes

1. Related work - the paper is missing some important references that propose very similar approaches, such as DOMiNO [2] and SMERL [1], which propose methods of combining task-rewards with learning diverse skills, using dual optimization with Lagrange multipliers, and DoMiNiC [3] which applies DOMiNO to quadruped tasks, in a very similar fashion to the proposed method.
I think a comparison with these methods would be highly beneficial to show the advantages of the proposed framework.


2. Hardware validation would make the contributions much stronger. One of the issues with skill discovery is that it can be potentially very jerky and jittery, and I think that showing whether such policies can safely translate to the hardware is important. For example, the wall-jump policy shown in the supplementary video looks very impressive, but I doubt it’d be easy to deploy on the hardware.

3. How smooth are the skills with respect to the behaviors? Are certain regions in the skill space more likely to cause failures than others, or is it entirely random?


4. Can METRA even work with goal-conditioned policies? The objective in METRA is formally defined as (phi(sT) - phi(s0))^T * z and is then reformulated as the telescopic sum that you have mentioned around line 220.  However, if you have a goal-conditioned objective (say reaching a point over the gap as in Fig. 5 a) or even a desired velocity), you’d essentially be constraining both the initial and final states (s0 and sT=s_goal). Since your initial state is fixed regardless of the skill, if you want to maximize the mutual information you’d need to vary your final state - and hence violate your goal objective. Other works get around that by maximizing the mutual information between the current state and skill (rather than the state difference as in METRA) or looking at the expected state distribution under skills
Can the proposed method then be seen as an approach for exploration rather than skill discovery? I think this should be made clearer in the paper. Since the skill discovery objective directly conflicts with the task objective, if I understand correctly, you would essentially be using skill discovery to explore diverse behaviors at the start of training, but still converge to more or less a single behavior at the end.


5. On that note, It would be interesting to see the METRA objective after training - is the alignment between latent transitions and skills still good?


6. Shouldn’t lambda start decreasing as you train your model further? After the agent has learned many skills, each with a different task return, wouldn’t the task reward be maximized if all skill-conditioned behaviors converge to the behavior corresponding to the skill with highest task return? From Fig 6/8 it seems that lambda is relatively stationary.

## Minor notes:

1. Section 3.1 mentions the action space as joint torques - is that really the case or is it a typo?


2. Fig. 6b) and Fig. 8 have “# of obstacles passed” as the y axis label - shouldn’t this be the value of lambda instead?


3. Does the METRA encoder use the same observations as the policy?

## Summary:
The paper proposes an interesting combination of skill discovery (through temporal distance maximization) and task rewards. The novelty seems to mostly come from the specific choice of skill discovery algorithm and the way the quality-diversity optimization is handled. The results overall seem promising, and I think some comparisons with competing methods like the ones mentioned above would strengthen the contributions. Furthermore, some hardware experiments would go a long way in showing how beneficial this approach is for robotics.

## Bibliography

[1]	S. Kumar, A. Kumar, S. Levine, and C. Finn, ‘One Solution is Not All You Need: Few-Shot Extrapolation via Structured MaxEnt RL’, in Advances in Neural Information Processing Systems, Curran Associates, Inc., 2020, pp. 8198–8210. doi: 10.48550/arXiv.2010.14484.

[2]	T. Zahavy et al., ‘Discovering Policies with DOMiNO: Diversity Optimization Maintaining Near Optimality’, presented at the The Eleventh International Conference on Learning Representations, Sep. 2022. 

[3]	J. Cheng, M. Vlastelica, P. Kolev, C. Li, and G. Martius, ‘Learning Diverse Skills for Local Navigation under Multi-constraint Optimality’, in 2024 IEEE International Conference on Robotics and Automation (ICRA), Yokohama, Japan: IEEE, May 2024, pp. 5083–5089. doi: 10.1109/ICRA57147.2024.10611629.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730694243940

thread_type:
official_review

<thread_object>
creator: reviewer_45ih

<content>
summary:
This paper combines reinforcement learning and unsupervised skill discovery to learn agile locomotion. A bi-level optimization is proposed to train a parameter to balance the task reward and the diversity reward. The proposed approach can achieve impressive locomotion task, such as a simulated wall-jump.

strengths:
- This paper clearly presents the contributions and results.
- The achieved wall-jump in simulation looks impressive.

weaknesses:
- The novelty of this paper is limited. 
- More baselines should be included, e.g., a pure RL policy with more complex task reward such as those parkour policies.
- Some of the learned motions look a bit unrealistic. Real-world experiments would strengthen the contribution of the proposed approach.
- More task-related plots should be included, e.g., body orientation plots, joint torque plots etc.
- For videos, it would be better to include failure cases, baselines. The video shows diverse solutions to one task, but should also show something like jumping over or crawling under the same hanging obstacle as in Fig. 2.

questions:
- Detailed formulation of task reward is missing. Mathematical formulations and the desired values for calculating the task reward for each task need to be provided.
- Why is the standard deviation of the training curve of the proposed approach so large, e.g., Fig. 4a & 4b?
- The standard deviation of the lambda curve for leaping task in Fig. 8 is very large. Why the authors conclude it is always within the range of 0.2-0.4? Besides, y-axis name is not correct for some lambda curves throughout the paper, e.g., Fig. 6b.
- When comparing the proposed approach with fixed lambda values, the constant lambda values should include the converged values as well.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730721790455

thread_type:
official_review

<thread_object>
creator: reviewer_rntf

<content>
summary:
This study introduces a method for teaching robots agile locomotion skills using unsupervised reinforcement learning (RL). The proposed RL approach combines a task-specific reward with an unsupervised skill discovery reward, scaled dynamically. This combination allows the robot to acquire skills such as jumping, running, and flipping without specially designed rewards or expert demonstrations. The approach was evaluated in simulation, showing reasonable performance improvements over existing baselines.

strengths:
1. The method is intuitive and appears straightforward to implement, with detailed information provided for reproducing results in simulation.
2. The approach successfully learned complex locomotion skills, including wall-jumping, within a simulated environment.
3. The ablation study shows automatically tune the weight for unsupervised rl objective helps on improving the sample efficiency.

weaknesses:
1. The method is tested solely in simulation without detailing specific simulation physics parameters. Given recent advancements in robot learning for quadrupedal robots, a real-world evaluation would help strengthen the impact and credibility of the results.

2. The motions generated by the learned model appear unnatural, which may hinder transferability to real-world applications.

3. Experimental setup and evaluation raise several concerns:

* Evaluations were conducted in three relatively simple environments (Leap, Climb, Crawl), none of which are as challenging as the demonstrated wall-jump skill.
* The study compares only with limited baselines (Task-Only, Div-Only, RND), whereas it would be informative to include diversity-oriented baselines such as DIAYN, as seen in previous work like METRA (Park, 2023).
* The task scope closely resembles that of Robot Parkour Learning (Zhuang 2023), so comparing it with methods that use manually tuned reward functions would help clarify the specific contributions of this work.
* The wall-jump task appears to have been run with a single seed and without robust baseline comparisons across other tasks, so further clarification is needed here.

4. Some related literature is missing:

Agile Locomotion:

"Lifelike agility and play on quadrupedal robots using reinforcement learning and generative pre-trained models"

"Generalized Animal Imitator: Agile Locomotion with Versatile Motion Prior"

Unsupervised RL for Learning Locomotion Skills:

"ASE: Large-scale reusable adversarial skill embeddings for physically simulated characters"

Using Auxiliary Rewards for Enhancing Robot Learning:

"Adversarial Motion Priors Make Good Substitutes for Complex Reward Functions"

questions:
Since the proposed method is applying existing unsupervised RL work to the robotics tasks, more experiments should be conducted to make it a promising submission. Please refer to the weakness mentioned above
</content>
</thread_object>

</thread>


Metareview:
metareview:
(a) The paper introduces a framework combining reinforcement learning with unsupervised skill discovery to enable legged robots to learn agile locomotion skills. The core claim is that by using a learnable parameter to balance task-specific rewards with a diversity reward, robots can learn complex motions without extensive reward engineering, expert demonstrations, or curriculum learning. The method can train robots to perform highly agile motions, such as crawling, jumping, leaping, and a wall-jump, achieving performance on par with or better than manually tuned reward systems, without intensive reward engineering.

(b) Strengths:
- The method successfully trains robots to perform complex, agile motions such as wall-jumping with good performance in sim.
- The paper is generally well-written and easy to follow, with detailed information for reproducing results in simulation.
- The methodology description is comprehensive.

(c) Weaknesses:
- Limited Novelty: Reviewers noted that the method combines existing techniques, and thus the overall novelty is limited.
- No real-world experiment. Some motions appear unnatural, which may limit the transferability to the real world.

(d) The decision is to reject the paper. For locomotion research paper, the weaknesses in the lack of real-world experiment and methodological novelty are significant.