Title: PEAC: Unsupervised Pre-training for Cross-Embodiment Reinforcement Learning
Venue: NeurIPS 2024
Accepted: yes
Keywords: cross-embodiment reinforcement learning

unsupervised reinforcement learning

cross-embodiment exploration

cross-embodiment skill discovery
Abstract: Designing generalizable agents capable of adapting to diverse embodiments has achieved significant attention in Reinforcement Learning (RL), which is critical for deploying RL agents in various real-world applications. Previous Cross-Embodiment RL approaches have focused on transferring knowledge across embodiments within specific tasks. These methods often result in knowledge tightly coupled with those tasks and fail to adequately capture the distinct characteristics of different embodiments. To address this limitation, we introduce the notion of Cross-Embodiment Unsupervised RL (CEURL), which leverages unsupervised learning to enable agents to acquire embodiment-aware and task-agnostic knowledge through online interactions within reward-free environments. We formulate CEURL as a novel Controlled Embodiment Markov Decision Process (CE-MDP) and systematically analyze CEURL's pre-training objectives under CE-MDP. Based on these analyses, we develop a novel algorithm Pre-trained Embodiment-Aware Control (PEAC) for handling CEURL, incorporating an intrinsic reward function specifically designed for cross-embodiment pre-training. PEAC not only provides an intuitive optimization strategy for cross-embodiment pre-training but also can integrate flexibly with existing unsupervised RL methods, facilitating cross-embodiment exploration and skill discovery. Extensive experiments in both simulated (e.g., DMC and Robosuite) and real-world environments (e.g., legged locomotion) demonstrate that PEAC significantly improves adaptation performance and cross-embodiment generalization, demonstrating its effectiveness in overcoming the unique challenges of CEURL. The project page and code are in https://yingchengyang.github.io/ceurl.

PEAC: Unsupervised Pre-training for Cross-Embodiment Reinforcement Learning

Abstract

Designing generalizable agents capable of adapting to diverse embodiments has achieved significant attention in Reinforcement Learning (RL), which is critical for deploying RL agents in various real-world applications. Previous CrossEmbodiment RL approaches have focused on transferring knowledge across embodiments within specific tasks. These methods often result in knowledge tightly coupled with those tasks and fail to adequately capture the distinct characteristics of different embodiments. To address this limitation, we introduce the notion of Cross-Embodiment Unsupervised RL (CEURL), which leverages unsupervised learning to enable agents to acquire embodiment-aware and task-agnostic knowledge through online interactions within reward-free environments. We formulate CEURL as a novel Controlled Embodiment Markov Decision Process (CE-MDP) and systematically analyze CEURL's pre-training objectives under CE-MDP. Based on these analyses, we develop a novel algorithm Pre-trained Embodiment-Aware Control (PEAC) for handling CEURL, incorporating an intrinsic reward function specifically designed for cross-embodiment pre-training. PEAC not only provides an intuitive optimization strategy for cross-embodiment pre-training but also can integrate flexibly with existing unsupervised RL methods, facilitating cross-embodiment exploration and skill discovery. Extensive experiments in both simulated (e.g., DMC and Robosuite) and real-world environments (e.g., legged locomotion) demonstrate that PEAC significantly improves adaptation performance and cross-embodiment generalization, demonstrating its effectiveness in overcoming the unique challenges of CEURL. The project page and code are in https://yingchengyang.github.io/ceurl .

1 Introduction

Cross-embodiment reinforcement learning (RL) involves designing algorithms that effectively function across various physical embodiments. The fundamental goal is to enable agents to apply skills and strategies learned from some embodiments to other embodiments, which may own different physical dynamics, action-effectors, shapes, and so on [28, 70, 58, 52, 12, 69, 60]. This capability significantly enhances the generalization of RL agents, reducing the necessity for embodiment-specific training. By adeptly adapting to new and shifting embodiment, cross-embodiment RL ensures that agents maintain reliable performance in unpredictable real-world scenarios, thereby benefiting the deployment process and reducing the need for extensive data collection for each new embodiment.

Cross-Embodiment Reinforcement Learning

（

under different state/action spaces, or dynamics

）

Reward free

world

Pre-trained Agent

Pre-train to extract generalized knowledge

Fine-tune

Pre-trained Embodiment-Aware

Control (PEAC)

One of the primary challenges in this area is the transfer of knowledge across embodiments that have vastly different physical dynamics and environmental interactions. This requires the agent to abstract knowledge in a way that is not overly specialized to a single embodiment or some downstream tasks. However, directly training cross-embodiment agents under some given tasks will cause the learned knowledge highly related to these tasks rather than only to embodiments themselves.

Inspired by the transformative effects of unsupervised learning in natural language processing and computer vision [6, 21], which has demonstrated efficiency in extracting generalized knowledge independent of downstream tasks, we propose a natural question: Can we pre-train cross-embodiment agents in an unsupervised manner, i.e., online cross-embodiment pre-training in reward-free environments, to capture generalized knowledge only related to embodiments? Existing unsupervised RL techniques, including exploration [45, 38] and skill discovery [10, 26] ones, typically involve pre-training agents by engaging a single embodiment within a controlled Markov Decision Process (MDP) that lacks extrinsic reward signals. These pre-trained agents are then expected to quickly fine-tune to any downstream tasks characterized by extrinsic rewards using this specific embodiment. This approach of unsupervised RL fosters the development of policies that are not overly specialized to specific tasks or reward structures but are rather driven by intrinsic motivations of embodiments, which shows the potential for discovering more generalized knowledge across different embodiments.

In this work, we adapt the unsupervised RL paradigm to the cross-embodiment setting, introducing the concept of Cross-Embodiment Unsupervised RL (CEURL). This setting involves pre-training with a distribution of embodiments in reward-free environments, followed by fine-tuning to handle specific downstream tasks through these embodiments. These embodiments may own similar structures so that we can abstract generalized knowledge from them. To analyze CEURL and design corresponding algorithms, we formulate it as a Controlled Embodiment Markov Decision Process (CE-MDP), which comprises a distribution of controlled MDPs, each defined by its unique embodiment context. Compared to the traditional single-embodiment setting, the CE-MDP framework addresses the additional complexity caused by the inherent variability among embodiments. We then extend the information geometry analyses of the controlled MDP [11] to better explain the complexity of CEMDP. Our findings indicate that skill vertices within CE-MDP may no longer be simple deterministic policies and the behaviors across different embodiments can display substantial variability.

To address the complexities of CE-MDP, we undertake an in-depth analysis of the pre-training objective in CE-MDP. We aim to enable our pre-trained agent to quickly fine-tune for any downstream tasks denoted as R ext, especially under the worst-case reward scenarios. Thus, our pre-training objective involves minimizing across R ext while maximizing the fine-tuned policy π ∗ , leading to a complex min-max problem (Eq. 3). We further introduce a novel Pre-trained Embodiment-Aware Control (PEAC) algorithm to optimize this objective and handle CE-MDP, which improves the agent's robustness and adaptability across various embodiments by employing a cross-embodiment intrinsic reward R CE. This reward is complemented by an embodiment discriminator, which distinguishes between different embodiments. During fine-tuning, the pre-trained policy is further enhanced under the extrinsic reward, R ext with limited timesteps. Moreover, PEAC can integrate flexibly with existing single-embodiment unsupervised RL methods to achieve cross-embodiment exploration and skill discovery, resulting in two combination algorithm examples PEAC-LBS and PEAC-DIAYN.

To verify the versatility and effectiveness of our algorithm, we extensively evaluate PEAC in both simulated and real-world environments. In simulations, we choose state-based / image-based DeepMind Control Suite (DMC) environments extending Unsupervised RL Benchmark (URLB) [27] and different robotic arms in Robosuite [73]. Under these settings, PEAC demonstrates superior few-shot learning ability to downstream tasks, and remarkable generalization ability to unseen embodiments, surpassing existing state-of-the-art unsupervised RL models. Besides, we have evaluated PEAC in real-world Aliengo robots by considering practical joint failure settings based on Isaacgym [37], verifying PEAC's strong adaptability on different joint failures and various real-world terrains.

In summary, the main contributions are as follows:

We propose a novel setting CEURL to enhance agents' adaptability and generalization across diverse embodiments, and then we introduce the Pre-trained Embodiment-Aware Control (PEAC) algorithm for handling CEURL.

We integrate PEAC with existing exploration and skill discovery techniques, designing practical methods and facilitating efficient cross-embodiment exploration and skill discovery.

Extensive experiments show that PEAC not only excels in fast fine-tuning but also effectively generalizes across new embodiments, outperforming current SOTA unsupervised RL models.

2 Related Work

Cross-Embodiment RL. Designing generalizable agents simultaneously controlling diverse embodiments has achieved significant attention in RL. A common strategy involves using expert trajectories [70, 49, 5, 66, 60], internet-scale human videos [3, 58, 13], or offline datasets [29, 59, 8, 41] to train a generalist agent that can handle various tasks across different embodiments. However, these methods are often limited by the need for large-scale, costly datasets and the availability of expert trajectories. Additionally, the discrepancy between open-loop training and closed-loop testing may lead to distribution shifts [32], adversely affecting the final performance. An alternative line of research [28, 36, 64, 4, 66, 52, 12, 68] focuses on training general agents through online interaction across diverse environments. However, these methods treat the embodiment and task as a unified training environment and overlook the role of proprioception, i.e., the internal understanding of an agent's embodiment, which has recently proven to be beneficial for representation learning and optimization in RL [23, 15]. Thus these methods may not fully capture the intrinsic properties of different embodiments by linking knowledge to specific tasks. Emerging research suggests the potential of decoupling the training of embodiment characteristics from task execution, aiming to develop a unified cross-embodiment model. This involves unsupervised pre-training across a variety of embodiments, followed by task-aware fine-tuning, enabling a single agent to adeptly manage both roles effectively.

Unsupervised RL. Unsupervised RL leverages interactions with reward-free environments to extract useful knowledge, such as exploratory policies, diverse skills, or world models [17, 18]. These pre-trained models are utilized to fast adapt to downstream tasks within specific embodiments and environments. Unsupervised RL methods can be categorized into two main types: exploration and skill discovery. Exploration methods aim to maximize state coverage, typically through intrinsic rewards that encourage uncertainty [45, 7, 46, 51, 38, 40, 67] or state entropy [30, 47, 35, 34]. The resulting exploratory trajectories benefit pre-training actor-critic or world models, thereby enhancing fine-tuning efficiency [27]. Skill discovery methods focus on learning an array of distinguishable skills, often by maximizing the mutual information between states and acquired skills [10, 54, 20, 55, 25, 26, 72, 24, 61]. This approach benefits from theoretical insights into the information geometry of skill state distributions, emphasizing the importance of maximizing distances between different skills [11, 22, 42, 43, 44]. Recent efforts also explore incremental skill learning in dynamic environments [53, 31]. Unlike these methods generally focus on single embodiments, we aim to develop generalizable models capable of handling downstream tasks across a variety of embodiments.

3 Cross-Embodiment Unsupervised RL

In this section, we analyze the cross-embodiment RL in an unsupervised manner, which is formulated as our Controlled Embodiment MDP. Then we propose a novel algorithm PEAC to optimize CE-MDP.

3.1 Controlled Embodiment Markov Decision Processes

Cross-embodiment RL can be formulated by contextual MDP [19] with a distribution of Markovian decision processes (MDPs) of {M e } . Cross-embodiment RL hopes to learn shared knowledge from this distribution of MDPs, which is crucial for enhancing the adaptability or generalization of agents across embodiments. However, directly optimizing agents by online interacting with {M e } or utilizing offline datasets sampled from {M e } may learn knowledge not only related to these embodiments but also highly related to these task reward functions in {M e } . This phenomenon may have negative impacts on learning the general knowledge across embodiments or improving the agent's generalization ability. For example, as the agent is required to handle {M e } , it will less explore the trajectories with low rewards. These trajectories, although not optimal for the embodiment in this task, might also include embodiment knowledge and be useful for other tasks. Without extrinsic task rewards, the agent is encouraged to learn embodiment-aware and task-agnostic knowledge, which can effectively adapt to any downstream task across embodiments.

In this paper, we propose to pre-train cross-embodiment agents in reward-free environments to ensure that the agent can learn knowledge only specialized in these embodiments themselves. In other words, we introduce unsupervised RL into cross-embodiment RL as a novel setting: cross-embodiment unsupervised RL (CEURL). As shown in Fig. 1, in CEURL, we first pre-train a general agent by interacting with the reward-free environment through varying embodiments sampled from an unknown embodiment distribution. Given any downstream task represented by the extrinsic reward R ext, the pre-trained agent is subsequently fine-tuned to control these embodiments, and other unseen embodiments from the distribution, to complete this task within limited steps (like one-tenth of the pre-training steps). Formally, we formulate CEURL as the following controlled embodiment MDP (CE-MDP):

Definition 3.1 ( Controlled Embodiment MDP (CE-MDP) ) . A CE-MDP includes a distribution of controlled MDPs defined as M c e = ( S e , A e , P e , γ ) , where e ∼ E and E represents the embodiment distribution. Each embodiment may have different state spaces S e and action spaces A e . P e : S e × A e → ∆( S e ) denoting the transition dynamics for embodiment e and γ is the discount factor. We define the state space S = ∪ e S e and adopt a unified action embedding space A with corresponding action projectors ϕ e : A → A e , which can be fixed or learnable.

Thus we can establish a unified policy π : S → ∆( A ) across all embodiments. For any embodiment e , we sample an action a from π ( ·| s ) for a state s ∈ S e and execute the projected action ϕ e ( a ) . Without loss of generality, we assume ϕ e is fixed and focus our analysis on the policy π . To explain the complexities of CE-MDP with varying embodiment contexts, we extend the single-embodiment information geometry analyses [11] into our cross-embodiment setting. First, we consider the discount state distribution of π within M c e at state s as d e π ( s ) = (1 -γ ) ∑ ∞ t =0 [ γ t P e ( s t = s )] . It is well known that the trajectory return of the state-based reward function can be computed as

Thus, the properties of d e π are significant in determining useful initializations for downstream tasks. We consider the set D e = { d e π ∈ ∆( S ) | ∀ π } , which includes all feasible d e π over the probability simplex. As shown in [11], for each e , D e is a convex set, and any useful policy, which can be optimal for certain downstream tasks under embodiment e , must be a vertex of D e , typically corresponding to deterministic policies.

However, in the context of CEURL, the unknown embodiment context e introduces partial observability [14] and significant differences in the corresponding points of the same skill across different embodiments. In CE-MDP, we consider the entire embodiment space and define d E π ( s ) = E e ∼E [ d e π ( s )] , with D E = { d E π ∈ ∆( S ) | ∀ π } . The primary challenge lies in the high variability of embodiments, which complicates the process of learning a policy that generalizes well across different embodiments. We demonstrate that the vertices of D E may no longer correspond to deterministic policies, as they need to handle all embodiments in the distribution. This significantly heightens the challenge of the pre-training process in CE-MDP, making it more difficult to find useful cross-embodiment skills (proofs and discussion in Appendix A.1).

To solve CEURL under the paradigm of CE-MDP, the agent will collect reward-free trajectories τ = ( s 0 , a 0 , s 1 , ... ) with probability p M c e ,π ( τ ) = P e ( s 0 ) ∏ t =0 π ( a t | s t ) P e ( s t +1 | s t , a t ) via some

sampled embodiments e during the pre-training. These trajectories are then used in CEURL methods to design intrinsic rewards R int for pre-training agents. During fine-tuning, we will sample several embodiments e from E and combine M c e with a downstream task represented by extrinsic rewards R ext, and agents are required to maximize the task return over all embodiments, i.e., E e ∼E [ J M c e , R ext ( π ) ] , within limited steps (like one-tenth or less of the pre-training steps).

3.2 Pre-trained Embodiment-Aware Control

We primarily focus on the pre-training objective of CEURL, specifically determining the optimal pre-trained policy π for CEURL. In the fine-tuning stage, given any downstream task characterized by extrinsic reward R ext, the pre-trained policy π will be optimized into the fine-tuned policy π ∗ with limited steps to handle R ext via some RL algorithms like PPO [50]. Consequently, it is widely assumed that π ∗ will remain close to π during fine-tuning due to constraints on limited interactions with the environment [11]. Our cross-embodiment fine-tuning objective thus combines policy improvement under R ext and a policy constraint evaluated via KL divergence

where β > 0 is the unknown trade-off parameter related to the fine-tuning steps (when fine-tuning steps tend towards infinity, β tends to 0 and this objective converges to the original RL objective), and ¯ M represents the 'average embodiment MDP' satisfying that p ¯ M ,π ( τ ) = E e ∼E [ p M c e ,π ( τ ) ] . During fine-tuning, we hope to optimize π ∗ by maximizing F , i.e., the fine-tuned result is max p M c e ,π ∗ ( τ ) F ( π, π ∗ , R ext , e ) . As the pre-trained policy π needs to handle any downstream task, we consider the worst-case extrinsic reward function across the embodiment distribution, and our cross-embodiment pre-training objective can be formally represented as maximizing

This objective is a min-max problem that is hard to optimize. Fortunately, we can simplify it as below Theorem 3.2 (Proof in Appendix A.2) . The pre-training objective Eq. (3) of ( π, E ) satisfies

Here p ( e ) and p π ( e | τ ) are embodiment prior and posterior probabilities, respectively. This result simplifies our pre-trained objective as a form easy to calculate and optimize. Also, although β is an unknown parameter, the optimal pre-trained policy is independent of β . Based on these analyses, we propose a novel algorithm named Pre-trained Embodiment-Aware Control (PEAC). In PEAC, we first train an embodiment discriminator q θ ( e | τ ) to approximate p π ( e | τ ) , which can learn the embodiment context via historical trajectories. For cross-embodiment pre-training, PEAC then utilizes our cross-embodiment intrinsic reward, which is defined following Eq. (4) as

Assuming the embodiment prior p ( e ) is fixed, R CE encourages the agent to explore the region with low log q θ ( e | τ ) . In these trajectories, the embodiment discriminator is misled, where the agent may not have explored enough or different embodiment posteriors are similar. Thus, the embodiment discriminator can boost itself from these trajectories and learned embodiment-aware contexts that can effectively represent different embodiments, which benefit generalizing to unseen embodiments.

In practice, R CE needs to be calculated for each state s rather than the whole trajectory τ , also, the embodiment discriminator needs to classify the embodiment context for every state. For RL backbones that encode historical information as the hidden state h like Dreamer [17, 18, 64], we directly train q θ ( e | h , s ) as the discriminator and further calculate R CE . For RL algorithms with Markovian policies like PPO [50], we encode a fixed length historical state-action pair to the hidden state h and also train q θ ( e | h , s ) , following [28]. For a fair comparison, our policy still uses Markovian policy and does not utilize encoded historical messages. PEAC's pseudo-code is in Appendix C.

A

4 Cross-Embodiment Exploration and Skill Discovery

As shown above, PEAC pre-trains the agent for the optimal initialization to few-shot handle downstream tasks across embodiments. Besides, although PEAC does not directly explore or discover skills, it is flexible to combine with existing unsupervised RL methods, including exploration and skill discovery ones, to achieve cross-embodiment exploration and skill discovery. Below we will discuss in detail the specific combination between PEAC and these two classes respectively, exporting two practical combination algorithms, PEAC-LBS and PEAC-DIAYN, as examples.

Embodiment-Aware Exploration. Existing exploration methods mainly encourage the agent to explore unseen regions. As PEAC suggests the agent explores the region where the embodiment discriminator is wrong, it is natural to directly combine R CE and exploration intrinsic rewards to achieve cross-embodiment exploration, i.e., balancing embodiment representation learning and unseen state exploration. As an example, we take LBS [38], of which the intrinsic reward is the KL divergence between the latent prior and the approximation posterior, as the PEAC-LBS. As R CE and R LBS are both related to some KL divergence, we can directly add up these two intrinsic rewards with the same weight in PEAC-LBS, of which the detailed pseudo-code is in Appendix C.

Embodiment-Aware Skill Discovery. Single-embodiment skill-discovery mainly maximizes the mutual information between trajectories τ and skills z as I ( τ ; z ) = D KL ( p ( τ, z ) ∥ p ( τ ) p ( z )) [10], which has been shown as optimal initiation to some skill-based adaptation objective [11]. We combine it and our cross-embodiment fine-tuning objective Eq. (2) to propose a unified cross-embodiment skill-based adaptation objective as

Similar to Theorem 3.2, we can define our pre-training objective and simplify it as

The proof of Eq. (7) is in Appendix A.3, where we also show it is a general form of Theorem 3.2 and the single-embodiment skill-discovery result [11]. The result of Eq. (7) includes two terms for handling cross-embodiment and discovering skills respectively. In detail, the first term is the same as the objective in Eq. (4), thus we can directly optimize it via PEAC. As the second term is similar to the classical skill-discover objective I ( τ ; z ) but only embodiment-aware, we can extend existing skill-discovery methods into an embodiment-aware version for handling it.

We take DIAYN [10] as an example, resulting in PEAC-DIAYN. Overall, In the pre-training stage, given a random skill z and an embodiment e , we will sample trajectories with the policy π θ ( a | s , z , e ) that is conditioned on z and the predicted embodiment context. Then we will train a neural network p ( z , e | τ ) to jointly predict the current skill and the embodiment. For training the policy, we combine R CE and R DIAYN as the intrinsic reward. During fine-tuning, we utilize the embodiment discriminator, mapping observed trajectories to infer the embodiment context. We then train an embodiment-aware meta-controller π ( z | e , τ ) , which inputs the state and predicted context and then outputs the skill. It extends existing embodiment-agnostic meta-controller [39] and directly chooses from skill spaces rather than complicated action spaces. The pseudo-code of PEAC-DIAYN is in Appendix C.

0.45

0.60

PEAC (Ours)

BeCL

CIC

APS

SMM

DIAYN

LBS

ProtoRL

Disagreement

RND

ICM

Median (

)

0.30

0.45

0.60

0.75

IQM (

)

0.4

0.5

0.6

0.7

Mean (

)

0.3

0.4

0.5

0.6

Optimality Gap (

)

Expert Normalized Score

Expert Normalized Score

0.60

0.75

0.90

PEAC-LBS (Ours)

Choreographer

LBS

APT

Plan2Explore

RND

ICM

PEAC-DIAYN (Ours)

CIC

LSD

APS

DIAYN

Median (

)

0.60

0.75

0.90

IQM (

)

0.60

0.75

0.90

Mean (

)

0.15

0.30

0.45

Optimality Gap (

)

5 Experiments

We now present extensive empirical results to answer the following questions:

Does PEAC enhance the cross-embodiment unsupervised pre-training for handling different downstream tasks? (Sec. 5.2)

Can CEURL benefit cross-embodiment RL and effectively generalize to unseen embodiments? (Sec. 5.3)

Does CEURL advantage to real-world cross-embodiment applications? (Sec. 5.4)

5.1 Experimental Setup

To fully evaluate PEAC in CEURL, we choose extensive benchmarks (Fig. 2), including state-based / image-based Deepmind Control Suite (DMC) [56] in URLB [27], Robosuite [73, 69] for robotic manipulation, and Isaacgym [37] for simulation as well as real-world legged locomotion. Below we will introduce embodiments, tasks, and baselines for these settings, with more details in Appendix B.

State-based DMC & Image-based DMC. These two benchmarks extend URLB [27], classical single-embodiment unsupervised RL settings. Based on basic embodiments, we change the mass or damping to conduct three distinct embodiment distributions: walker-mass, quadruped-mass, and quadruped-damping, following previous work with diverse embodiments [28, 65]. All downstream tasks follow URLB. These two settings take robot states and images as observations respectively.

In state-based DMC, we compare PEAC with 5 exploration and 5 skill-discovery methods: ICM [45], RND [7], Disagreement [46], ProtoRL [62], LBS [38], DIAYN [10], SMM [30], APS [34], CIC [26], and BeCL [61], which are standard and SOTA for this setting. For all methods, we take DDPG [33] as the RL backbone, which is widely used in this benchmark [27]. In image-based DMC, we take 5 exploration baselines: ICM, RND, Plan2Explore [51], APT [35], and LBS; as well as 4 skill-discovery baselines: DIAYN, APS, LSD [42], and CIC. Also, we choose a SOTA baseline Choreographer [39], which combines exploration and skill discovery. For all methods, we take DreamerV2 [18] as the backbone algorithm, which has currently shown leading performance in this benchmark [48].

Robosuite. Wefurther consider embodiment distribution with greater change: different robotic arms for manipulation tasks from Robosuite [73]. We pre-train our agents in robotic arms Panda, IIWA, and Kinova3. Besides, we take robotic arm Jaco for evaluating generalization. Following [69], we take DrQ [63] as the RL backbone and choose standard task settings: Door, Lift, and TwoArmPegInHole.

Table 1: Results of

Robosuite and

Isaacgym

.

Isaacgym. To explore CEURL in realistic environments, we design embodiment distributions based on the Unitree A1 robot in Isaacgym simulation [37], which is widely used for the real-world legged robot control [1, 74]. As A1 owns 12 controllable joints, we design A1-disabled, a uniform distribution of 12 embodiments, each with a joint failure, respectively. It is realistic as robots may damage some joints when deploying in the real world, and they are still required to complete tasks to their best. We choose standard RL backbone PPO [50] and five downstream tasks: run, climb, leap, crawl, and tilt, following [74]. We take classical baselines for Robosuite and Isaacgym: ICM, RND, and LBS. Besides, we have deployed Aliengo robots with different failure joints to evaluate the effectiveness of PEAC in real-world applications.

5.2 Evaluation of PEAC

State-based DMC. We first report results in state-based DMC to show that PEAC can facilitate crossembodiment pre-training. All algorithms, repeated 10 random seeds, are pre-trained 2M timesteps in reward-free environments with different embodiments, followed by fine-tuned downstream tasks for all these embodiments with 100k timesteps. We train DDPG agents for each downstream task 2M steps to get the expert return and calculate the expert normalized score for each method. Following [2], in Fig. 3, we report mean, median, interquartile mean (IQM), and optimality gap (OG) metrics along with stratified bootstrap confidence intervals. Fig. 3 demonstrates that PEAC substantially outperforms other baselines on all metrics, indicating that our cross-embodiment intrinsic reward contributes positively to downstream tasks across different embodiments. Notably, compared with BeCL and CIC which get the second and third scores, PEAC not only has higher performance but also a smaller confidence interval, highlighting its stability. Appendix B.4 reports detailed results of these statistics (Table 8) and individual results for each downstream task (Table 9).

Image-based DMC. As described in Sec. 4, PEAC can flexibly combine with existing unsupervised RL methods. To verify it, we evaluate PEAC-LBS and PEAC-DIAYN in image-based DMC. The pre-training and fine-tuning steps are still 2M and 100k respectively. Also, we present four metrics: Median, IQM, Mean, and OG with stratified bootstrap confidence intervals in Fig. 4. Taking IQM as our primary metric, PEAC-LBS not only has a higher value but also a relatively smaller confidence interval, indicating its better stability. As mentioned in [39], pure skill-discovery methods like DIAYN struggle on this benchmark with a certain gap compared to exploratory method. The phenomenon seems more pronounced in cross-embodiment setting than single-embodiment setting, which might be because of the increased difficulty of finding consistent skills across embodiments. As PEAC-DIAYN discovers skills across-embodiment, it consistently leads in performance compared with all other pure skill discovery methods across all four statistics. In Appendix B.5, we report detailed results of these statistics in Table 10 and detailed results for all downstream tasks in Table 11.

100k

500k

1M

2M

Pretraining Steps

0.3

0.4

0.5

0.6

0.7

0.8

0.9

Expert Normalized Score

PEAC-LBS (Ours)

Choreographer

LBS

APT

Plan2Explore

RND

ICM

100k

500k

1M

2M

Pretraining Steps

0.30

0.35

0.40

0.45

0.50

0.55

0.60

0.65

0.70

Expert Normalized Score

PEAC-DIAYN (Ours)

CIC

LSD

APS

DIAYN

Cross-Embodiment: different failure joints

Robosuite. Besides, we validate PEAC in a more challenging setting Robosuite where different embodiments own different robotic arms (subfigures 3-6 in Fig. 2). As shown in Table 1, PEAC still significantly outperforms all baselines in both training and testing embodiments, demonstrating its powerful cross-embodiment ability and better generalization ability. The detailed results of each robotic arm are in Table 12 of Appendix B.6.

Ablation studies. We do several ablation studies in image-based DMC to clarify the contribution of PEAC better. First, we evaluate the effectiveness of pre-trained steps in fine-tuned performance. We pre-train agents for 100k, 500k, 1M, and 2M steps and then fine-tune them for 100k steps. As shown in Fig. 6, all algorithms improve with pre-training timesteps increasing, indicating that cross-embodiment pre-training effectively benefits fast handling downstream tasks. PEAC-LBS becomes the best-performing method from 1M steps on and PEAC-DIAYN significantly exceeds skill discovery methods. This suggests that PEAC excels at handling cross-embodiment tasks with increased pre-training steps. Additional results are in Appendix B.7. Besides pre-training steps, we also do more ablations studies of different components in PEAC to verify their effectiveness in Appendix B.8. For example, we evaluate the stability of PEAC-LBS in DMC-image under different β (we set it as 1.0 in all main experiments), which is the trade-off parameter for balancing the policy improvement term and the policy constraint term. Moreover, we also do an ablation study on our embodiment discriminator to verify the contribution of each component in our PEAC. More results and analyses are in Appendix B.8.

5.3 Generalization to Unseen Embodiments

To answer the second question, we further assess the generalization ability of PEAC to unseen embodiments. First, we directly leverage pre-trained agents to zero-shot sample trajectories with different unseen embodiments and then visualize results through t-SNE [57] in Fig. 5, where different colored points represent states sampled via different embodiments. As shown in Fig. 5, PEAC-LBS can distinguish different embodiments' states more effectively compared to LBS, which is difficult to distinguish them (more results are in Appendix B.9). Furthermore, we evaluate the generalization ability of fine-tuned agents for all methods by zero-shot evaluating them with unseen embodiments and the same downstream task. In Table 2, we report the detailed generalization results of all 3 domains about state-based DMC and image-based DMC. The results demonstrate that the fine-tuned agents of PEAC can successfully handle the same downstream task with unseen embodiments, which illustrates that PEAC effectively learns cross-embodiment knowledge. Detailed results for each downstream task are in Appendix B.10 (Table 16-17).

5.4 Real-World Applications

To validate CEURL in more realistic settings, we conduct results based on legged locomotion in Isaacgym, which is widely used for real-world applications. First, we present simulation results of A1-disabled in Table 1, with 100M pre-train timesteps and 10M fine-tune timesteps. As shown in Table 1, PEAC effectively establishes a good initialization model across embodiments with different joint failures and quickly adapts to downstream tasks, especially for challenging climb and leap tasks.

Besides, we have deployed PEAC fine-tuned agents in real-world Aliengo-disabled robots, i.e., Aliengo robots with different failure joints. As shown in Fig. 7, due to joint failure, the movement ability of the robot is limited compared to normal settings, but the robot still demonstrates strong adaptability on various terrains not seen in simulators. More images and videos of real-world applications are in Appendix B.12.

5.5 Limitations and Discussion

In terms of limitations, we assume that different embodiments may own similar structures so that we can pre-train a unified agent for them. As a result, it might be challenging for PEAC to handle extremely different embodiments. Also, existing unsupervised RL methods still struggle to handle more challenging downstream tasks. In Appendix B.11, we take the first step to evaluate several more challenging downstream tasks and more different embodiment distributions, of which the results show that PEAC can still perform better than baselines. Designing more efficient cross-embodiment unsupervised algorithms for these more difficult and practical settings are interesting future directions. The Broader Impact os this work is discussed in Appendix D.

6 Conclusion

In this work, we propose to analyze cross-embodiment RL in an unsupervised RL perspective as CEURL, i.e., pre-training in an embodiment distribution. We formulate it as CE-MDP, with some more challenging properties than the single-embodiment setting. By analyzing the optimal cross-embodiment initialization, we propose PEAC with a principled intrinsic reward function and further show that PEAC can flexibly combine with existing unsupervised RL. Experimental results demonstrate that PEAC can effectively handle downstream tasks across embodiments for extensive settings, ranging from image-based observation, state-based observation, and real-world legged locomotion. We hope this work can encourage further research in developing RL agents for both task generalization and embodiment generalization, especially in real-world control.

Acknowledgments and Disclosure of Funding

This work was supported by NSFC Projects (Nos. 92248303, 92370124, 62350080, 62276149, 62061136001), BNRist (BNR2022RC01006), Tsinghua Institute for Guo Qiang, and the High Performance Computing Center, Tsinghua University. J. Zhu was also supported by the XPlorer Prize.

References

Ananye Agarwal, Ashish Kumar, Jitendra Malik, and Deepak Pathak. Legged locomotion in challenging terrains using egocentric vision. In Conference on robot learning , pages 403-415. PMLR, 2023.

Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Marc Bellemare. Deep reinforcement learning at the edge of the statistical precipice. Advances in neural information processing systems , 34:29304-29320, 2021.

Bowen Baker, Ilge Akkaya, Peter Zhokov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune. Video pretraining (vpt): Learning to act by watching unlabeled online videos. Advances in Neural Information Processing Systems , 35:24639-24654, 2022.

Michael Beukman, Devon Jarvis, Richard Klein, Steven James, and Benjamin Rosman. Dynamics generalisation in reinforcement learning via adaptive context-aware policies. In Thirty-seventh Conference on Neural Information Processing Systems , 2023.

Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15818 , 2023.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.

Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In International Conference on Learning Representations , 2018.

Yevgen Chebotar, Quan Vuong, Karol Hausman, Fei Xia, Yao Lu, Alex Irpan, Aviral Kumar, Tianhe Yu, Alexander Herzog, Karl Pertsch, et al. Q-transformer: Scalable offline reinforcement learning via autoregressive q-functions. In Conference on Robot Learning , pages 3909-3928. PMLR, 2023.

Thomas M Cover and Joy A Thomas. Elements of information theory. 1991.

Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. In International Conference on Learning Representations , 2018.

Benjamin Eysenbach, Ruslan Salakhutdinov, and Sergey Levine. The information geometry of unsupervised reinforcement learning. In International Conference on Learning Representations , 2021.

Gilbert Feng, Hongbo Zhang, Zhongyu Li, Xue Bin Peng, Bhuvan Basireddy, Linzhu Yue, Zhitao Song, Lizhi Yang, Yunhui Liu, Koushil Sreenath, et al. Genloco: Generalized locomotion controllers for quadrupedal robots. In Conference on Robot Learning , pages 1893-1903. PMLR, 2023.

Dibya Ghosh, Chethan Anand Bhateja, and Sergey Levine. Reinforcement learning from passive data via latent intentions. In International Conference on Machine Learning , pages 11321-11339. PMLR, 2023.

Dibya Ghosh, Jad Rahme, Aviral Kumar, Amy Zhang, Ryan P Adams, and Sergey Levine. Why generalization in rl is difficult: Epistemic pomdps and implicit partial observability. Advances in Neural Information Processing Systems , 34:25502-25515, 2021.

Kevin Gmelin, Shikhar Bahl, Russell Mendonca, and Deepak Pathak. Efficient rl via disentangled environment and agent representations. In International Conference on Machine Learning , pages 1152511545. PMLR, 2023.

Agrim Gupta, Silvio Savarese, Surya Ganguli, and Li Fei-Fei. Embodied intelligence via learning and evolution. Nature communications , 12(1):5721, 2021.

Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In International Conference on Learning Representations , 2019.

Danijar Hafner, Timothy P Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. In International Conference on Learning Representations , 2020.

Assaf Hallak, Dotan Di Castro, and Shie Mannor. Contextual markov decision processes. arXiv preprint arXiv:1502.02259 , 2015.

Steven Hansen, Will Dabney, Andre Barreto, David Warde-Farley, Tom Van de Wiele, and Volodymyr Mnih. Fast task inference with variational intrinsic successor features. In International Conference on Learning Representations , 2019.

Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 16000-16009, 2022.

Shuncheng He, Yuhang Jiang, Hongchang Zhang, Jianzhun Shao, and Xiangyang Ji. Wasserstein unsupervised reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 6884-6892, 2022.

Edward S Hu, Kun Huang, Oleh Rybkin, and Dinesh Jayaraman. Know thyself: Transferable visual control policies through robot-awareness. In International Conference on Learning Representations , 2021.

Zheyuan Jiang, Jingyue Gao, and Jianyu Chen. Unsupervised skill discovery via recurrent skill training. Advances in Neural Information Processing Systems , 35:39034-39046, 2022.

Jaekyeom Kim, Seohong Park, and Gunhee Kim. Unsupervised skill discovery with bottleneck option learning. In International Conference on Machine Learning , pages 5572-5582. PMLR, 2021.

Michael Laskin, Hao Liu, Xue Bin Peng, Denis Yarats, Aravind Rajeswaran, and Pieter Abbeel. Unsupervised reinforcement learning with contrastive intrinsic control. Advances in Neural Information Processing Systems , 35:34478-34491, 2022.

Michael Laskin, Denis Yarats, Hao Liu, Kimin Lee, Albert Zhan, Kevin Lu, Catherine Cang, Lerrel Pinto, and Pieter Abbeel. Urlb: Unsupervised reinforcement learning benchmark. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2) , 2021.

Kimin Lee, Younggyo Seo, Seunghyun Lee, Honglak Lee, and Jinwoo Shin. Context-aware dynamics model for generalization in model-based reinforcement learning. In International Conference on Machine Learning , pages 5757-5766. PMLR, 2020.

Kuang-Huei Lee, Ofir Nachum, Mengjiao Sherry Yang, Lisa Lee, Daniel Freeman, Sergio Guadarrama, Ian Fischer, Winnie Xu, Eric Jang, Henryk Michalewski, et al. Multi-game decision transformers. Advances in Neural Information Processing Systems , 35:27921-27936, 2022.

Lisa Lee, Benjamin Eysenbach, Emilio Parisotto, Eric Xing, Sergey Levine, and Ruslan Salakhutdinov. Efficient exploration via state marginal matching. arXiv preprint arXiv:1906.05274 , 2019.

Sang-Hyun Lee and Seung-Woo Seo. Unsupervised skill discovery for learning shared structures across changing environments. In International Conference on Machine Learning , pages 19185-19199. PMLR, 2023.

Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 , 2020.

Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971 , 2015.

Hao Liu and Pieter Abbeel. Aps: Active pretraining with successor features. In International Conference on Machine Learning , pages 6736-6747. PMLR, 2021.

Hao Liu and Pieter Abbeel. Behavior from the void: Unsupervised active pre-training. Advances in Neural Information Processing Systems , 34:18459-18473, 2021.

Xin Liu, Yaran Chen, Haoran Li, Boyu Li, and Dongbin Zhao. Cross-domain random pre-training with prototypes for reinforcement learning. arXiv preprint arXiv:2302.05614 , 2023.

Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, et al. Isaac gym: High performance gpu based physics simulation for robot learning. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2) , 2021.

Pietro Mazzaglia, Ozan Catal, Tim Verbelen, and Bart Dhoedt. Curiosity-driven exploration via latent bayesian surprise. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 7752-7760, 2022.

Pietro Mazzaglia, Tim Verbelen, Bart Dhoedt, Alexandre Lacoste, and Sai Rajeswar. Choreographer: Learning and adapting skills in imagination. In The Eleventh International Conference on Learning Representations , 2022.

Mirco Mutti, Mattia Mancassola, and Marcello Restelli. Unsupervised reinforcement learning in multiple environments. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 7850-7858, 2022.

Fei Ni, Jianye Hao, Yao Mu, Yifu Yuan, Yan Zheng, Bin Wang, and Zhixuan Liang. Metadiffuser: Diffusion model as conditional planner for offline meta-rl. In International Conference on Machine Learning , pages 26087-26105. PMLR, 2023.

Seohong Park, Jongwook Choi, Jaekyeom Kim, Honglak Lee, and Gunhee Kim. Lipschitz-constrained unsupervised skill discovery. arXiv preprint arXiv:2202.00914 , 2022.

Seohong Park, Kimin Lee, Youngwoon Lee, and Pieter Abbeel. Controllability-aware unsupervised skill discovery. arXiv preprint arXiv:2302.05103 , 2023.

Seohong Park, Oleh Rybkin, and Sergey Levine. Metra: Scalable unsupervised rl with metric-aware abstraction. arXiv preprint arXiv:2310.08887 , 2023.

Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International conference on machine learning , pages 2778-2787. PMLR, 2017.

Deepak Pathak, Dhiraj Gandhi, and Abhinav Gupta. Self-supervised exploration via disagreement. In International conference on machine learning , pages 5062-5071. PMLR, 2019.

Vitchyr Pong, Murtaza Dalal, Steven Lin, Ashvin Nair, Shikhar Bahl, and Sergey Levine. Skew-fit: State-covering self-supervised reinforcement learning. In International Conference on Machine Learning , pages 7783-7792. PMLR, 2020.

Sai Rajeswar, Pietro Mazzaglia, Tim Verbelen, Alexandre Piché, Bart Dhoedt, Aaron Courville, and Alexandre Lacoste. Mastering the unsupervised reinforcement learning benchmark from pixels. In International Conference on Machine Learning , pages 28598-28617. PMLR, 2023.

Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gómez Colmenarejo, Alexander Novikov, Gabriel Barth-maron, Mai Giménez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al. A generalist agent. Transactions on Machine Learning Research , 2022.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017.

Ramanan Sekar, Oleh Rybkin, Kostas Daniilidis, Pieter Abbeel, Danijar Hafner, and Deepak Pathak. Planning to explore via self-supervised world models. In International Conference on Machine Learning , pages 8583-8592. PMLR, 2020.

Milad Shafiee, Guillaume Bellegarda, and Auke Ijspeert. Manyquadrupeds: Learning a single locomotion policy for diverse quadruped robots. arXiv preprint arXiv:2310.10486 , 2023.

Nur Muhammad Mahi Shafiullah and Lerrel Pinto. One after another: Learning incremental skills for a changing world. In International Conference on Learning Representations , 2021.

Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics-aware unsupervised discovery of skills. In International Conference on Learning Representations , 2019.

DJ Strouse, Kate Baumli, David Warde-Farley, Volodymyr Mnih, and Steven Stenberg Hansen. Learning more skills through optimistic exploration. In International Conference on Learning Representations , 2021.

Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv:1801.00690 , 2018.

Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research , 9(11), 2008.

Mengda Xu, Zhenjia Xu, Cheng Chi, Manuela Veloso, and Shuran Song. Xskill: Cross embodiment skill discovery. In Conference on Robot Learning , pages 3536-3555. PMLR, 2023.

Yifan Xu, Nicklas Hansen, Zirui Wang, Yung-Chieh Chan, Hao Su, and Zhuowen Tu. On the feasibility of cross-task transfer with model-based reinforcement learning. In The Eleventh International Conference on Learning Representations , 2022.

Jonathan Yang, Catherine Glossop, Arjun Bhorkar, Dhruv Shah, Quan Vuong, Chelsea Finn, Dorsa Sadigh, and Sergey Levine. Pushing the limits of cross-embodiment learning for manipulation and navigation. arXiv preprint arXiv:2402.19432 , 2024.

Rushuai Yang, Chenjia Bai, Hongyi Guo, Siyuan Li, Bin Zhao, Zhen Wang, Peng Liu, and Xuelong Li. Behavior contrastive learning for unsupervised skill discovery. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Research , pages 39183-39204. PMLR, 23-29 Jul 2023.

Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Reinforcement learning with prototypical representations. In International Conference on Machine Learning , pages 11920-11931. PMLR, 2021.

Denis Yarats, Ilya Kostrikov, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. In International conference on learning representations , 2020.

Chengyang Ying, Zhongkai Hao, Xinning Zhou, Hang Su, Songming Liu, Dong Yan, and Jun Zhu. Task aware dreamer for task generalization in reinforcement learning. arXiv preprint arXiv:2303.05092 , 2023.

Chengyang Ying, Xinning Zhou, Hang Su, Dong Yan, Ning Chen, and Jun Zhu. Towards safe reinforcement learning via constraining conditional value-at-risk. arXiv preprint arXiv:2206.04436 , 2022.

Chen Yu, Weinan Zhang, Hang Lai, Zheng Tian, Laurent Kneip, and Jun Wang. Multi-embodiment legged robot control as a sequence modeling problem. In 2023 IEEE International Conference on Robotics and Automation (ICRA) , pages 7250-7257. IEEE, 2023.

Mingqi Yuan, Bo Li, Xin Jin, and Wenjun Zeng. Automatic intrinsic reward shaping for exploration in deep reinforcement learning. arXiv preprint arXiv:2301.10886 , 2023.

Zhecheng Yuan, Tianming Wei, Shuiqi Cheng, Gu Zhang, Yuanpei Chen, and Huazhe Xu. Learning to manipulate anywhere: A visual generalizable framework for reinforcement learning. arXiv preprint arXiv:2407.15815 , 2024.

Zhecheng Yuan, Sizhe Yang, Pu Hua, Can Chang, Kaizhe Hu, and Huazhe Xu. Rl-vigen: A reinforcement learning benchmark for visual generalization. Advances in Neural Information Processing Systems , 36, 2024.

Kevin Zakka, Andy Zeng, Pete Florence, Jonathan Tompson, Jeannette Bohg, and Debidatta Dwibedi. Xirl: Cross-embodiment inverse reinforcement learning. In Conference on Robot Learning , pages 537-546. PMLR, 2022.

Amy Zhang, Shagun Sodhani, Khimya Khetarpal, and Joelle Pineau. Learning robust state abstractions for hidden-parameter block mdps. arXiv preprint arXiv:2007.07206 , 2020.

Andrew Zhao, Matthieu Lin, Yangguang Li, Yong-Jin Liu, and Gao Huang. A mixture of surprises for unsupervised reinforcement learning. Advances in Neural Information Processing Systems , 35:2607826090, 2022.

Yuke Zhu, Josiah Wong, Ajay Mandlekar, Roberto Martín-Martín, Abhishek Joshi, Soroush Nasiriany, and Yifeng Zhu. robosuite: A modular simulation framework and benchmark for robot learning. arXiv preprint arXiv:2009.12293 , 2020.

Ziwen Zhuang, Zipeng Fu, Jianren Wang, Christopher G Atkeson, Sören Schwertfeger, Chelsea Finn, and Hang Zhao. Robot parkour learning. In 7th Annual Conference on Robot Learning , 2023.

A Proof of Theorems

In this section, we will provide detailed proof of theorems in the paper.

A.1 Properties and Challenges of D E

We first construct an example to show that vertices of D E may no longer be deterministic policies.

Considering a simple embodiment distribution with only 2 embodiments e 1 , e 2 with the embodiment probability p ( e 1 ) = p ( e 2 ) = 1 2 . For each embodiment, there are two states s 1 , s 2 and two actions a 1 , a 2 and the dynamic is

In this setting, there are four deterministic policies:

For any policy µ , we denote that ρ 1 ,µ , ρ 2 ,µ are the state distribution of µ under the environment P e 1 or P e 2 respectively. Then we can calculate that

As the embodiment probability is p ( e 1 ) = p ( e 2 ) = 1 2 , all these four policy share the same state distribution as

Furthermore, we consider a stochastic policy π satisfies that

Next we will calculate ρ 1 ,π and ρ 2 ,π . For ρ 1 ,π , at the timestep 0, we have the initial state distribution as p 0 ( s 1 ) = p 0 ( s 2 ) = 1 2 , assume that at timestep t we have corresponding p t ( s 1 ) , p t ( s 2 ) , we can naturally get the recurrence relation as

Naturally, we have p t ( s 1 ) = 1 -1 2 t , p t ( s 2 ) = 1 2 t and thus the discount state distribution of s 2 is

And we have

Similarly, we can calculate ρ 2 ,π . At the timestep 0, we have the initial state distribution as p 0 ( s 1 ) = p 0 ( s 2 ) = 1 2 , assume that at timestep t we have corresponding p t ( s 1 ) , p t ( s 2 ) , we can naturally get the recurrence relation as

As p t ( s 1 ) + p t ( s 2 ) = 1 , we can solve this recurrence relation via

Thus the discount state distribution of s 1 is

And we have

As the embodiment probability is p ( e 1 ) = p ( e 2 ) = 1 2 , the state distribution of π is

Taking any γ ∈ (0 , 1) , it is obvious that ρ π is not within the closure composed of ρ π 1 , ρ π 2 , ρ π 3 , ρ π 4 (actually the point (1/2,1/2)). Thus we have explained that the vertices of D E might no longer be simple deterministic policies.

A.2 Proof of Theorem 3.2

Proof. Recall that

We set a functional f satisfying that

Using the calculus of variations, we can calculate its optimal value at the point p ∗ satisfying that

here b is a constant not related to p ∗ , and we have p ∗ ( τ ) = p ¯ M ,π ( τ ) e R ext ( τ ) β -b . As ∫ p ∗ ( τ ) = 1 , we can calculate that

Consequently, we have

Similarly, we set a functional g satisfying that

Using the calculus of variations, we can calculate its optimal value at the point r ∗ satisfying that

Thus we can calculate that

Thus we have proven this result.

A.3 Detailed Discussion and Proof about Embodiment-Aware Skill Discovery

Here we discuss our cross-embodiment skill-based adaptation objective .

We begin by proving Eq. (7) . First, we show that

Our proof is similar to the proof in Appendix A.2. Recall that

Similar to Eq. (22)-Eq. (25), we have

Also, similar to Eq. (26)-Eq. (28), we have

Thus we have

where the last equality holds from the fact that the maximum is achieved when putting all the probability weight on the input z maximizing D KL ( p M c e ,π ( τ | z ) ∥ p ¯ M ,π ( τ ) ) .

Next, we will show that D KL ( p M c e ,π ( τ | z ) ∥ p ¯ M ,π ( τ ) ) is a general form of our Theorem 3.2 and the results in the single-embodiment setting [11]. Naturally, when we ignore z , F s ( π, π ∗ , R ext , e ) will degenerate into F ( π, π ∗ , R ext , e ) , and Eq. (7) will also degenerate into Eq. (4), i.e., the results in Theorem 3.2. On the other hand, if we change Eq. (7) into the single-embodiment setting, i.e., E is a Dirac distribution with the probability p ( e ) = 1 for some fixed e , then we have

the last approximation simplifies the complex coupling relationship between π and z , following [11]. Furthermore, by Lemma 6.5 in [11] (proof in Theorem 13.1.1 from [9]), we have

which is the objective of existing single-embodiment skill-discovery methods.

Finally, we will Eq. (7) , which further indicates that our cross-embodiment skill-based objective can be decomposed into two terms: one for handling cross-embodiment while the other aims at discovering skills. Actually, we have

B Experimental Details

In this section, we will introduce more detailed information about our experiments. In Sec. B.1, we introduce the detailed environments and tasks used in our experiments. In Sec. B.2, we will illustrate all the baselines compared in experiments. Also, all hyper-parameters of experiments are in Sec. B.3. Moreover, we supplement more detailed experimental results about state-based DMC, image-based DMC, and Robosuite in Sec. B.4, Sec. B.5, and Sec. B.6, respectively. Then we conduct detailed generalization results of pre-trained models and fine-tuned models in Sec. B.9 and Sec. B.10, respectively. Finally, we report more detailed real-world experiments in Sec. B.12.

B.1 Embodiments and Tasks

State-based DMC. This benchmark is based on DMC [56] and URLB [27] with state-based observation. Each domain contains one robot and four downstream tasks. We extend it into the cross-embodiment settings: Walker-mass, Quadruped-mass, and Quadruped-damping. Walkermass extends the Walker robot in DMC, which is a two-leg robot, and designs a distribution with different mass m , i.e., m times the mass of a standard walker robot. Similarly, Quadruped-mass also considers quadruped robots with different mass m . Quadruped-damping, on the other hand, changes the damping of the standard quadruped robot with l times. The detailed parameters of training embodiments and generalization embodiments are in Table 3.

Image-based DMC. This benchmark is the same with state-based DMC but with image-based observation. Thus we consider similar three embodiment distributions: Walker-mass, Quadrupedmass, and Quadruped-damping.

Robosuite. This benchmark utilizes the environment in [73] and follows the experimental setting in RL-Vigen [69], of which the cross-embodiment setting includes Panda, IIWA, and Kinova3. Here different embodiments may own different shapes (observations), and dynamics. Similarly, we pretrain cross-embodiments in all these three embodiments and fast fine-tune the pre-trained agents to downstream tasks. Besides these three embodiments, we also directly fine-tune our pre-trained models in one unseen embodiment: Jaco, to validate the cross-embodiment generalization ability of CEURL. For task sets, we consider three widely used tasks: Door, Lift, and TwoArmPegInHole. Noticing that although these three tasks can be finished by the same robots, their demand for robotic arms varies a lot. For example, TwoArmPegInHole needs two robotic arms but the other two tasks only need one. Consequently, we pre-train cross-embodiment agents for each single task, for all methods.

Isaacgym. We first design a setting in simulation based on Unitree A1 in Isaacgym, which is a challenging legged locomotion task and is widely used for real-world legged locomotion. The action space of A1 is a 12-dimension vector, representing 12 joint torque. Thus we consider our A1-disabled benchmark, including 12 embodiments, each of which owns a joint torque failure, i.e., the torque output of this joint is always 0 in this embodiment. This setting is practical as our robot may experience partial joint failure during use, and we still hope that it can complete the task as much as possible.

Moreover, we deploy PEAC into real-world Aliengo robots with failure joints. Similarly, we consider the embodiment distribution Aliengo-disabled, which owns 12 embodiments, each of which owns a joint torque failure respectively. We first pre-train a unified agent across these 12 embodiment in reward-free environments. During fine-tuning, for each embodiment, we utilize the same pre-trained agent to fine-tune the given moving task through this embodiment. Finally, we deploy the fine-tuned agent into the real-world setting to evaluate its movement ability under different kinds of terrains with joint failure.

B.2 Baselines and Implementations

ICM [45]. Intrinsic Curiosity Module (ICM) designs intrinsic rewards as the divergence between the projected state representations in a feature space and the estimations made by a feature dynamics model.

RND [7]. Random Network Distillation (RND) utilizes a predictor network's error in imitating a randomly initialized target network to generate intrinsic rewards, enhancing exploration in learning environments.

Disagreement [46] / Plan2Explore [51]. The Disagreement algorithm leverages prediction variance across multiple models to estimate state uncertainty, guiding exploration towards less certain states. The Plan2Explore algorithm employs a self-supervised, world-model-based framework, using model disagreement to assess environmental uncertainty and incentivize exploration in sparse-reward scenarios.

ProtoRL [62]. Proto-RL combines representation learning and exploration through a selfsupervised learning framework, using prototype representations to pre-train task-independent representations in the environment, effectively improving policy learning in continuous control tasks.

APT [35]. Active Pre-training (APT) estimates entropy for a given state using a particle-based estimator based on the K nearest-neighbors algorithm.

LBS [38]. Latent Bayesian Surprise (LBS) applies Bayesian surprise within a latent space, efficiently facilitating exploration by measuring the disparity between an agent's prior and posterior beliefs about system dynamics.

Choreographer [39]. Choreographer is a model-based approach in unsupervised skill learning that employs a world model for skill acquisition and adaptation, distinguishing exploration from skill learning and leveraging a meta-controller for efficient skill adaptation in simulated scenarios, enhancing adaptability to downstream tasks and environmental exploration.

DIAYN [10]. Diversity is All You Need (DIAYN) autonomously learns a diverse set of skills by maximizing mutual information between states and latent skills, using a maximum entropy policy.

SMM [30]. State Marginal Matching (SMM) develops a task-agnostic exploration strategy by learning a policy to match the state distribution of an agent with a given target state distribution.

APS [34]. Active Pre-training with Successor Feature (APS) maximizes the mutual information between states and task variables by reinterpreting and combining variational successor features with nonparametric entropy maximization.

LSD [42]. Lipschitz-constrained Skill Discovery (LSD) adopts a Lipschitz-constrained state representation function, ensuring that maximizing this objective in the latent space leads to an increase in traveled distances or variations in the state space, thereby enabling the discovery of more diverse, dynamic, and far-reaching skills.

CIC [26]. Contrastive Intrinsic Control (CIC) is an unsupervised reinforcement learning algorithm that leverages contrastive learning to maximize the mutual information between state transitions and latent skill vectors, subsequently maximizing the entropy of these embeddings as intrinsic rewards to foster behavioral diversity.

BeCL [61]. Behavior Contrastive Learning (BeCL) utilizes contrastive learning for unsupervised skill discovery, defining its reward function based on the mutual information between states generated by the same skill.

Next, we will introduce the implementations of baselines for all experimental settings.

For state-based DMC , almost all baselines (ICM, RND, Disagreement, ProtoRL, DIAYN, SMM, APS) combined with RL backbone DDPG are directly following the official implementation in urlb ( https://github.com/rll-research/url_benchmark ). For LBS, we refer the implementation in [48] ( https://github.com/mazpie/mastering-urlb ) and combine it with the code of urlb. For other more recent baselines, we also follow their official implementations, including CIC ( https://github.com/rll-research/cic ) and BeCL ( https://github.com/ Rooshy-yang/BeCL ).

For image-based DMC , almost all baselines (ICM, RND, Plan2Explore, APT, LBS, DIAYN, APS) combined with RL backbone DreamerV2 are directly following the official implementation in [48] ( https://github.com/mazpie/mastering-urlb ), which currently achieves the leading performance in image-based DMC of urlb. For CIC, we combine its official code ( https://github. com/rll-research/cic ), which mainly considers state-based DMC, and the DreamerV2 backbone in [48]. Similarly, for LSD, we refer to its official code ( https://github.com/seohongpark/LSD ) and combine it with the code of [48]. For Choreographer, of which the backbone is DreamerV2, we directly utilize its official code ( https://github.com/mazpie/choreographer ).

For Robosuite , our code is based on the code of RL-Vigen [69] ( https://gemcollector.github. io/RL-ViGen ), including the RL backbone DrQ. For Isaacgym , our code is based on the official code of [74] ( https://github.com/ZiwenZhuang/parkour ), which implements five downstream tasks (run, climb, leap, crawl, tilt). For these two settings (Robosuite and Isaacgym), as there are few works considering unsupervised RL in such a challenging setting, we implement classical baselines (ICM, RND, LBS) by referring their implementations in urlb ( https://github.com/ rll-research/url_benchmark ) and [48] ( https://github.com/mazpie/mastering-urlb ).

B.3 Hyper-parameters

Baseline hyper-parameters are taken from their implementations (see Appendix B.2 above). Here we introduce PEAC's hyper-parameters. For all settings, hyper-parameters of RL backbones (DDPG, DreamerV2, PPO) follow standard settings.

First, for PEAC in state-based DMC with RL backbone DDPG, our code is based on urlb ( https://github.com/mazpie/mastering-urlb ) and inherits hyper-parameters of DDPG. For completeness, we list all hyper-parameters as

Next, for PEAC-LBS and PEAC-DIAYN in image-based DMC with RL backbone DreamerV2, our code is based on [48] ( https://github.com/mazpie/mastering-urlb ). Hyper-parameters of

Table 5: Details of hyper-parameters used for image-based DMC.

Then, for PEAC in Robosuite, our code follows RL-Vigen [69] ( https://gemcollector.github. io/RL-ViGen ). PEAC's hyper-parameters, inheriting DrQ's hyperparameters, include

Finally, for PEAC in A1-disabled of Isaacgym with RL backbone PPO, our code follows [74] ( https://github.com/ZiwenZhuang/parkour ). PEAC's hyper-parameters, inheriting PPO's hyperparameters, include

B.4 Detailed results in state-based DMC

In Table 8, we present detailed results in state-based DMC of all four statistics (medium, IQM, mean, OG) for baselines and our PEAC. The results indicate that PEAC performs the best in all these four metrics, while BeCL and CIC perform second and third respectively. Moreover, we report individual results for each downstream task of state-based DMC in Table 9. PEAC performs comparably to BeCL as well as CIC in the Walker-mass tasks and best on most Quadruped-mass and Quadruped-damping tasks. Especially, in the challenging Quadruped-damping setting, PEAC can complete cross-embodiment downstream tasks and significantly outperforms BeCL and CIC.

B.5 Detailed results in image-based DMC

In Table 10, we present detailed results in state-based DMC of all four statistics (medium, IQM, mean, OG) for baselines and our PEAC-LBS as well as PEAC-DIAYN. Besides these statistics, in Table 11, we further report the detailed results for the 12 downstream tasks, averaged across all embodiments and seeds. Overall, PEAC-LBS's performance is steadily on top, outperforming existing methods, especially in Walker-mass. Also, compared with other pure skill discovery methods, PEAC-DIAYN performs more consistently on all tasks and achieves higher average rewards.

B.6 Detailed results in Robosuite

In Table 12, we report detailed results in Robosuite with all tasks and robotic arms. Overall, PEAC performs better in more tasks and owns better generalization ability to unseen robot Jaco.

B.7 Ablation of timesteps in image-based DMC

In Figure 8, we show additional results about the performance in three domains of image-based DMC for different algorithms and pre-training timesteps. Overall, PEAC-LBS outperforms all methods, while Choreographer and LBS are still competitive on the Quadruped-mass. Also, PEAC-DIAYN outperforms all other pure skill discovery methods.

Walker-mass

Quadruped-damping

Overall

Pretraining on 100k steps

0

10

20

30

40

50

60

70

80

90

100

Expert Normalized Score (%)

PEAC-DIAYN (Ours)

CIC

LSD

APS

DIAYN

PEAC-LBS (Ours)

Choreographer

LBS

APT

Plan2Explore

RND

ICM

Walker-mass

Quadruped-damping

Overall

Pretraining on 1M steps

0

10

20

30

40

50

60

70

80

90

100

Expert Normalized Score (%)

PEAC-DIAYN (Ours)

CIC

LSD

APS

DIAYN

PEAC-LBS (Ours)

Choreographer

LBS

APT

Plan2Explore

RND

ICM

LSD

DIAYN

APS

PEAC-LBS (Ours)

Choreographer

LBS

Quadruped-damping

Pretraining on 500k steps

LSD

DIAYN

Choreographer

APS

PEAC-LBS (Ours)

LBS

Quadruped-damping

Pretraining on 2M steps

B.8 More Ablation Studies

In this part, we conclude more ablation studies to better clarify the contribution of each component in PEAC. First, we supplement ablation studies of the hyperparameter β in Eq. 2 ( β is set to 1.0 in all our experiments). As discussed in the paper, β is a parameter that is negatively related to the fine-tuning timesteps and is for balancing the policy improvement term and the policy constraint term. When the fine-tuning timestep tends to the infinity, β tends to 0. Unfortunately, the relationship between β and the fine-tuning timesteps is complicated. Thus we evaluate PEAC-LBS under different β as below

As shown in Table 13, when β is large, the performance of PEAC-LBS decreases more than β is small, but PEAC-LBS is overall stable with different β .

Moreover, to clarify the effectiveness of our embodiment discriminator, we supplement LBS-Context and DIAYN-Context, i.e., combining LBS and DIAYN with the embodiment discriminator in PEAC, which utilizes embodiment information during the pre-training stage. Our results in state-based DMC and Image-based DMC are in Table 14 and Table 15, respectively.

PEAC-DIAYN (Ours)

CIC

100

90

80

70

60

50

40

30

20

10

0

Walker-mass

PEAC-DIAYN (Ours)

CIC

100

90

80

70

60

50

40

30

20

10

0

APT

Plan2Explore

Overall

APT

Plan2Explore

Overall

RND

ICM

RND

ICM

Expert Normalized Score (%)

Expert Normalized Score (%)

Walker-mass

As shown in these two tables, LBS-Context and DIAYN-Context own comparable or superior performance compared with LBS and DIAYN respectively, and PEAC still significantly outperforms them. Consequently, this ablation study highlights that both the embodiment discriminator and cross-embodiment intrinsic rewards R CE are effective for handling CEURL.

B.9 Generalization results of pre-trained models

In Fig. 9, we evaluate the generalization ability of pre-trained models to unseen embodiments of all exploration methods in Walker-mass of image-based DMC. After pre-training on several embodiments, we zero-shot utilize these agents to sample trajectories via two different unseen embodiments. Given the trajectories, we reduce the dimension of the hidden states calculated by the world model via t-SNE [57], where points with different colors represent data generated by different embodiments. As shown in Fig. 9, all the baselines can not distinguish different embodiments, while our PEAC-LBS can roughly divide them into two regions, indicating the pre-trained model of PEAC-LBS own strong generalization ability to unseen embodiments.

ICM

RND

Plan2Explore

APT

LBS

Choreographer

PEAC-LBS

B.10 Generalization results of fine-tuned models

In this part, we evaluate the generalization ability of the fine-tuned agents to unseen embodiments of state-based DMC and image-based DMC. In these two settings, we pre-train and fine-tune the agent with the sampled training embodiments ( Train column in Table 3) and zero-shot evaluate the performance of the fine-tuned agents in the same task but with unseen in-distribution embodiments ( Generalization column in Table 3). The detailed generalization results of all downstream tasks in state-based DMC and image-based DMC are in Table 16-17, respectively. As shown in Table 16,

Similarly, Table 17 shows that PEAC-LBS not only outperforms baselines but also owns a greater leading advantage than baselines, compared with the trained embodiments. Moreover, PEAC-DIAYN exceeds other pure-exploration methods and demonstrates strong generalization ability.

B.11 More challenging tasks and varying embodiments

In this section, we will consider CEURL in much more challenging tasks and more varying embodiment distributions, which are significant future directions for unsupervised cross-embodiment agents in more challenging real-world applications.

We first consider more complicated tasks including locomotion in complicated terrain. Following previous work [16], we design locomotion tasks in incline terrains and the results are below.

As shown in Table 18, the performance of LBS and PEAC-LBS decreases when locomoting in the incline terrain due to its complexity. PEAC-LBS still significantly outperforms LBS, expressing that our method, especially the cross-embodiment intrinsic rewards, benefits cross-embodiment unsupervised pre-training for handling more complicated tasks.

Besides more complicated tasks, one possible future direction is to consider more different, or even exactly different embodiments. We take the first step by designing several settings with more varying and challenging embodiment distributions:

Walker-Cheetah: includes two Walker robots with a mass of 0.4 and 1.6 times the normal mass, as well as two Cheetah robots with a mass of 0.4 and 1.6 times the normal mass.

Walker-Humanoid: includes one Walker robot and one Humanoid robot. Their robot properties, robot shapes, and action spaces are all different.

Walker-length and Cheetah-torsolength [71]: The former includes walker robots with different foot lengths while the second one includes cheetah robots with different torso lengths. Thus robots' properties and morphologies are different. The figures of these embodiments are in Fig. 10.

A

1

1

We mainly compare our PEAC-DIAYN and PEAC-LBS with DIAYN, LBS, and Choreographer in embodiment distributions: Walker-Cheetah, Walker-Humanoid, Walker-length, and Cheetahtorsolength. of which the results are in Table 19, Table 20, and Table 21 respectively.

As shown in these tables, PEAC can achieve much greater performance compared to baselines. These experiments indicate that PEAC has powerful abilities to handle various kinds of embodiment differences, including different morphologies. Unfortunately, when the embodiments vary a lot (like Walker-Humanoid), the performance of PEAC is still limited, thus designing more effective methods for handling complicated embodiments like Humanoid is a promising future direction for further considering cross-embodiment settings.

B.12 Real-World Applications

As a supplement of Sec. 5.4, we provide more detailed images of real-world robot deployments. As shown in Fig. 11, our method can fast fine-tune to different embodiments and handle different terrains, which are unseen in the simulation. A detailed video is provided on the paper homepage.

Passing through Shrubs

Climbing a Slope

Descend Stairs

Robustness

B.13 Computing Resource

In experiments, all the agents are trained by GeForce RTX 2080 Ti with Intel(R) Xeon(R) Silver 4210 CPU @ 2.20GHz. In Image-based DMC / state-based DMC / Robosuite / Isaacgym, pre-training each algorithm (each seed, domain) takes around 2 / 0.5 / 1.5 / 2 days respectively.

C Pseduo-codes of Algorithms

Algorithm 1 Pre-trained Embodiment-Aware Control (PEAC)

Require: M training embodiments { e m } M m =1 , M replay buffers {D m } M m =1 , N testing embodiments { e M + n } N n =1 , initialize neural network parameters of the policy

1: // Pre -Training

2: while is unsupervised phase do

3: // Data Collection

4: for m = 1 , 2 , ..., M do

5: Sample state-action pairs { ( s t m , a t m ) } t with the policy by controlling the embodiment e m and store them into D m .

6: end for

7: // Model Training

8: for update step = 1 , 2 , ..., U do

9: Sample state-action pairs form each replay buffer { ( s i t , a i t ) T t =1 } ∼ D i , i = 1 , 2 , ..., M

10: Update the embodiment discriminator via these data.

11: Compute the cross-embodiment intrinsic reward R CE for each state-action pair and concatenate them together.

12: Update the policy by RL backbones (like PPO, DDPG, DreamerV2, and so on) with these data and R CE .

13: end for

14: end while

15: // Fine -Tuning

16: while is supervised phase do

17: Sample state-action-reward pairs with extrinsic rewards R ext via embodiment e m and store them into D m .

18: Update the policy by jointly training data from different replay buffers via RL backbones.

19: end while

20: // Evaluation

21: Evaluate fine-tuned policy with downstream task R ext via { e m } M m =1 and unseen embodiments { e M + n } N n =1 .

Algorithm 2 PEAC-LBS

Require: M training embodiments { e m } M m =1 , M replay buffers {D m } M m =1 , N testing embodiments { e M + n } N n =1 , initialize neural network parameters of the policy

1: // Pre -Training

2: while is unsupervised phase do

3: // Data Collection (the same as PEAC)

4: ...

5: // Model Training

6: for update step = 1 , 2 , ..., U do

7: Sample state-action pairs form each replay buffer { ( s i t , a i t ) T t =1 } ∼ D i , i = 1 , 2 , ..., M

8: Update the embodiment discriminator via these data.

9: Update the components of LBS, including the Latent Prior model, the Latent Posterior model, and the Reconstruction model (In DreamerV2 backbone, we can directly utilize its prior model and posterior model).

10: Compute the intrinsic reward R CE + R LBS for each state-action pair and concatenate them together.

11: Update the policy by RL backbones (like PPO, DDPG, DreamerV2, and so on) with these data and R CE + R LBS .

12: end for

13: end while

14: // Fine -Tuning(the same as PEAC)

15: ...

16: // Evaluation

17: Evaluate fine-tuned policy with downstream task R ext via { e m } M m =1 and unseen embodiments { e M + n } N n =1 .

Algorithm 3 PEAC-DIAYN

Require: M training embodiments { e m } M m =1 , M replay buffers {D m } M m =1 , N testing embodiments { e M + n } N n =1 , initialize neural network parameters of the behavior policy conditioned on skill and embodiment context π ( ·| s, z, e ) , initialize neural network parameters of the embodiment-aware skill policy π ( z | e, τ )

1: // Pre -Training

3: // Data Collection (the same as PEAC)

2: while is unsupervised phase do 4: ... 5: // Model Training 6: for update step = 1 , 2 , ..., U do

7: Sample state-action pairs form each replay buffer { ( s i t , a i t ) T t =1 } ∼ D i , i = 1 , 2 , ..., M

8: Update the embodiment discriminator via these data.

9: Update the skill discriminator of DIAYN via these data.

10: Compute the intrinsic reward R CE + R DIAYN for each state-action pair and concatenate them together.

11: Update the behavior policy conditioned on skill and embodiment context by RL backbones (like PPO, DDPG, DreamerV2, and so on) with these data and R CE + R DIAYN .

12: end for

13: end while

14: // Fine -Tuning

15: while is supervised phase do

16: Sample state-action-reward pairs with extrinsic rewards R ext via embodiment e m and store them into D m .

17: Update the embodiment-aware skill policy by jointly training data from different replay buffers via RL backbones.

18: end while

19: // Evaluation

20: Evaluate fine-tuned agents with downstream task R ext via { e m } M m =1 and unseen embodiments { e M + n } N n =1 .

D Broader Impact

Designing generalizable agents for varying tasks and embodiments is a major concern in reinforcement learning. This work focuses on cross-embodiment unsupervised reinforcement learning and proposes a novel algorithm PEAC, which leverages trajectories from different embodiments for pre-training, subsequently broadly enhancing performance on downstream tasks. Such advancements provide the potential for future real-world cross-embodiment control. One of the potential negative impacts is that algorithms using deep neural networks, which lack interoperability and face security and robustness issues. There are no serious ethical issues as this is basic research.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: As mentioned in the abstract and introduction, this work mainly integrates the unsupervised RL paradigm into cross-embodiment RL, as a novel concept CEURL (Sec. 3). Then we theoretically derive a novel algorithm PEAC (Sec. 3-4) and widely evaluate PEAC in a large number of environments (Sec. 5). All these contributions are described in detail in the paper.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Yes, we have discussed the limitations of this work in Sec. 5.5

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Yes, we have included the detailed proofs of all results in the Appendix A, including properties of D E (Appendix A.1), proof of Theorem 3.2 (Appendix A.2), and proof about Embodiment-Aware Skill Discovery (Appendix A.3).

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We have provided the code in the paper homepage. Also, we have provided detailed information on our experiments, including hyper-parameters, codebase, and pseudocode of our algorithms in Appendix B and Appendix C.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We have provided the source code in our paper homepage.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We have provided details for our experiments in Appendix B, including our environments, hyper-parameters, optimizers, and so on, as well as our source code in the paper homepage.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes] ,

Justification: To mitigate the effectiveness of the randomness, we repeat several random seeds for all experiments (10 for state-based DMC and 3 for the others, following previous works' settings). When reporting the results, besides reporting mean ± std performance, we

also report IQM and OG in state-based DMC and image-based DMC for better evaluating different algorithms.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Yes, we have reported the computing source in Appendix B.13, including the type of computing CPU/GPU, computing time, and so on.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our research does not involve human subjects or participants. And we have discussed the potential societal impact in Appendix. D. As this is basic research, there are no serious social issues.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Yes, we have discussed the Border Impact of this work in Appendix. D.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: There is no such risk in this paper.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have cited the original paper that produced the code/environment package and included the URL of these codebases in Appendix B.2.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: Yes, we have provided documentation of our source code in the supplementary materials.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a new unsupervised reinforcement learning (URL) algorithm called Pre-trained Embodiment-Aware Control (PEAC). PEAC is designed to tackle cross-embodiment tasks, explicitly considering the influence of different embodiments to facilitate exploration and skill discovery across embodiments. Experimental results demonstrate that PEAC notably enhances adaptation performance.\n\nSTRENGTHS:\n1. The paper is well-written and easy to follow. The intuitive figures and logically structured text make PEAC clear and understandable.\n2. The author(s) conduct comprehensive experiments and present a solid work.\n3. As embodied intelligence gains increasing attention, cross-embodiment research is indeed an important direction. PEAC undoubtedly contributes to the community in this regard.\n\nWEAKNESSES:\n1. Although the author(s) compare various standard and SOTA baselines in the experiments, they lack strong persuasiveness. Since the author(s) focus on cross-embodiment tasks, it is apparent that the baselines, which do not utilize embodiment information $e$, exhibit poorer performance. Therefore, it is unclear whether PEAC's high performance is due to the use of embodiment information $e$ or the utilization of cross-embodiment intrinsic rewards $\\mathcal R_{\\text{CE}}$. To highlight the paper's contribution, I suggest that the author(s) modify some baselines to naively incorporate embodiment information and then compare with them. For example, conditioning the policy and reward of DIAYN on embodiment $e$, etc.\n2. Experiments conducted on Robosuite effectively showcase \"cross-embodiment,\" while other environments may not do so as prominently. By modifying parameters like \"mass\" and \"damping\" in DMC tasks or introducing joint torque failures in Unitree A1, the setting becomes more like a Meta RL, where the robot's morphology remains relatively unchanged. In locomotion tasks, \"cross-embodiment\" could be exemplified by robots like Gym-MuJoCo's Walker2d, DMC's Walker, and Humanoid, where they all use two legs for alternating walking and aim to enhance mobility. However, they differ in morphology. Demonstrating whether PEAC can learn the same shared knowledge of bipedal walking from these diverse morphologies may better illustrate the topic of \"cross-embodiment.\"\n\nQUESTIONS:\nPlease see Weaknesses.\n\nLIMITATIONS:\nMy primary concern is that the baselines in the experiments do not utilize embodiment information $e$. Please refer to the first point in Weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis papaer introduces a novel setting, cross-emodiment unsupervised RL, which deals with pre-training good policies in reward free environments across different embodiments in order to perform well on downstream tasks on unseen embodiments. The authors propose the algorithm PEAC for unsupervised learning in this setting. Their method proposes a novel intrinsic reward which is derived from the minmax objective of maximizing the improvement in the downstream fine-tuned policy over the pre-trained policy minus a regularizing KL term, which ensures that the fine-tuned policy stays close to the pre-trained policy, under the worst-case extrinsic reward. From this objective, they derive an intrinsic reward for the unsupervised cross-embodiment setting with maximizes the (fixed) log prior over embodiments minus the (learned) log posterior over embodiments, which encourages the agent to seek areas of the state space that are common accross embodiments, or underexplored. They conduct extensive experiments in both simulated and real environments, including both state-based and image-based DMC tasks, and compare against a large number of pre-existing skill-based and exploratoin based unsupervised RL methods. Their results are impressive, showing improvement in almost all cases over the baselines. Moreover, they conduct real-world experiments in which their algorithm similarly outperforms.\n\nSTRENGTHS:\n- Introduces a novel setting: cross-embodiment unsupervised RL\n- Convincing motivation - the idea that most cross-embodiment training which focuses on a single task may not learn generalizable embodiment-specific skills rather than task-specific skills\n- Extensive experiments in both simulation and real-world settings with very strong results compared to baselines\n\nWEAKNESSES:\n- The paper is hard to follow and key details on the implementation are not fully explained. For example in line 68 a \"embodiment context encoder\" is mentioned but I do not see this explained any where else in the paper. How is the discriminator trained?\n- The extension of the information geometry analyses in Sec 3 do not seem particularly related to the main claims of the paper. This section serves primarly to highlight that policies may be stochastic, however it does not seem that the this information is utilized in the algorithm presented\n- The stated goal of the paper is to better learn embodiement-aware skills, but the policy is optimized to maximize the expectation over all embodiments. Would this encourage embodiment-aware skills or rather embodiment-agnostic skills?\n\nQUESTIONS:\n1. It is not clear to me how the cross-embodiment policy handles different state spaces. For example in the cheetah-walker experiments in the appendix, how is the policy architecture designed such that it can take in different size state vectors?\n2. What does super script c in the MDP M represent?\n3. Can you please explain the equality in the last line of Eq 29?\n4. It is not clear what is unique about the cross-embodiment setting for unsupervised RL versus the more generic contextual MDP setting. Why do embodiments needed to be treated differently?\n5. If log(p(e)) is fixed, does it need to be in the reward?\n\nLIMITATIONS:\n- As the authors note, their experiments are limited to settings where the embodiments are very similar. It is not clear how their method extends to dissimilar embodiments, particularly with different sized state vectors.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the challenge of designing generalizable agents capable of adapting to diverse embodiments. The authors propose the CEURL setting as a novel framework for this problem and introduce the PEAC algorithm to address it. Recognizing that CEURL requires minimizing across different downstream tasks while maximizing the fine-tuned policy, PEAC tackles the issue through policy improvement and policy constraint mechanisms. PEAC can integrate with existing unsupervised reinforcement learning methods designed for single-embodiment settings, such as LBS and DIAYN. Through experiments conducted in both simulation and real-world settings, the effectiveness of PEAC is demonstrated.\n\nSTRENGTHS:\nOriginality: This paper introduces a novel problem formulation, CEURL, and presents a new algorithm, PEAC.\n\nQuality:\n1. The authors thoroughly discuss the challenges inherent in the CEURL problem definition, supported by mathematical proofs. This elevates CEURL from a simple definition to a high-quality problem formulation.\n2.Convincing experiments involving real robots are presented.\n\nClarity: The paper is well-written and easy to follow.\n\nSignificance: This paper has the potential to significantly impact cross-embodiment control through its introduction of a new problem formulation and an innovative unsupervised pretraining framework.\n\nWEAKNESSES:\n1. Among all the experiments presented, tasks for DMC and Issacgym are relatively limited, with only simple movements. I recommend adding more complex tasks like [1].\n2. In ablation study, the authors only show the results of different training timesteps. More ablation studies can help demonstrate how the algorithm works.\n3. From my perspective, this paper is closely related to [2], which proposes an unsupervised skill discovery method through contrastive learning. While [2] focuses on single-embodiment scenarios, PEAC extends this approach to multiple embodiments, addressing the additional challenges posed by varying state and action spaces via introducing additional embodiment prior and posterior probabilities. \n\n[1] Gupta, Agrim, et al. \"Embodied intelligence via learning and evolution.\" Nature communications 12.1 (2021): 5721.\n[2] Yang, Rushuai, et al. \"Behavior contrastive learning for unsupervised skill discovery.\" International Conference on Machine Learning. PMLR, 2023.\n\nQUESTIONS:\n1. For objective function (2), can you provide an ablation study using only the Policy Improvement part and only the Policy Constraint part?\n2. In Figure 6 (right), PEAC-DIAYN shows the worst performance at 100k training steps. What is the cause?\n3. In Appendix B.6, the performance of PEAC in Robosuite is fair comparing to that in DMC or Issacgym. Does PEAC work better for simple tasks (like walk, stand) than the tasks in Robosuite?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}