PEAC: Unsupervised Pre-training for Cross-Embodiment Reinforcement Learning
Authors: Chengyang Ying
Zhongkai Hao
Xinning Zhou
Xuezhou Xu
Hang Su
Xingxing Zhang
Jun Zhu
Venue: NeurIPS 2024 poster (2024)
Keywords: cross-embodiment reinforcement learning
unsupervised reinforcement learning
cross-embodiment exploration
cross-embodiment skill discovery
Primary area: reinforcement_learning
TL;DR: We propose an unsupervised pre-training method named Pre-trained Embodiment-Aware Control (PEAC) for handling Cross-Embodiment Reinforcement Learning

Abstract: Designing generalizable agents capable of adapting to diverse embodiments has achieved significant attention in Reinforcement Learning (RL), which is critical for deploying RL agents in various real-world applications. Previous Cross-Embodiment RL approaches have focused on transferring knowledge across embodiments within specific tasks. These methods often result in knowledge tightly coupled with those tasks and fail to adequately capture the distinct characteristics of different embodiments. To address this limitation, we introduce the notion of Cross-Embodiment Unsupervised RL (CEURL), which leverages unsupervised learning to enable agents to acquire embodiment-aware and task-agnostic knowledge through online interactions within reward-free environments. We formulate CEURL as a novel Controlled Embodiment Markov Decision Process (CE-MDP) and systematically analyze CEURL's pre-training objectives under CE-MDP. Based on these analyses, we develop a novel algorithm Pre-trained Embodiment-Aware Control (PEAC) for handling CEURL, incorporating an intrinsic reward function specifically designed for cross-embodiment pre-training. PEAC not only provides an intuitive optimization strategy for cross-embodiment pre-training but also can integrate flexibly with existing unsupervised RL methods, facilitating cross-embodiment exploration and skill discovery. Extensive experiments in both simulated (e.g., DMC and Robosuite) and real-world environments (e.g., legged locomotion) demonstrate that PEAC significantly improves adaptation performance and cross-embodiment generalization, demonstrating its effectiveness in overcoming the unique challenges of CEURL. The project page and code are in https://yingchengyang.github.io/ceurl.

PEAC: Unsupervised Pre-training for Cross-Embodiment Reinforcement Learning
Chengyang Ying'
Zhongkai Hao! Xinning Zhou' Xuezhou Xu
Hang Su Xingxing Zhang' Jun Zhu Department of Computer Science & Technology, Institute for AI, BNRist Center, Tsinghua- Joint MI. Center, THBI Lab, Tsinghua University Pazhou Lab (Huangpu), Guangzhou, China ycy21 1 @mails. tsinghua. edu.
Abstract
Designing generalizable agents capable of adapting to diverse embodiments has achieved significant attention Reinforcement Learning (RL), which criti- cal for deploying RL agents in various real-world applications. Previous Cross- Embodiment RL. approaches have focused on transferring knowledge across em- bodiments within specific tasks. These methods often result in knowledge tightly coupled with those tasks and fail to adequately capture the distinct characteristics of different embodiments. To address this limitation, we introduce the notion of Pross-Embodiment Unsupervised RL (CEURL), which leverages unsupervised learning to enable agents to acquire embodiment-aware and task-agnostic knowl- edge through online interactions within reward-free environments. We formulate CEURL novel Controlled Embodiment Markov Decision Process (CE-MDP) and systematically analyze CEURL's pre-training objectives under CE-MDP. Based these analyses, we develop novel algorithm Pre-trained Embodiment-AAarc Control (PEAC) for handling CEURL, incorporating an intrinsic reward func- tion specifically designed for cross-embodiment pre-training PEAC not only provides an intuitive optimization strategy for cross- eoosseembodiment pre-training but also can integrate flexibly with existing unsupervised RL methods, facilitating cross-embodiment exploration and skill discovery. Extensive experiments in both simulated (e.g., DMC and Robosuite) and real-world environments (e.g., legged locomotion) demonstrate that PEAC significantly improves adaptation perfor- mance and cross-embodiment generalization, demonstrating its effectiveness overcoming the unique challenges of CEURL. The project page and code are in gi github. io/ccerl.
Introduction
Cross-embodimen reinforcement learning (RL) involves designing algorithms that effectively func- tion across various physical embodiments. The fundamental goal is to enable agents to apply skills and strategies learned from some embodiments to other embodiments, which may physical dynamics, action-effectors, shapes, and so on [28, 70, 58, 52, 12, 69, 60]. This capability sig- nificantly enhances the generalization of RL agents, reducing the necessity for embodiment-speer specific training. By adeptly adapting to new and shifting embodiment, cross-embodiee embodiment RL ensures that agents maintain reliable performance in unpredictable real-world scenarios, thereby benefiting the deployment process and reducing the need for extensive data collection for each new embodiment.
different
Cooresponding author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Rex(r) A lng log gu(elt)
Reward free world
Pre-trained Agent
Dewastrcam Ticks
Crev--Enboddmeet
(rnder soselaction spaces, dysamios)
Pre-train to extract generalized knowledge
Fine-tume
Figure I: Overview of Crross-Embodinee Unsupervised Reinforcement Learning (CEURL). The left subfigure illustrates the cross-embodiment setting with various possible embodiment changes. Directly training RL agents across embodiments under given tasks may result in task-aware rather than embodiment-aware knowledge. CEURL pre-trains agents in reward-free environments extract embodiment-aware knowledge. The center subfigure shows the Pre-trained Embodiment-Aww Aware Control (PEAC) algorithm, using our coos-embbdimmnt intrinsic reward function ReE (T). The right subfigure demonstrates the fine-tuning phase, where pre-trained agents fast adapt to different downstream tasks, improving adaptation and generalization.
One of the primary challenges in this area is the transfer of knowledge across embodiments that have vastly different physical dynamics and environmental interactions. This requires the agent abstract knowledge in a way that is not overly specialized to a single embodiment or some downstream tasks. However, directly training cross-emin eessodmebod agents under some given tasks will cause the learned knowledge highly related to these tasks rather than only to embodiments themselves.
Inspired by the transformative effects of unsupervised learning in natural language processing and computer vision [6, 21], which has demonstrated efficiency in extracting generalized knowledge independent of downstream tasks, we propose natural question: Can we pre-train tross-embodiment agents in an unsupervised manner, i.e., online cross-embodiment pre-training in reward-free envi- romments, capture generalized knowledge only related to embodiments? Existing unsupervised RL techniques, including exploration [45, 38] and skill discovery [10, 26] ones, typically involve pre-training agents by engaging a single embodiment within a controlled Markov Decision Process (MDP) that lacks extrinsic reward signals. These pre-trained agents are then expected to quickly fine-tune to any downstream tasks characterized by extrinsic rewards using this specific embodiment. This approach of unsupervised RL fosters the development of policies that are not overly specialized to specific tasks or reward structures but are rather driven by intrinsic motivations of embodiments, which shows the potential for discovering more generalized knowledge across different embodiments.
In this work, adapt the unsupervised RL paradigm to the cross enoodeembod setting, introducing the concept of coos-Emmbodiennt Unsupervised RL (CEURL). This setting involves pre-training with distribution of embodiments in reward-free environments, followed by fine-tuning handle specific downstream tasks through these embodiments. These embodiments may own similar structures so that we can abstract generalized knowledge from them. To analyze CEURL and design corresponding algorithms, we formulate it as a Controlled Embodiment Markov Decision Process (CE-MDP), which comprises distribution of controlled MDPs, each defined by its unique embodiment context. Compared to the traditional eendie-embodimeet setting, the CE-MDP framework addresses the additional complexity caused by the inherent variability among embodiments. We then extend the information geometry analyses of the controlled MDP [11] to better explain the complexity of CE- MDP. Our findings indicate that skill vertices within CE-MDP may no longer be simple deterministic policies and the behaviors across different embodiments can display substantial variability.
To address the complexities of CE-MDP, we undertake in-depth analysis of the pre-training objective in CE-MDP. We aim to enable our pre-trained agent to quickly fine-tune for any downstream tasks denoted as Rext. especially under the worst-case reward scenarios. Thus, our pre-training objective involves minimizing across Ren while maximizing the fine-tuned policy π." leading to a complex min-max problem (Eq. 3). We further introduce a novel Pre-traines Embodiment-Aww -ware Control (PEAC) algorithm to optimize this objective and handle CE-MDP, which improves the agent's s robustness and adaptability across various embodiments by employing a cross-embodiment intrinsic reward ReE- This reward is complemented by an embodiment discriminator, which distinguishes between different embodiments. During fine-tuning, the pre-trained policy is further enhanced under the extrinsic reward, Rex with limited timesteps. Moreover, PEAC can integrate flexibly with existing single-embodiment unsupervised RL methods to achieve crosseembodiment exploration and skill discovery, resulting in two combination algorithm examples PEAC-LBS and PEAC-DIAYN.

---

To verify the versatility and effectiveness of our algorithm, we extensively evaluate PEAC in both simulated and wealdworld environments. In simulations, we choose state-based/ image-based Deep- Mind Control Suite (DMC) environments extending Unsupervised RI. Benchmark (URL.B [27] and different robotic arms in Robosuite [73]. Under these settings, PEAC demonstrates superior few-shot learning ability to downstream tasks, and remarkable generalization ability to unseen embodiments, surpassing existing tate-of-the-ar unsupervised RL models. Besides, we have evaluated PEAC in real-world Aliengo robots by considering practical joint failure settings based on Isaacgym [37], verifying PEAC's strong adaptability on different joint failures and various real-world terrains.
In summary, the main contributions are as follows:
We propose novel setting CEURL enhance agents' adaptability and generalization across diverse embodiments, and then we introduce the Pre-trained Embodiment-Aware Control (PEAC) algorithm for handling CEURL.
We integrate PEAC with existing exploration and skill discovery techniques, designing practical methods and facilitating efficient enbosimmboi exploration and skill discovery.
Extensive experiments show that PEAC not only excels in fast fine-tuning but also effectively generalizes across new embodiments, outperforming current SOTA unsupervised RL models.
2
Related Work
Froos-Embddiment RL. Designing generalizable agents simultaneously controlling diverse em- bodiments has achieved significant attention in RL. A common strategy involves using expert trajec- tories [70, 49, 5, 66, 60]. internet-scale human videos [3, 58, 13]. or offline datasets [29, 59, 8. 41] to train a generalist agent that can handle various tasks across different embodiments. However, these methods are often limited by the need for large-scale, costly datasets and the availability of expert trajectories. Additionally, the discrepancy between open-loop training and closed-loop testing may lead to distribution shifts [32], adversely affecting the final performance. An alternative line of research [28, 36, 64, 4, 66. 52, 12, 68] focuses training general agents through online interaction across diverse environments. However, these methods treat the embodiment and task as a unified training environment and overlook the role of proprioception, i.e., the internal understanding of an agent's 's embodiment, which has recently proven to be beneficial for representation learning and optimization in RL [23, 15]. Thus these methods may not fully capture the intrinsic properties of different embodiments by linking knowledge to specific tasks. Emerging research suggests the potential of decoupling the training of embodiment characteristics from task execution, aiming to develop a unified cross-embodiment model. This involves unsupervised pre-training across a variety of embodiments, followed by task-aware fine-tuning, enabling a single agent to adeptly manage both roles effectively.
Unsupervised RL. Unsupervised RL leverages interactions with reward-free environments to extract useful knowledge, such as exploratory policies, diverse skills, or world models [17, 18). These pre-trained models are utilized fast adapt downstream tasks within specific embodiments and environments. Unsupervised RL methods can be categorized into two main types: exploration and skill discovery. Exploration methods aim maximize state coverage, typically through intrinsic rewards that encourage uncertainty [45, 7, 46, 51, 38, 40, 67] or state entropy [30, 47, 35, 34j. The resulting exploratory trajectories benefit pre-training actor-critic or world models, thereby enhancing fine-tuning efficiency [27]. Skill discovery methods focus on learning an array of distinguishable skills, often by maximizing the mutual information between states and acquired skills [10, 54, 20, 55, 25, 26, 72, 24, 61]. This approach benefits from theoretical insights into the information geometry of skill state distributions, emphasizing the importance of maximizing distances between different skills [11, 22, 42, 43, 44]. Recent efforts also explore incremental skill learning in dynamic environments [53, 31]. Unlike these methods generally focus on single embodiments, we aim to develop generalizable models capable of handling downstream tasks across a variety of embodiments.
3 Cross-Embodiment Unsupervised RL
In this section, we analyze the cross-embodiment RL in an unsupervised manner, which is formulated as our Controlled Embodiment MDP. Then we propose a novel algorithm PEAC to optimize CE-MDP.
3

---

3.1 Controlled Embodiment Markov Decision Processes
coosseembodiment RL can be formulated by contextual MDP [19] with a distribution of Markovian decision processes (MDPs) of (Ma). Cross coosseembodiment RL hopes to learn shared knowledge from this distribution of MDPs, which is crucial for enhancing the adaptability or generalization of agents across embodiments. However, directly optimizing agents by online interacting with (Me) or utilizing offline datasets sampled from {Me} may learn knowledge not only related to these embodiments but also highly related to these task reward functions in {Mee. This phenomenon may have negative impacts on learning the general knowledge across embodiments or improving the agent's generalization ability. For example, as the agent required to handle {Me). it will less explore the trajectories with low rewards. These trajectories, although not optimal for the embodiment in this task, might also include embodiment knowledge and be useful for other tasks. Without extrinsic task rewards. the agent is encouraged to learn embodiment-awwan and task -aggosictt knowledge, which can effectively adapt any downstream task across embodiments.
In this paper, we propose to pre-train cross-embodiment agents in reward-free environments to ensure that the agent can learn knowledge only specialized in these embodiments themselves. In other words, we introduce unsupervised RL into cross-embodiment RL as a novel setting: cross-embodiment unsupervised RL (CEURL). As shown in Fig. 1, in CEURL, we first pre-train general agent by interacting with the reward-free environment through varying embodiments sampled from an unknown embodiment distribution. Given any downstream task represented by the extrinsic reward Rex. the pre-trained agent is subsequently fine-tuned to control these embodiments, and other unseen embodiments from the distribution, to complete this task within limited steps (like one-tenth of the pre-training steps). Formally, we formulate CEURL as the following controlled embodiment MDP (CE-MDP):
Definition 3.1 (Controlled Embodiment MDP (CE-MDP)). A CE-MDP includes a distribution of controlled MDPs defined as M Ae, Pe, T). where e ~ E and & represents the embodiment distribution. Each embodiment may have different state spaces Se and action spaces Ae. Pe Se X Ae - A(Se) denoting the transition dynamics for embodiment and is the discount factor. We define the state space = UeSe and adopt a unified action embedding space A with corresponding action projectors &e A + Ae. which can be fixed learnable.
Thus establish a unified policy S + A(A) across all embodiments. For any embodiment e, we sample an action a from (.-(s) for a state s E Se and execute the projected action de (a). Without loss of generality, we assume de is fixed and focus analysis the policy To explain the complexities of CE-MDP with varying embodiment contexts, we extend the singlc-embodiment cross-embodiment setting. First, we consider the
information geometry analyses [11] into
discount state distribution of π within M at state s as de (s) = (1 - 7) Eiz ty:Pe(s: =)]] )). It is well known that the trajectory return of the state-based reward function can be computed
J JMg. (T) A ET [Reu(T)] 1 EE Eg [Rex
(I)
Thus, the properties of d= are significant in determining useful initializations for downstream tasks. We consider the set De {de E ASS) Vr), which includes all feasible de over the probability simplex. As shown in [11], for each e, D is a convex set, and any useful policy, which can be optimal for certain downstream tasks under embodiment e, must be vertex of De typically corresponding
to deterministic policies.
However, in the context of CEURL,, the unknown embodiment context introduces partial observabil- ity [14] and significant differences in the corresponding points of the same skill across different em- bodiments. In CE-MDP,
consider the entire embodiment space and define de (s) Ee- [d_ (s)], with DE = {de E (SS) VTT. The primary challenge lies in the high variability of embodiments, which complicates the process of learning policy that generalizes well across different embodiments. We demonstrate that the vertices of DE may no longer correspond to deterministic policies, as they need to handle all embodiments in the distribution. This significantly heightens the challenge of the pre-training process in CE-MDP, making more difficult to find useful emooddmebod skills (proofs and discussion in Appendix A.)).
To solve CEURL under the paradigm of CE-MDP, the agent will collect reward-free trajectories (8o, ao .....) with probability p.g. (T) (Pe II (aa. Pe a,) via some

---

sampled embodiments during the pre-training. These trajectories are then used in CEURL meth- ods design intrinsic rewards Rint. for pre-training agents. During fine-tuning, we will sample several embodiments from & and combine M with downstream task represented by extrin- sic rewards Rex: and agents are required to maximize the task return over all embodiments, i.e., Be- [MMg.Ro Roi (r)]. within limited steps (like one-tenth or less of the pre-training steps).
3.2 Pre-trained Embodiment-Aware Control
We primarily focus on the pre- pre-training objective of CEURL, specifically determining the optimal pre-trained policy π for CEURL. In the fine-tuning stage, given any downstream task characterized by extrinsic reward Rex. the pre-trained policy л will be optimized into the fine-tuned policy x² with limited steps to handle Rex via some RL algorithms like PPO [50]. Consequently, it is widely assumed that n will remain close to ı during ane-tuning due to constraints on limited interactions with the environment [11]. Our cross- vros-eebbdiiment fine-tuning objective thus combines policy improvement under ext and a policy constraint evaluated via KL. divergence
F(T,, x. Rex, e) 2 Bppg (r) (xx( (T) - (r) [Rex(T]] 3DKL(PMM.... (P) (T)):
Policy Improvement
Policy Constraint
where 3 > is the unknown trade-off parameter related to the fine-tuning steps (when fine-tuning steps tend towards infinity, tends to 0 and this objective converges to the original RL objective), and M represents the "average embodiment MDP" satisfying that P PM (T) Ee- [p.vg. (7)] During fine-tuning,
hope to optimize Λ² by maximizing F, i.e., the fine-tuned result is
maxppgg.* (+) F(T,T", T", ,eet, e). As the pre-trained policy needs to handle any downstream task, we consider the worst-case extrinsic reward function across the embodiment distribution, and our cross-embodiment pre-training objective can be formally represented as maximizing
u(T, E) BENE
min max F(T,, nT,,, Reai, e) Rea(T) Pagg.a
This objective is min-max problem that is hard to optimize. Fortunately, we can simplify it as below Theorem 3.2 (Proof in Appendix A.2). The pre-training objective Eq. (3) of (T, satisfies
U(n, E) = Re [KL (r)ee(()))llxx = -e E Bropag. log p(e) Pr (e(
(4)
Here p(e) and Pr (elT) are embodiment prior and posterior probabilities, respectively. This result simplifies our pre-trained objective as a form easy to calculate and optimize. Also, although 3 is an unknown parameter, the optimal pre-trained policy is independent of (3. Based these analyses, we propose a novel algorithm named Pre-trained Embodiment-Awaz first train embodiment discriminator qo (elr) approximate Pr (e(T)) which learn the embodiment context via historical trajectories. For cross-embodiment pre-training, PEAC then utilizes our cross-embodiment intrinsic reward, which is defined following Eq. (4) as
Control (PEAC). In PEAC,
RCE(T) - log p(e) - log go(e)T).
(5)
Assuming the embodiment prior p(e) is fixed, RcF encourages the agent to explore the region with low log (elr)) In these trajectories, the embodiment discriminator is misled, where the agent may not have explored enough or differennt embodiment posteriors are similar. Thus, the embodiment discriminator can boost itself from these trajectories and learned embodiment-awwe contexts that can effectively represent different embodiments, which benefit generalizing to unseen embodiments.
In practice, Res needs to be calculated for each state s rather than the whole trajectory т, also, the embodiment discriminator needs to classify the embodiment context for every state. For RL backbones that encode historical information as the hidden state h like Dreamer [17, 18, 64], we directly train (e(e/h. h. 8) as the discriminator and further calculate ReF. For RL algorithms with Markovian policies like PPO [50], we encode fixed length historical state-action pair to the hidden state l and also train go (e) he, 2), following [28]. For a fair comparison, our policy still uses Markovian policy and does not utilize encoded historical messages. PEAC's pseudo-code is in Appendix C.
5

---

Figure 2: Benchmark environments, including DMC [56], Robosuite [73]. Isaacgym [37].
Cross-Embodiment Exploration and Skill Discovery
As shown above, PEAC pre-trains the agent for the optimal initialization to few-shot handle down- stream tasks across embodiments. Besides, although PEAC does not directly explore or discover skills, it is flexible to combine with existing unsupervised RL methods, including exploration and skill discovery ones, to achieve cross-embodiment exploration and skill discovery. Below we will discuss in detail the specific combination between PEAC and these two classes respectively, exporting two practical combination algorithms, PEAC-LBS and PEAC-DIAYN, as examples.
Embodiment-Aware Exploration. Existing exploration methods mainly encourage the agent to explore unseen regions. As PEAC suggests the agent explores the region where the embodiment discriminator is wrong, it is natural to directly combine ReE and exploration intrinsic rewards to achieve cross-embodiment exploration, i.c., balancing embodiment representation learning and unseen state exploration. As example, we take LBS [38], of which the intrinsic reward is the KL divergence between the latent prior and the approximation posterior, as the PEAC-LBS. As RcE and RLBS both related to some KL divergence, with the same weight in PEAC-LBS, of which the detailed pseudo-code in Appendix C.
directly add up these two intrinsic rewards
Embodinent-Awar Skill Discovery. single-embodiment diilo-iiscovery mainly maximizes the mutual information between trajectories T and skills # as I(T; x) = DRI. (p(T, x) ()((())p(()) [10], which has been shown optimal initiation to some skill-based adaptation objective [11]. We combine and our cross-embodimen fine-tuning objective Eq. (2) to propose a unified cross-embodiment skill-based adaptation objective
Fo(TT x², Rex, e) E p (1) [Re)(T)) max E pag in (T(E)) *'
(6)
-BDKL (T))),,,,
Similar to Theorem 3.2, we can define our pre-training objective and simplify it as
U,( A EENE min Raa(F) PMg.x (r)
max F. F. x;, Rex, e)
(7)
(elT) (T, Dellootr-AMMMMllatMMMMMM. Pn (e)
Ble max p(E) ME)
E Exppper log
The proof of Eq. (7) is in Appendix A.3, where we also show it is general form of Theorem 3.2 and the single-embe enggleeemodimmnt disll-disyovrry result [11]. The result of Eq. (7) includes two terms for handling cross-embodiment and discovering skills respectively. In detail, the first term is the same as the objective Eq. (4), thus we can directly optimize via PEAC. As the second term similar to the classical skill-discover objective T(r; z) but only embodiment-aware, we can extend existing skill-discovery methods into an cmbodiment-awwerr aware version for handling it.
We take DIAYN [10] as an example, resulting in PEAC-DIAYN, Overall, In the pre-training stage, given a random skill and an embodiment e, we will sample trajectories with the policy ( 8, z, e) that is conditioned on 2 and the predicted embodiment context. Then we will train neural network p(z,e// to jointly predict the current skill and the embodiment. For training the policy, we combine ReE and RDILAN as the intrinsic reward. During fine-tuning we utilize the embodiment discriminator, mapping observed trajectories to infer the embodiment context. We then train an ambodiment-awaan meta-controller (2)ee, 7). which inputs the state and predicted context and then outputs the skill. It extends existing embodiment-agnoote meta-controller [39] and directly chooses from skill spaces rather than complicated action spaces. The pseudo-code of PEAC- DIAYN is in Appendix C.

---

Madian
IOM (1)
Moan (1)
Optimality Gap (4)
ICM
RND
Disagreement
ProtoRL
LBS
DIATN
SMM
APS
CIC:
BeCL
PEAC (Durs)
0.45
0.30
Export Normalized Score
Figure 3: Aggregate metrics [2] in state-based DMC. Each statistic for every algorithm has 120 runs
(3 embodiment settings downstream tasks x 10 seeds).
Median
1QM
Mean
Optimality (4)
DIAYN
APS
LSD
CIC
PEAC-DIAYN (Ours)
KCM
RND-
Plan2Explore
L85
Chomagrapher
PEAC-LBS (Ours)
Expert Normalized Score
Figure 4: Aggregate metrics [2] image-based DMC. Each statistic for every algorithm has 36 runs
(3 embodiment settings x 4 downstream tasks x 3 seeds).
5 Experiments
We now present extensive empirical results to answer the following questions:
Does PEAC enhance the cross-embodiment unsupervised pre-training for handling different downstream tasks? (Sec. 5.2)
Can CEURL benefit cross-embodiment RL and effectively generalize to unseen embodi- ments? (Sec. 5.3)
Does CEURL advantage to real-world cross-embodiment apppiiiciiiiiiiiiiiiiiiiinn (Sec. 5.4)
5.1 Experimental Setup
To fully evaluate PEAC in CEURL, choose extensive benchmarks (Fig. 2), including base- / image-based Deepmind Control Suite (DMC) [56] in URLB [27], Robosuite [73, 69] for robotic manipulation, and Isaacgym [37] for simulation as well real-world legged locomotion. Below we will introduce embodiments, tasks, and baselines for these settings, with more details in Appendix B.
State-based DMC & Imsge-based DMC. These two benchmarks extend URLB [27], classical single- single-embodiment unsupervised RL settings. Based on basic embodiments, we change the mass damping conduct three distinct embodiment distributions: walker-mass, quadruped-mass, and quadruped-dampinn following previous work with diverse embodiments [28, 65]. All downstream tasks follow URLB. These two settings take robot states and images observations respectively.
In state-based DMC, we compare PEAC with exploration and skill-discovery methods: ICM [45], RND [7], Disagreement [46]. ProtoRL [62]. LBS [38], DIAYN [10]. SMM [30], APS [34], CIC [26].
and BeCL [61], which are standard and SOTA for this setting. For all methods, we take DDPG [33]
as the RL backbone, which is widely used in this benchmark [27]. In image- based DMC, we take exploration baselines: ICM, RND, Plan2Explore [51], APT [35], and LBS; as well as skill-discovery baselines: DIAYN, APS, LSD [42], and CIC. Also, choose SOTA baseline Chorcographer [39], which combines exploration and skill discovery. For all methods, we take Dreamer [18] as the backbone algorithm, which has currently shown leading performance in this benchmark [48].
Robosuite. We further consider embodiment distribution with greater change: different robotic arms for manipulation tasks from Robosuite [73]. We pre-train our agents in robotic arms Panda, IIWA, and Kinova3. Besides, we take robotic arm Jaco for evaluating generalization. Following [69], we take DrQ [63] the RL backbone and choose standard task settings: Door, Lift, and TwoArmPegInHolc.
7

---

Domains
Truin
Robosuite
Test
climb
Al Al-disabled
leup
crawl
PEACILSS
ICM
174.4
178.3
6.7
5.7
4,0
8.5
13.9
RND
155.0
22
LBS
157.7
106.6
0.4
1.7
1.4
1.1
2.1
PEAC (Ouns)
190.7
200.8
19.2
10.3
10.0
20.3
17.3
Table 1: Results of Robosuite and Isaacgym.
Figure 5: Generalization Visualization
Domsins
Walker- mass
Quadnupod- mass
Quadruped- camping
Domains
Walker
Quadruped
Quadruped damping
mus
KCM
391.5
224.9
227.1
163.6
150.7 +: 129.7
DIAYN
4636 250.8
199.6
1837
659.
187.5
RND
361.8 1: 172.9
588.0
1: 164.8
139.5 1: 119.9
APS
555.5
566.9
155.0
192.6
Disigreement
321.6
152.6
434.2
176.7
1403
S16
$10.7 173.2
ProtoLL
DIAYN
LBS
267.6 + 155.3
380.3
227.0
212.7
456.6
508.2
421.6
222.7
209.0
17533
325-2
397.0
350.4
+ 159,9
1: 226.2
PEAC-DIAYN
RND
ICM
658-2 = 238.8
648.1 252.4
621.9
M99.4
527.4
536.1
605.7 + 170.5
695
180.1
179.4
590.2
558.4 = 175.8
557.2
1607
168.9
1695
SMM
451.2
196.6
217.3
145.9
162.5
1: 119.2
Phan2laploree
6774 245.2
660.2
162.1
608.2
157.7
APS
222,0
464.6
206.0
285
AFT
643.9 242.6
617.7 160.5
60012 149.7
CIC
205.
200.6
002.2
193.
166.2 + 126.6
LBS
6583 319.7
710.7 162.3
712.7
142.5
BeCL
544.5 1: 258,7
425.6
228.5
421.9 1: 246.9
Charesgmpher
687.8
6223 159.4
1166
PEAC (Ours)
491.3 +
631.0
235.7
573.7
PEAC-LES 7548
Table 2: Generalization results of unseen embodiments in state-based DMC (left) and image- based DMC (right). For each domain, we report the average return of each different algorithm and bold the best performance.
Isaacgym. To explore CEURL in realistic environments, we design embodiment distributions based on the Unitree Al robot in Isaacgym simulation [37], which is widely used for the real-world legged robot control [1, 74]. As Al owns 12 controllable joints, we design Al-disabled, uniform distribution of 12 embodiments, each with joint failure, respectively. It is realistic as robots may damage some joints when deploying in the real world, and they are still required to complete tasks to their best. We choose standard RL backbone PPO [50] and five downstream tasks: run, climb, leap, crawl, and tilt, following [74]. We take classical baselines for Robosuite and Isaacgym: ICM, RND, and LBS. Besides, we have deployed Aliengo robots with different failure joints to evaluate the effectiveness of PEAC in real-world applications.
5.2 Evaluation of PEAC
State-based DMC. We first report results in state-based DMC to show that PEAC can facilitate cross- embodiment pre-training. All algorithms, repeated 10 random seeds, are pre-trained 2M timesteps in reward-free environments with different embodiments, followed by fine-tuned downstream tasks for all these embodiments with 100k timesteps. We train DDPG agents for each downstream task 2M steps to get the expert return and calculate the expert normalized score for each method. Following [2], in Fig. 3, we report mean, median, interquartile mean (IQM), and optimality gap (OG) metrics along with stratified bootstrap confidence intervals. Fig. 3 demonstrates that PEAC substantially outperforms other baselines on all metrics, indicating that our cross- cross-embodiment intrinsic reward contributes positively to downstream tasks across different embodiments. Notably, compared with BeCL and CIC which get the second and third scores, PEAC not only has higher performance but also smaller confidence interval, highlighting its stability. Appendix B.4 reports detailed results of these statistics (Table 8) and individual results for each downstream task (Table 9).
Image-based DMC. As described in Sec. 4, PEAC can flexibly combine with existing unsupervised RL. methods. To verify it, we evaluate PEAC-LBS and PEAĆ-DIAYN in image-based DMC. The pre-training and finettuning steps are still 2M and 100k respectively. Also, we present four metries: Median, IQM, Mean, and OG with stratified bootstrap confidence intervals in Fig. 4. Taking IQM as our primary metric, PEAC-LBS not only has a higher value but also a relatively smaller confidence interval, indicating its better stability. As mentioned in [39], pure skill- dkill-discovery methods like DIAYN struggle on this benchmark with a certain gap compared to exploratory method. The phenomenon seems more pronounced in cross cros--ebodiment setting than single-embodinees setting, which might be because of the increased difficulty finding consistent skills across embodiments. As PEAC-DIAYN discovers skills eeoosseebbodiment, it consistently leads in performance compared with all other pure skill discovery methods across all four statisties. In Appendix B.5, we report detailed results of these statisties in Table 10 and detailed results for all downstream tasks in Table 11.
8

---

0.70
Ra Jam
0.8
0.61
0.90
CrossEEbbodinee defferent Bularo jbants
Choneographr
0.4
LB3
PEAC CPPN (Jun)
0.40
joint
0.3
13M
0.50
033
APS
52Ek
10kc
500k
Figure Ablation studies on pre-training timesteps.
Figure Real-world results.
Robosuite. Besides, we validate PEAC more challenging setting Robosuite where different embodiments own different robotic arms (subfigures 3-6 in Fig. 2). As shown in Table 1, PEAC still significantly outperforms all baselines in both training and testing embodiments, demonstrating its powerful cross-embodee eoos-emmbddmment ability and better generalization ability. The detailed results of each robotic arm are in Table 12 of Appendix B.
Ablation studies. We do several ablation studies in image-based DMC to clarify the contribution of PEAC better. First, evaluate the effectiveness of pre-trained steps in fine-tuned performance. We pre-train agents for 100k, 500k, 1M, and 2M steps and then fine-tune them for 100k steps. As shown in Fig. 6, all algorithms improve with pre-training timesteps increasing, indicating that cross-embodiment pre-training effectively benefits fast handling downstream tasks. PEAC-LBS becomes the best- perforrrfrnngg method from 1M steps on and PEAC-DIAYN significantly exceeds skill discovery methods. This suggests that PEAC excels at handling cross- coss-embbddimen tasks with increased pre-training steps. Additional results are Appendix B.7. Besides pre-training steps, we also do more ablations studies of different components in PEAC to verify their effectiveness in Appendix B.8. For example, we evaluate the stability of PEAC-LBS in DMC-image under different (we set it as 1.0 in all main experiments), which is the trade-off parameter for balancing the policy improvement term and the policy constraint term. Moreover, we also do an ablation study on our embodiment discriminator to verify the contribution of each component in PEAC. More results and analyses are in Appendix B.8.
5.3
Generalization to Unseen Embodiments
To answer the second question, further assess the generalization ability of PEAC to unseen embodiments. First, we directly leverage pre-trained agents to zero-shot sample trajectories with different unseen embodiments and then visualize results through t-SNE [57] in Fig. 5, where different colored points represent states sampled via different embodiments. As shown in Fig. 5, PEAC-LBS can distinguish different embodiments' states more effectively compared LBS, which is difficult to distinguish them (more results are in Appendix B.9). Furthermore, we evaluate the generalization ability of fine-tuned agents for all methods by zero-shot evaluating them with unseen embodiments and the same downstream task. In Table 2, we report the detailed generalization results of all domains about state-based DMC and image-based DMC. The results demonstrate that the fine-tuned agents of PEAC can successfully handle the same downstream task with unseen embodiments, which illustrates that PEAC effectively learns cross-embodiment knowledge. Detailed results for each downstream task are in Appendix B. 10 (Table 16-17).
5.4 Real-World Applications
To validate CEURL in more realistic settings, we conduct results based on legged locomotion in Isaacgym, which is widely used for real-world applications. First, we present simulation results of Al Al-disabled Table I, with 100M pre-train timesteps and 10M fine-tune timesteps. As shown in Table 1, PEAC effectively establishes good initialization model across embodiments with different joint failures and quickly adapts to downstream tasks, especially for challenging climb and leap tasks.
Besides, we have deployed PEAC fine-tuned agents in real-world Aliengo-disabled robots, i.e., Aliengo robots with different failure joints. As shown in Fig. 7, due to joint failure, the movement ability of the robot is limited compared to normal settings, but the robot still demonstrates strong adaptability on various terrains not seen in simulators. More images and videos of real-world applications are in Appendix 1...

---

5.5 Limitations and Discussion
In terms of limitations, we assume that different embodiments may own similar structures so that we can pre-train unified agent for them. As a result, it might be challenging for PEAC to handle extremely different embodiments. Also, existing unsupervised RL methods still struggle to handle more challenging downstream tasks. In Appendix B.11, we take the first step to evaluate several more challenging downstream tasks and more different embodiment distributions, of which the results show that PEAC can still perform better than baselines. Designing more efficient mros-eebbddiment unsupervised algorithms for these more difficult and practical settings are interesting future directions. The Broader Impact this work is discussed in Appendix D.
Conclusion
In this work, we propose to analyze cross-embodiment RL in an unsupervised RL perspective as CEURL, i.e., pre-training in an embodiment distribution. We formulate it as CE-MDP, with some more challenging properties than the single-embodiment setting. By analyzing the optimal cross-embodiment initialization, we propose PEAC with a principled intrinsic reward function and further show that PEAC can flexibly combine with existing unsupervised RL. Experimental results demonstrate that PEAC can effectively handle downstream tasks across embodiments for extensive settings, ranging from image-based observation, state-based observation, and real-world legged locomotion. We hope this work can encourage further research in developing RL agents for both task generalization and embodiment generalization, especially in real-world control.
Acknowledgments and Disclosure of Funding
This work supported by NSFC Projects (Nos. 92248303, 92370124, 62350080, 62276149,
036611) BNRist (BNR2022RC01066, Tsinghua Institute for Guo Qiang, and the High Performance Computing Center, Tsinghua University. J. Zhu was also supported by the XPlorer Prize.
References
[1] Ananye Agarwal, Ashish Kumar, Jitendra Malik, and Deepak Pathak. Legged locomotion in challenging terrains using egocentric vision. In Conference on robot learning, pages 403 415. PMLR, 2023.
[2] Rishahh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C. Courville, and Mare Bellemare. Deep reinforcement learning at the edge of the statistical precipice. Advances in neural information processing systems, 14:29304-29322 2021.
[3] Bowen Baker, 1lge Akkaya, Peter Zhokov, Joost Huizinga, Jic Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune. Video pretraining (vpt): Learning act by watching unlabeled online videos. Advances in Neural Information Processing Systems, 35:24639-24654, 2022.
[4] Michael Beukman, Devon Jarvis, Richard Klein, Steven James, and Benjamin Rosman. Dynamics generalisation in reinforcement learning via adaptive context-aware policies. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[5] Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski. Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15816 2023.
language-actin models
[6] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subhiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learnrr. Advances in neural information processing systems, 33:1877-1901, 2020.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In International Conference on Learning Representations, 2018
[8] Yevgen Chebotar, Quan Vuong. Karol Hausman, Fei Xia, Yao Lu, Alex Irpan. Aviral Kumar, Tianhe Yu, Alexander Herzog, Karl Pertsch, et al. Qrtansformer:: Scalable offline reinforcement learning via autoregressive u-functions. In Conference Robot Learning, pages 3909-3925 PMLR, 2023.
10

---

[9] Thomas M Cover and Joy A Thomas. Elements of information theory. 1991.
[10] Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. In International Conference on Learning Representations, 2018.
[11] Benjamin Eysenbach. Ruslan Salakhutdinov, and Sergey Levine. The information geometry unsupervised reinforcement learning. In International Conference on Learning Representations. 2021.
[12] Gilbert Feng, Hongbo Zhang, Zhongyu Li, Xue Bin Peng. Bhuvan Basireddy, Linzhu Yue, Zhitao Song, Lizhi Yang. Yunhui Liu, Koushil Sreenath, et al. Genloco: Generalized locomotion controllers for quadrupedal robots. In Conference Robot Learning, pages 1893 1903. PMLR, 2023.
[13] Dibya Ghosh, Chethan Anand Bhateja, and Sergey Levine. Reinforcement learning from passive data via latent intentions. International Conference on Machine Learning, pages 11321-11339, PMLR, 2023.
[14] Dibya Ghosh, Jad Rahme, Aviral Kumar, Amy Zhang, Ryan Adams, and Sergey Levine. Why generaliza- tion in rl is difficult: Epistemic pomdps and implicit partial observability. Advances in Neural Information Processing Systems, 34:25502-25515 2021.
[15] Kevin Gmelin, Shikhar Bahl, Russell Mendonca. and Deepak Pathak. Efficient rl via disentangled environment and agent representations. International Conference on Machine Learning, pages 11525- 11545. PMLR, 2023.
[16] Agrim Gupta, Silvio Savarese, Surya Ganguli, and Li Fei-Fei. Embodied intelligence via learning and evolution. Nature communications, 12(1):5721, 2021.
[17] Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In International Conference on Learning Representations, 2019.
[18] Danijar Hafner, Timothy P Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. In International Conference on Learning Representations, 2020.
[19] Assaf Hallak, Dotan Di Castro, and Shie Mannor. Contextual markov decision processes. arXiv preprint arXiv:1502.022 1502.0029,, 2015.
[20] Steven Hansen, Will Dabney, Andre Barreto, David Warde-Farley, Tom Van de Wiele, and Volodymyr Mnih. Fast task inference with variational intrinsic successor features. In International Conference on Learning Reprexentations, 2019.
[21] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 16000-16009, 2022.
[22] Shuncheng He, Yuhang Jiang, Hongchang Zhang, Jianzhun Shao, and Xiangyang Ji. Wasserstein unsuper- vised reinforcement lcarning. Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 6884 6892, 2022.
[23] Edward S Hu, Kun Huang, Olch Rybkin, and Dinesh Jayaraman. Know thyself: Transferable visual control policies through awarenss..ssss. In International Conference on Learning Representations, 2021.
[24] Zheyuan Jiang. Jingyue Gao, and Jianyu Chen. Unsupervised skill discovery via recurrent skill training. Advances in Neural Information Processing Systems, 099999044439946..0 2022.
[25] Jaekyeom Kim, Seohong Park, and Gunhee Kim. Unsupervised skill discovery with bottleneck option learning. In International Conference on Machine Learning, pages 5572-5582, PMLR, 2021.
[26] Michael Laskin, Hao Liu, Xue Bin Peng, Denis Yarats, Aravind Rajeswaran, and Pieter Abbeel. Unsuper- vised reinforcement learning with contrastive intrinsic control. Advances in Neural Information Processing Systems, 35:34478-34491, 2022.
[27] Michael Laskin, Denis Yarats, Hao Liu, Kimin Lee, Albert Zhan, Kevin Lu, Catherine Cang, Lerrel Pinto, and Picter Abbecl. Urlb: Unsupervised reinforeement learning benchmark. In Thirty-fithh Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
[28] Kimin Lee, Younggyo Seo, Seunghyun Lee, Honglak Lee, and Jinwoo Shin. Context-awan dynamics model for generalization in model-basee reinforcement learning. In International Conference on Machine Learning, pages 5757-5766. PMLR. 2020.

---

[29] Kuang-Huei Lee. Ofir Nachum. Mengjiao Sherry Yang, Lisa Lee. Daniel Freeman, Sergio Guadarrama, lan Fischer, Winnie Xu, Eric Jang. Henryk Michalewski, al. Multi-game decision transformers. Advances in Neural Information Processing Systems, 35:27921-27936, 2022.
[30] Lisa Lee, Benjamin Eysenbach, Emilio Parisotto, Eric Xing, Sergey Levine, and Ruslan Salakhutdinov. Efficient exploration via state marginal matching. arXiv preprint 196.000906.02774 2019.
[31] Sang-Hyun Lee and Scung-Wor Woo Sco. Unsupervised skill discovery for learning shared structures across changing environments. In International Conference Machine Learning, pages 19185-19199 PMLR, 2023.
[32] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offfine reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:200S.0 2020.
[33] Timothy Lillicrap. Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv: 000.1/509999711 2015.
[34] Hao Liu and Picter Abbeel. Aps: Active pretraining with successor features. In International Conference on Machine Learning, pages 6736 6747. PMLR, 2021.
[35] Hao Liu and Pieter Abbeel. Behavior from the void: Unsupervised active pre-training Advances in Neural Information Processing Systems, 34: 10111885188755 2021.
[36] Xin Liu, Yaran Chen, Haoran Li, Boyu Li, and Dongbin Zhao. Cross-domain random pre-training with protolypes for reinforcement learning. arXiv preprint arXiv:2302.05611 2023.
[37] Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, al. Isaac gym: High performance gpu based physics simulation for rohot learning. Thitty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
[38] Pietro Mazzaglia, Ozan Catal, Tim Verbelen, and Bart Dhoedt, Curiosity-drive exploration via latent bayesian surprise. In Proceedings of the AAAI Conference Artificial Intelligence, volume 36, pages 7752-7760, 2022.
[39] Pietro Mazzaglia, Tim Verbelen, Bart Dhoedt, Alexandre T. Lacoste, and Sai Rajeswar. Choreographer: Learning and adapting skills imagination. The Eleventh International Conference on Learning Representations, 2022.
[40] Mirco Mutti, Mattia Mancassola, and Marcello Restelli. Unsupervised reinforcement learning in multiple environments. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 7850-7858, 2022.
[41] Fei Ni, Jianye Hao, Yao Mu, Yifu Yuan, Zheng. Bin Wang, Zhixuan Liang. Metadiffuser: Diffusion model as conditional planner for offline meta-rl. In International Conference on Machine Learning, pages 26087-26105. PMLR, 2023.
[42] Scohong Park, Jongwook Choi, Jackyeom Kim, Honglak Lee, and Gunhee Kim. Lipschitz-consttriine unsupervised skill discovery. arXiv preprint arXiv:2202.00914, 2022.
[43] Seohong Park, Kimin Lee. Youngwoon Lee, and Pieter Abbeel. Controllabilit--ware unsupervised skill discovery. arXiv preprint arXiv:2302.05100 2023.
[44] Seohong Park, Oleh Rybkin, and Sergey Levine. Metra: Scalable unsupervised rl with metric-aware abstraction. arXiv preprint 2rXiv:2310.08887, 2023.
[45] Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International conference machine learning, pages 2778-2787. PMLR, 2017.
[46] Deepak Pathak, Dhiraj Gandhi, and Abhinav Gupta. Self-supervised exploration via disagreement. In International conference on machine learning, pages 5062-5071. PMLR, 2019.
[47] Vitchyr Pong, Murtaza Dalal, Steven Lin, Ashvin Nair, Shikhar Bahl, and Sergey Levine. Skew-fit: State-coverte cate--oovering self supervised reinforcement learning. In International Conference on Machine Learning. pages PMLR, 2020.
12

---

[48] Sai Rajeswar. Pietro Mazzaglia, Tim Verbelen, Alexandre Piché, Bart Dhoedt, Aaron Courville. and Alexandre Lacoste. Mastering the unsupervised reinforcement learning benchmark from pixels. In International Conference on Machine Learning, pages 28598-28617. PMLR, 2023.
[49] Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gómez Colmenarejo, Alexander Novikov, Gabriel Barth-maron, Mai Giménez. Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al. A generalist agent. Transactions on Machine Learning Research, 2022.
[50] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 2017.
[51] Ramanan Sekar, Oleh Rybkin, Kostas Daniilidis, Picter Abbeel, Danijar Hafner, and Deepak Pathak. Planning to explore via self-supervised world models. In International Conference on Machine Learning. pages 8583-8592. PMLR, 2020.
[52] Milad Shafice, Guillaume Bellegarda, and Auke ljspeert. Manyquadrupeds; Learning single locomotion policy for diverse quadruped robots. arXiv preprint arXiv:2310. 10466, 2023.
[53] Nur Muhammad Mahi Shafiullah and Lerrel Pinto. One after another: Learnnng incremental skills for changing world. In International Conference on Learning Representations, 2021.
[54] Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics-aware unsupervised discovery of skills. In International Conference on Learning Representations, 2019.
[55] DJ Strouse, Kate Baumli, David Warde-Farley, Volodymyr Mnih, and Steven Stenberg Hansen. Learning more skills through optimistic exploration. In International Conference on Learning Representations, 2021.
[56] Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Frez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefraneq, et al. Deepmind control suite. arXiv preprint arXiv:1801.00660. 2018.
[57] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne, Journal of machine learning research, 911), 2008.
[58] Mengda Xu, Zhenjia Xu, Cheng Chi, Manuela Veloso, and Shuran Song. Xskill: Cross embodiment skill discovery. In Conference on Robot Learning, pages 3536-3555. PMI.R, 2023.
[59] Yifan Xu, Nicklas Hansen, Zirui Wang, Yung-Chich Chan, Hao Su, and Zhuowen Tu. On feasibility of cross-tase task transfer with model-based reinforeement lcarning. In The Eleventh International Conference on Learning Representations, 2022.
[60] Jonathan Yang. Catherine Glossop. Arjun Bhorkar, Dhruv Shah, Quan Vuong, Chelsea Finn, Dorsa Sadigh, and Sergey Levine. Pushing the limits of cross-embodiment learning for manipulation and navigation. arXiv preprint arXiv:2402.1944 19432. 2024.
[61] Rushuai Yang, Chenjia Bai, Hongyi Guo, Siyuan Li, Bin Zhao, Zhen Wang. Peng Liu, and Xuelong Li. Behavior contrastive learning for unsupervised skill discovery. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning. volume 202 of Proceedings of Machine Learning Research, pages 39183 39204. PMLR, 23 Jul 2023.
[62] Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Reinforcement learning with prototypical representations. In International Conference on Machine Learning, pages 11920-11931. PMLR, 2021.
[63] Denis Yarats, Ilya Kostrikov, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. In International conference on learning representations, 2020.
[64] Chengyang Ying, Zhongkai Hao, Xinning Zhou, Hang Su, Songming Liu, Dong Yan, and Jun Zhu. Task aware dreamer for task generalization in reinforcement learning. arXiv preprint arXiv:2303.00092 2023.
[65] Chengyang Ying, Xinning Zhou, Hang Su, Dong Yan, Ning Chen, and Jun Zhu. Towards safe reinforcement leamning via constraining conditional valuc-at-risk. arXiv preprint ar Xiv: 04436, 2022.
[66] Chen Yu, Weinan Zhang, Hang Lai, Zheng Tian, Laurent Kneip, and Jun Wang. Multi-enbodiment legged robot control as sequence modeling problem. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 7250-7257 IEEE, 2023.
13

---

[67] Mingqi Yuan, Bo Li, Xin Jin, and Wenjun Zeng. Automatic intrinsic reward shaping for exploration in deep reinforcement learning. arXiv preprini wrXiv:2301.10880 2023.
[68] Zhecheng Yuan, Tianming Wei, Shuiqi Cheng, Gu Zhang, Yuanpei Chen, and Huazhe Xu. Learning to manipulate anywhere: A visual generalizable framework for reinforcement learning. arXiv preprint xrXiv2200.15815.2222 2024.
[69] Zhecheng Yuan, Sizhe Yang, Pu Hua, Can Chang, Kaizhe Hu, and Huazhe Xu. Rl-vigen: reinforcement learning benchmark for visual generalization. Advances in Neural Information Processing Systems, 36, 2024.
[70] Kevin Zakka, Andy Zeng, Florence, Jonathan Tompson, Jeannette Bohg. Debidatta Dwibedi. Xirl: Cross-embodiment inverse reinfoncement learning. In Conference on Robot Learning, pages 537-546. PMLR, 2022.
[71] Amy Zhang. Shagun Sodhani, Khimya Khetarpal, and Joelle Pineau. Learning robust state abstractions for hidden- paremmerr block mdps. arXiv preprint arXiv:2007.0720 2020.
[72] Andrew Zhao, Matthieu Lin, Yangguang Li, Yong-Jin Liu, and Gao Huang. A mixture of surprises for unsupervised reinforcement learning. Advances in Neural Information Processing Systems, 35:26078 26090, 2022.
[73] Yuke Zhu, Josiah Wong, Ajay Mandlekar, Roberto Martín-Martinn Abhishek Joshi. Soroush Nasiriany, and Yifeng Zhu. robosuite: A modular simulation framework and benehmark for robot learning. arXiv preprint wxXiv:2099112290 2020.
[74] Ziwen Zhuang, Zipeng Fu, Jianren Wang. Christopher G Atkeson, Sören Schwertfeger, Chelsea Finn, and Hang Zhao. Robot parkour learning. Annual Conference on Robot Learning, 2023.
14

---

A Proof of Theorems
In this section, we will provide detailed proof of theorems in the paper.
A.1 Properties and Challenges of DE
We first construct an example to show that vertices of may no longer be deterministie policies.
Considering a simple embodiment distribution with only embodiments é E2 with the embodiment probability p(e1)= p(e2) For each embodiment, there two states 81; 82 and two actions a1 02 and the dynamic is
Pej (81 81, 01) = P.p (82 81, 01) = Pe (8) 81, a2) = 0, Pe, (82 81, 02) = 1
Pe: (81 82, a1) = =0.Pes S2; a Pe ($) 82, a2) 1, Pei 82, a2) 0
(8)
Pe, (81 $1, a1) = =0. (22 81, 01 = Pep ($1 81, a2) = Pe, (82 81, 02) 0
Poz (81 82, a1) Pe, (82 82, 01) Peg (81 82, (2)) 0, Peg (2, 02) 1,
St
81
S2
a2
al
el
e2
In this setting, there are four deterministic policies:
Ti (11) a1, (82) a1
(2)) a1 T2(22) a2. T4 (81) = a22 (82) a2.
(9)
33 (1) = a2, (3( = an,
For any policy μi, we denote that Pr,u P2,μ are the state distribution of μu under the environment Pe. or Pe, respectively. Then we can calculate that
1
P1xx
2' 2
P2.m
22'2)
1 1+7
7
1 7-7 1 1+?)
P1.
P2,ma
2
2
(10)
1
1-7
177
1+7 1 1
P1.x3
.P2,m
2
2
1
P1.Za
2' 2'2)
P2,
2'
As the embodiment probability is p(e1) = p((2) = 2' all these four policy share the same state distribution as
Pm. = Pr. Pr. Pr. 2'
(11)
Furthermore, we consider a stochastic policy ग satisfies that
1
1
t (()))8)) = 1, (a2ls)) = 0, (aI182) = 5 ग. (a2 (a2)))))
(12)
Next we will calculate Pr,m and P2 For Pl, at the timestep 0, we have the initial state distribution as (s1) = Po(s2) = 1 naturally get the recurrence relation
2. assume that at timestep we have corresponding (r())), (82), we can
(s+1(s1) (1() +
2 P((s)),
(e41(82) = p())
(13)
Naturally, we have (s1) = -1 (s and thus the discount state distribution of $2
1 1-7
(1- 7)
(14)
2
2-7
t=u
15

---

And we have
1 2 2
P1,x
(15)
Similarly, we can calculate P2,=- At the timestep 0, we have the initial state distribution as Po P81) = Po(*2) 2. assume that at timestep we have corresponding (ss1), we can naturally get the recurrence relation
1 Pt+1(s1) p)((s), P(+1(82) pr((s) pr((s),
(16)
As p: + px(82) = 1, we can solve this recurrence relation via
1
((11 ()(())) 2 P.(())),
(-2) (81) ((-2)'p:(8)) (-2)²
=.. (-2)) pa(s1) ((-2)* + (-2))
+ + (-2)9)
(17)
1- (-2)
(-2)
1
Pt+1 (81)
6 (-2)
3
Thus the discount state distribution of 81 is
(6
1
1
1-7
1
(1
6 (-2)
+
3
=
6
2
3
+
6
14
2
(18)
t=0
t=
And we have
1+
P2.x
(19)
12 7 2+7
As the embodiment probability is p(e1) = p(ez) 2. the state distribution of is
2 Pr --
(20)
Taking any 7 (0, 1), it obvious that Per is not within the closure composed of (r1: Praz Pwa: Pr. (actually the point 1/2)). Thus we have explained that the vertices of DE might no longer be simple deterministic policies.
A.2 Proof of Theorem 3.2
Proof. Recall that
F(z,T" n", Rea; e) a (G) Epng. (r) (r) [Rex(T)) - DD. (F)
(21)
We set functional f satisfying that
f(p(T)) = Ep()) - (r) - 3DxL (p()))l
(22)
Using the calculus of variations, we can calculate its optimal value at the point p* satisfying that
(²(T) + b3,
Real =B=0o PMt,x (T)
(23)
here is a constant not related to p", and we have (T( = (T)e Ray(r) b As fp = we can calculate that
PM. Rafh1
b 0=1ogg log / "M. Rayle) dr, p²(r)
(24)
P dr
16

---

Consequently, we have
max F(T, T,, Rex. e) = Ep' (r) [Txx -Bp (t BDL (F (T)llmm
PMg.a (7)
-f p" (T)) - Epncs (r) (r) [Txx(T)] - p²()) log 1²( dr
(T) Reu(T)
= i - (e())Rea)T)d - Epucs (+) Read - 3 p²(+t)
dr + B Blog P. Raylo) dr
B
log / (T Baylo) dr E--pp (r) (T)((T)).
(25)
Similarly, set functional 9 satisfying that
g(r(r)) Blog / (T), -(2 dr Peng.-())((() (r())
(26)
Using the calculus of variations, calculate its optimal value at the point r satisfying that
B PM. (T)e (1))
("((T) P.n + log f (P)) dt) dr. (27)
-PMg,*(T),
log
f (T) dr
B
(T)
Thus we can calculate that
min max F(T, Rex, e)= log PMM, (T) F dr (1)()(())r(())) Rei(T) (M)
r((r)
P log / (T) -00 dr -BEpgparrr1 log (T) (T)
- 3 log (M - 2d dr (28)
=- BDI. (PMe:
i.c.
Beng min max Rai(T) PM (T)
F(Ir, n², Rea, e) B.ae [- DRL (FME. (T))
log (x(()
log (T)
-e Br
= Bleee By BTNPME
Pr (T[T|ME)
(r(er))/p(e)
(29)
p(e)
log p(e)
log
BE B.e
Pa (e, T))
Pa (elT)
Thus we have proven this result.
A.3 Detailed Discussion and Proof about Embodiment-Aware Skill Discovery
Here discuss cross-embodiment skill-based adaptation objective.
We begin by proving Eq. (7). First, we show that
Ex min max Rea(T) PMg.-- (r) Fa(m, π.,, Rest, e)
(30)
Eg
Ex)D(=)MM) [3DKL (1xc,((r1))lllxx..((())))) p(=(M{)
Our proof is similar to the proof in Appendix A.2. Recall that
F. xt" Rexi e)
E Ppgg -- (T) max Sp [Rea(T))
(31)
DDK (7) (7)
Similar to Eq. (22)-Eq, (25), we have
max
F
[Rea([T)] (re))
max F Pugg (r|x*) (Rea - Dr
PM (7|z)
(TM,( Bayl) dr max E
=3 log
- (7) = (
(T)
B Blog / (M)
Rey(t) dr E PPMgg=(T1= (Rexa))
= min 2"
(32)
17

---

Also, similar to Eq. (26)-Eq. (28), we have
B log / (Pa
Regit) pp x Rexi(
min (a)(F)
min *"
3 Blog / (T
Realt) dr -E prg (7/2*) Rexi (T))
(33)
min min Rea(7)
min [-3DKI.
Thus we have
Ee min
max Fa( n,; Rex: e)
Rei(T) (r)
= -Ee min [- (-8DKL (Tl
(34)
Ee max [8DKL (pm - (T)=") (T))]
Eh max Re [8DeL. p(=lMs)
where the last equality holds from the fact that the maximum is achieved when putting all the probability weight on the input 2 maximizing DRL. (T)))
Next, we will show that DEL (TM))=(T|=))llMMM... (T) is a general form our Theorem 3.2 and the results the single-embodiment setting [11]. Naturally, when we ignore z, FT x², Reas e) will degenerate into F(T, TT, Rest: e), and Eq. (7) will also degenerate into Eq. i4), the results in
Theorem 3.2. On the other hand, if we change Eq. (7) into the single-embodiment setting, i.e., & is Dirac distribution with the probability p(e)
for some fixed then have
max min max Fo Fa (,,,""' x", Rex,, e)
Ra)(T) P.M... (T)
max E Be-p(z/Mg) [8DKL (p.g. (TM.)r((==))llmmm... (T)] p(z(Mg)
max
(35)
min max E Ez)p(x) 13DKL (PM (Tl)llc(ll)llllxx....
If p(=)
min max E E [L (p(T)=)]lP(T)))
P p(z)
the last approximation simplifies the complex coupling relationship between and following [11]. Furthermore, by Lemma 6.5 in [11] (proof in Theorem 13.1.1 from [9]), we have
min max Ez~p(z) [DL (p(r [p(T|=)]lp(T)] max I(T;²). p(z)
(36)
p(*)
which is the objective existing single- emgeddeembo -iiccovery methods.
Finally, we will Eq. (7), which further indicates that our cross-embodiment skill-based objective can be decomposed into two terms: one for handling cross-embodiment while the other aims at discovering skills. Actually, we have
Be min max Fa( n², Rest e)
PMg
= =Bexp(x/MM)
[DLL.
Pa (,,
Pr(2, dzdr log [T( dzdt= = / (r( log
p(e)
Pa (T)
(r 2)
-f (
Pa (zle, dzdr
(37)
(=(e)p((z(e))
-f (=(e))) (C dr + Px (r( (=(e))) dzdr
PT(T, z|e)
(e)
log (e(TT) + Dwu.ffm(trzeeellpseeeee
-E
(e(e)
18

---

B Experimental Details
In this section, we will introduce more detailed information about experiments, In Sec. B. we introduce the detailed environments and tasks used in our experiments. In Sec. B.2, we will illustrate all the baselines compared in experiments. Also, all hyper-parameters of experiments in Sec. B.3. Moreover, we supplement more detailed experimental results about state-based DMC, image-based DMC, and Robosuite in Sec. B.4, Sec. B.5, and Sec. B.6, respectively. Then we conduct detailed generalization results of pre-trained models and fine-tuned models in Sec. B.9 and Sec. B.10, respectively. Finally, we report more detailed real-world experiments in Sec. B.12.
B.1 Embodiments and Tasks
State-based DMC. This benchmark is based on DMC [56] and URLB [27] with state-based
observation. Each domain contains one robot and four downstream tasks. We extend it into the cross-embodiment settings: Walker-mass, Quadruped-mass, and Quadruped-dampae mass extends the Walker robot in DMC, which is a two-leg robot, and designs a distribution with different mass m, i.e., m times the mass standard walker robot. Similarly, Quadruped-mass also considers quadruped robots with different mass Quadruped-damping on the other hand, changes the damping of the standard quadruped robot with / times. The detailed parameters of training embodiments and generalization embodiments are in Table 3.
Walker-
Image-based DMC. This benchmark is the same with state-based DMC but with image-based based observation. Thus we consider similar three embodiment distributions: Walker-mass, Quadruped- mass, and Quadnuped-damping.
Train
Generalization
Walker-mass
m E {0.2, 0.6, 1.,, 1.8 (0.4, 0.8, 1.O, 1.4
m E (0.4, 0.8, 1.2, 1.6]
Quadruped-mass
{0.6, 1.2)
pandrrupd-dampiing E (0.2,0.6, 1.0, 14, 1.8)
LE (0.4, 0.8, 1.2, 1.6)
Table 3: Environment parameters used for state-based DMC and image-based DMC.
Robosuite. This benchmark utilizes the environment in [73] and follows the experimental setting in RL-Vigen [69], of which the cross-embodiment setting includes Panda, IIWA, and Kinova3. Here different embodiments may own different shapes (observations), and dynamics. Similarly, we pre- train cross-embodiments in all these three embodiments and fast fine-tune the pre-trained agents to downstream tasks. Besides these three embodiments, models in one unseen embodiment: Jaco, to validate the cross- cooodieebod generalization ability of CEURL. For task sets, we consider three widely used tasks: Door, Lift, and TwoArmPegInHole Noticing that although these three tasks can be finished by the same robots, their demand for robotic arms varies lot. For example, IwooArmPelIHHole needs two robotic arms but the other two tasks only need one. Consequently, we pre-train cross-embodiment agents for each single task, for all methods.
also directly fine-tune our pre-trained
Isaacgym. We first design a setting in simulation based on Unitree Al in Isaacgym, which is challenging legged locomotion task and is widely used for real-world legged locomotion. The action space of A1 a 12-dimension vector, representing 12 joint torque. Thus we consider our A1 Al-disabled benchmark, including 12 embodiments, each of which owns joint torque failure, i.e., the torque output of this joint is always 0 in this embodiment. This setting is practical as our robot may experience partial joint failure during use, and we still hope that it can complete the task as much as possible.
Moreover, we deploy PEAC into real-world Aliengo robots with failure joints. Similarly, we consider the embodiment distribution Aliengo-disabled, which owns 12 embodiments, each of which owns a joint torque failure respectively. We first pre-train a unified agent across these 12 embodiment in reward-free environments. During fine-tuning, for each embodiment, we utilize the same pre-trained agent to fine-tune the given moving task through this embodiment. Finally, we deploy the fine-tuned agent into the real-world setting to evaluate its movement ability under dif different kinds of terrains with joint failure.
19

---

B.2 Baselines and Implementations
ICM [45]. Intrinsic Curiosity Module (ICM) designs intrinsic rewards as the divergence between the projected state representations in feature space and the estimations made by a feature dynamics model.
RND [7]. Random Network Distillation (RND) utilizes a predictor network's error in imitating randomly initialized target network to generate intrinsic rewards, enhancing exploration in learning environments.
Disagreement Plgp/lleeExee [51]. The Disagreement algorithm leverages prediction variance across multiple models to estimate state uncertainty, guiding exploration towards less certain states. The Plan2Explore algorithm employs self-supervised, world-model-basee framework, using model disagreement assess environmental uncertainty and incentivize exploration in sparse-reward scenarios.
ProtoRL [62]. Proto-RL combines representation learning and exploration through a self- supervised learning framework, using prototype representations pre-train fusk-independent repre- sentations in the environment, effectively improving policy learning in continuous control tasks.
APT [35]. Active Pre-training (APT) estimates entropy for a given state using a particle-basee based estimator based on the K neirest-neighhors algorithm.
LBS [38]. Latent Bayesian Surprise (LBS) applies Bayesian surprise within latent space, effi- ciently facilitating exploration by measuring the disparity between an agent's prior and posterior beliefs about system dynamics.
Choreographer [39]. Choreographer is model-based approach in unsupervised skill learning that employs a world model for skill acquisition and adaptation, distinguishing exploration from skill learning and leveraging meta-controller for efficient skill adaptation in simulated scenarios, enhancing adaptability to downstream tasks and environmental exploration.
DIAYN [10].
Diversity is All You Need (DLAYN) autonomously learns a diverse set of skills by maximizing mutual information between states and latent skills, using maximum entropy policy.
SMM [30]. State Marginal Matching (SMM) develops agssosgnottic exploration strategy by learning policy to match the state distribution of an agent with a given target state distribution.
APS [34]. Active Pre-training with Successor Feature (APS) maximizes the mutual information between states and task variables by reinterpreting and combining variational successor features with nonparametric entropy maximization.
LSD [42]. Lipschitz-conee constraind Skill Discovery (LSD) adopts a sipschitz-constrainne state repre- sentation function, ensuring that maximizing this objective in the latent space leads to increase in traveled distances or variations in the state space, thereby enabling the discovery of more diverse, dynamic, and far-reaching skills.
CIC [26]. Contrastive Intrinsic Control (CIC) is unsupervised reinforcement learning algorithm that leverages contrastive learning to maximize the mutual information between state transitions and latent skill vectors, subsequently maximizing the entropy of these embeddings intrinsic rewards to foster behavioral diversity.
BeCL [61]. Behavior Contrastive Learning (BeCL) utilizes contrastive learning for unsupervised skill discovery, defining its reward function based on the mutual information between states generated by the same skill.
Next, we will introduce the implementations of basclines for all experimental settings.
20

---

For state-based DMC, almost all baselines (ICM. RND, Disagreement, ProtoRL, DIAYN, SMM, APS) combined with RL backbone DDPG are directly following the official implementation in urlb (https: //g thtb.com//ll--rssearrhhhe b_t__cbbre_rr)) For LBS, we refer the imple- and combine with the code of urlb. For other more recent baselines, we also follow their official implementations, in- cluding CIC (https:/ cttchuu..ommmllll// com/rll wwwerrreeeeeerhhleee cic) and BeCL (https:/ /tttp:://ithub... com/ Rooshy-yang/Bede
mentation in [48] (https://gi thub.
For image-based DMC, almost all baselines (ICM, RND, Plan2Explore, APT, LBS, DIAYN, APS) combined with RL backbone DreamerV2 are directly following the official implementation in [48] (https: / gi thub. com/ commmazpie// /tappiiimmttttinngg... urlb). which currently achieves the leading perfor- mance in image-based based DMC of urlb. For CIC, we combine its official code (https: / github. com/ril-reseeach/ cic), which mainly considers state-based DMC, and the DreamerV2 backbone in [48]. Similarly, for LSD, we refer to its official code (https: //gi thub. com/ mmmm/s/oooogarr//LD and combine it with the code of [48]. For Choreographer, of which the backhone is DreamerV2, we directly utilize its official code (https / thub. cwm
For Robosuite, our code is based on the code of RL-Vigen [69] (https:/ http///mmoollccec github. io/RL- Vii-V,,,), including the RL backbone DrQ. For Isaacgym, our code is based on the official code of [74] (https: /gg thub. com/ /i Zhuang parkour), which implements five downstream wwwennhuannaaarourrrrrr
tasks (run, climb, leap, crawl, tilt). For these two settings (Robosuite and Isaacgym), as there are few works considering unsupervised RL in such a challenging setting, we implement classical com/ baselines (ICM, RND, LBS) by referring their implementations in urlb (https: rll ell-research/ url_ wawpaarrc/urrebececheeee chmarkk and [48] (https: //gi /gtbt con./mappie/mastrrrngg-gg
B.3 Hyper-parameter
Baseline typer-parameters are taken from their implementations (see Appendix B.2 above). Here we introduce PEAC's hyper-parameters. For all settings, papeapparaeeters of RL backbones (DDPG, DreamerV2, V2, PPO) follow standard settings.
First, for PEAC in state-based DMC with RL backbone DDPG, our code is based on urlb (https: /ttp:://githbb com/ com/mazpion e/ /ww.ttriinnn....///e unrbi) and inherits hyper-parameter of DDPG. For completeness, we list all hyyper-parametery
DDPG Hyper-parann parametrrrrttrr
Value
Replay buffer capacity
10
Action repeat
Seed frames
4000
n-step returns
3
Mini-batch size
1024
Seed frames
4000
Discount 7
0.99
Optimizer
Adam
Learning rate
lc-4
Agent update frequency
Critic target EMA rate TQ
0.01
Features dim.
1024
Hidden dim.
1024
Exploration stddev clip
0.3
Exploration stddev value
0.2
Number pre-training frames
2 1x
Number fine-turning frames
I 1x10
PEAC Hyper-paramette
Value
Historical information encoder
GRU (dim(S) + din -> 1024)
Encoded historical information length
10
Embodiment context model
MLP (1024 =mmboddiment context dim)
Table 4: Details of hyper-parameterr used for state-based DMC.
Next, for PEAC-LBS and PEAC-DIAYN imsee DMC with RI. backbone DreamerV2, our code is based on [48] (uttps://ee ://gtt/ub.t/t//// com comm/naapie//msttrring urlb). parea--arameters of
21

---

PEAC-LBS and PEAC-DIAYN inherit DreamerV2's hyper-parametery as well as inherit hyper- parameters of LBS and DIAYN, respectively.
Dreamer Hyper-parameet
Value
Environment trames/update
MLP number of layers
4
MLP number of units
400
Hidden layers dimension
400
Adam epsilon
x10
Weight decay
x x10
Gradient clipping
100
World Model
Batch size
50
Sequence length
50
Discrete latent state dimension
32
Discrete latent classes
32
GRU cell dimension
200
KL free nats
1
KL balancing
0.8
Adam learning rate
x10-4
Slow critic update interval
100
Actor-Critic
Imagination horizon
Discount 7
0.99
GAE A
0.95
Adam learning rate
x10-
Actor entropy loss scale
x 100
PEAC-LBS crrper-paramete
Value
Embodiment context model
MI.P (DreamerV2 encoder dim + 200 + 200 + Emboddment context dim)
LBS model
MLP (DreamerV2 encoder dim + 200 + 200 - 200 200 -+ 1)
PEAC-DIAYN Hyper-parameter
Value
Embodiment context model
MLP (DreamerV2 encoder dim + 200 + + Embodiment context dim)
DIAYN model
MLP (DreamerV2 encoder dim + 200 - 200 -> skill dim)
Table 5: Details of hyper-purameterss used for image-based DMC.
Then, for PEAC in Robosuite, our code follows RL-Vigen [69] (tttpps:/gemcoolectorr wwt.encoollttooiitubb........... io/RL- DLGnn.. PEAC's syper-parameters, inheriting DrQ's hyperparameters, include
DrQ Hyper-parameter
Value
Discount Tactor
0.99
Optimizer
Adam
Learning rate
le-4
Action repeat
N-step return
Hidden dim
1024
Frame stack
Replay Buffer size
1000000
Feature dim
50
PEAC Elyper-paramett
Value
Historical information encoder
GRU (Encoder Feature Dim dim + 50)
Encoded historical information length
10
Embodiment context model
MLP (50 Emmbodimmnt context dim)
Table 6: Details of hyper parameters used for Robosuite.
Finally, for PEAC in Al-disabled of Isaacgym with RL backbone PPO, our code follows [74] (uttps://github www/ZivenZhuang////oooooeaoo (atpieennZZuaang/kkrororrr. PEAC's hyper-parameters,, inheriting PPO's hyperparameters, include
22

---

PPO Hyper-parameter
Value
PPO clip range GAE A Learning rate
0.2
0.95
le-4
Reward discount factor Minimum policy std
0.99
0.2
Number of environments
4096
Number of environment steps training batch
24
Learning epochs per training batch
5
Number of mini-batches per training batch
4
PEAC Hyper-parameler
Value
Historical information encoder
GRU (dim(S) + dim (A) - 128)
Encoded historical information length
24
Embodiment context model
MLP (128 Emboodmenn context dim)
Table 7: Details of typer-parametent used for Isaacgym.
B.4 Detailed results in state-hased DMC
In Table 8, we present detailed results in state-based DMC of all four statistics (medium, IQM, mean, OG) for baselines and our PEAC. The results indicate that PEAC performs the best in all these four metrics, while BeCL and CIC perform second and third respectively. Moreover, we report individual results for each downstream task of state-based DMC in Table 9. PEAC performs comparably to BeCL as well as CIC in the Walker-mass tasks and best on most Quadruped-man and Quadruped-damping tasks. Especially, in the challenging Quadruped-damping setting, PEAC can complete cooss-embodmment downstream tasks and significantly outperforms BeCL and CIC.
Metrics
Median
IQM
Mean
Optimality Gap
ICM
0.37
0.35
0.37
0.63
RND
0,48
0.47
0.47
0.53
Disagreement
0.40
0.39
0.38
0.62
ProtoRI.
0.52
0.51
0.52
0.48
LBS
0.51
0.48
0.50
0.50
DIAYN
0.47
0.43
0,46
0.54
SMM
0.38
0.36
0.39
0.61
APS
0.48
0.47
0.49
0.51
CIC
0.54
0.46
BeCL
0.60
0.62
0.61
0.39
PEAC (Ours)
0.67
0.69
0.67
0.33
Table Aggregate metrics [2] state-based DMC. For every algorithm, there are embodiment settings, each trained with 10 seeds and fine-tuned under 4 downstream tasks, thus each statistic for every method has 120 runs.
Demains Taols
Quadniped-mas
Quslnipot-lamping
Normilized Avenge
stand
walk
mi
Bip
stand
walk
ran
juma
stand
walk
run
jump
ICM
665.3
418.0
146.2
246.6
460
229.5
215.6
323.5
365.8
182.4
180.2
203.1
0.37
RND
588.9
386.7
176.4
253.8
820.6
563.7
409.6
589.5
325.4
166.2
156.0
235.8
0.47
549.3
250.0
372.4
274.0
199.1
0.38
731.6
325.8
498.3
LBS
618.0
370.3
1368
343.1
160.8
899.1
383.7
517.2
574.0
302.0
258.4
335.8
0.50
DIAYN
302.1
265.
105.3
212.7
682.
651.3
371.0
16%
353.1
356.7
331.8
391.8
0.16
673.5
APS
629.8
429.8
129.4
291.4
653.1
474.1
325.3
533.7
479.
254.9
302.4
403.7
0.49
CIC
824.8
536.6
220.7
327.7
762.5
610.9
442.7
617.5
335.9
194.
166 (
267.5
0.54
BeCL.
838.6
623.6
238.5
348.1
729
445.0
349.4
557.1
553.7
485.8
292.0
509.8
0.61
PEAC (Oas)
823.8
330.5
7860
Table Detailed results state-based DMC. Average cumulative reward (mean of seeds) of the best policy.
23

---

B.5 Detailed results in image-based DMC
In Table 10, we present detailed results in state-based DMC of all four statistics (medium, IQM, mean, OG) for baselines and our PEAC-LBS as well as PEAC-DIAYN, Besides these statistics, in Table 11, we further report the detailed results for the 12 downstream tasks, averaged across all embodiments and seeds. Overall, PEAC-LBS's performance is steadily on top, outperforming existing methods, especially in Walker-mass Also, compared with other pure skill discovery methods, PEAC-DIAYN performs more consistently on all tasks and achieves higher average rewards.
Metries
Median
IQM
Mean
Optimality Gap
DIAYN
0.58
0.53
0.56
0.44
APS
0.66
0.64
0.66
0.34
LSD
0.64
0.63
0.64
0.36
CIC
0.67
0.66
0.67
0.33
PEAC-DIAYN (Ours)
0.72
0.69
0.71
0.29
ICM
0.77
0.76
0.77
0.24
RND
0.74
0.75
0.75
0.26
Plan2Explore
0.78
0.80
0.78
0.22
APT
0.75
0.75
0.75
0.25
LBS
0.84
0.85
0.84
0.17
Choreographer
0.84
0.86
0.84
0.17
PEAC-LBS (Ours)
0.93
0.93
0.92
0.10
Table 10: Aggregate metries [2] in image-based DMC. For every algorithm, there are 3 embodiment settings, each trained with 3 seeds and fine-tuned under downstream tasks, thus each statistic for every method has 36 runs.
Walker mans
Quadruped
Quadruped damping
Normilized
Taskss
stand
walk
run
Gip
stand
walk
run
jump
stand
walk
rua
jump
Average
103.8
311.9
411.8
410.8
367.4
APS
906.2
554.1
228.9
473.2
814.9
414.8
413.5
677.2
830.5
417.0
379.1
560.7
0.66
LSD
912.8
664.1
227.6
401.9
769
409.2
401.3
555.5
634.9
447.9
481.4
608.5
0.64
259.8
341.8
416.5
DIAYN (Ouns)
854.5
731.8
305.9
491.1
720.5
420.1
446.6
548.8
867.3
500.6
440.8
671.6
0.71
I.M
946.5
797.0
304.6
492.8
937.4
610.4
401.0
S09.
$34.9
458.0
433.77
683.5
0.77
444.4
405.5
708.
PlanZExplore
342.2
518.8
470.9
421.2
APT
914.2
781.1
332.8
485.5
833.6
513.4
489.2
718.3
863.8
494.4
430.1
639.1
0.75
LBS
937.9
754.4
365.1
531.2
900.1
732.3
535.1
777.4
853.4
731.7
511.1
758.5
0.84
Choreogragher
819.4
PEAC-LBS (Cuun))
418.3
673.5
744.5
607.7
814.9
775.9
648.2
784.1
Table 11: Detailed results in image-based DMC. Average cumulative reward (mean of seeds) of the best policy trained by different algorithms. We bold the best performance of each task. The six baselines above are exploration-ban methods (Choreographer utilizes both exploration and skill-discovery techniques), while the following four baselines are skill-discovery methods.
B.6 Detailed results Robosuite
In Table 12, we report detailed results in Robosuite with all tasks and robotic arms. Overall, PEAC performs better in more tasks and owns better generalization ability unseen robot Jaco.
Dumains
Phnda
ITWA
Doar
Kinonn
Jazn
Panda
IIWA
Kinnvi
Ineo
Panda
IwoArmPosinHole Kinovas
IIWA
Jaon
ICM
156.2
134.4
32.2
107.7
134,1
151.6
85.9
89.5
288.4
282.8
304.1
337.8
RND
128.4
150.5
148.0
127.2
75.0
92.7
$4.4
64.2
272.7
277.6
312.8
363..5
LOS
120.4
128.7
79.6
105.0
89.7
66.7
$7.9
268.0
271.5
314.9
308.0
PEAC (Curs))
158.1
109.8
321.9
Table 12: Detailed results in Robosuite.
24

---

B.7 Ablation of timesteps in image-based DMC
In Figure 8, we show additional results about the performance three domains of image-based DMC for different algorithms and pre-training timesteps. Overall, PEAC-LBS outperforms all methods, while Choreographer and LBS are still competitive on the Quadruped-mass. Also, PEAC-DIAYN outperforms all other pure skill discovery methods.
#CCACORM 50un LCD In 24N1
DAO
1PAC-00P% LULED In D6AN
SOCAACC BRATT
Export bareetz Store (50
G000
Sopa Pa
CHea
Frenanning on 100% SONDS
Prenasiing 50%K steps
3000
BPAACORNN SOUAL
BRAA (3hm))
DEWY
IEAP
DOWN
Chind BLAAX
MOChangga MRATT
LSA
Expert Norrotoos Scord
Export Nomottos Soon
Ceea
alaps
Figure 8: Ablation study of pre-training steps in image-based DMC.
B.8 More Ablation Studies
In this part, we conclude more ablation studies to better clarify the contribution of each component in PEAC. First, we supplement ablation studies of the hyperparameter in Eq. 2 (3 is set to 1.0 in all our experiments). As discussed in the paper, /3 is a parameter that negatively related to the fine-tuning timesteps and is for balancing the policy improvement term and the policy constraint term. When the fine-tuning timestep tends to the infinity, 33 tends to O. Unfortunately, the relationship between 3 and the fine-tuning timesteps complicated. Thus we evaluate PEAC-LBS under different (3 as below
Walker mass
Quadroped-mas
Quadruped damping walk
Normilized
Tacks
stand
walk
run
CTip
stand
walk
rum
jump
stand
rum
jump
Averuee
1.0
418.3
673.5
744.5
607.7
775.0
648.2
784.1
0.1
963.8
877.6
404.8
604.4
905.3
830.
477.5
797.0
903.0
757.6
648.5
807.6
0.90
0.5
958.5
896.4
416.5
640.8
929.6
794.3
593.9
306.7
921.2
746.3
540.5
794.7
0.90
2.0
967.6
542.2
499.6
3.0
961.7
901.6
399.6
634.9
651.0
728.1
436.9
466.9
644.5
Table 13: Ablation for / of PEAC-LBS in image-based DMC. Average cumulative reward (mean of 3 seeds) of the best policy trained by different algorithms.
As shown in Table 13, when 3 is large, the performance of PEAC-LBS decreases more than B is small, but PEAC-LBS is overall stable with different B.
Moreover, to clarify the effectiveness of our embodiment discriminator, we supplement LBS-Context and DIAYNNContext, i.e., combining LBS and DIAYN with the embodiment discriminator in PEAC, which utilizes embodiment information during the pre-training stage. Our results state-based DMC and Image-based DMC are in Table 14 and Table 15, respectively.
25

---

Domsiss
Walker-mass
Qundrupedmmas
Quaduped-damping
Nonuilized
Tasks
stand
walk
flip
stand
walk
jump
stand
walk
jump
Average
LBS
618.0
570.3
136.8
343.1
740.8
49%.1
368.7
517.2
574.0
302.0
258.4
335.8
0.50
LBS-Cantext
207.6
389.0
630.1
308.1
425.1
478.3
300.4
372.2
106.8
212.7
469.1
553.4
394.8
DIAYN-Comest
657.0
541.1
153.1
301.8
735.4
495.2
415.5
581.5
688 4
525.5
290.1
477.0
0.56
PEAC (Churs)
823.8
499.9
210.6
320.5
786.
754.5
288.3
645.6
712.3
644.1
393.5
541.8
0.67
Table 14: Ablation study for baselines w/ our embodiment discriminator in state-based DMC.
Domains Tasks
Walker mass
Quadruped mass
Quadruped damping
Normilized Average
stand
walk
Biip
stand
walk
jump
stand
walk
jump
DIAYN
772.6
515.1
193.8
365.7
583.9
425.9
311.9
431.8
791.8
410.8
367.4
$36.
0.36
465.2
423.1
350.2
309.7
544.9
FRAC-DIAYN (Ouns)
954.5
731.8
305.9
191.1
720.5
420.1
446.6
548.8
867.3
500.6
440,8
671.6
0.71
LBS
937.
754.4
306.1
531.2
900.1
752.3
535.1
777.4
883.4
731.7
311.1
258.5
0.84
LBS Contest
933.3
792.4
305.4
530.7
601.9
477.7
776.7
797.7
627.3
808.8
0,84
PEAC-LBS (Churs)
418.3
673.5
744.5
607.7
814.9
775.9
648.2
784.1
Table 15: Ablation study for baselines w/ our embodiment discriminator in image-based DMC.
As shown in these two tables, LBS-Context and DIAYN-Context own comparable or superior performance compared with LBS and DIAYN respectively, and PEAC still significantly outperforms them. Consequently, this ablation study highlights that both the embodiment discriminator and cross-embodiment intrinsic rewards Ree are effective for handling CEURL.
B.9 Generalization results of pre-trained models
In Fig. 9, we evaluate the generalization ability of pre-trained models to unseen embodiments of all exploration methods in Walker mass of image-bases based DMC. After pre-training on several embodiments, we zero-shot utilize these agents to sample trajectories via two different unseen embodiments. Given trajectories, we reduce the dimension of the hidden states calculated by world model via t-SNE [57], where points with different colors represent data generated by different embodiments. As shown in Fig. 9, all the baselines can not distinguish different embodiments, while our PEAC-LBS roughly divide them into two regions, indicating the pre-trained model of PEAC-LBS own strong generalization ability to unseen embodiments.
ICM
RND
ART
LB5
Chareographer
PEAC-LBS
Figure 9: Visualization of the pre-trained model generalization to unseen embodiments.
B.10 Generalization results of fine-tuned models
In this part, we evaluate the generalization ability of the fine-tuned agents to unseen embodiments of state-based DMC and image- based DMC. In these two settings. we pre-train and fine-tune tune the agent with the sampled training embodiments (Train column in Table 3) and zero-shot evaluate performance of the fine-tuned agents in the same task but with unseen in-distribution embodiments (Generalization column Table 3). The detailed generalization results of downstream tasks in based DMC and image-based DMC are in Table 16-17, respectively. As shown in Table 16.
26

---

PEAC still significantly outperforms all baselines in normalized average return and there is even greater leading advantage than baselines, compared with the trained embodiments. This indicates that PEAC can effectively generalize to unseen embodiments and effectively handle downstream tasks.
Domains
Walker-miss walk
Quadruped-cm walk
Quadruped- dampupdd-ddmpingg
Normilized Average
Tools
stand
Bip
stand
jump
stand
walk
jump
702.0
246.6
321.3
165.7
259.8
112.3
0.32
RND
609.6
421:2
183.5
244.9
810.2
563.2
413.3
583.1
223.5
110.7
87.0
218.3
0.45
Disagreement
537.6
331.6
139,8
250.0
555.7
354.7
323.4
303.2
200.3
118.4
110.0
131.4
0.36
ProtRL
742.1
694.0
203.9
320.5
626.5
420.9
343.2
495.7
545.4
299.1 1
236.6
293.1
0.51
LBS
628.0
412.0
142.0
339,4
147.7
462.1
452.4
553.8
290.4
245.6
312,0
DIAYN
497.4
257.8
107.5
207.5
677.9
202.0
366.3
451.1
547.9
361.4 4
338.1
387.1
0.45
SMM
680.8
561.4
252.6
315.6
309
144.5
171.2
244.0
278.1
116.8
115.2
211,0
0.36
663.7
859.5
607.
2359
312.4
763.7
6
630.9
234.3
130.8
112.1
179.4
0.52
BeCl.
874.2
693.4
255.9
354.5
683.0
369.1 7
349.6
517.5
522.0
425 1
285.7
491.4
0.59
PEAC (Ours)
860,0
551.3
225
324.
776.3
741.7
734.6
641.9
385.7
537.0
0.68
Table 16: Detailed results in state-based DMC in evaluation embodiments. Average cumulative reward (mean of 10 seeds) of the best policy trained by different algorithms.
Similarly, Table 77 shows that PEAC-LBS not only outperforms baselines but also owns a greater leading advantage than baselines, compared with the trained embodiments. Moreover, PEAC-DIAYN exceeds other pure-exploration methods and demonstrates strong generalization ability.
Domains Thick
Waller--ass
Quadripal-mas
Quadnupra-laappin
Normilized Avarage
walk
Iun
flip
walk
risn
jump
walk
nun
jump
DIAYN
793.5
537.7
158.1
370.5
380.5
333.2
365.3
401.7
365.1
499.2
APS
927.8
601.8
238.6
473.2
781.6
442.2
430.2
706.3
840.9
409.8
377.1
550.4
0.67
LSD
921.4
706.9
239.4
362.4
757.2
401.8
300.2
534.6
620.6
444.2
487.3
601.6
0.63
CIC
756.1
308.9
421.4
868.1 3
355.1
502.8
453.1
403.6
562.3
PEAC-DIAYN (Chrss)
964.1
770.1
340.3
485.7
412.1
422.3
510.8
540.1
436.8
656.6
0.70
ICM
955.3
7938
335.6
487.9
907.5
397.0
4,0.22
786.2
5601
467.9
107.9
668.1
0.77
RND
963.6
825.5
360.7
206.8
843.8
433.1
429.3
743.6
449.2
407.3
714.7
Plan2Explore
$62.4
366.2
517.8
648.6
457.5
653.2
550.9
419.8
671.1
APT
938.0
811.1
357.5
467.0
820.1
485.7
484.8
689.2
777.0
526.0
431.3
645.7
0.74
LBS
942.5
789.7
387.3
529.1
898.8 8
896.4
542.6
765.8
575.7
770.4
524.1
261.5
0.85
Choreogragber
956.8
849.9
408.4
542.0
921.1
648.4
446.4
748.5
884.3
723.8
592.8
227.5
0,84
PEAC-LBS (Ouns)
750.6
596.8
799.9
798.7
Table 17: Detailed results in image-based DMC in evaluation embodiments. Average cumulative reward (mean of 3 seeds) of the best policy trained by different algorithms.
B.11 More challenging tasks and varying embodiments
In this section, we will consider CEURL in much more challenging tasks and more varying embodi- ment distributions, which are significant future directions for unsupervised cross-embodiment agents in more challenging real-world applications.
We first consider more complicated tasks including locomotion in complicated terrain. Following previous work [16], we design locomotion tasks in incline terrains and the results are below.
Domains Task
Walker- incline
siand
walk
run
Aii
LBS
189.1
156.0
248.4
493.0
PEAC-LBS (Ouns)
$57.9
245.0
748.8
681.7
Table 18: Detailed results of Walker-mas--incling in image-based DMC.
As shown in Table 18, the performance of LBS and PEAC-LBS decreases when locomoting in the incline terrain due to its complexity. PEAC-LES still significantly outperforms LBS, expressing that our method, especially the cross- cooosdembod intrinsic rewards, benefits cross-embodiment unsupervised pre-training for handling more complicated tasks.
Besides more complicated tasks, one possible future direction is to consider more different, or even exactly different embodiments. We take the first step by designing several settings with more varying and challenging embodiment distributions:
27

---

Walker-Cheetall includes two Walker robots with mass of 0.4 and 1.6 times the normal mass, as well as two Cheetah robots with mass of 0.4 and 1.6 times the normal mass.
Halaer-Humanoid: includes Walker robot and one Humanoid robot. Their robot properties, robot shapes, and action spaces are all different.
Walker-length and Cheetah-torsolength [71]: The former includes walker robots with differ- ent foot lengths while the second one includes cheetah robots with different torso lengths. Thus robots' properties and morphologies are different. The figures these embodiments are in Fig. 10.
Figure 10: Benchmark environments of Walker-length and Cheetah-torsolength. In Walker-length. the length of the left foot sole of different robots is different. In Cheetah Cheetah-torsolengthh torsolength, the length of the torso is different.
We mainly compare our PEAC-DIAYN and PEAC-LBS with DIAYN, LBS, and Choreographer in embodiment distributions: Walker-Cheetah, Palker-Humanote Walker-length, and Cheetah- torsolength. of which the results are in Table 19, Table 20, and Table 21 respectively.
Wollier-Chesah
Task
Waller-stand Chectal-nu
Walker-ma & Chectabbraa
Walker-fiip Cheetalettn
Walker-fip & Cheetah-tlip
DIAYN PEAC DLAYN (Ours)
414.3
246.2
346.6
4483
632.6
297.5
442.2
$27.5
611.S
401.0
Chonoographer
681.4
374.2
446.9
PEAC (Ours)
671.2
390.7
452.3
679.2
Table 19: Detailed results of Walker-Cheetah in image-based DMC.
Domains
Waller-FFummnoid walk-wwalk
Tisk
stand-stadd
stand-walk
stand-rtnn
walk-stand
walk--nn
run-stard
minwwsit
ran-nin
DIAYN PEAC-DIANN (Ours)
445.1
437.7
423.7
$31.5
335.9
339.3
115.3
139.3
127.0
470.4
447.2
476.3
409.6
355.6
363.1
135.6
126.9
135.9
476_1
Chuneugrahhr
471.0
479.9
483.7
4093
413.6
216.4
233.1
1603
LBS (Ours)
468.4
480.3
482.3
4608
470.5
466.1
196.8
234.4
242.8
Table 20: Detailed results of Halker-Humanoid in image-based DMC.
Domains
Walker-lenhh
Cheetah-isma length run backward
Task
sband
walk
nun
flip
run
Rlip
flip backward
DEAYN
748.5
764.0
328.7
532.3
721.1
723,6
634.2
502.4
PEAC-DIAYN (Ooss)
962.7
955.9
564.3
90066
695.4
664.5
689.2
507.8
Chorcu
961.4
958.5
356.3
918.0
708.1
7003
619.0
PEAC-LBS (Ouas)
966.1
956.6
573.4
899.3
731.8
704.4
747.5
515.4
Table 21: Detailed results of Walker-length and Cheetah-torso_ Jength in image-based DMC.
As shown in these tables, PEAC can achieve much greater performance compared to baselines. These experiments indicate that PEAC has powerful abilities to handle various kinds of embodiment differences, including different morphologies. Unfortunately, when the embodiments vary a lot (like Walker- Walker-Humanoid), the performance of PEAC is still limited, thus designing more effective methods for handling complicated embodiments like Humanoid is promising future direction for further considering cross-embodiment settings.
28

---

B.12 Real-World Applications
As supplement of Sec. 5.4, we provide more detailed images real-world robot deployments. As shown in Fig. 11, our method can fast fine-tune to different embodiments and handle different terrains, which unseen in the simulation. detailed video provided on the paper homepage.
Multiple Terrains
Crossing Weeds
Passing through Shrubs
Climbing Stope
Desceind Stairs
Robustriss
Figure 11: Real-world results for Aliengo robot with different joint failure in different terrains.
B.13 Computing Resource
In experiments, all the agents trained by GeForce RTX 2080 Ti with Intel(R) Xeon(R) Silver 4210 CPU @ 2.20GHz. In Image-based DMC state-based DMC / Robosuite / Isaacgym, pre-training each algorithm (each seed, domain) takes around / 0.5 1.5 2 days respectively.
C Pseduo-codes of Algorithms
Algorithm Pre-trained Embodinent--warre Control (PEAC)
Require: M training embodiments {em LM M replay buffers (Dm J JM=1 N testing embodiments
N (eM++) m== initialize neural network parameters of the policy
1: // Pre- vee-Trainin
2: while is unsupervised phase do
3: // Data Collection
4: for m = 1, ..., M do
5:
Sample state-action pairs {(77', aa""))) with the policy by controlling the embodiment em and store them into D.n-
6:
end for
7:
// Model Training
8:
for update step 1, 2... U do
9:
Sample state-action pairs form each replay buffer ((s af) Dí 1,2, M Update the embodiment discriminator via these data. Compute the cross-embodiment intrinsic reward ReE for each state-action pair and con- catenate them together.
10:
12:
Update the policy by RL backbones (like PPO, DDPG, Dreamer and so on) with these data and RCE.
13:
end for
14: end while
15: // Fine-Tuning
16: while is supervised phase do
17: Sample state-action-rewaard pairs with extrinsic rewards Rext via embodiment em and store them into DD..-
18:
Update the policy by jointly training data from different replay buffers via RL backbones. end while
19:
20:
// Evaluation
JM and unseen embodiments Evaluate fine-tuned policy with downstream task Roxt via {M
(eM+n) )
29

---

Algorithm 2 PEAC-LBS
Require: M training embodiments {em 1M m==1 M replay buffers (Dm J
{Dm) m== N testing embodiments
(eM++ Mast+nn=111 1==1- initialize neural network parameters of the policy
Pre-Training
2: while is unsupervised phase do
3: // Data Collection (the same PEAC)
4:
5: 7/ Model Training
6: for update step 1, 2... U do
Sample state-action pairs form each replay buffer df] Di,i = ..,...., Update the embodiment discriminator via these data.
8:
9:
Update the components of LBS, including the Latent Prior model, the Latent Posterior model, and the Reconstruction model (In DreamerV2 backbone, we can directly utilize its prior model and posterior model). Compute the intrinsic reward RoE + RLBS for each state-action pair and concatenate them together.
10:
11:
Update the policy by RL backbones (like PPO, DDPG, DreamerV2, and so on) with these data and RCE + RLBS-
12: end for
13:
end while
14: Fine-Tuniin Fine-Tuningg(th) (hhe same PFAC)
15:
16:
// Evaluation
17:
Evaluate fine-tuned policy with downstream task Rext via {Em m=1 and unseen embodiments
(EM+n)H=1. JA
Algorithm 3 PEEC-DIAYN
Require: M training embodiments {em J.M M replay buffers (Dm JM 1m=1- N testing embodiments {eas4n) initialize neural network parameters of the behavior policy conditioned on skill and embodiment context क (-118,,, e),, initialize neural network parameters of the embodiment-awwaee skill policy r)
1: // /Pre-Training
2: while is unsupervised phase do
3: // Data Collection (the same PEAC)
4:
5: 7/ Model Training
6:
for update step 1, 2,.,. do
7:
Sample state-action pairs form each replay buffer of). D,, 1,2, Update the embodiment discriminator via these data.
8:
9:
Update the skill discriminator of DIAYN via these data.
10:
Compute the intrinsic reward RRE + RDIAYN for each state-action pair and concatenate them together.
11:
Update the behavior policy conditioned on skill and embodiment context by RL backbones (like PPO, DDPG, DreamerV2, and so on) with these data and ReE + RDIAYN.
12: end for
13: end while
14: // Fine-Tuning
15: while is supervised phase do
16: Sample coate-action-reweee pairs with extrinsic rewards Rext via embodiment em and store
them into D,n-
17:
Update the embodiment-awan skill policy by jointly training data from different replay buffers via RL backbones.
18:
end while
19:
// Evaluation
20:
Evaluate fine-tuned agents with downstream task Rext via MM= and unseen embodiments (eM+n) J
30

---

D
Broader Impact
Designing generalizable agents for varying tasks and embodiments major concern in reinforcement learning. This work focuses on cross-embodiment unsupervised reinforcement learning and proposes a novel algorithm PEAC, which leverages trajectories from different embodiments for pre-training. subsequently broadly enhancing performance on downstream tasks. Such advancements provide the potential for future real-world cross-embodiment control. One of the potential negative impacts is that algorithms using deep neural networks, which lack interoperability and face security and robustness issues. There are no serious ethical issues as this basic research.
31

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: As mentioned in the abstract and introduction, this work mainly integrates the unsupervised RL paradigm into cross- eoosdiembodmeent RL, novel concept CEURL (Sec. 3). Then we theoretically derive a novel algorithm PEAC (Sec. 3-4) and widely evaluate PEAC in a large number of environments (Sec. 5). All these contributions are described in detail in the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Yes, we have discussed the limitations of this work in Sec. 5.5
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
32

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: Yes, we have included the detailed proofs of all results in the Appendix A, including properties of DEE (Appendix A.I), proof of Theorem 3.2 (Appendix A.2), and proof about Embodiment-Aware Skill Discovery (Appendix A.3).
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We have provided the code in the paper homepage. Also, we have provided hyper-parameters, parameters, codebase, and pseudo- code of our algorithms in Appendix B and Appendix C.
detailed information on our experiments, including hyper-
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
33

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification:
We have provided the source code in our paper homepage.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nipp cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ /nips.
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have provided details for our experiments in Appendix B, including environments, hyper-parametee optimizers, and so on, as well as our source code in the paper homepage.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes].
Justification: To mitigate the effectiveness of the randomness, repeat several random seeds for all experiments (10 for state-based DMC and for the others, following previous works' settings). When reporting the results, besides reporting mean performance,
34

---

also report IQM and OG in based DMC and image-based DMC for better evaluating different algorithms.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi-
dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Yes, we have reported the computing source in Appendix B.13, including the type of computing CPU/GPU, computing time, and so on.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https /tt///nerrip../// wwwturpp.ccpuuuuue wwtppblic//a/eecccccccccccccctct wtcppublic/ttiisttieeeeeieee www.hhhiiiiieeeeeeeeeee
Answer: [Yes]
Justification: Our research does not involve human subjects or participants. And we have discussed the potential societal impact in Appendix, D. As this basic research, there are no serious social issues.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
35

---

10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Yes, have discussed the Border Impact of this work in Appendix. D.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymeent of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out, For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: There is such risk in this paper.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have cited the original paper that produced the code/environment package and included the URL of these codebases in Appendix B.2.
36

---

Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Yes, we have provided documentation of our source code in the supplementary materials.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
37

---

Answer:
[NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
38


Reviews:
[{"review_id": "LSY0DGP172", "reviewer": "Reviewer_3BHi", "summary": "This paper addresses the challenge of designing generalizable agents capable of adapting to diverse embodiments. The authors propose the CEURL setting as a novel framework for this problem and introduce the PEAC algorithm to address it. Recognizing that CEURL requires minimizing across different downstream tasks while maximizing the fine-tuned policy, PEAC tackles the issue through policy improvement and policy constraint mechanisms. PEAC can integrate with existing unsupervised reinforcement learning methods designed for single-embodiment settings, such as LBS and DIAYN. Through experiments conducted in both simulation and real-world settings, the effectiveness of PEAC is demonstrated.", "questions": "1. For objective function (2), can you provide an ablation study using only the Policy Improvement part and only the Policy Constraint part?\n2. In Figure 6 (right), PEAC-DIAYN shows the worst performance at 100k training steps. What is the cause?\n3. In Appendix B.6, the performance of PEAC in Robosuite is fair comparing to that in DMC or Issacgym. Does PEAC work better for simple tasks (like walk, stand) than the tasks in Robosuite?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality: This paper introduces a novel problem formulation, CEURL, and presents a new algorithm, PEAC.\n\nQuality:\n1. The authors thoroughly discuss the challenges inherent in the CEURL problem definition, supported by mathematical proofs. This elevates CEURL from a simple definition to a high-quality problem formulation.\n2.Convincing experiments involving real robots are presented.\n\nClarity: The paper is well-written and easy to follow.\n\nSignificance: This paper has the potential to significantly impact cross-embodiment control through its introduction of a new problem formulation and an innovative unsupervised pretraining framework.", "weaknesses": "1. Among all the experiments presented, tasks for DMC and Issacgym are relatively limited, with only simple movements. I recommend adding more complex tasks like [1].\n2. In ablation study, the authors only show the results of different training timesteps. More ablation studies can help demonstrate how the algorithm works.\n3. From my perspective, this paper is closely related to [2], which proposes an unsupervised skill discovery method through contrastive learning. While [2] focuses on single-embodiment scenarios, PEAC extends this approach to multiple embodiments, addressing the additional challenges posed by varying state and action spaces via introducing additional embodiment prior and posterior probabilities. \n\n[1] Gupta, Agrim, et al. \"Embodied intelligence via learning and evolution.\" Nature communications 12.1 (2021): 5721.\n[2] Yang, Rushuai, et al. \"Behavior contrastive learning for unsupervised skill discovery.\" International Conference on Machine Learning. PMLR, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your supportive review and valuable suggestions. Below we address the detailed comments for all your questions.\n\n**W1:** About more complex tasks.\n\n**A:** In our experiments, tasks chosen in DMC and Isaacgym are **standard** and **widely used in unsupervised RL evaluation** [1, 2, 3]. These tasks are basic and important to evaluate agents' locomotion ability, and they can also combine to handle much more complex tasks like parkour [2]. Thanks for your suggestion and reference, referring to [4] including tasks of locomotion in complicated terrain like inclines, we have designed more complicated tasks for Walker-mass in DMC to evaluate their locomotion ability in complicated terrain like inclines: **Walker-mass-incline**. The results are shown below:\n\n| Walker-mass-incline | stand |walk|run| flip| average|\n|-|-|-|-|-|-|\n| LBS | 725.2 | 597.7 | 109.3 | 430.6 | 465.7|\n| PEAC-LBS | **793.8** | **670.3** | **268.9** | **509.1** | **560.5** |\n\nAs shown in this table, the performance of LBS and PEAC-LBS decreases when locomoting in the incline terrain due to its complexity. **PEAC-LBS still significantly outperforms LBS**, expressing that our method, especially the cross-embodiment intrinsic rewards, benefits cross-embodiment unsupervised pre-training for handling more complicated tasks.\n\n**W2 \\& Q1:** More ablation studies, like only using one of the two parts in Eq. 2.\n\n**A:** Thanks for your suggestion. There are two terms in Eq.2: the policy improvement part and the policy constraint part, representing that in the fine-tuning stage, we need to **maximize the policy performance** within **limited training steps**, respectively (Lines 182-188 in the paper). Thus they need to be considered together and it is unreasonable to use only one of these two parts for this setting. Moreover, we introduce a trade-off parameter $\\beta$ in Eq.2 to **balance these two terms**, **which is set as 1.0 in all our experiments**. Consequently, we have supplemented ablation of $\\beta$, of which the results are: \n\n|Ablation of $\\beta$|Walker-mass|Quadruped-mass| \n|-|-|-|\n|PEAC-LBS($\\beta$=1.0, default)| **727.6** | **755.2** |\n|PEAC-LBS($\\beta$=0.5)|**726.9** |728.8| \n|PEAC-LBS($\\beta$=2.0)|713.2|**750.1**| \n\nAs shown in this table, when $\\beta$ is smaller, the performance of PEAC-LBS decreases a little, and overall PEAC-LBS is stable with different $\\beta$.\n\nBesides $\\beta$, we have also supplemented more ablation studies, especially about our cross-embodiment intrinsic rewards $R_{\\text{CE}}$, we consider **LBS-Context** and **DIAYN-Context** i.e., combining LBS and DIAYN with our embodiment discriminator in PEAC but without $R_{\\text{CE}}$. Our results are as below:\n\n| State-based DMC | Walker-mass | Quadruped-mass | Quadruped-damping |\n|-|-|-|-|\n| DIAYN  | 266.7  | 501.8| 416.7|\n| DIAYN-context  | 364.4  | 564.2 | 499.5|\n| LBS | 367.1 | 536.5 | 367.6 |\n| LBS-context | 365.9 | 514.5  | 417.4|\n| PEAC  | **463.7** | **643.6** | **572.9**|\n\n| Image-based DMC | Walker-mass | Quadruped-mass | Quadruped-damping |\n|-|-|-|-|\n| DIAYN | 461.8 | 438.4  | 526.5  |\n| DIAYN-context  | 572.0 | 457.6 | 546.6 |\n| PEAC-DIAYN | 587.3 | 578.4 | 572.8 |\n| LBS | 647.2 | 736.2 | 721.2 |\n| LBS-context | 655.1  | 739.1 | 722.5  |\n| PEAC-LBS | **727.6** | **755.2** | **740.2** |\n\nAs shown in these two tables, LBS-Context and DIAYN-Context own superior performance compared with LBS and DIAYN respectively, and PEAC still significantly outperforms these baselines. Consequently, this ablation study **highlights that both the embodiment discriminator and cross-embodiment intrinsic rewards $R_{\\text{CE}}$ are necessary for handling CEURL** (more details of results are in the global response).\n\n**W3:** About the relation with the BeCL [2].\n\n**A:** BeCL [2] mainly focuses on encouraging the agent to learn diverse skills related to corresponding behaviors via applying contrastive learning for single-embodiment skill discovery. As you have mentioned, our PEAC considers the cross-embodiment setting and thus is **orthogonal to these skill-discovery methods**. \n\nWe have discussed and compared BeCL as well as other skill discovery methods in the paper. Especially, as shown in Sec.4, we have discussed the relationship between the objective of skill discovery (including BeCL) and our cross-embodiment objective, with a result of **a unified cross-embodiment skill-based adaptation objective** (Eq. 6-7). Thus our PEAC can naturally combine with these skill-discovery methods, including BeCL, and we propose PEAC-DIAYN as an example. Thanks again for your question, applying contrastive learning methods like BeCL to our PEAC for handling CEURL is indeed an interesting future direction.\n\n**Q2:** About the performance of PEAC-DIAYN in 100k pre-training steps.\n\n**A:** Thanks for your question. When the pre-training steps are small (like 100k), it is difficult to learn useful knowledge during the pre-training stage (like world models or our embodiment discriminator), and the uncertainty of results is relatively high, thus current research mainly considers the fine-tuning results after pre-training 2M in DMC [1-2].\n\n**Q3:** About the performance of PEAC in Robosuite.\n\n**A:** In Robosuite, the overall performance of PEAC is still 10\\% higher than all baselines, indicating the effectiveness of PEAC. Compared with DMC, the morphologies in Robosuite vary a lot more, thus baselines may also distinguish different embodiments directly via their morphology observation. Consequently, PEAC's leading advantage may slightly decrease, but still significantly outperforms baselines (We have also supplemented experiments in DMC with varying morphologies in the global rebuttal). Thanks for your question, we will discuss it more in the revised versions.\n\nReference:\n\n[1] URLB: Unsupervised Reinforcement Learning Benchmark\n\n[2] Behavior contrastive learning for unsupervised skill discovery\n\n[3] Robot Parkour Learning\n\n[4] Embodied intelligence via learning and evolution", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "viGvgTq0jQ", "reviewer": "Reviewer_xr64", "summary": "This papaer introduces a novel setting, cross-emodiment unsupervised RL, which deals with pre-training good policies in reward free environments across different embodiments in order to perform well on downstream tasks on unseen embodiments. The authors propose the algorithm PEAC for unsupervised learning in this setting. Their method proposes a novel intrinsic reward which is derived from the minmax objective of maximizing the improvement in the downstream fine-tuned policy over the pre-trained policy minus a regularizing KL term, which ensures that the fine-tuned policy stays close to the pre-trained policy, under the worst-case extrinsic reward. From this objective, they derive an intrinsic reward for the unsupervised cross-embodiment setting with maximizes the (fixed) log prior over embodiments minus the (learned) log posterior over embodiments, which encourages the agent to seek areas of the state space that are common accross embodiments, or underexplored. They conduct extensive experiments in both simulated and real environments, including both state-based and image-based DMC tasks, and compare against a large number of pre-existing skill-based and exploratoin based unsupervised RL methods. Their results are impressive, showing improvement in almost all cases over the baselines. Moreover, they conduct real-world experiments in which their algorithm similarly outperforms.", "questions": "1. It is not clear to me how the cross-embodiment policy handles different state spaces. For example in the cheetah-walker experiments in the appendix, how is the policy architecture designed such that it can take in different size state vectors?\n2. What does super script c in the MDP M represent?\n3. Can you please explain the equality in the last line of Eq 29?\n4. It is not clear what is unique about the cross-embodiment setting for unsupervised RL versus the more generic contextual MDP setting. Why do embodiments needed to be treated differently?\n5. If log(p(e)) is fixed, does it need to be in the reward?", "limitations": "- As the authors note, their experiments are limited to settings where the embodiments are very similar. It is not clear how their method extends to dissimilar embodiments, particularly with different sized state vectors.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Introduces a novel setting: cross-embodiment unsupervised RL\n- Convincing motivation - the idea that most cross-embodiment training which focuses on a single task may not learn generalizable embodiment-specific skills rather than task-specific skills\n- Extensive experiments in both simulation and real-world settings with very strong results compared to baselines", "weaknesses": "- The paper is hard to follow and key details on the implementation are not fully explained. For example in line 68 a \"embodiment context encoder\" is mentioned but I do not see this explained any where else in the paper. How is the discriminator trained?\n- The extension of the information geometry analyses in Sec 3 do not seem particularly related to the main claims of the paper. This section serves primarly to highlight that policies may be stochastic, however it does not seem that the this information is utilized in the algorithm presented\n- The stated goal of the paper is to better learn embodiement-aware skills, but the policy is optimized to maximize the expectation over all embodiments. Would this encourage embodiment-aware skills or rather embodiment-agnostic skills?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks a lot for the supportive review and constructive suggestions. Below we address the detailed comments.\n\n**W1:** About the embodiment context encoder in line 68.\n\n**A:** The embodiment context encoder is the embodiment discriminator in the main text of the paper, and we will polish our paper to make it consistent in context. As for the training of the discriminator, we utilize cross-entropy loss of the one-hot embodiment vector during the pre-training stage, following previous works [1]. We will clarify it in the revised versions.\n\nTo better clarify the impact of the embodiment discriminator and the cross-embodiment intrinsic reward $R_{\\text{CE}}$, we have supplemented ablation studies of **LBS-Context** and **DIAYN-Context**, i.e., combining LBS and DIAYN with our embodiment discriminator in PEAC but without $R_{\\text{CE}}$. Our results are as below:\n\n|State-based DMC| Walker-mass|Quadruped-mass | Quadruped-damping |\n|-|-|-|-|\n|DIAYN|266.7| 501.8| 416.7|\n|DIAYN-context| 364.4|564.2| 499.5|\n|LBS |367.1|536.5| 367.6|\n|LBS-context | 365.9 | 514.5| 417.4|\n|PEAC| **463.7** | **643.6** | **572.9** |\n\n|Image-based DMC|Walker-mass | Quadruped-mass | Quadruped-damping |\n|-|-|-|-|\n|DIAYN|461.8| 438.4| 526.5  |\n|DIAYN-context| 572.0| 457.6 | 546.6 |\n|PEAC-DIAYN | 587.3| 578.4 | 572.8 |\n|LBS| 647.2|736.2|721.2 |\n|LBS-context|655.1|739.1 | 722.5|\n|PEAC-LBS|**727.6**|**755.2**|**740.2**|\n\nAs shown in these two tables, LBS-Context and DIAYN-Context own superior performance compared with LBS and DIAYN respectively, and PEAC still significantly outperforms these baselines. Consequently, this ablation study **highlights that both the embodiment discriminator and cross-embodiment intrinsic rewards $R_{\\text{CE}}$ are effective for handling CEURL** (more details and results are in the global response and the attached PDF).\n\n**W2:** About the information geometry analyses in Sec. 3.\n\n**A:** Yes, our extension of the information geometry analyses does not directly guide the design of PEAC. As the major contributions of this paper include the novel setting **CEURL** and our algorithm **PEAC**, **our information geometry analyses are mainly related to CEURL** by revealing its difficulty. Moreover, as you have mentioned, our information geometry analyses highlight that policies may be stochastic or history-related, thus our PEAC chooses to encode historical state-action pairs to the embodiment context in practice (Lines 213-216). We will emphasize it in the revised version.\n\n**W3:** About embodiment-aware or embodiment-agnostic skills?\n\n**A:** The goal of **embodiment-aware skill discovery** (Lines 232-252) is to learn similar skills for all these embodiments during the pre-training stage. However, as different embodiments own different properties/morphologies, their actions for achieving similar skills may differ a lot. Thus we name our method embodiment-aware skill discovery. Thanks again for your question, we will discuss the concept more clearly in the revised version.\n\n**Q1:** How to handle different action/state spaces?\n\n**A:** When handling state/action spaces with different dimensions, we will take the maximal dimension of these spaces $d$ and pad the states/actions into $d$ dimension by filling in zeros, following previous works [2].\n\n**Q2:** What does $c$ in the MDP $\\mathcal{M}^c$ represent?\n\n**A:** We use $\\mathcal{M}$ to represent original MDP, $\\mathcal{M}_e$ represent MDP with embodiment $e$, **$\\mathcal{M}^c$ represent controlled MDP**, i.e., an MDP without rewards in the pre-training stage, and $\\mathcal{M}_e^c$ represent controlled MDP with embodiment $e$.\n\n**Q3:** About the equality in the last line of Eq. 29.\n\n**A:** The last equality of Eq. 29 can be derived by the property of conditional probability and we abbreviate $\\mathcal{M}_e^c$ as $e$:\n\n$$\\log \\frac{p_{\\pi}(\\tau)}{p_{\\pi}(\\tau|e)} = \\log \\frac{p_{\\pi}(\\tau)}{p_{\\pi}(\\tau, e) / p_{\\pi}(e)} = \\log \\frac{p_{\\pi}(e) }{p_{\\pi}(\\tau, e) / p_{\\pi}(\\tau)} = \\log \\frac{p(e)}{p_{\\pi}(e|\\tau)}.$$\n\nThanks for your question, we will provide the detailed derivation in the revised versions.\n\n**Q4:** About the connection between CEURL and contextual MDP.\n\n**A:** As we have mentioned in Lines 125-126, cross-embodiment RL can be formulated by contextual MDP. Moreover, as the cross-embodiment setting is practical for real-world applications and owns several unique properties, it introduces various new problems like cross-embodiment transfer [3] and has received widespread attention in embodied intelligence [3-5] (as mentioned by Reviewer RG3H). In this work, considering that different embodiments may own similar structures and learn similar skills, we propose unsupervised pre-train across embodiments to learn knowledge only specialized in these embodiments themselves, i.e., CEURL, which is a novel setting that existing contextual MDP can not cover and thus we combine contextual MDP with controlled MDP as our CEMDP to formulate our problem.\n\n**Q5:** Does $\\log p(e)$ used in the reward?\n\n**A:** No, as we have mentioned in Lines 205-206, our cross-embodiment intrinsic reward eliminates $\\log p(e)$ as it is fixed.\n\n**Limitation:** About experimental settings that can better highlight cross-embodiment\n\n**A:**  We have supplemented more cross-embodiments with different morphologies in DMC: Walker-Humanoid, Walker-length, and Cheetah-torsolength. **More details and results are in the global response, where PEAC shows greater performance compared to baselines**. The results indicate that PEAC has powerful abilities to handle various kinds of embodiment differences, including different morphologies.\n\nReference:\n\n[1] Multi-task reinforcement learning with soft modularization\n\n[2] Learning to Modulate pre-trained Models in RL\n\n[3] Cross-Embodiment Robot Manipulation Skill Transfer using Latent Space Alignment\n\n[4] Xirl: Cross-embodiment inverse reinforcement learning\n\n[5] Pushing the limits of cross-embodiment learning for manipulation and navigation", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gqAZZueTYN", "reviewer": "Reviewer_RG3H", "summary": "This paper presents a new unsupervised reinforcement learning (URL) algorithm called Pre-trained Embodiment-Aware Control (PEAC). PEAC is designed to tackle cross-embodiment tasks, explicitly considering the influence of different embodiments to facilitate exploration and skill discovery across embodiments. Experimental results demonstrate that PEAC notably enhances adaptation performance.", "questions": "Please see Weaknesses.", "limitations": "My primary concern is that the baselines in the experiments do not utilize embodiment information $e$. Please refer to the first point in Weaknesses.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and easy to follow. The intuitive figures and logically structured text make PEAC clear and understandable.\n2. The author(s) conduct comprehensive experiments and present a solid work.\n3. As embodied intelligence gains increasing attention, cross-embodiment research is indeed an important direction. PEAC undoubtedly contributes to the community in this regard.", "weaknesses": "1. Although the author(s) compare various standard and SOTA baselines in the experiments, they lack strong persuasiveness. Since the author(s) focus on cross-embodiment tasks, it is apparent that the baselines, which do not utilize embodiment information $e$, exhibit poorer performance. Therefore, it is unclear whether PEAC's high performance is due to the use of embodiment information $e$ or the utilization of cross-embodiment intrinsic rewards $\\mathcal R_{\\text{CE}}$. To highlight the paper's contribution, I suggest that the author(s) modify some baselines to naively incorporate embodiment information and then compare with them. For example, conditioning the policy and reward of DIAYN on embodiment $e$, etc.\n2. Experiments conducted on Robosuite effectively showcase \"cross-embodiment,\" while other environments may not do so as prominently. By modifying parameters like \"mass\" and \"damping\" in DMC tasks or introducing joint torque failures in Unitree A1, the setting becomes more like a Meta RL, where the robot's morphology remains relatively unchanged. In locomotion tasks, \"cross-embodiment\" could be exemplified by robots like Gym-MuJoCo's Walker2d, DMC's Walker, and Humanoid, where they all use two legs for alternating walking and aim to enhance mobility. However, they differ in morphology. Demonstrating whether PEAC can learn the same shared knowledge of bipedal walking from these diverse morphologies may better illustrate the topic of \"cross-embodiment.\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks a lot for your review, especially the praise for PEAC's contributions to embodied intelligence. Below we will address all your concerns.\n\n**W1:** About baselines utilizing embodiment information $e$.\n\n**A:** Thank you for recognizing that we have included various standard and SOTA baselines. Following your valuable suggestions of adding baselines incorporating embodiment information, we have supplemented experiments of **LBS-Context** and **DIAYN-Context**, which inherit LBS and DIAYN by incorporating embodiment information $e$, to better highlight the paper's contribution. Our results are below:\n\n| State-based DMC | Walker-mass | Quadruped-mass | Quadruped-damping |\n|-|-|-|-|\n| DIAYN  | 266.7  | 501.8| 416.7 |\n| DIAYN-context  | 364.4  | 564.2 | 499.5  |\n| LBS | 367.1 | 536.5 | 367.6 |\n| LBS-context | 365.9 | 514.5  | 417.4|\n| PEAC  | **463.7** | **643.6** | **572.9** |\n\n| Image-based DMC | Walker-mass | Quadruped-mass | Quadruped-damping |\n|-|-|-|-|\n| DIAYN | 461.8 | 438.4  | 526.5  |\n| DIAYN-context  | 572.0 | 457.6 | 546.6 |\n| PEAC-DIAYN | 587.3 | 578.4 | 572.8 |\n| LBS | 647.2 | 736.2 | 721.2 |\n| LBS-context | 655.1  | 739.1 | 722.5  |\n| PEAC-LBS | **727.6** | **755.2** | **740.2** |\n\nAs shown in these two tables, LBS-Context and DIAYN-Context own superior performance compared with LBS and DIAYN respectively, and PEAC still significantly outperforms these baselines. Especially, in state-based DMC, the improvement in the performance of LBS-Context and DIAYN-Context is more remarkable than in image-based DMC. It may be because, in image-based DMC, we take DreamerV2 as the RL backbone, which has encoded historical information for making the decision and is possible to distinguish different embodiments through historical information to some degree. But in state-based DMC, we take DDPG as the RL backbones following previous works [1], which are Markovian policies and thus utilizing embodiment information during the pre-training is helpful (Lines 210-216). Consequently, our supplemented results **highlight that both the embodiment discriminator and cross-embodiment intrinsic rewards $\\mathcal{R}_{\\text{CE}}$ are effective for handling CEURL** (more details, including figures with aggregate metrics of these methods, are in the global response and the corresponding PDF).\n\n**W2:** About experimental settings that can better highlight cross-embodiment.\n\n**A:** Thank you for your recognition of our experiments on Robosuite. In DMC and Isaacgym, we mainly modify the property parameters or introduce joint torque failures because embodiments with different properties are **one of the most basic cross-embodiment settings**, which is worth paying attention to especially when we first research CEURL. Moreover, although these embodiments only modify property parameters, **their ability to handle tasks may differ a lot**, e.g. in our real-world experiments, when the front foot joint of the A1 robot fails, it can only drag its legs forward, while when the rear foot joint fails, it will lift its rear legs to move forward (Fig. 7 and video in the supplementary material).\n\nIn experiments of DMC, besides settings with different properties like Walker-mass, we have also included settings with different morphology in **Appendix B.10**: **Walker-Cheetah**, where Walker and Cheetah both own two legs but their morphology differs a lot. Experiments in Appendix B.10 demonstrate that PEAC can also significantly outperform baselines and reveal PEAC's potential in handling exactly different embodiments.\n\nMoreover, following your constructive suggestions, we have also supplemented extensive experiments in DMC considering settings with greater differences across embodiments, **especially differing in morphology**:\n\n- **Walker-Humanoid**: Following your constructive suggestions, we consider the cross-embodiment setting that includes Walker robots and Humanoid robots.  Their robot properties, robot shapes, and action spaces are all different.\n- **Walker-length** and **Cheetah-torsolength** [2]: The former includes walker robots with different foot lengths while the second one includes cheetah robots with different torso lengths. Thus robots' properties and morphologies are different  (Figs of these embodiments are in the attached PDF of the global response).\n\nThe results in these settings are below:\n\n| Walker-Humanoid | stand-stand | stand-walk | stand-run | walk-stand | walk-walk | walk-run | run-stand | run-walk | run-run | average      |\n|-|-|-|-|-|-|-|-|-|-|-|\n| DIAYN  | 286.8 | 320.2  | 307.2 | 185.4 | 175.4 | 209.7 | 75.4  | 61.1 | 55.4 | 186.3  |\n| PEAC-DIAYN | 231.6 | 430.3  | 371.7  | 247.4 | 210.7 | 245.2 | 88.3 | 85.1 | 69.9  | 220.0  |\n| LBS | 477.8 | 463.2   | 475.7 | **407.1** | 343.7 | 401.1 | 150.8  | 112.3 | 134.6 | 329.6  |\n| PEAC-LBS | **486.4** | **484.9**  | **483.6**  | **406.9**   | **373.6**  | **432.5** | **180.4** | **145.7** | **188.2** | **353.6**  |\n\n| Walker-length | stand | walk | run| flip | average  |\n|-|-|-|-|-|-|\n| LBS | **977.5** | 909.1 | **557.5** | 630.4 | 768.6 |\n| PEAC-LBS | **970.0**  | **976.4** | 544.2 | **764.9** | **813.9**  |\n\n| Cheetah-torsolength | run | run_backward | flip  | flip_backward | average  |\n|-|-|-|-|-|-|\n| LBS | 625.9 | **512.9** | 611.9 | 523.0 | 568.4  |\n| PEAC-LBS| **745.3** | 499.7 | **646.4** | **649.2** | **635.2** |\n\nAs shown in these tables, PEAC can achieve much greater performance compared to baselines. These experiments indicate that PEAC has powerful abilities to handle various kinds of embodiment differences, including different morphologies. Also, designing more effective methods for handling complicated embodiments like Humanoid is a promising future direction.\n\nReference:\n\n[1] URLB: Unsupervised Reinforcement Learning Benchmark\n\n[2] Learning Robust State Abstractions for Hidden-Parameter Block MDPs", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for their valuable comments which help to further improve our paper. Here we first address the common concerns on **baselines/ablation studies** and **more complicated cross-embodiment settings**. Then, we provide a detailed response to the comments of each reviewer respectively.

**Q1:** More baselines and ablation studies.

**A:** We appreciate the reviewer's recognition of the comprehensiveness of experiments (Reviewer RG3H) and the adequacy of baselines (Reviewer xr64). **To better highlight the contribution of this paper**, we have supplemented various ablation studies following the valuable suggestions of Reviewer 3BHi and RG3H. 

First, we supplement **LBS-Context** and **DIAYN-Context**, i.e., combining LBS and DIAYN with the embodiment discriminator in PEAC, which utilizes embodiment information during the pre-training stage. Our results are as below:

| State-based DMC | Walker-mass | Quadruped-mass | Quadruped-damping |
|-|-|-|-|
| DIAYN  | 266.7  | 501.8| 416.7 |
| DIAYN-context  | 364.4  | 564.2 | 499.5  |
| LBS | 367.1 | 536.5 | 367.6 |
| LBS-context | 365.9 | 514.5  | 417.4|
| PEAC  | **463.7** | **643.6** | **572.9** |

| Image-based DMC | Walker-mass | Quadruped-mass | Quadruped-damping |
|-|-|-|-|
| DIAYN | 461.8 | 438.4  | 526.5  |
| DIAYN-context  | 572.0 | 457.6 | 546.6 |
| PEAC-DIAYN | 587.3 | 578.4 | 572.8 |
| LBS | 647.2 | 736.2 | 721.2 |
| LBS-context | 655.1  | 739.1 | 722.5  |
| PEAC-LBS | **727.6** | **755.2** | **740.2** |

**The figures with aggregate metrics of these methods are in the attached PDF**. As shown in these two tables and figures, LBS-Context and DIAYN-Context own superior performance compared with LBS and DIAYN respectively, and PEAC still significantly outperforms them. Especially, in state-based DMC, the improvement in the performance of LBS-Context and DIAYN-Context is more remarkable than in image-based DMC. It may be because, in image-based DMC, we take DreamerV2 as the RL backbone, which has encoded historical information for making the decision and is possible to distinguish different embodiments through historical information to some degree (Lines 210-216). Consequently, **this ablation study highlights that both the embodiment discriminator and cross-embodiment intrinsic rewards $\mathcal{R}_{\text{CE}}$ are effective for handling CEURL**.

Moreover, we supplement ablation studies of the hyperparameter $\beta$ in Eq.2 ($\beta$ is set to 1.0 in all our experiments), which is for balancing the policy improvement term and the policy constraint term.

| Ablation of $\beta$ | Walker-mass | Quadruped-mass | 
|-|-|-|
| PEAC-LBS  | **727.6** | **755.2**  | 
| PEAC-LBS($\beta$=0.5) | **726.9** | 728.8  | 
| PEAC-LBS($\beta$=2.0) | 713.2 | **750.1**  | 

As shown in this table, when $\beta$ is small, the performance of PEAC-LBS decreases a little, but PEAC-LBS is still stable with different $\beta$.

**Q2:** About experiments with more complicated cross-embodiment settings.

**A:** To the best of our knowledge, this is **the first work to consider unsupervised pre-training across different embodiments**. Consequently, when designing experimental settings, we hope to cover diverse kinds of cross-embodiment settings to fully validate algorithms in this setting, including different embodiment properties, different morphologies, different actions, and so on. In DMC, we mainly consider embodiments with different embodiment properties like mass and damping as this is **one of the most basic cross-embodiment settings** and is worth paying attention to especially when we first research CEURL. Besides these, there are also settings with **different morphologies** in **Appendix B.10**: **Walker-Cheetah**, of which the results show that PEAC can also handle embodiments with different morphologies. 

Following reviewers' valuable suggestions, we have also supplemented more cross-embodiments with **different morphologies** in DMC: 

- **Walker-Humanoid**: As suggested by Reviewer RG3H, we consider the cross-embodiment setting that includes Walker robots and Humanoid robots. Their robot properties, robot shapes, and action spaces are all different.
- **Walker-length** and **Cheetah-torsolength** [1]: The former includes walker robots with different foot lengths while the second one includes cheetah robots with different torso lengths. Thus robots' properties and morphologies are different. The figures of these embodiments are in the attached PDF.

The results in these settings are below:

| Walker-Humanoid | stand-stand | stand-walk | stand-run | walk-stand | walk-walk | walk-run | run-stand | run-walk | run-run | average      |
|-|-|-|-|-|-|-|-|-|-|-|
| DIAYN  | 286.8 | 320.2  | 307.2 | 185.4 | 175.4 | 209.7 | 75.4  | 61.1 | 55.4 | 186.3  |
| PEAC-DIAYN | 231.6 | 430.3  | 371.7  | 247.4 | 210.7 | 245.2 | 88.3 | 85.1 | 69.9  | 220.0  |
| LBS | 477.8 | 463.2   | 475.7 | **407.1** | 343.7 | 401.1 | 150.8  | 112.3 | 134.6 | 329.6  |
| PEAC-LBS | **486.4** | **484.9**  | **483.6**  | **406.9** | **373.6**  | **432.5** | **180.4** | **145.7** | **188.2** | **353.6**  |

| Walker-length | stand | walk | run| flip | average  |
|-|-|-|-|-|-|
| LBS | **977.5** | 909.1 | **557.5** | 630.4 | 768.6  |
| PEAC-LBS | **970.0**  | **976.4** | 544.2 | **764.9** | **813.9**  |

| Cheetah-torsolength | run | run_backward | flip  | flip_backward | average  |
|-|-|-|-|-|-|
| LBS | 625.9 | **512.9** | 611.9 | 523.0 | 568.4  |
| PEAC-LBS| **745.3** | 499.7 | **646.4** | **649.2** | **635.2** |

As shown in these tables, PEAC can achieve much greater performance compared to baselines. These experiments indicate that PEAC has powerful abilities to handle various kinds of embodiment differences, including different morphologies. Also, designing more effective methods for handling complicated embodiments like Humanoid is a promising future direction.

Reference:

[1] Learning Robust State Abstractions for Hidden-Parameter Block MDPs

Meta-review: This work studies an area of cross-embodiment unsupervised RL, which is of growing interest in the community. The pros outweigh the cons for this paper and the analysis and discussion of the problem space is a fair contribution to the community. We encourage the authors to continue to consider the comments from the reviewers to improve the final version of the work.
Decision: Accept (poster)