Cross-Embodiment Offline Reinforcement Learning for Heterogeneous Robot Datasets
Authors: Haruki Abe
Takayuki Osa
Yusuke Mukuta
Tatsuya Harada
Venue: ICLR 2026 Poster (2026)
Keywords: Offline Reinforcement Learning
Cross-Embodiment Learning
Multi-Task Learning
Primary area: reinforcement learning

Abstract: Scalable robot policy pre-training has been hindered by the high cost of collecting high-quality demonstrations for each platform. In this study, we address this issue by uniting offline reinforcement learning (offline RL) with cross-embodiment learning. Offline RL leverages both expert and abundant suboptimal data, and cross-embodiment learning aggregates heterogeneous robot trajectories across diverse morphologies to acquire universal control priors. We perform a systematic analysis of this offline RL and cross-embodiment paradigm, providing a principled understanding of its strengths and limitations. To evaluate this offline RL and cross-embodiment paradigm, we construct a suite of locomotion datasets spanning 16 distinct robot platforms. Our experiments confirm that this combined approach excels at pre-training with datasets rich in suboptimal trajectories, outperforming pure behavior cloning. However, as the proportion of suboptimal data and the number of robot types increase, we observe that conflicting gradients across morphologies begin to impede learning. To mitigate this, we introduce an embodiment-based grouping strategy in which robots are clustered by morphological similarity and the model is updated with a group gradient. This simple, static grouping substantially reduces inter-robot conflicts and outperforms existing conflict-resolution methods. Project page: https://haruki-abe.github.io/cross_embodiment_offline_rl_website

Published as a conference paper at ICLR 2026
CROSS-EMBODIMENT OFFLINE REINFORCEMENT LEARNING FOR HETEROGENEOUS ROBOT DATASETS
Haruki Abel.2, Takayuki Osa?, Yusuke Mukuta Tatsuya Harada
'Tee University of Tokyo. Tokyo, Japan
PRIKEN Center for Advanced Intelligence Project, Tokyo, Japan {abe, mukuta, harada @mmi t. w-w.kyoooo... aac.y..c...... ac. jp, {takayuki, oraayukk.
ABSTRACT
Scalable robot policy pre-training has been hindered by the high cost of collecting high-quality demonstrations for each platform. In this study, we address this is- sue by uniting offline reinforcement learning (offline RL) with cross-embodiment learning. Offfine RL leverages both expert and abundant suboptimal data, and cross-embodiment learning aggregates heterogeneous robot trajectories across di- verse morphologies to acquire universal control priors. We perform systematic analysis of this offline RL and coos-emmboimmen paradigm. providing princi- pled understanding of its strengths and limitations. To evaluate this offline RI and cross-embodiment paradigm, we construct suite of locomotion datasets span- ning 16 distinct robot platforms. Our experiments confirm that this combined approach excels at pre-training with datasets rich in suboptimal trajectories, out- performing pure behavior cloning. However, the proportion of suboptimal data and the number of robot types increase, we observe that conflicting gradients across morphologies begin to impede learning. To mitigate this, introduce an embodiment-based grouping strategy in which robots are clustered by morpho- logical similarity and the model is updated with group gradient. This simple, static grouping substantially reduces inter-robot conflicts and outperforms exist- ing resoutton methods.
INTRODUCTION
In recent years, breakthroughs across many areas of machine learning have been driven by scaling up both the size the model and the dataset. In particular, large language models (LLM) and vision-language models (VLM) pre-trained on the web-scale, diverse and massive data have become capable of solving a wide variety of linguistic and visual tasks (OpenAl 2023 Gemini Team 2023). Likewise, generative models for image and video generation, well music generation, can
produce outputs of unprecedented quality (Rombach et al. 2022 Singer et al. 2022).
This trend has also begun to influence robotics, where scaling transformer-based architectures and training them on large, heterogeneous robot datasets have produced "robot foundation models" capa- ble of multiple tasks within a single model [Black al., [2044] Open X-Embodiment Collaboration 2024 Octo Model Team et al. 22024] Kim et al.) (024). However, despite the promise of foundation models for robotics, they face critical limitation. Available demonstration data remain extremely limited compared with the vast text and image corpora that fuel today's 's foundation models. Col- lecting manipulation data time-consuming and expensive, and each new task requires careful teleoperation, specialized hardware, and often manual labeling, making data scaling difficult.
To overcome this data bottleneck, researchers have turned to cross-embodiment learning, where single model is pre-trained on demonstrations from many different robot platforms rather than just one. By pooling heterogencous data, the model can learn more generalizable control primitives. Building this, the key practical benefit is adaptability, because cross-embodiment pre-trained policy can be adapted to new robots with only modest additional data, avoiding per-robot data col- lection and from-scratcl training and thereby lowering costs and speeding deployment. Recent VLA work supports this direction, showing that -efteeterr-ffiienttt and optimized fine-tune tinettuning can achieve
strong performance and substantial speedups with limited data (Kim al. 2024 2025).

---

Published as a conference paper at ICLR 2026
However, these foundation models for robotics have so far been confined to imitation learning, which requires high-quality demonstration data, typically gathered via expert telcoperation, and therefore remains costly for each individual robot. To overcome this limitation, there is growing attention on offline reinforcement learning (offline RI) as a pre-training method. Offline RI. can leverage not only expert demonstrations but also suboptimal data, enabling policy learning that surpass pure imitation learning in settings where high-quality data are difficult to acquire, such as robotics (Levine al., 2020). For example, Q-Transformer (Chebotar et al. 2023) demonstrates that, when offline RL is applied to large-scale robotic datasets containing suboptiaal trajectories, the model can outperform behavior cloning (BC), one of the most widely used approaches to imitation learning- This suggests a natural complementarity. By combining cross-embodiment learning with offline RL, one could exploit heterogeneous robot data and both expert and suboptimal data to dramatically ex- pand the pool of data usable for pre-training. However, this promising combination remains largely unexplored, and there is no established methodology or systematic analysis.
The contribution of this paper is an empirical study that systematically investigates the challenges and potential of cross-embodiment offline RL. We present a new benchmark and analysis of pre- training combining offline RL with cross-embodiment learning on data collected from up to 16 different robot platforms. Our experiments reveal two key findings: (i) under conditions with sub- stantial suboptimal data, the combined cross-enbodiment and offline RL approach outperforms imi- tation learning, and (ii) as the proportion of suboptimal data and the number of robot types increase, improving performance becomes difficult for particular embodiments under naive offline RL. We analyze this phenomenon and find that, in cross- tross-embodiment learning, each robot's gradients con- flict and cause some robots' gradients vanish. This phenomenon leads degraded performance. To address this, we propose a novel group-task update strategy based on robot embodiment informa- tion. We encode each robot as a morphology graph, compute pairwise distances, and cluster robots accordingly. We then update the policy group by group, sequentially applying an actor update us- ing only the samples of the group. This grouping mitigates gradient conflicts and yields substantial performance gains in settings with high suboptimal data ratios where standard offline RL stalls.
This paper makes the following contributions:
We introduce and analyze the new benchmark that combines offline RL with cross- embodiment learning across up to 16 distinct robot platforms.
2. We show that offline RL with cross-embodiment outperforms BC when suboptimal tra- jectories dominate, and that the pre-trained model can learn representations that markedly accelcrate fine-tuning on previously unseen robots.
We find positive transfer among morphologically similar robots, while demonstrating that increasing the suboptimal-data ratio and the number/diversity of robots amplifies inter-robot gradient conflicts, which turn induces negative transfer.
4. To mitigate these conflicts, we propose a simple, static grouping strategy that represents each robot morphology graph and clusters robots by graph-based distances; perform- ing group-wise policy updates reduces gradient interference and yields large gains, up to 39.8% improvement, especially under high suboptimal-date conditions.
RELATED WORKS
2.1
OFFLINE RL
Offline RL aims to learn policy that maximizes cumulative reward using only static dataset of environment interactions without further online interaction (Levine et al. 2020). Compared to BC, which simply mimics the trajectories of the dataset, offline RL can stitch together suboptimal seg- ments to produce better performance when the dataset contains a mix of expert and suboptimal data (Kostrikov et al. 2021 Kumar et al. (2020.. For example, implicit Q-learning (IQL) (Kostrikov et al.) 2021) first fits a state value function (s((s) via expectile regression to capture an upper expectile of the return distribution; it then uses the learned Vs'(s') as the TD target for updating state sitte-action value function Qu(s, a). Finally, the policy [a(a s) is extracted via advantage-weight
BC, avoiding any need to evaluate diitr--iistiiuutionn actions. This pipeline enables IQL. to leverage sub-
2

---

Published as a conference paper at ICLR 2026
optimal trajectories effectively and achieve strong performance without explicit policy constraints. In this work, we adopt offline RL as our pre-training paradigm to broaden robotic data, especially suboptimal data that can be exploited for foundation model learning in robotics.
2.2 BROSS-EMBODIMENT LEARNING
Cross-embodimenn learning trains single network on data collected from multiple robot morpholo- gies, enabling transfer of control priors across platforms. Since collecting large datasets for any single robot is costly, pre-training on heterogeneous robot data has become popular strategy to im- prove generalization capability (Open X-Fmbodiment Collaboration, 2024 Octo Model Team al.) 2024 Black et al.] 2024; Kim et al. 2024). However, existing eoosseembodiment foundation mod- els rely almost exclusively on imitation learning (Open X- X-Embodiment Collaboration 2024 Octo Model Team et al. 2024), and there has been little work combining cross-embodiment pre-training with offline RL. While Nakamoto et al. (2024) applied offline RL to data from two robot platforms, they did not analyze any cross- cross-embodiment effects. Similarly, Sprrineeber et al. (2024) conducted offline RL on a dataset comprising two manipulators and several toy tasks but did not investigate the benefits challenges of learning from many distinct embodiments simultaneously. To fill this gap, we introduce the new benchmark that systematically combines offline RI. with cross-embodiment learning, analyze the interactions between these paradigms, and propose methods mitigate the challenges that arise when pooling heterogencous and often suboptimal robot data.
3
EXPERIMENTAL SETUP
3.1 PROBLEM SETTING
We study multi-embodiment offline RL, where single policy must control multiple robot mor- phologies under a common aate-actioon interface. Unlike the standard multi-task RL setting, where a single robot embodiment solves multiple tasks with different rewards or goals, here mul- tiple robot embodiments solve common locomotion objective and reward functions that share the same components but may use embodimeen--ppeciff weights. Such settings have been re- ferred to tros-eembodimeet or multi-embodimmen learning (Open X-Embodiment Collaboration, 2024 Bohlinger et al., 2024). In this work, we follow this terminology and refer to our set- ting cross-embodiment offline RI. Concretely, let TT denote finite of robot embodiments (e.g.. different quadrupeds, bipeds, hexapods) and let fmorph : T -> Rdd map each embodi- ment index F to a morphology descriptor (morph(r). Each embodiment 7 T induces an MDP Pu.- (- a,,,,,r, 7), where S C Rdd, is the shared observation space (joint angles, velocities, etc.), A C Rdd is the continuous control space, (o. the embodiment- specific distribution over initial states, (=(s' a) is the embodiment--pecce ies, (s, a) is a dense sabbodimen--speciine reward and y (0, 1) is the discount factor. Episodes may end according to a terminal condition encoded in the transition tuples through termination indicator de {0, 1).
M,
environment dynam-
We assume access to pooled offline dataset D Urer((se, ae 8t+1; Tt, d generated by an unknown behavior policy (a 8, (7)) Our goal is to learn a single parameterized policy (a 8, fmorph (T)) that maximizes the expected cumulative discounted return over all embodiments R = Σ, "rr"rr(s (s)) using only D. Rather than providing the policy with one-hot embodiment index, we rely on morphology features (size, mass, link lengths, etc.), which formalize as the descriptor fanorph()). In this way, the same policy aa(a s, pmorpph (T)) can generalize across embodiments by conditioning on these universal state features.
3.2 ENVIRONMENTS AND DATASET
To facilitate cross-embodiment pre-training under an offline RL paradigm, we constructed a new locomotion dataset within the MuJoCo (Todorov al. 2012) simulation environment, following the walking tasks of Bohlinger et al. (004). Our dataset includes 16 distinct robot platforms, including nine quadrupeds, six bipeds, and hexapod, each trained by Proximal Policy Optimization (PPO) (Schulman et al. 2017). During training, we record the tuple (sr, ag, St+1: Tt: d,) at each time step capture the state, action, next state, reward, and termination signals.
3

---

Published as a conference paper at ICLR 2026
For each robot, curate six variants of 1 M step datasets, divided by data quality:
Expert data: M steps collected by rolling out the fully converged PPO policy,
Expert Replay data: all interaction steps from training start until expert-level performance
(~> 500 M steps in total), uniformly subsampled
M steps to bound dataset size.
70% Suboptimal Replay data: k steps drawn from the early (suboptimal) phase of PPO training, mixed with 300 k steps from the late (expert-like) phase, totaling M steps.
Each of these Expert, Expert Replay, and 70% Suboptimal Replay datasets is provided in two walking-direction variants, Forward and Backward. In the Forward variant, the robot is commanded to walk forward at 1 m/s, whereas in the Backward variant the commanded base velocity is -1 m/s. The reward in both cases is the same dense locomotion reward, and the tasks differ only in the com- manded walking direction. See Appendix B for dataset construction details and Appendix C for reward distributions.
3.3 NETWORK ARCHITECTURE
In this section, present approach to cross-cmbodiment learning in offline RL setting. The central challenge to train single network across robots whose state and action dimensions differ. To address this, adopt the URMA architecture (Bohlinger et al. 2024), which enables multiple robots to share a single policy and state value function. URMA factorizes each observation into an embodiment-agee magooiment-agnostic general part o, and a robot-specific part. For locomotion, the robot- specific stream is further split into variable-length sets of joint and foot observations, {oz (o)))e)())) and ser()) Descriptor-condition concatenated with Og form morphology-agnosti we further extend URMA by introducing a state-action value function (Q-function). Specifically, we encode each action with an action encoder to obtain a latent action vector, which we then concatenate with the latent representation of the URMA encoder. See Appendix D for architectural details.
attention aggregates each set into fixed-size latents, which are
representation. To facilitate offline RL,
ANALYSIS OF STANDARD OFFLINE RL ALGORITHM IN CROSS-EMBODIMENT OFFLINE RL
4.1 COMPARISON OF BEHAVIOR CLONING AND OFFLINE RL
Here, we compare BC, widely used in cross-embodiment training of foun- dational robot models, with offline RL. To date, applications of offline RL to robot foundation models have been rare, owing to the difficulty of learning from unlabeled interaction data; thus, rigorous evaluation is necessary. Table reports the perfor- mance of BC and implicit Q-learning (QQL) Kostrikov et al. 202I), the commonly used offline RL method. On datasets with relatively uniform behavioral quality (for example, For- ward Expert and Backward Expert), BC and offline RL achieve comparable results. In contrast, on datasets containing predominantly suboptimal trajectories, specifically 70% Suboptimal Replay Forward and Expert Replay data, offline RL methods surpass BC. This finding mirrors the results reported on benchmarks like D4RL (Fu et al.) 2020) and confirms that offline RL remains robust when datasets include significant suboptimal data in cross-embodiment context.
Table 1: BC vs. IQL performance across datasets (mean + standard error over seeds).
Dataset
BC
IQL
Expert Forward Expert Backward Expert Replay Forward
63.31 0.10
63.39 - 0.05
67.17 0.01
6 0.01
49.71 + 1.06
54.61
+ 0.12
Expert Replay Backward
42.87 + 1.32
51.86 - 1.56
70% Suboptimal Forward
30.52 3310
36.62
1.02
70% Suboptimal Backward
41.42 + 0.71
38.69 + 0.89
Mean
49.17
52.05
4.2 EFFECTS OF CROSS-EMBODIMEN PRE-TRAINING
In the next experiment, we now evaluate how cross-embodiment pre-training impacts the perfor- mance of single-embodiment fine-tuning with offline RL. Figure shows the learning curves for
4

---

Published as a conference paper at ICLR 2026
(a) Badger
(b) Unitree G1
Cassie
60
80
80
Eval/Epikode Retturn
60
60
40
40
15-
20
20:
Single robot XE pretrained
15 Update steps
25
Update steps
x10
Update steps
Figure 1: Comparison of learning curves between cross-embodiment pre-trained networks and net- works trained without cross-embodiment pre-training for Badger, Unitree GI, and Cassie.
"leave-one-out" experiment. We pre-train via offline RL on a dataset excluding one robot, then fine- tune that robot with pre-trained networks, comparing it to model trained without cross-embodiment pre-training. The model pre-trained with cross-embodiment data converges markedly faster. The re- sults show that, for the quadrupedal robot Badger as well as for the bipedal robots Unitree G1 and Cassie, the network pre-trained via cross- embodiment dataset is able to learn effective policy more rapidly. These results demonstrate that cross-embodiment learning serves as a highly effective pre-training strategy in offline RL.
4.3 POSITIVE AND NEGATIVE TRANSFER IN CROSS EMBODIMET LEARNING WITH SUBOPTIMAL DATA
In this section, first compare the final performance of models trained each robot in isolation with a model trained via cross-embodiment learning. Table shows the final rewards achieved by IQL when trained via cross-embodiment on the Expert Forward and 70% Suboptimal Replay Forward datasets, along with the rewards obtained by models trained separately on each robot.
In the Expert Forward dataset, the cross-embodiment models learn just effectively the single- robot models. However, an examination of the 70% Suboptimal Replay Forward dataset, which contains a large proportion of suboptimal data, reveals different result. Although the average performance of cross-embon cross-embodiment models falls below that of isolated models, certain quadrupedal robots, Unitree AI, Unitree Gol. Unitree Go2, and Badger, show substantial performance gains. This suggests that positive transfer occurs among the quadrupeds, likely because they contribute the largest share of data to the dataset.
By contrast, bipedal robots such as Unitree H1 and Unitree GI, which have relatively little cimilar-embodiment data in the dataset, suffer pronounced performance degradation under cross- embodiment learning compared to their isolated training. Because this negative transfer does not ap- pear when using suboptimal data without cross-embodiment or when training on the Expert dataset which contains fewer suboptimal trajectories, conclude that negative transfer emerges only when a dataset combines large amounts of suboptimal data with the enboddmmbod training regime.
Together these results show that the combination of cross-embodiment learning and offline RL can yield both positive and negative transfer. Negative transfer is most likely when suboptimal tra- jectories dominate and embodiment diversity is high. These findings indicate that effective cross- embodiment learning requires methods that minimize negative transfer while maximizing positive transfer.
5 GRADIENT CONFLICTS AND RESOLUTION VIA EMBODIMENT-BASED GROUPING
In this section, we analyze why combining large amounts of suboptimal data with cross-embost embodimend learning fails to yield performance improvements. Our empirical results indicate that gradient flict a primary cause of negative transfer. To address this issue, we propose a novel mitigation strategy that groups robots according to their embodiment, thus reducing gradient conflicts.
5

---

Published as a conference paper at ICLR 2026
Expert vs. 70% Suboptimal IQL performance across robots and avg. gradient cosine similarity C' on the 70% subop- timal dataset. Cells shaded blue ( indicate large positive transfer (CE exceeds Single by 10). while cells shaded light red indicate large negative transfer (CE falls below Single
Table 2:
Figure 2: Fraction of negative pairwise gradient cosine sim- ilarities.
0.90
0.323
by > 10).
Rupotonn 0
0.25
6.268
0.20
Robot
Exp Single
Exp CE
70%
Single
70%
CE
Avg. C
0.139
unitrec_al
53.78
53.69
14.55
27.38
0.114
unitree-go
54.03
54.05
14.46
40.05
0.118
unitree go2 go2
53.50
53.54
13.76
52.39
0.102
Exsul
30% aoSpar
anymal_h
49.84
49.73
48.00
47.81
0.070
Suboptimal data fraction
anymal_c
44.46
44.46
43.63
42.73
0.082
of C'
barkour_v0
46.28
46.68
46.25
46.71
0.115
barkour.vb
53.25
53.51
55.09
54.98
0.128
badger
53.00
52.89
15.98
10.53
0.118
bittle
36.19
36.82
45.68
44.92
0.080
unitree_hl
54.06
53.43
54.47
6.00
0.097
unitree gl
79.06
78.81
78.93
0.86
0.088
talos
108.37
108.40
3.26
10.17
0.088
robotis
89.31
88.56
102.63
98.25
0.103
Uniros
Al quedhuoed
All raba
nao_v5
83.56
83.42
91.56
86.38
0.107
cassie
79.13
79.15
1.19
1.64
0.074
(h) Embodiment diversity vs.
hexapod
74.94
76.40
1.28
0.37
0.075
fraction of C <.0.
mean
63.30
63.35
39.42
37.57
0.097
5.1 GRADIENT CONFLICTS AMONG ROBOTS
We hypothesize that when suboptimal data dominate the training set, simultaneous cross- embodiment learning induces negative transfer through conflicting policy gradients across robots. In particular, bipedal robots (e.g.. Unitree HI and Unitree GI) most affected, since the dataset contains relatively little data from similarly embodied robots; these gradient conflicts can effectively cancel useful updates and stall learning.
To quantify this effect, we analyze the actor gradients arising from addaanaage-eiigheee regression (Kostrikov et 2021 Nair al. 2020). For each embodiment v, we define the actor objective
C (0) (x.o) [w[s, (u(s, a) log (a s)], w(s,a) )) = cxp(3(Q(s. a) V(s))). (1)
in which Q(s, a) denotes the learned iate-action value function and (s)) denotes the learned state value function. We denote the per-embodiment actor gradient by gr VoC5(0) and measure entormemboimment alignment via the pairwise cosine similarity
(gr,, 9r,)
ClT,,
||g., [19t,
Figure 2a reports. during training, the proportion of pairwise cosine similarities that are negative for three datasets: Expert Forward and 30% and 70% Suboptimal Replay Forward. As the proportion of suboptimal data increases, the share of negative cosines increases, indicating more frequent gradient conflicts between robots.
(C 7y.
To further investigate the relationship between transfer and gradient conflicts, we compute the cor- relation between each robot's 's transfer gain and its average gradient cosine similarity with all other robots in the 70% Suboptimal Replay dataset. We define the transfer gain as the change in aver- age return induced by cross-embodiment training compared single-embodiment training. We use robots with substantial transfer gain (absolute change in return greater than 10), highlighted in Table 2 The resulting correlation, r 0.815, indicates a strong positive relationship: robots that exhibit positive transfer have more aligned gradients, whereas those with large negative transfer hibit greater gradient conflict. These findings confirm that gradient conflicts underlie the negative transfer observed when applying cross-ent cross-embodiment learning datasets rich in suboptimal data.
Next, we examine how gradient conflicts change as the number and diversity of robots increase (Fig- ure 2hp. Using the 70% Suboptimal Forward dataset, we progressively expanded the set of included

---

Published as a conference paper at ICLR 2026
Correlation = 0.630
10
uniaityy Jeresiinee
Embociment similarty
0.8
0.5
0.4
0.0
0.6
Mean _in_einer
(a) Embodinent-basee matrix
similarity
(b) Average gradient cosine simi-
(c) Sinbodiment-basee similarity
larity matrix
vs. mean gradient cosine similarity
Figure 3: (a) Embodiment-based similarity matrix (1 mon-mmax-normllized FGW distance between robot pairs); (b) Gradient cosine similarity matrix in Expert Forward dataset from Section 5.1 with the color scale compressed for readability using symlog normalization; (c) Scatter plot of embodiment-baid based similarity and mean gradient cosine similarity for all robot pairs.
robots: a relatively similar group (Unitree A1, Gol. Go2), then all nine quadrupeds, and finally all 16 robots. As we include more diverse robots, each robot exhibits a higher fraction of negative pairwise cosine similarities with the others (C < 0). In other words, greater embodiment diversity leads to more negative cosine similarities and more frequent gradient conflicts, making negative transfer more likely. Additional distributional analysis of these effects is provided in Appendix F and Appendix G. shows that the same qualitative relationships between the amount of suboptimal data, the number and diversity of embodiments, and how often gradient conflicts occur also hold for another offline RL algorithm, TD3+BC (Fujimoto & Gu 2021). suggesting that these phenomena arise more generally in cross-embodiment offline RI..
5.2 CORRELATION BETWEEN EMBODIMENT DISTANCE AND THE GRADIENT CONFLICTS
We now analyze how gradient alignment across robots captured by morphological similarity. We represent each robot's embodiment as graph to quantify inter-robot distances. Nodes correspond to the torso, joints, and feet. Edges connect the torso to adjacent joints, adjacent joints each other, and each terminal joint to its foot. Node features include relative positions from the torso and control parameters, capturing morphology and actuation. We then compute pairwise distances between these graphs using the Fused Gromov-Wasserster (FGW) distance (Vayer et al. 22020). Because FGW jointly accounts for both graph structure and node features when computing distances, is well suited for measuring dissimilarities between these robot embodiment graphs. For visualization and subsequent correlation analysis, we linearly normalize the FGW distance matrix so that all entries lie in the range [0, 1]. then convert distances into similarities by plotting 1 draw. that larger values correspond to more similar robot pairs. The resulting similarity matrix is shown in Figure Bal Quadrupedal robots cluster closely, for example Unitree Gol, A1, Go2, and Anymal B/C, matches intuition.
We next analyze how these embodiment based distances relate to the gradient conflicts characterized in Section 5.I and find that they are strongly correlated. Figure 33) presents the average gradient cosine similarity matrix from Section [.II Figure 3e further visualizes this relationship as scatter plot between embodiment similarity and mean gradient cosine similarity. The Pearson correlation coefficient between these two quantities r 0.63 (p 1.26 x 10 10-14), indicating a statistically significant positive correlation: morphologically similar robot pairs tend to have more aligned policy gradients, whereas morphologically dissimilar pairs exhibit greater gradient conflicts. Repeating the same embodiment similarity analysis with TD3+BC in Appendix G.2 yields comparable Pearson correlation between embodiment similarity and mean gradient cosine similarity, further supporting that the morphology dependent structure of gradient conflicts is general property. Motivated by these observations, the next subsection introduces an embodiment grouped offline RL update that explicitly exploits this structure to mitigate gradient conflicts.

---

Published as a conference paper at ICLR 2026
Cross Embodiment Dataset
Embodiment Graph
Embodiment Grouping
Group
Encoding
Grouping
Group
Update
Figure Overview of Embodiment Grouping (EG) for cross-embodiment offline RL..
5.3 SRRODIMENT-GROUREE OFFLINE RL UPDATE
Based on the analysis in the previous subsec- tion, we hypothesized that preventing gradi- conflict would yield more efficient learn- ing. Inspired by selective group updates (Jcong
Algorithm
Emboodinent-Grroppel Offlie RI.
Require:
Robot groups [GI..... Gm dataset D>
Ensure: Policy dz; critics/targets 6.
& Yoon 2025). we partition robots into groups before applying actor updates. Concretely, fol- lowing Section 5.2, we represent each robot as graph, compute pairwise FGW distances be- tween robot graphs, and create an embodiment-
1: for each outer iteration do
2: Sample global minibatch B ~
3: Compute Leritie (B); update Og
4: Update target networks
5:
P shuffle([1,.
distance matrix. We then apply hierarchical
6:
for m do
clustering to this matrix to obtain M robot groups, which remain fixed throughout train-
7:
B,n - far robot(x) 9m Compute I., (Bm(): update 0x
8:
ing. Training then proceeds with one global critic update, followed by sequential group- wise actor updates. Figure Saare/jillsse illustrates this full Embodiment Grouping (EG) workflow. Details on the robot graph construction and FGW distance hyperparameters are given in Appendix[E] In the experiments, we also evaluate alternative groupings such random partitioning and a heuristic split, and we report the comparisons Section 6.2 This pre-specified embodiment-basee grouping is simple to implement yet highly effective. The algo- rithm summarized in Algorithm and can be easily integrated with standard offline RL methods.
9: end for
10: end for
EXPERIMENTAL EVALUATION
Our experiments are designed to validate the effectiveness of Embodiment Grouping in two parts. First, we evaluate performance improvements in cross-embodiment offline RL on six datasets con- taining varying proportions of suboptimal data. We compare eight methods. As single-method baselines, we include Behavior Cloning (BC) using the same network architecture as our offline RL backbones, TD3+BC (Fujimoto Gui 2021 as strong and widely used offline continuous- control baseline, and implicit Q-learning (IQL). To evaluate existing approaches for mitigating negative transfer, we also test two mandient-confict-waare IQL variants: IQL+PCGrad, which solves gradient conflicts via gradient projection YYu et al. 2020). and IQL+SEL, which selectively groups tasks based on dynamic task affinity (Jeong & Yoon (025). Finally, assess the effect of our embodiment-baat
grouping strategy across different learning backbones, report Embodi- ment Grouping (EG) counterparts of BC, TD3+BC, and IQL. We denote these variants as BC+EG, TD3+BC+EG, and IQL+EG (ours), respectively. All methods trained offline and in aggregate use approximately 100 million transition samples per robot for gradient updates.
Second, conduct ablation studies to isolate the contributions of (i) grouping strategy, comparing random grouping, an intuitive biped quadruped split and our embodiment-basse grouping: (ii) the number of groups M; and (iii) update-count control, for which report compute- normtlizoddddddd comparison that matches both total optimizer steps and processed samples across methods, Detailed hyperparameters and training configurations provided in Appendix
8

---

Published as a conference paper at ICLR 2026
Table 3: Each Algorithm Performance across Dataset (# Standard Error, seeds)
Datnset
BC
TD34BC
IQL
IQL SEL
IQL.
PCGrad
BC EG
TD3+BC' BG
IQL 4 EG (ours)
Forward
63.31 0.10
52.14 1: 1.39
63.39
0.05
63:37
0:07
63,37 0:04
63.47
59.34
63.52 0.04
Expent Rockward
47.94
0.48
67.10
0.01
67.24
0.02
67.05 0.02
Expent Reglay Forward
1.06
55.66
0.84
54.61
0.32
55.01
0.55
$3.84 0:67
57.04
Replay Buckward
1:22
55.13
1.06
Suhopttinal Forward
30.52
15.74
56.02
1.00
59
2.00
Suboptimal Backward
1.75
Mian
49.17
46,43
$2.05
55.07
53:48
53:42
$1.39
57.29
6.1 GROUPING ROBOTS BY EMBODIMENT MAKES BROSS--EMOODIMEN LEARNING BETTER
In this experiment, we evaluate, on our cross-embodiment offline RL benchmark, how much the proposed Embodiment Grouping (EG) improves implicit Q-learning (IQL) under cooss-embodiment training. We compare across six datasets (Expert Expert-Replay / 70% Suboptimal Backward). Table 3 reports the final returns, averaged over five random seeds with + indicating standard error. The variant that combines IQL with Embodiment Grouping achieves the best average performance. The gains are especially large when the dataset contains more suboptimal trajectories, in the replay and 70% Suboptimal splits. Relative methods that suppress gradient conflicts such as PCGrad and Selective Grouping (SEL), IQL+EG performs better. Compared to the IQL. cross-embodiment baseline, the average improvement in the Suboptimal datasets 70% is 7.15% for PCGrad, 18.33% for SEL and 33.99% for EG.
Forward /
EG yields consistent benefits across range of offiine learning baselines. On the 70% Subop- timal datasets, applying EG to TD3+BC improves performance by +19.5% on average, raising the overall mean from 46.43 51.39. Similarly, BC+EG improves BC by +26.3% on the same suboptimal splits. These results indicate that embodiment-aware grouping remains broadly effec- tive across different offline objectives. These improvements are most pronounced exactly in the regimes where Section 5. revealed severe inter-robot gradient conflicts, supporting our hypothesis that embodiment-bart based grouping primarily helps by mitigating such conflicts across embodiments.
6.2 ABLATIONS & COMPUTE-NORMALIZEA ANALYSIS
This section disentangles design choices in Embodiment Grouping (EG) and the impact of compute. We examine: (i) grouping strategy, (ii) sensitivity to the number groups M, and (iii) comparisons where optimization steps and data usage are normalized.
(i) Grouping strategy We compare clus- tering based on embodiment distance (FGW; Sec(5.2) (EG) against random partitioning (Random) and an intuitive four-group split (Heuristic: bipeds, quadrupeds, hexapods, and torso-less bipeds). Tably reports the mean final return and variance on the 70% Suboptimal Forward dataset.
Table Ablation on grouping strategy (70% Subop- timal Forward). Values are mean final return + SEM; Relative is vs. cross-embodiment IQL.
Method
Final return
Relative %
IQL (baseline)
37.57 0.78
0.00%
Random grouping Heuristic EG (ours)
38.73 2.03
+3.08%
34.45 1.97
-8.31% %
From the table, EG achieves the most sta- ble and substantial improvement on the 70% Suboptimal Forward dataset (+14.41, +38.34%) In contrast, Random yields only small gain (+1.16, +3.08%). and the intuitive four- way split Heuristic actually degrades performance (-3. 14, -8.31%). This trend consistent with the correlation between reward gains and inter-robot gradient alignment shown in Sec. [5.1 indi- cating that EG, which groups by FGW-based distance, better aligns updates during training. It is somewhat surprising that the Heuristic split did not improve performance. A likely reason is that coarse categories such as leg count cannot capture praddeettrreevan factors like actuator placement, link lengths, mass distribution, and joint couplings. By contrast. EG based on FGW distance forms groups that reflect structural and actuation similarity while maintaining sufficient data per group, thereby preserving within-group update consistency and suppressing gradient conflicts.
51.98 1.70
+38.34 %

---

Published as a conference paper at ICLR 2026
(ii) Sensitivity to the group count M We evaluate the effect of the number of Embod- iment Grouping clusters M by sweeping M over {1, 2, 4, 7, 10, 13). The final performance function of M is plotted in Fig. perfor- mance peaks at small to moderate M. whereas excessive partitioning leads
140
52.5
Wall-clok time (h) (4)
120
50.0
100
47.5
80
45.0
Final
42.5
60
slight degra- dation (e.g,, on the 70% Forward dataset the best score is attained at M-7). Because M also determines the number of policy updates per batch, smaller M reduces the number of updates and thus improves computational effi- ciency. As shown in Fig. the wall-clock train- ing time grows substantially with M. Taken to- gether, these results suggest that relatively small values, M-2 ~ 4, already yield strong gains; a practical strategy is to start with small M and increase it gradually while balancing the trade-off between training time and performance.
37.5
10
13
Group count
Figure 5: Effect of the number of Embodiment Grouping clusters M on final return and wall- clock training time (mean + s.e.).
(iii) Compute- and data-normalized comparison EG performs M policy updates per outer iteration, so naive setup increases the number of updates by factor of M. The results above matched the total number of data steps but not the number of policy's 's optimizer up- dates, which may be unfair. To remove this effect, we also run a compute-normalized comparison in which, for the IQL baseline, optimizer steps K by to match EG, and reduce the batch size by 1/M that the total number of pro- cessed samples remains constant. The results are sum- marized in Tab. 5 We observe that with the 70% Suboptimal Forward dataset, the advantage of EG persists even after normalization. This indicates that the benefit of EG does not stem merely from increasing the number of policy updates, but rather from grouping strategy that mitigates gradient interference.
Table 5: Normalized IQL vs. EG (70% Suboptimal Forward). AR EG nor- malized IQL.
Method
(mean + SEM)
Normalized IQL
44.20 + 2.22
multiply the total number of
IQL + EG
51.98 + 1.70
AR
+7.78
7 CONCLUSION
We presented the systematic study of combining offline RL with cross-embodiment pre-training across 16 robot platforms. Our results show that this paradigm is particularly effective when datasets contain substantial suboptimal trajectories. On such datasets achieves higher returns than BC, and for downstream adaptation it learns faster than training without cross- coos-eembodiment pre-training. We also identified a core failure mode, inter-robot gradient conflicts, whose incidence grows with both the proportion of suboptimal data and the number of embodiments. Quantifying gradient alignment showed a consistent association with transfer performance, and we found that this alignment is well captured by embodiment distance. This motivates our proposed Embodiment Grouping (EG), which clusters robots by morphology-awart distances and performs group-wise policy updates. EG consistently reduces interference and delivers the largest gains under high-suboptimal conditions.
This study has limitations. Our evaluation restricted to MuJoCo locomotion in simulation, which leaves sim-to-real transfer and broader domains such as manipulation and mobile manipulation untested. Our proposed Embodiment Grouping is static and derived from embodiment graphs us- ing FGW distances. works stably offline RI., but may not adapt when learning dynamics or data quality change, in offline-to-online RL settings. Promising directions include validation on real robots and on manipulation tasks. Another direction is to improve Embodiment Grouping with dynamic grouping that uses embodiment information together with learning progress and data characteristics. In addition, once compatible groups are identified, learning group- embod- iment task representations within each group, for example using contrastive objectives (Yap & Koon Ng 2024), could further strengthen -ithin sharing. We leave this combined direction for future work.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was partially supported by JST Moonshot R&D Grant Number JPMJPS2011, CREST Grant Number JPMJCR2015 and Basic Research Grant (Super AI) of Institute for AI and Beyond of the University of Tokyo. T.O. was supported by JSPS KAKENHI Grant Number JP25K03176.
REPRODUCIBILITY STATEMENT
We ensure reproducibility via clear pointers. The grouped-update procedure appears in Algorithm The architecture and training details in Appendix [D] Hyperparameters for BC, IQL., TD3+BC.
T
and PPO are in Appendix [ and Tables 11 and 12 Dataset construction, including Expert, Expert
Replay, and X% Suboptimal. is in Appendix B with distributions of rewards in Appendix C Em-
bodiment distance and clustering, including robot graphs and FGW settings, are in Appendix Appendix E.2
and
REFERENCES
Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian lchter, et al. A sissio-languag--cttinn flow model for general robot control. arXiv preprint arXiv:2410.2416,, 2024.
Nico Bohlinger, Grzegorz Czechmanowski, Maciej Krupka, Piotr Kicki, Krzysztof Walas, Jan Pe- ters, and Davide Tateo. One policy to run them all: an end-to-end learning approach to multi- embodiment locomotion. Conference on Robot Learning. 2024.
Yevgen Chebotar, Quan Vuong, Alex Irpan, Karol Hausman, Fei Xia, Yao Lu, Aviral Kumar, Tianhe Yu. Alexander Herzog, Karl Pertsch, Keerthana Gopalakrishnan, Julian Ibarz, Ofir Nachum, Sumedh Sontakke, Grecia Salazar, Huong T Tran, Jodilyn Peralta, Clayton Tan, Deeksha Manju- nath, Jaspiar Singht, Brianna Zitkovich, Tomas Jackson, Kanishka Rao, Chelsea Finn, and Sergey Levine. Q-transformer: Scalable offline reinforcement learning via autoregressive q-functions. In 7th Annual Conference on Robot Learning, 2023.
Rémi Flamary, Cédric Vincent-Cuaz, Nicolas Courty, Alexandre Gramfort, Oleksii Kachaiev, Huy Quang Tran, Laurène David, Clément Bonet, Nathan Cassereau, Théo Gnassounou, Eloi Tanguy, Julie Delon, Antoine Collas, Sonia Mazelet, Laetitia Chapel, Tanguy Kerdoneuff, Xizheng Yu, Matthew Feickert, Paul Krzakala, Tianlin Liu, and Eduardo Fernandes Montesuma. Pot python optimal transport (version 0.9.5), 2024. URL https: withtttthhhhh...g com/ Pwt.oyy/oooooooo
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforeement learning. arXiv preprint arXiv:2004.07219, 2020. 2020.
Scott Fujimoto and Shixiang Shane Gu. minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34:20132-20142 201444,, 2021.
Gemini Team. Gemini: a family of highly capable multimodal models. 11Xi::2312.11805. 2023.
arXiv preprint
Wooseong Jeong and Kuk-Jin Yoon. Selective task group updates for multi-task optimization. arXiv preprint 202. 2025.
Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pamnag Sanketi, Quan Vuong, Thomas Kollar, Ben- jamin Burchfiel, Russ Tedrake, Dorsa Sadigh, Sergey Levine, Percy Liang, and Chelsea Finn. Openvla: An open-source vision-language-aaii
model. arXiv preprint 24060094006 2024.
Moo Jin Kim, Chelsea Finn, and Percy Liang- Fine-tuning vision-languagg--ction models: Opti-
mizing speed and success. arXiv preprint 1966 2025.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q- learning. arXiv preprint arXiv:2/ 10.06169, 2021.

---

Published as a conference paper at ICLR 2026
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in neural information processing systems, 33:1179- 11919 2020.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto-
rial, review, and perspectives on open problems. arXiv preprini arXiv:2005.01644, 2020.
Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online rein- forcement learning with offiin datasets. arXiv preprint arXiv:2006.09359) 2020.
Mitsuhiko Nakamoto, Oier Mees, Aviral Kumar, and Sergey Levine. Steering your generalists: Improving robotic foundation models via value guidance. Conference on Robot Learning (CoRL), 2024.
Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Charles Xu, Jianlan Luo, Tobias Kreiman, You Liang Tan, Lawrence Yunliang Chen, Pannag Sanketi, Quan Vuong. Ted Xiao, Dorsa Sadigh, Chelsea Finn, and Sergey Levine. Octo: An open-source generalist robot policy. Proceedings of Robotics: Science and Systems, Delft, Netherlands, 2024.
Open X-Embodiment Collaboration. Open X-embodiment: Robotic learning datasets and rt-x mod- cls: Open x-embodiment collaboration In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 6892-6903, IEEE, 2024.
OpenAl. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp.
2022.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 2017.
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang. Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation data. arXiv preprint arXiv:2209. 14792, 2022.
geneerattodoognnattto without text-video
Jost Tobias Springenberg, Abbas Abdolmalcki, Jingwei Zhang, Oliver Groth, Michael Bloesch, Thomas Lampe, Philemon Brakel, Sarah Bechtle, Steven Kapturowski, Roland Hafner, et al. Offline actor-critic reinforcement learning scales to large models. In International Conference on Machine Learning (ICML), 2024.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 JEEE/RSJ international conference on intelligent robots and systems, pp. 5026-5033. IEEE, 2012.
Titouan Vayer, Lactitia Chapel, Rémi Flamary, Romain Tavenard, and Nicolas Courty. Fused gromov-wassersee distance for structured objects. Algorithmss, 13(9):212, 2020.
Boon Peng Yap and Beng Koon Ng. Group-wise contrastive bottleneck for weakly-supervisee visual representation learning. 2024 IEEF/CVF Winter Conference on Applications of Computer Vision ( (WACV), pp. 2235-2244 2024. doi: 10. (U19//AAVV57001.222..4.0
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. Advances in neural information processing systems, 33: 5824-5836, 2020.
A USE OF LARGE LANGUAGE MODELS (LLMs)
We used [.LMs polish the writing of the paper and perform grammar checks.
12

---

Published as a conference paper at ICLR 2026
B DATASET CONSTRUCTION DETAILS
This appendix describes how each dataset used in our experiments was constructed.
B.I EXPERT DATASET
The Expert dataset was collected using an expert model that can walk almost perfectly according given command, For each robot, we independently trained a PPO policy to conver- gence and used it to generate data. Starting from the envi- ronment's default initial state, we sampled actions from the Gaussian distribution predicted by the trained policy log trajectories. We illustrate the Forward and Backward locomo- tion commands in Figure 6 For the Forward dataset, we is- sued command to walk forward at 1 m/s; for the Backward dataset, we issued a command to walk backward at m/s.
Backward
Forward
Figure 6: Illustration of the Forward and Backward locomotion com- mands used in our datasets.
B.2 EXPERT REPLAY DATASET
The Expert Replay dataset contains interaction data collected from the beginning training up to the point which the model can walk according to the given command (e.g., move forward at m/s for the Forward condition). Storing all PPO interaction data impractical due to PPO's ' low sample efficiency, which would result in extremely large data volume and step counts. We therefore constructed a uniformly thinned M-step dataset via the following procedure:
(1) Environment selection and full logging. Among the 48 parallel environments used for train- ing, we selected one and fully recorded all rollouts (approximately 10 M steps) in that environment, thereby capturing trajectories from the initial exploration phase through to near convergence.
(2) Extract data up to just before convergence. From the saved logs, we reconstructed episode boundaries and computed each episode's 's return and length. We then applied moving average to episode returns and identified the first point at which performance reached 90% of the final perfor- mance. Data up to just before this point were retained as the candidate set, while the subsequent steps, during which performance increases only slowly toward full convergence, were omitted.
(3) Uniform thinning to 1 M steps. If the total number of steps in the candidate set exceeded M, we down-sampled by discarding episodes at equal intervals with respect to cumulative steps. This preserved the overall distribution while reducing the dataset to approximately
steps.
Using this shared procedure, we created the replay datasets for all robots. For the Forward datasets of Unitree A1, Gol, and Go2, the default PPO hyperparameters led local optima. To encourage exploration and avoid such local optima, trained these policies with increased policy entropy coefficient, roty_coof = 0.1, and used the resulting data.
B.3 X SUBOPTIMAL DATASET
The X% Suboptimal dataset is constructed so that X% of the data are suboptimal. particular, the 70% Suboptimal dataset used in our experiments contains relatively large proportion of suboptimal trajectories. Whereas the Replay dataset samples evenly from early to late training phases, the X% Suboptimal dataset is formed by sampling X% from the early training phase and 100 -0 from the late training phase.
B.4 REWARD FUNCTION
We construct our datasets by following the walking-tass setup of Bohlinger et al. (024) and adopt dense locomotion reward that encourages accurate tracking of commanded base velocities while pe- nalizing undesirable gait patterns. The reward is composed of two command uammand-tracking terms defined with respect the target velocities c, together with several penalty terms that shape the gait. Each
13

---

Published as a conference paper at ICLR 2026
term is weighted by its own coefficient, and the total reward is obtained by summing all weighted terms and clipping the result from below at zero. Tables and list the equations for each compo- nent of the reward function and the corresponding coefficients.
Term
Name
Equation
(TI) Xy velocity tracking
exp
Usy
Cryl /0.25))
(T2) Yaw velocity tracking
exp
Wyaw
Gyaw/ /2/025))
(T3) Z. velocity penalty
(T4) Pitch-roll velocity penalty
Wittch roll 12
(T5) Pitch-roll position penalty
pitch, roll 12
(T6) Joint nominal differences penalty
Inominal 2
(17) Joint limits penalty
(0. 9qmin < 0.94max)
(T8) Joint accelerations penalty
1
2
(T9) Joint torques penalty
(TI0) Action rate penalty
[ 1
(T11) Walking height penalty
11
Aenomina
(T12) Collisions penalty
collisions
(T13) Air time penalty
Σ, (p) -0.5)
(T14) Symmetry penalty
Σ, 1(p) right)
Table 6: Reward terms composing the reward function.
Coefficients for each robot are listed in
Table
Robot
T1
12
T3
T4
T5
T6
T7
T8
T9
T10
T11
T12
T13
T14
T
ANYmalB
2.0
1.0
2.0
0.05
0.2
0.0
10.0
2.5c-7
2c-4
0.01
30.0
1.0
0.1
0.5
20e6
ANYmaIC
2.0
1.0
10.0
2.5c-7
2c-4
0.01
30.0
1.0
0.1
20c6
Barkour
3.0
1.5
0.05
10.0
2.5e-7
2e-4
0.01
30.0
1.0
15e6
Barkour
2.0
1.0
0.05
10.0
2.5c-7
2c-4
0.01
30.0
1.0
15c6
Silver Badger
2.0
1.0
2.0
0.05
0.2
0.0
10.0
2.5c-7
2c-4
0.01
30.0
1.0
0.1
0.5
12c6
Bittle
5.0
2.5
2.0
0.05
0.2
0.0
10.0
2.5e=7
2e-4
0.01
30.0
1.0
0.1
0.5
40e6
A1
2.0
1.0
2.0
0.05
0.2
0.2
10.0
2.5c-7
2c-5
0.01
30.0
1.0
0.1
0.5
12c6
Gol
2.0
1.0
10.0
2.5c-7
2c-4
0.01
30.0
1.0
0.1
12c6
Go2
2.0
10.0
2.5e-7
2e-4
0.01
30.0
1.0
0.1
Cassie
3.0
1.5
0.05
10.0
2.5c-7
2c-5
0.01
30.0
1.0
0.1
0.5
50c6
Talos
4.0
2.0
2.0
0.05
0.2
0.2
10.0
2.5e:7
2e-5
0.01
30.0
1.0
0.1
0.5
80e6
OP3
4.0
2.0
2.0
0.1
0.2
0.4
10.0
1.2e-6
4e-4
6e-3
30.0
1.0
0.1
0.5
40e6
Nao VS
4.0
2.0
2.0
0.1
0.2
0.15
10.0
1.2c-6
4c-4
6e-3
30.0
1.0
0.1
0.5
40e6
3.0
1.5
2.0
0.05
0.2
0.2
10.0
2.5e-7
5e-5
0.01
30.0
1.0
0.1
0.5
50e6
2.0
10.0
2.5e-7
2e-5
0.01
30.0
1.0
0.1
Hexapod
4.0
2.5c-7
20-4
0.01
30.0
0.5
15c6
Table 7: Reward coefficients re and curriculum length T for each robot.
C DATASET DETAIL
Figure 7 overlays histograms of the total reward per episode for the Forward datasets, comparing the three data quality levels used throughout the paper: Expert Forward, Expert Replay Forward, and 70% Suboptimal Forward.
Overall, three consistent patterns emerge: (i) Expert datasets are sharply concentrated at higher returns, indicating that most episodes achieve near-target performance; (ii) Expert Replay exhibits broad spread that reflects a mixture of early failures and late competent behavior accumulated during training; (iii) 70% Suboptimal shifts mass toward lower returns, with many episodes clustered near the low-reward region. Notably, while the 70% Suboptimal dataset contains only a small fraction of high-return episodes, those episodes tend to be considerably longer; when weighting by time steps, they account for approximately 30% of all steps. We observe qualitatively similar distributions for the Backward datasets.
These distributional differences clarify why offline RL tends to outperform pure imitation when suboptimal data abundant: objectives that reweight actions by estimated advantages discount
14

---

Published as a conference paper at ICLR 2026
low-quality behaviors while still leveraging recoverable structure present in Replay and Suboptimal corpora.
Epicade proportion
1.0
unitree al
unitree gol
1.0
Unitree go2
1.0
anymal b
0.5
0.5
0.5
0..5
0.0
0.0
0.0
0.0
anymal
1.0
1.0
1.0
1.0
bittle
unitree h1
unitree g1
talos
0.25
0.00
0.0
0.0
0.0
robotis, op3
nao v5
cassie
hexapod
3.0
1.0
0.75
Expert
Replay
0.50
Suboptimal (100
0.5
0.5
0.5
0.00
25
100
20
80
60
80
25
75
Episode Total Reward
Episode Total Reward
Episode Total Reward
Episede Total Reward
Figure 7: Overlaid histograms of per-episode total reward (x-axis) vs. episode proportion (y-axis) for Forward datasets across all robots. Each panel corresponds to robot; colors denote Expert, Expert replay, and 70% Suboptimal. Expert concentrates high returns, Replay spans wide range, and 70% Suboptimal places substantial mass on low returns. Vertical solid line: median; vertical dashed lines: 25th and 75th percentiles.
D ARCHITECTURE DETAILS
Encoder (URMA). We split each observation into general o, and robot-specific streams. Forloco- motion, the observation vector is subdivided into joint- and foot-level sets {o; Jje((r), {or)ser(r) with per-item descriptors d,, dy. Descriptors and observations are encoded by MLPs ʃø : R -R and fø TR. La URMA uses deseriitor-conditioneed attention that gates each latent dimension:
fa(dg)
exp
Ejoints Σ Ejo
T +
(3)
fd(d,
JEJ
exp
+
La
In the same way, we obtain Seet from dy,, and finally form a single latent vector by concate- nation ī = concat Oy Zjointa, Zfeet]
Actor network. A core MLP he maps the encoder output action latent:
Eaction
h2))).
Per joint, the action head decodes Gaussian parameters from the tuple (encoded descriptor, joint latent, action latent) and samples an action:
a NMp Eactions 2,), oz
d' = (d,).
Here My and 0, ML.ss
15

---

Published as a conference paper at ICLR 2026
State value network. The state value network uses the encoder and predicts state value from the latent:
V²(2) (2))),
(5)
implemented as an MLP (e.g., [512, 256, 128)).
State-action value Network For offline RL, we add an action encoder fa IR -R and form joint latent for Q:
Zo fa(a),
(2, =
(concatt,, =al); k E (1,2).
To accommodate heterogeneous action sizes, we feed Sa a zero-padded action vector E Rdduxx (padded to the maximum action length across robots).
Jeints
Foots
Repeat
Encoder fø
Come Newwork
Z
URMA Encodeer
Encoder Fip
(a) URMA encoder.
Descriptor-conditon attention aggregates
(b) State-action value network: The action latent vector is concatenated with other latent vectors.
joint/foot latents into a fixed-size embedding.
Figure 8: Model overview: (a) URMA encoder; (b) State-action value network.
E
EMBODIMENT GROUPING DETAILS
This appendix gives detailed recipe for grouping robots when building the embodiment distance used by our method. We first define the rohot graph representation. We then list the node features and hyperparameters used to compute the Fused Kromov-Wasseree distance.
E. ROBOT GRAPH REPRESENTATION
We represent each robot as a graph. The nodes correspond to the torso the joints and the feet. The edges connect the torso adjacent joints adjacent joints each other and the terminal joint to its foot, Figure 9 shows images of all robots and the corresponding graph representation. Each node carries feature vector constructed from the local descriptors that URMA uses as encoder inputs. The graph features are derived from the same observation factorization that URMA assumes: the joint-level and foot-level sets {oyljeJ(t) and {or)seF(+) with their corresponding local descrip- tors {d, })e](>) and } together with a general observation stream. our embodiment graphs, only the local descriptors (d,) and {dy} } used node attributes; importantly, do not use URMA's learned latent embeddings directly as graph node features. In particular, joint nodes the joint descriptor dy. foot nodes use the foot descriptor d,. and the node receives zero vector of the same dimension. The shared observation decomposition between URMA and the embodiment graphs can be viewed as an inductive bias that helps Embodiment Grouping miti- gate gradient conflicts. When applying embodiment grouping with policy architectures other than URMA, may therefore be beneficial to design graph node features so that they respect the obser- vation factorization implied by the chosen policy architecture.
E.2 COMPUTING FUSED GROMOV-WASSERRS
DISTANCE
We compute pairwise distances between robot graphs using the Fused Gromov- caanov--ssseetteiin distance with the POT library (Flamary et al. 2024). Feature costs the Fuclidean norm standardized
16

---

Published as a conference paper at ICLR 2026
untiows
untrew, goi
unitree gez
erymal
anymal
barkour
badger
untraw
bis
robst's opd
Cossie
hecepod
Joint
A
Foot
torso
Figure 9: All robots with their images and the corresponding graph representation. Nodes are torso joints and feet. Edges follow kinematic adjacency and the connection from each terminal joint its foot.
descriptors. Structural costs use shortest path distances on each graph. Node weights are uniform over nodes each graph. Table lists the hyperparameters that were fixed for all pairs.
We next discuss how this computation scales with the number of embodiments. In our method, FGW used only to compute pairwise distances between embodiments before training, in order to build the embodiment distance matrix. As the number of robots N increases, the number of FGW computations grows with the number of pairs, which is (2) = N(N - 1)/2. Thus, the overall complexity for building the distance matrix is O(N3). In practice, this cost is small compared to policy training. For our 16-robot setup, computing the full 16 x 16 distance matrix with our POT- based implementation takes about 0.8 seconds on a standard machine. Even for substantially larger N, this one-time cost remains negligible relative to the offline RL training time.
Table 8: Essential hyperparameters for Fused -amov--Wssersttinn distance with POT
Parameter
Setting
Feature ground metric Structure ground metric
Euclidean distance
Shortest path distance
Fusion tradeoff
0.5
Regularization
le-3
E.3 RESULTING GROUP ASSIGNMENTS
Using the FGW distance matrix described above, we perform hierarchical agglomerative clustering and cut the dendrogram to obtain M groups. Figure I0 visualizes the resulting robot partitions for M = 2, 4. These assignments are computed once from the FGW distance matrix and kept fixed across all experiments.
17

---

Published as a conference paper at ICLR 2026
Gree
Genup
Once
Oroqg
umee ded
Unire der
Cantr
Pobon DE?
(a) M 2
(b) M 4
Figure 10: Embodiment-gree assignments obtained by cutting the same hierarchical clustering tree different numbers of groups.
0.40
70% suboptimal 30%- sulapiabal
All robot quadruped
Expert
Unitree A1, Gol. Ge2
0.035
Proportion
0.025
Proportion
0.020
0.001
0.000
0.000
-3.4
-07
0.0
0.2
0.6
0.6
0.000
0.6
-0.2
0.0
0.7
0.5
0.6
(a) Varying suboptimal-data proportion.
(b) Varying embodiment diversity.
Figure 11: Histograms of pairwise cosine similarities CiT Tyl aggregated over training. In both settings (a) higher suboptimal-data ratios and (b) greater embodiment diversity- the fraction with C' < 0 increases; within < 0, mass concentrates at more negative values. Solid lines indicate the mean C.
F DETAILED ANALYSIS OF GRADIENT CONFLICTS
We further analyze gradient conflicts by collecting the pairwise cosine similarities C(T,, Ty] for all robot pairs and training steps, and aggregating them into histograms (see Figure 11). Two consistent trends emerge:
(i) Effect of suboptimal data. As the proportion of suboptimal trajectories increases, the fraction of pairs with C < grows. Moreover, within the negative region (C' < 0), share of values close to -1 increases with the proportion of suboptimal data, indicating stronger misalignment per update and a higher probability negative transfer. Consistently, the mean cosine C' decreases as the suboptimal fraction increases. Overall, these results show that increasing suboptimal data makes gradient conflicts both more frequent and more severe.
(ii) Effect of embodiment diversity. As include more and more diverse embodiments, the frac- tion of pairs with C < 0 increases, while the share strongly aligned pairs (large positive C') diminishes. The mean cosine C also gradually declines the embodiment diversity increases. To- gether, these effects indicate that greater embodiment diversity amplifies gradient conflicts, thereby increasing negative transfer and making positive transfer less likely.
G ADDITIONAL CORRELATION ANALYSIS FOR TD3+BC
In this appendix, verify the generality of the IQL based analyses in Sections 5.1 and 5.2 which linked gradient conflicts to suboptimality, embodiment diversity, and embodiment similarity, by re- peating the same study with another offline RL algorithm, TD3+BC. Concretely, when apply
18

---

Published as a conference paper at ICLR 2026
Table 9: Expert vs. 70% Suboptimal TD3+BC performance across robots and avg. gradient cosine similarity C on the 70% suboptimal dataset. Cells shaded light red ( indicate large negative transfer (CE falls below Single by 10).
Figure 12: Fraction of nega- tive pairwise gradient cosine similarities (TD3+BC).
0.00
Robot
70% Single (TD3+BC) 70% CE (TD3+BC)
C
unitree_al
27.86
27.65
0.233
0.00
6.22
0.181
unitree gol
52.56
52.57
0.227
unitrec. go2
51.97
52.18
0.213
anymal.b b
32.53
12.50
0.159
pas
anymal.c
42.32
23.41
0.168
30%-oubopt
barkour v0
45.85
45.42
0.208
Suboptimal data fraction
barkour vh
54.78
54.61
0.235
of (TD3+BC).
badger
52.56
40.58
0.201
bittle
45.64
44.71
0.117
unitree.hl .l
33.44
1.42
0.149
0.200
unitree.gl
66.71
1.94
0.155
0,277
talos
73.97
6.14
0.123
ropotis.op3
101.38
94.35
0.146
0.1
0.195
nao_v5
87.60
85.40
0.148
cassie
3.49
2.91
0.146
hexapod
23.01
23.24
0.116
Unitee
acbot
A1. Ge
49.73
35.55
0.171
(h) Embodiment diversity vs. fraction of C' 0 OTT3++BC
gradient conflict analyses TD3+BC, find that (i) the fraction of negative pairwise gradient cosine similarities increases as the proportion of suboptimal data grows, (ii) the same fraction also increases we enlarge the number and morphological diversity of robots, and (iii) these gradient conflicts are strongly structured by embodiment similarity. since robots that are closer in the mor- phology graph exhibit more aligned policy gradients. Taken together, the IQL and TD3+BC results indicate that the observed relationships between suboptimality, embodiment diversity, embodiment similarity, and gradient conflicts are general properties of cross embodiment offline RI. rather than artifacts of particular actor update.
G.1 GRADIENT CONFLICTS UNDER TD3+BC.
For each embodiment т, we define the TD3+BC actor loss LT.TD34BC on the subset of tran- sitions belonging to robot and denote its gradient by 9r V (0)) As in Sec- tion 5.1 we measure inter embodiment alignment using pairwise cosine similarities C(T,, Tyl =
(g.. 9r, /llg..11 119,,
Figure 12a reports, over the course of training, the fraction of negative pairwise cosine similarities (C(Ts, Tyl < under the Expert Forward, 30% Suboptimal Replay Forward, and 70% Suboptimal Replay Forward datasets. We observe the same qualitative trend as the IQL. case: as the propor- tion of suboptimal data increases, the fraction of negative cosines also increases, indicating more frequent gradient conflicts between robots. This confirms that the emergence of strong gradient con- flicts in the presence of abundant suboptimal data occurs for TD3+BC. To further investigate the relationship between positive and negative transfer and gradient conflicts under TD3+BC, we compute the correlation coefficient between each robot's reward difference dcros--embodined minus single-robot training) and its average TD3+BC gradi- cosine similarity with all other robots in the 70% Suboptimal Replay Forward dataset, using robots with substantial performance changes (absolute reward difference greater than 10), which are highlighted in Table The resulting correlation, r = 0.766, indicates a strong positive relationship: robots that exhibit large negative transfer exhibit greater gradient conflict. These findings mirror the IQL results and further support that gradient conflicts underlie the negative transfer observed when applying cooss-embodmment learning to datasets rich in suboptimal data.
tied the particular choice of IQL and also
We also examine how gradient conflicts evolve with embodiment diversity under TD3+BC. Using the 70% Suboptimal Replay Forward dataset, gradually expand the set of robots included in
19

---

Published as a conference paper at ICLR 2026
Sealnis Nimes Drandioe
and
Carre alion G.711
Embodiment sim artyy
0.4
0.0
0.2
0.4
0.6
0.5
Mean grad inner
(a) Embodiment-baase matrix
similarity
(b) Average gradient cosine simi- larity matrix (ID3+BC)
(c) Emboolmennt- similar- ity vs. mean gradient cosine similarity (TD3+BC)
Figure 13: (a) Embodiment- similarity matrix (1 min-max-normalize FGW distance be- tween robot pairs); (b) Average gradient cosine similarity matrix for TD3+BC on the Expert Forward dataset, with the color scale compressed for readability using symlog normalization; (c) Scatter plot of embodiment- based similarity and mean gradient cosine similarity for all robot pairs under TD3+BC.
cross embodiment training, starting from a small, morphologically homogeneous quadruped subset (Unitree A1, Go1, Go2), then all nine quadrupeds, and finally all 16 robots. For each configuration, we compute, for every robot, the fraction of negative pairwise cosine similarities with the other robots and average this quantity across robots. Figure [2b shows that, as add more diverse embodiments, the fraction of negative TD3+BC cosine similarities increases, just as in the IQL case. In other words, greater embodiment diversity leads to more frequent TD3+BC gradient conflicts, making negative transfer more likely when the dataset is rich in suboptimal trajectories.
G.2 CORRELATION WITH EMBODIMENT SIMILARITY
In this section, we analyze how TD3+BC gradient alignment across robots is captured by the same embodiment similarity measure used in Section 5.2 We reuse the FGW based embodiment simi- larity matrix and construct the corresponding mean TD3+BC gradient cosine similarity matrix by averaging C(T,, Ty] over training for each robot pair.
Figure 13a shows the resulting embodiment-basee similarity matrix, while Figure 13b presents the corresponding matrix mean TD3+BC gradient cosine similarities. As the IQL case, morpho- logically similar quadrupeds form tight cluster in both matrices. Figure 13c further visualizes this relationship as a seatter plot between embodiment similarity and mean TD3+BC gradient co- sine similarity. The Pearson correlation coefficient between these two quantities is 0.711 with 8.89 x 10-20 indicating a strong positive correlation: robot pairs that are more similar in embodiment tend have more aligned TD3+BC policy gradients. Together with the IQL results in Section 5.2, this supports the validity of embodiment based grouping as general, morphology driven mechanism for reducing gradient conflicts in cross embodiment offline RI.
H APPLYING EMBODIMENT GROUPING TO THE CRITIC
Algorithm applies Embodiment Grouping (EG) to the actor updates, while the critic is updated once per outer iteration using the global minibatch. This design is motivated by our analysis that negative transfer in heterogeneous cross-embodiment offline RL is mainly caused by policy-gradient conflict. Here we examine whether extending EG to critic learning further improves performance.
Setup. We evaluate variant, denoted as EG-Actor+Critic, that applies EG the critic in addi- tion to the actor. Specifically, after sampling a global minibatch B, we split it into specifc minibatches {Bm) using the same embodiment clusters the actor update. We then update the
20

---

Published as a conference paper at ICLR 2026
Table 10: Ablation on applying Embodiment Grouping the critic. Returns are mean + s.c. over seeds on the 70% Suboptimal Replay Forward dataset.
Group count M
EG on Actor only
EG on Actor Critic
2
48.53 + 2.33
48.38 + 2.25 50.30 + 0.22
51.98 + 1.70
critic sequentially over each Bm (i.e., performing . critic updates per outer iteration), followed by the standard EG actor updates. All other training configurations follow Section
Results. Table 10 ropports results the 70% Suboptimal Replay Forward dataset. Applying EG to the critic yields no meaningful improvement over actor-only EG, and slightly degrades perfor- mance for M 4.
Discussion. Although extending EG to the critic is conceptually possible, the results in Table I0 show that does not bring additional return gains over actor-only variant our setting. More- over, this extension comes with clear practical drawback: the critic must be optimized separately within each group. leading to a substantial increase wall-clock training time. EG mainly intro- duced to mitigate policy-gradient confficts, and actor-only EG already provides strong performance improvements. Given the limited benefit and the extra cost of critic-side grouping, we apply EG only to the actor in the main method.
I HYPERPARAMETERS AND TRAINING DETAILS
This appendix summarizes the hyperparameters and training procedures used for reproducibility. Unless otherwise noted, values are shared across all robots.
1.1 IQL, BC, AND TD3+BC HYPERPARAMETERS FOR TRAINING
Hyperparameters for IQL, BC, and TD3+BC are listed below.
Table 11: IQL, BC, and TD3+BC hyperparameters (common)
Parameter
IQL
BC
TD3+BC
Batch size per robot Learning rate
1024
1024
1024
3e-4
3e-4
3e-4
Offline updates
le5
le5
le5
Max grad norm
0.5
0.5
0.5
Discount factor
0.99
0.99
Value expectile Texp
0.7
Policy temperature (AWAC) 3
3.0
Target network EMA Trarget
0.005
Policy update frequency Spoticy
2
Policy noise std policy
0.2
Policy noise clip Cpolicy
0.5
Behavior cloning weight a
2.5
Max action amax
5.0
I.2 PPO HYPERPARAMETERS FOR DATASET GENERATION
Representative PPO hyperparameters used for dataset collection (consistent with Appendix B).
21

---

Published as a conference paper at ICLR 2026
Table 12: PPO hyperparameters (dataset collection)
Parameter
Value
Total batch size / update
522240
(48 envs x 10880 steps)
Minibatch size
32640
SGD epochs per update
10
Learning rate (init + final)
0.0004 -> 0.0 (linear anneal over 100M steps)
Entropy coefficient
0.0 or 0.1
Discount factor y
0.99
GAE A
0.9
Clip range (PPO e)
0.1
Max gradient norm
5.0
Parallel environments
48
22


Reviews:
[{"review_id": "mQEDqFxYiw", "reviewer": "Reviewer_PQSJ", "summary": "This paper presents Embodiment Grouping (EG), a morphology-aware approach for cross-embodiment offline RL. The authors observe that gradient conflicts arise when training shared policies across diverse morphologies. They measure embodiment similarity via the Fused Gromov–Wasserstein (FGW) distance, which jointly considers kinematic structure and behavior embeddings, and use this to cluster robots into morphology-aligned groups for group-wise policy training. The result: better policy transfer, stability, and data efficiency across heterogeneous robot datasets.", "questions": "1.\tHow does EG perform as the number of embodiments scales? Does the FGW computation remain tractable?\n\n2.\tCould group-wise contrastive representation learning replace fixed clustering?\n\n3.\tIs there an optimal number of groups, or does performance plateau?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1.\tThe paper fills an important gap between robot foundation models and morphology-aware generalization. Prior work (e.g., URMA) has explored transfer across morphologies but typically ignores the destructive gradient interference that arises when training across incompatible embodiments. EG’s use of FGW distance to quantify morphology-level relationships and structure training groups is an elegant and novel solution.\n\n2.\tThe experimental setup is extensive and thoughtfully designed. Using 16 diverse morphologies (quadrupeds, bipeds, manipulators) provides a broad basis for claims. The analysis linking gradient cosine similarity and policy degradation offers convincing causal evidence. Ablation studies on group count and morphological embedding confirm robustness of results.\n\n3.\tThe paper explains why shared gradient updates can harm policies across dissimilar bodies, which is a phenomenon often observed but seldom quantified. Their visualizations of inter-group gradient alignments are insightful and elevate the reader’s understanding of multi-embodiment interference.", "weaknesses": "1.\tWhile FGW distance effectively captures morphological similarity, there is no formal argument linking FGW similarity to gradient alignment or loss landscape smoothness. Without such a bridge, the theoretical contribution remains descriptive rather than predictive.\n\n2.\tEmbodiment relationships evolve as policies adapt. Fixed grouping may lead to stale partitions that no longer reflect actual learning dynamics.\n\n3.\tThe morphological embeddings used to compute FGW distance come from URMA, which itself encodes behavioral information. This could inflate grouping performance.\n\n4.\tThe experiments focus exclusively on locomotion. It’s unclear if EG transfers to manipulation tasks, where control topology differs drastically.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QhhqI6aCzv", "reviewer": "Reviewer_PZLb", "summary": "The paper studies using cross-embodiment with offline RL on locomotion tasks. It finds that offline RL is more valuable when offline dataset is more sub-optimal. Furthermore, it notices that cross-embodiment transfer of some robots have negative impacts, owing to large morpholy gap and conflict gradients. The paper then proposes valuable method to group different groups to prevent conflict gradients. Experiment show that the proposed method has strong improvement over baseline offline RL, comparing to other multi-task gradient techniques.", "questions": "1. Can the results generalize to other imitaiton learning / offline RL algorithms.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The setting of cross-embodiment offline RL is novel in the community.\n2. The proposed method is easy to understand and is effective in practice.\n3. The comparison involve gradient projection techniques used in continual learning / multi-task literature,\n3. The evaluation of different embodiment is comprehensive.", "weaknesses": "1. Tasks only contain locomotion on walking. A comprehensive analysis / benchmark should involve more diverse tasks.\n2. The baseline offline RL only contains IQL. More algorithms like CQL e.t.c should be evaluated. It is unclear if the claim can be extended.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CfgJw5NHuA", "reviewer": "Reviewer_gRNY", "summary": "This study investigates how to apply offline reinforcement learning (Offline RL) on cross-embodiment robotic datasets containing a large amount of suboptimal data. This study finds that when robot embodiments are diverse, gradient conflicts across different types of robots , hinder learning and lead to negative transfer. To address this issue, this study proposes an embodiment grouping (EG) strategy based on morphological similarity, which clusters morphologically similar robots into groups and updates the policy sequentially per group, effectively mitigating gradient conflicts. Experiments demonstrate that this method achieves significant performance improvements on datasets dominated by suboptimal data, with an average gain as high as 39.8%.", "questions": "1.Robot locomotion can be considered to have three degrees of freedom: forward/backward, left/right, and turning left/right. This paper only addresses tasks involving the forward/backward degree of freedom. Could the authors provide results trained on all three degrees of freedom?\n\n2.In Algorithm 1, the grouping strategy is applied only to the actor and not to the critic. What is the rationale behind this design choice?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1.The experimental results are strong, with experiments conducted on as many as 16 different types of robots. The proposed method shows a clear improvement over the baseline.\n\n2.The paper employs a clear validation approach to demonstrate the impact of gradient conflicts caused by cross-embodiment data.", "weaknesses": "1.The work lacks real-robot experiments. All experiments are conducted in simulated environments, with no validation on physical robots.\n\n2.Certain acronym definitions appear after their first use in the paper; for example, “EG” appears in Table 1 before being formally introduced, which affects readability.\n\n3.The implementation only evaluates forward and backward motions across different robots, lacking validation on a broader range of tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EvvGXoICHd", "reviewer": "Reviewer_umLy", "summary": "The paper proposed cross-embodiment benchmark tasks and dataset construction paradigms for evaluating offline RL algorithms. Through empirical studies, the paper identifies both positive and negative knowledge transfer between robots with different morphology when training on suboptimal datasets. The author then hypothesized that the negative knowledge transfer happened because of morphological dissimilarity between groups of robots. Quantitatively, this can be observed from the gradient conflicts of the robot-dependent actor losses. To mitigate this issue, the paper proposed a graph-based grouping strategy using the morphology of different robots, showing better cross-embodiment offline RL performances.", "questions": "- Sec. 3.1: What’s the purpose of the reward function? Or what’s the task of each MDP? Are different robotics solving the same task?\n- Sec. 3.3: The explanation of the action encoder is not clear. Do different robots share the same action support? Although they share the same action space, some action dimensions might be redundant for some robots.\n- line 174: What are “Forward” and “Backward” variants of the dataset?\n- Sec. 4.1: The meaning of each column in Table 1 is not clear from the text in Sec 4.1. The introductions of each baseline are deferred into Sec 6, which makes the columns in Table 1 confusing.\n- Sec. 4.2 and Sec. 4.3: If I understand correctly, the single robot variant does not have a pre-training stage. Is it a fair comparison between an algorithm with cross-embodiment pre-training and task-specific fine-tuning with an algorithm only involving task-specific training? Intuitively, the number of gradient updates are different.\n- The experiment results for the texts between line 311 - line 318 are missing.\n- Is there a quantitative relationship or a metric between the embodiment distance (Fig 3 (a)) and the gradient cosine similarity (Fig 3 (b))?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- Training offline RL algorithms on cross-embodiment datasets is an important step towards scalable robotic foundation models. The paper proposed a cross-embodiment dataset collection paradigm to evaluate offline RL algorithms.\n\n- Through empirical studies, the paper relates the negative knowledge in cross-embodiment learning to the gradient conflicts of robot-dependent actor losses.\n\n- The paper then proposed a graph-based grouping strategy using the morphology similarity of different robots to mitigate this issue.", "weaknesses": "- Although the paper discussed a set of optimal and suboptimal datasets, the set of rewards / tasks are somewhat limited. Specifically, the paper only considers simple locomotion rewards (following velocities) for different robots.\n\n- One of the motivations of the paper is to compare the performance of offline RL algorithms trained on cross-embodiment datasets. However, the paper primarily focuses on the IQL algorithm and its variants, which limit the conclusions from experiments.\n\n- Since the paper focuses on empirical studies, including the dataset and open-source code in the submission would strengthen the conclusions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)