Meta-Controller: Few-Shot Imitation of Unseen Embodiments and Tasks in Continuous Control
Authors: Seongwoong Cho
Donggyun Kim
Jinwoo Lee
Seunghoon Hong
Venue: NeurIPS 2024 poster (2024)
Keywords: few-shot learning
imitation learning
behavior cloning
transformers
meta-learning
Primary area: reinforcement_learning

Abstract: Generalizing across robot embodiments and tasks is crucial for adaptive robotic systems. Modular policy learning approaches adapt to new embodiments but are limited to specific tasks, while few-shot imitation learning (IL) approaches often focus on a single embodiment.
In this paper, we introduce a few-shot behavior cloning framework to simultaneously generalize to unseen embodiments and tasks using a few (e.g., five) reward-free demonstrations. Our framework leverages a joint-level input-output representation to unify the state and action spaces of heterogeneous embodiments and employs a novel structure-motion state encoder that is parameterized to capture both shared knowledge across all embodiments and embodiment-specific knowledge. A matching-based policy network then predicts actions from a few demonstrations, producing an adaptive policy that is robust to over-fitting. Evaluated in the DeepMind Control suite, our framework termed Meta-Controller demonstrates superior few-shot generalization to unseen embodiments and tasks over modular policy learning and few-shot IL approaches.

Meta-Controller Few-Shot Imitation of Unseen Embodiments and Tasks in Continuous Control
Seongwoong Cho' Donggyun Kim*
Jinwoo Lee
Seunghoon Hong
School of Computing, KAIST
{seongwoongjo, kdgyun425, bestgenius10, seunghoon seunghoon. hong:hkaist..acc......oo
Abstract
Generalizing across robot embodiments and tasks is crucial for adaptive robotic systems. Modular policy learning approaches adapt new embodiments but are limited to specific tasks, while few-shot imitation learning (II.) approaches often focus single embodiment. In this paper, introduce few-shot behavior cloning framework to simultaneously generalize to unseen embodiments and tasks using a few (e.g., five) reward-free demonstrations. Our framework leverages a joint-level input-output representation to unify the state and action spaces of heterogeneous embodiments and employs a novel structure-motion state encoder that is parameterized to capture both shared knowledge across all embodiments and seoodddmnnt-ppeiifie knowledge. A matching- based policy network then predicts actions from a few demonstrations, producing an adaptive policy that is robust to over-fitting. Evaluated in the DeepMind Control suite, our framework termed Meta- Controller demonstrates superior few-shot generalization to unseen embodiments and tasks over modular policy learning and few-shot II. approaches. Codes are available at
Introduction
Generalizing across different robot embodiments and tasks with only a few demonstrations is a fundamental challenge in continuous control |6 [1 40 29 developing versatile and adaptive robotic systems that can operate effectively in diverse and dynamic environments. The high diversity of embodiments and tasks, however, makes this particularly challenging. Robot embodiments vary widely in their morphological (e.g.. number and connectivity of joints) and dynamic (e.g., motor gear ratios, damping coefficients) configurations, complicating the design unified architecture capable of handling heterogeneous input states and output actions. Furthermore, the variety of tasks-such as locomotion, object manipulation, and the learning of transferable skills that can efficiently generalize across different tasks.
33 [4]. This capability is crucial for
Despite significant advancements in reinforcement learning (RL) and imitation learning (IL), achiev- ing simultaneous generalization across diverse embodiments and tasks with a few demonstrations remains largely underexplored. Modular policy learning approaches [34] 88, 16 12 [0] have shown promise by learning modular policies that can be shared across embodiments with different morpholo- gies. However, these methods primarily focus specific task types such as locomotion, and lack the flexibility to adapt to a wide range of control tasks, limiting their broader applicability. Conversely, few-shot IL approaches 8 39 3 37 36) aim to learn novel tasks from few demonstrations. These techniques excel in scenarios with sparse training data but typically concentrate on a single embodiment with fixed morphological and dynamic structures, restricting their ability to generalize across various embodiments and tasks. As a result, these two fields have developed independently, and their integration remains an open area of research.
To address these challenges, we propose a novel framework that flexibly learns arbitrary control tasks on unseen embodiments using a few (e.g. five) reward-free demonstrations. To handle hetero-
Equal contribution
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

geneous embodiments within unified architecture, we tokenize states and actions into joint-level representations, since joints serve as the fundamental building blocks of robots and provide modular representation for compositional generalization to unseen embodiments [12] [0] Given this unified I/O, we employ a state encoder to capture both embodiment-specifee knowledge about morphology and dynamics and shared knowledge about the physics governing the environment. Then we design matching- based policy network that predicts actions from the encoded states, conditioned on a few demonstrations. Our model is trained using episodic meta-learning on dataset comprising various embodiments and tasks, followed by few-shot behavior cloning on unseen embodiments and tasks using a few reward-fire free demonstrations.
Our key contributions are as follows: (1) We design a novel structure motion encoder that operates on joint-level state representations, efficiently disentangling gebbodiment-speciffe and task-specific knowledge. (2) We propose a matching-
meta-learning framework that efficiently transfers knowledge of local motions to quickly learn unseen tasks with a few demonstrations. (3) We evaluate our framework in various environments within the DeepMind Control suite, encompassing diverse embodiments and tasks, demonstrating superior few-shot learning performance over existing baselines.
2 Problem Setup
reinforcement learning (RL) problem involves an agent interacting with an environment, typically modeled as a Markov Decision Process (MDP). An MDP is represented by the tuple AS P. R), where S is the state space, A is the action space, P : S x A + R is the transition probability, and R: : S x A + R is the reward function. In conventional RL, agent learns policy π : S + A that maximizes expected cumulative rewards [ )t)], where 7 E [0, 1] is discount factor. Training such an RL agent typically requires numerous interactions with the environment and carefully designed reward function, making it burdensome to learn new tasks. Behavior cloning (BC) addresses these challenges by using supervised learning techniques to imitate expert policy from offline demonstrations. We focus on the few-shot BC setting, where the training data consists of a few reward-free demonstrations D (ri jig. with each demonstration T; {(st, hs) being temporal sequence of states and actions performed by an expert model.
In this paper, consider continuous control problems of multi-joint robots that involve various embodiments and tasks. An embodiment & refers to the physical configuration of robots, which includes (I) the morphology, i.e., the shape, size, and arrangement of the components such as limbs, joints, and sensors, and (2) the dynamics parameters that affect the robot's 's behavior, e.g., motion inertia, mass, gear ratios, and damping. In terms of MDP, different embodiments
have different dimensionality of state and action spaces, e.g., dim (Se, ) * dim (Sg,) ) and (Am(AA.) ) & dim(Ag,) ) for E, * Ej. as well distinct transition probabilities P& that determine the kinematics of the robot. A task T is defined by a specific goal or objective that the robot must achieve within its environment, characterized by a reward function Ry. Tasks can vary widely, ranging from locomotion and manipulation to complex interactions with dynamic environments. The combination of different embodiments & and tasks T creates a broad class of continuous control problems.
Our objective is to achieve simultaneous generalization to unseen embodiments and tasks of contin- uous control with a few-shot behavior cloning framework. In other words, the model has to learn a policy for a novel continuous control problem from only a few demonstrations D, where both embodiment& E and task T can be arbitrary and previously unseen.
2.1 Challenges and Desiderata
Despite achieving the simultancous few-shot generalization to unseen embodiments and tasks is crucial for developing versatile and adaptive robotic systems, this problem remains less explored. We characterize two distinct challenges and desiderata to address each challenge.
Handling Heterogeneous Embodiments. To generalize to arbitrary embodiments in continuous control, the model must possess an architecture capable of universally handling heterogeneous states and actions of various embodiments. This necessitates unified input/output (1/O) representation for states and actions, allowing for the sharing of structural characteristics (e.g., dimensionality) and semantics (c.g., input attributes output control types) across different embodiments. Additionally,

---

Joint-Level 1/O Representation
St
per-joint state token
angle
velocity
axis
per-joint action token
command
(torque or force)
Action
S<t
State Encoder f
Actustor
Time
Shared
State Encoder
aLT
22T
State Encoder
Matching- based Policy Network π
azr
State Encoder
a
Few-shot Demonstrations
fTilieN [(sgg algT)ssx
Figure 1: The overall framework of Meta-Contues Controonnrolllr. First, the states and actions of various robot embodiments are tokenized into joint-level representations. The state tokens are then encoded by the state encoder to capture knowledge about the embodiments. Finally, matching -based policy network uses few-shot demonstrations with the encoded state features to predict per-joint actions.
the encoder to extract state features should be able to capture transferable knowledge across different embodiments as well as embodiment-specifii knowledge flexibly adapt to distinct morphologies and dynamics of each embodiment.
Few-shot Policy Adaptation. To achieve robust few-shot learning unseen tasks, efficient and flexible policy adaptation mechanism is essential. Given the diverse behaviors required by continuous control problems across various embodiments and tasks, the policy network must learn transferable skills shared by different tasks. Simultaneously, the model must dynamically adapt its policy by inferring the task based on few demonstrations, ensuring it captures the underlying structure of previously unseen tasks. Additionally, the adaptation mechanism must be robust to overfitting.
3 Method
In this section, we introduce Ceta-Controller, few-shot behavior cloning framework for simultane- ous generalization embodiments and tasks of continuous control. Figure I illustrates the overall framework.
addresses heterogeneous embodiments by tokenizing the states and actions into joint-level I/O (Section 3.I) and employing a state encoder that captures knowledge about the structure and dynamics of the embodiment (Section 3.2.1p. Then, matching-basee based policy network (Section 3.2.2) predicts the action by leveraging a few given demonstrations. The training protocol of ControCorr consists of episodic weta-learning and few-shot fine-tuning (Section 3.3)
3.1 Joint-Level I/O Representation
To unify the state and action spaces of different embodiments, adopt joint-level tokenization. Joints are fundamental components of robots, and their primary source of action is the torque or force generated by actuators attached to each joint. This allows standardize the states and actions of a robotic agent into per-joint observations and control commands. Consequently, joint-level states and actions provide natural modular representation, facilitating the compositional generalization of various robot embodiments?
We consider pre-detined set of joints whose compositions differ per embodiment.

---

Joint-level state features mi
f. Stectiee Enooler
fm
Innader
DC
posiiional embeaddngg
fm
m
0
Enilodinesii specille porameter.
Endodinnni and babasperite panmettex
di-directional Transformer
Causal Transformer
Pro
Endediinnu Ausosti pollioaal cubbedling
7
D
P
joint
Joint-level state tokens S.çt
timestep
The state encoder f consists of two component transformers. Joint-level state tokens first encoded by the structure encoder fe along the joint axis, where the positional embedding and part of backbone parameters adapt the model to each embodiment. The features are then passed to the motion encoder fm. which computes causal attentions of per-joint features along the temporal axis, where part of backbone parameters adapt the model both to the embodiment and task.
Figure 2:
For any given embodiment E, we represent the corresponding states 8, and actions a, as arrays of joint-level tokens follows:
= [Sjrrlgsee E Rexd a. [ag,1 E Re
where Jg is the number of joints in and t indicates the time. The state token sj. E IR represents per- joint information, such as position, velocity, movement axis, and motion types (i.e. linear angular). The action token aj.r E [-1, 1] represents the control command of j-th joint, where we assign zero value for free joints. This joint-level representation ensures consistency across heterogeneous robotic embodiments, enabling unified learning of different continuous control problems.
3.2 Meta-Controller
Given the tokenized representation of joint states and actions, we aim build an adaptive controller for arbitrary robot embodiments to perform arbitrary continuous control tasks based on set of few demonstrations D (r' Jss... To address the challenge discussed in Section 2.1 we employ an encoder f to extract the embodiment-awwer aware state features and an adaptive policy network that efficiently leverages the demonstrations to predict the actions.
3.2.1
State Encoder for Embodiment Generalization
The state encoder f encodes the tokens of the current state with history sig into state features m,
m, f(Sg( 0).
To effectively encode the states of each embodiment, decompose state encoder into two components: a structure encoder fa that captures morphological knowledge, and a motion encoder fm that captures dynamics knowledge.
Structure Encoder. The structure encoder models the relationships among the joints within each embodiment. As shown in Figure 2 we use bi-directional transformer on the joint-level state tokens 8; at each timestep / to extract the structure features Z,:
2, = fa(St + p 0E).
where the embodiment-speccfie positional embedding O is added to the input tokens. Note that we decompose the parameters of fa into adaptive parameters (pf.05 that capture embodiment-speccfic knowledge and shared parameters 0₂ that captures common knowledge across embodiments. The

---

Current State SCc
Fev-shot Demonstrations
Predicted Action
g Artinntaesda:
jumn ④
17
Aation Denodar
Podttond taionding
tinedeeg
Demonstration action features
Joint-level action tokens &
ayy
9
V
h
fv
Cansal Transformer
Causal Transformer
m,j.
j.t
Rjuce
Ps
Da
FIL
Joint-wise State Similarity
Weighted Sum of Actions
Joint level action tokens age
matching output vlc
(a) Matching- Policy Network
(b) Action Encoder and Decoder
Figure 3: An illustration of the matching-based policy network (a) Each state and action token in few-shot demonstrations is encoded by the corresponding encoders f and g. where we use the same encoder used for the current state. A matching module then computes the weighted sum of action features based on the joint-wise similarity between state features. Finally, an action decoder decodes the joint-wise matching output predict the current action. (b) Both the action encoder and decoder are causal transformers operating along the temporal axis of action tokens and features.
positional embedding p is crucial for adapting to local configurations (e.g., length, movement range) of joints in E not explicitly given in the state st, Global configurations (e.g., control timestep) shared by all joints are handled through the embodiment-specifi, parameters 6e in the transformer backbone. Shared parameters 0g capture common knowledge, such the physics governing the environment.
To enable efficient yet flexible adaptation during few-shot behavior cloning, we designate only small portion of the backhone parameters to be embodiment-specinee Inspired by parameter-effiiient fine-tuning (PEFT) approaches that effectively modulate transformers with only few parameters, we employ bias parameters (2), low-rank projection matrices [17). and also layer-scale parameters for 05. As explained Section 3.3 only the adaptive parameters (p.- 0²) are updated during few-shot learning unseen embodiments, ensuring robustness to overfitting the few demonstrations.
Motion Encoder. While the state encoder fa encodes structural information about the embodiments, it does not model the temporal dynamics of states which is crucial for understanding continuous control tasks. Therefore, we introduce motion encoder fm, which a causal transformer that encodes the state features along the temporal axis. As illustrated in Figure 2 fm rearranges the encoded structure features nęt into separate temporal sequences of joint-level features 2j,tt, then produce the motion features for each joint m,,, auto-regressively:
mj. fm (z,,5t Pm; By, (0.00/..7 ((C,T)). Vj Je
where Pm denotes the positional embedding for specifying the timesteps t. Additionally, we introduce
small portion of adaptive parameters ((E.T) in the causal transformer backbone, which both specific to embodiment & and task T. These parameters help the model understand the unique motions in the few-shot demonstrations that are specific each task and embodiment. We use the same PEFT techniques employed in the structure encoder for the adaptive parameters.
3.2.2 Few-shot Policy Adaptation for Task Generalization
To learn unseen tasks T, we design adaptive policy network with matching framework 20 that incorporates the demonstrations D = {( at agr))ss to produce an action å
à (m Ø))
(5)
Figure illustrates the architecture of the policy network, which consists of an action encoder g. matching module a,, and an action decoder h.
5

---

Matching- based Policy Network. To incorporate the demonstrations, we encode the state and action tokens in the demonstrations using the state encoder f introduced Section 3.2. and an additional action encoder y, respectively.
mi.r = f(5).ge; j,t = |(a]__(s + Pgi Pgi
Vj S Jg, At <T, Vi N, vj S Je. A1 - T, vi
(6)
where we employ a causal transformer for g with temporal position embedding Py- Then н incorpo- rates the motion features m, of the current state (Eq. (2) and the encoded demonstration features (Eq. (6) and (7)) via joint-wise matching as follows:
Σ Σ PSTISN
(mm. (m m 1 ja Vj S Jg,
(8)
where o is similarity function (e.g., cosine similarity). Finally, we employ an action decoder h, causal transformer that decodes the joint-wise matching output VJ, into j-th action token á
á (V Φn),
Vj Jg-
The matching- matee policy network offers significant benefits for few-shot behavior cloning, par- ticularly when dealing with unseen tasks and embodiments. Its robust adaptation capabilities stem from effectively incorporating demonstration data through similarity function, which dynami- cally matches current state features with those from demonstrations. This non-parametric approach minimizes overfitting, enhancing generalization from limited examples.
Hierarchical IL Interpretation. Eq. (8 can also be interpreted as hierarchical imitation learning that generalizes to unseen tasks using a transferable skill set. Since the action encoder extracts pool of temporal action features that are composed to produce the current action a, in the feature space. treat the action features v local motor skills, i.e., building blocks of joint behavior for various control tasks. This modular approach lets the network recombine these skills to efficiently tackle new challenges. By assigning high similarity scores to relevant demonstrations, the policy network ensures accurate imitation of expert behavior, improving performance on novel tasks.
3.3 Training & Inference
The training protocol Meta-Controller consists of two stages: episodic meta-learning and few-shot fine-tuning, where we train whole parameters (0, Φ) of the model during the first stage while we train only the adaptive parameters (p. 6 ((f.TT during the second stage.
Episodic Meta-Learning. During episodic meta-learning, the model acquires general knowledge of continuous control through number of episodes that mimic few-shot behavior cloning scenarios ensure effective adaptation to unseen embodiments and tasks. To this end, we leverage a meta-training dataset that consists of demonstrations B(E.T) by expert agents with various embodiments & and tasks T. At each episode, first sample continuous control problem (E, T), then sample two subsets of ((E,T): a set of support data Dz and query data Dq. Then the model trained to imitate query data using the demonstrations.
árl? |].
min BX(ETT) (0,0)
EDS,
Σ
||a
(10)
(Dal
(s..a,)e
where å, is produced by Eq. problems within the dataset.
5 using Ds, and p(E,T) is a uniform distribution over all control
Few-shot Fine-Tuning. apply our unseen embodiments and tasks with a few demonstrations D. To this end, we randomly split D into two disjoint subsets, and fine-tune the model with Eq. (10) but with only respect to the embodiment- specific and task-speeific parameters (pf, 0 ((E.T)) while freezing the rest. After fine-tuning, the model uses the whole demonstrations D in the policy network to produce the actions at evaluation.
After acquiring the meta-knowledge about continuous control problems, in few-shot behavior cloning setup, where it should adapt to both

---

4 Related Work
Modular Policy Learning. Modular policy learning aims to develop modular policies for multi-joint robots that are adaptable to various morphologies. NerveNet 34 uses a Graph Neural Network (GNN) to model structural relationships between joint-level features. SWAT [66 leverages graph features like the normalized graph Laplacian for improved structural learning in reinforcement learning (RL). Amorpheous [23] and MetaMorph [1] use transformer architectures, treating the morphological graph fully connected to exploit transformers' capabilities. These works focus on RL and zero-shot generalization to specific locomotion tasks. Furuta et al. [TO] proposes an imitation learning framework with benchmark environment for extensive morphologies, showing promising zero-shot generalization but limited to specific tasks like reaching a goal position, and not handling general continuous control tasks.
Few-shot Imitation Learning, Few-shot Imitation Learning (IL) approaches focus on generalizing novel RL tasks with only a few demonstrations. Gradient-based meta-learning algorithms like MAML 8 and Bayesian MAML [39] have been explored for rapid task adaptation. Duan et al. [ addressed one-shot imitation learning by incorporating an attention model over the query state and demonstrations. FIST [13] is a hierarchical skill transition model that learns to extract transferable high-level skill sets from demonstrations and leverages this skill knowledge during adaptation. Recently, PDT [37] introduced a transformer-bee bbssed few-shot learner, using the few-shot demonstration a prompt token. Similarly, HDT (36) trains a hyper-network generate adapter parameters for pre-trained decision transformer, adapting to few-shot demonstrations. Despite notable generalization abilities, these few-shot IL. approaches have only been studied within single embodiment, limiting their applicability to real-world scenarios with heterogeneous embodiments.
5 Experiments
5.1 Experimental Setup
Environment and Dataset. We evaluate the few-shot behavior cloning of unseen embodiments and tasks within the DeepMind Control (DMC) suite (31, which includes continuous control tasks featuring diverse kinematic structures. We select 30 tasks from 10 embodiments as training tasks and 8 tasks from 4 embodiments as held-out evaluation tasks. The evaluation tasks include three unseen embodiments (hopper, reacher- fothee--folle -our,, wolf) and one seen embodiment (walker), with the wolf being an additional embodiment introduced in [3.] Our meta-training dataset is constructed using a replay buffer of an expert agent [38], consisting of up to 2000 demonstration trajectories for each task and embodiment, Each demonstration consists of state-action pairs over T 500 timesteps, with rewards discarded for both episodic meta-learning and few-shot behavior cloning. For N-shot few-shot behavior cloning, we use the last N demonstrations from the buffer. A full detail of the dataset is included in Appendix B.I
Evaluation Protocol. We evaluate all models with 20 different initial states and report the mean and standard error. For evaluation metric, we use normalized score [9] calculated by Boree-radom score
experi scure- random score* where each score represents the average cumulative rewards during the evalua- tion. The random score is obtained by evaluating random agent using uniform distribution policy over the action spaces. We evaluate the models every 1000 iterations of few-shot behavior cloning and report the best score. For models that use task-specific low-rank projection matrices, we search the rank parameter over (4, 8, 16} and report the best one. Throughout the experiments, we present 5-shot learning results (N = 5) unless otherwise specified.
Baselines. We compare our model with various Decision Transformer 4 (Ds)- few-shot imitation learning approaches and two transformer-bast based modular policy learning approaches. In the ba-based models, we exclude return-to-go tokens from the input tokens simulate behavior cloning. Since the DT architecture is not inherently designed to handle heterogeneous state and action spaces, we modify its input and output linear heads in an embodiment-speer specific manner and fine-tune mune them for unseen embodiments.
DTT-baeed Models. From-Scratch Decision Transformer (FS-DT) is a decision transformer that trains a downstream task directly from randomly initialized weights. Multi-Task Decision Trans- former (MT-DT) is variant of the decision transformer that trains multiple tasks with task-specific
7

---

Table 1: 5-shot behavior cloning results DeepMind Control (DMC) suite.
Unseen Emb.
Socn Emm
Emb. (E)
hopper hop buid.
xolf
reacher- four
xalker
Avg.
Task (7)
hop
stand
valk
easy
hard
walk bud.
FS (
56.916.22
9.8109
44.71.10.8
3588172
00.71.422
5.514,9
3.2
64.311.0
10.111111
53.41188
46.21:0222
1022174
8.415.1
5.04
PDT 37
0.9
29.749..
6.5419
47.0=1144
35.748.3
3.34-77
5.8
5.0+1.5
16.7
PDT+PEFT
1.3
61.747777
12.34.00
549484
52.74:6.4
-1,743.3
9.14:59
74.1
33.2
HDT 66
0.840.4
51.6479
1377177
56.4111.
38.749.11
124553
8.5
3.3::1.0
21.8
L2M
4.5
45.1
4.0
10.6=10.7
65.943.88
3,746.
13.547.
40.0:19.8
28.7
MetaMorph
12
33,44:49
81.8422
54.1#4.7
73.04:4.
29.74446
-4112266
3.8:33.1
31.8+8.6
37.9
MTGv2
21.6.13.3
83.1122
40.716.3
68.2.17.3
29.11443
-1412.7
4,6.14.8
35.5
35.2
49.116.1
87.2.166
82.5.199
91.7.5.11
67.313.11
56.11888
50.81
84.31.5.7
parameters and fine-tunes only these parameters for few-shot adaptation. For the task-specific parameters, we use the same parameters as E used in our motion encoder. Prompt- based Decision Transformer (PDT) (37] adapts its policy by conditioning on the few-shot demonstrations through prompting. We also report the performance of a variant of PDT that fine-tunes task-specifie parameters (PDT+PEFT), similar to MT-DT. Hyper Decision Transformer (HDT) [36] employs a hyper-network conditioned on few-shot demonstrations generate parameters of the Adapter [25] module applied to DT. Learning tuning (PEFT) techniques to DT architecture. While L2M ís not directly proposed for few-shot imitation learning, we include this baseline since uses similar PEFT technique as ours.
Modulate (L2M) (8] incorporates parameter-efficien fine-
Modular Policy-based Models. We include two modular policy learning approaches, Meta- Morph [1) and MTG which utilize transformer architecture to encode joint-level states. Originally designed for zero-shot learning of locomotion tasks, we adapt these approaches by incorporating task-specific linear heads and fine-tuning them on few-shot demonstrations. Both models are trained using the behavior cloning (BC) objective as described [00
Implementation Details, We implement both the structure encoder and the motion encoder using a 6-layer transformer with attention heads and hidden dimension of 512. Due to the quadratic computation cost of the transformer, we set the maximum history size of causal attention layers in the encoders to 10. The baseline models use the same transformer backbone, All models are trained for 200,000 iterations on the meta-training dataset and fine-tuned for 10,000 iterations on the few-shot demonstrations. For downstream embodiments that are structurally similar to training task (e.g., reacher-thee three and reacher-four using the trained parameters during line-tuning. More details are included in Appendix B.2
initialize the encoder's 's embodiment-specifi parameters
5.2 Main Results
Table shows the 5-shot behavior cloning results of the models on both unseen and seen embodiments. We observe that Ceta--ontrolllr consistently outperforms all baselines across various continuous control problems, demonstrating its effectiveness. Existing few-shot imitation learning approaches that lack embodiment generalization mechanism, such as PDT and HDT, struggle to adapt to unseen embodiments, often performing comparably to the naive from-scratch baseline (DT-FS). In contrast, modular policy learning approaches like MetaMorph and MTGv2 show better adaptation to unseen embodiments (e.g., hopper). However, their performance is still inferior to Meta- Ceta--ontrolllrr, which can be attributed to the absence of few-shot adaptation mechanism for unseen tasks.
Notably, our model significantly outperforms all baselines in the challenging reacher-foor embod- iment, where models must understand the notion of a goal position given only five demonstrations. Figure44showe shows that while baseline models converge the pose the demonstrations regardless of the goal position, our model successfully reaches the goal position and converges with unique pose. We attribute this success to the modular nature of our model. Our state encoder effectively shares knowledge about joint-level motions thanks to its parametrization, and the aatching-based policy network flexibly exploits the local motor skills from a few demonstrations, We provide more results and analysis in Appendix
8

---

5-shot Demonstrations
Expert
DT-FS
MT-DT
PDT
PDF+PEFT
Ours
HDT
L2M
MetaMorph
MTGv2
Figure 4: Qualitative comparison on the hard task of the reacher-four embodiment, visualizing the final states of the demonstrations and the rollout trajectories of each model. In this task, the robot must move its limb tip to the goal position (visualized as a red ball). While most of the baselines converge to one of the poses in the demonstrations and ignore the goal position, our model accurately solves the task with distinct pose from the demonstrations.
Ours
Metpaorph
PDT+PEPT
hoppec,
hopper, hop bwd.
hopper, stand
walker, walk bwl.-bwdd
Average
Score
70
Number Thems.
20
Namberr Demo
2)
Number of Demo
Numbee l8 Domo. x
50
wolf, walk
wolf, run
reaches- four, easy
reacber four,
30
Saom
ScoRt
Spoe
10
Number of Demu.
Number Demo
Number of Damn.
Number of Domos
Number df Demo.
Figure 5: Ablation study the number of demonstrations. We plot the normalized scores for each pair of embodiment and task (E, T) and their average, varying the number of shots as 5, 10, and 20.
5.3 Ablation Studies
Ablation on the Architecture. To verify the effectiveness of each architectural component introduced in Section we conduct an ablation study by progressively replacing each module the structure encoder fsr the motion encoder fm. and the matching module o o-with linear layer. To isolate the effect of adaptation parameters, we replace f. with task task-specific linear layers and fn with embodiment- and task-specife linear layers. Table summarizes the results. We observe that the structure encoder fa crucial for generalization drastically when it is removed (row 1 vs. row 3). This indicates that the structure encoder captures modular knowledge about various morphologies, transferrable unseen embodiments. Combined with the structure encoder, the motion encoder further improves performance (row 2 vs. row 3), especially on the seen embodiment (walker). This shows that modeling the temporal relationships of joints is beneficial when the model understands the embodiment. Finally, the matching module consistently improves performance (row 3 vs. row 4), particularly on challenging tasks such as the hop task of hopper and the tasks of reacher-four This demonstrates the effectiveness of the matching architecture preventing overfitting with few demonstrations. We provide more ablation studies architectural components in Appendix D.I
unseen embodiments, as performance drops
Ablation on the Parametrization. To analyze the impact of the embodiment-specinn and task- specific parametrization introduced in the state encoder, we conduct ablation study by removing the adaptive parameters of 0 in the structure encoder and (!E.T) the motion encoder. In this study, the matching module is used in all models, where we provide additional results without using it in Appendix [D.2] The results show that the model without adaptive parameters in the structure encoder (row 1) performs well in many tasks, likely due to the universality of joint-level input/output representation, which allows for compositional generalization. However, adding embodimeen-speecfiic parameters consistently improves performance across all tasks, indicating the benefit of capturing embodiment- specific knowledge. The model without adaptive parameters the motion encoder (row
2) remains competitive with the model including them (row 3) in many tasks but fails in the hop task

---

Table 2: Ablation study the architectural components.
Unseon Emb.
Soen Emb
fm
hopper
wolf
reacher-foon four
walker
Avg.
hop
bup.bud.
stand
walk
easy
hard
salk-bod.
3.313.88
46.7119.3
14.915.9
53.5180
3.2.1.66
4.31.5.1
60.6.110.5
13.115.6
88.1117
75.0.5.77
4.127.2
48.11.5.4
22.84:5.9
86.3::16
8344442
79.7464
57346.77
54,949.5
24.8=8.2
73.44:500
60.3
/
49.1:16.1
87.2=116
82,5:449
91.75.11
67.34311
$6.128888
50.8:::0.66
84.3:15.7
71.1
Table 3: Ablation study on the adaptive parameters in the state encoder.
Unscen Embb
Soon Emb
(Ce.T)
hopper
wolf
reacher-four
walker
Avg.
hop
hop-bod.
stand
valk
run
easy
hard
xalk-bbd.
3..6.1.5.6
3..8
6.601.5.6
85.4.14.5
47.2.12.9
17.4.8.99
0.410099
59.218..1
49.0
x
16.42.6.4
8.77-2.7
3..0
89.2
71.01.311
70.8.1.99
48.1 1:944
5.34:111
58.7
49.1
1..2
82.514.9
91.7-1.11
67.343.1
56.11888
50.81 10.6
84.31.577
71.1
of hopper and the walk-bwd. task of walker. This failure, even seen embodiment (walker), suggests that certain unique motions cannot be adequately captured without adaptive parameters. Overall, introducing adaptive parameters in both encoders yields the best performance.
Ablation on the Number of Demonstrations, In Figure 5 we plot the performance of our model and the two best-performing baselines- one from modular policy learning and the other from few- shot IL-by varying the number of demonstrations. As expected, the performance of our model consistently increases with the number of demonstrations provided. The consistent superiority of our model in few-shot behavior cloning, across varying numbers of demonstrations, highlights the necessity of (1) a powerful encoder capable of handling unseen embodiments, and (2) few-shot policy adaptation mechanism that enables robust few-shot learning unseen tasks. These components are crucial for achieving simultaneous few-shot generalization to both unseen embodiments and tasks.
We provide more ablation studies on meta-training task composition in Appendix D.3
Conclusion
We addressed the challenging problem of few-shot behavior cloning with unseen embodiments and tasks in continuous control. Our framework, Meta-Controller, effectively handles diverse embodiments using two key components: the state encoder and the matching-bor policy network. Leveraging the modular nature of joint-level input/output representations, our state encoder extracts transferable features about the morphology and dynamics the embodiment, capturing both specific and shared knowledge. The matching- based policy network uses these features to infer task structures from few-shot demonstrations, enabling robust imitation without overfitting. Experiments showed that our model generalizes well to unseen embodiments and tasks with only five demonstrations,
Acknowledgments and Disclosure of Funding
This work was in part supported by the National Research Foundation of Korea (RS-2024-0035121 and RS-2024-004361655 the Institute of Information & communications Technology Planning & Evaluation (ITP) (RS-2022-11220926, RS-2024-005092279 SS-2021-11211066 RS-2022-11220959, and RSS-001--111900 funded by the Korea government (MSIT), and NAVER-Intel Co-Lab.
10

---

References
[1] Pytorch lightning. GitHub, 3, 2019.
[2] E. Ben Zaken, Y. Goldberg, and S. Ravfogel. BitFit: Simple parameter-efficien fine-tuning for transformer-basee masked language-models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguisties (Volume 2: Short Papers), 2022.
[3] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, X. Chen, K. Choromanski, T. Ding, D. Driess, A. Dubey, C. Finn, et al. Rt-2: Vision- lasion-languae-acctii models transfer web knowledge to robotic control. arXiv preprint arXiv:2307. 15818, 2023.
[4] L. Chen, K. Lu, A. Rajeswaran, K. Lee, A. Grover, M. Laskin, Abbeel, A. Srinivas, and I. Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:15084-15099 15097, 2021.
[5] K. M. Choromanski, V. Likhosherstov, D. Dohan, X. Song, A. Gane, T. Sarlos, P. Hawkins, J. Q. Davis, A. Mohiuddin, L. Kaiser, et al. Rethinking attention with performers. In International Conference Learning Representations, 2020.
[6] C. Devin, A. Gupta, T. Darrell, P. Abbeel, and S. Levine. Learning modular neural network policies for multi-task and multi-robot transfer. 2017 IEEE international conference on robotics and automation (ICRA), pages 2169-2176. IEEE, 2017.
[7] Y. Duan, M. Andrychowicz, Stadie, O. Jonathan Ho, J. Schneider, I. Sutskever, Abbeel, and W. Zaremba. One-shot imitation learning. Advances in neural information processing systems, 30, 2017.
[8] C. Finn, P. Abbeel, and S. Levine. Model-agnostic meta-learning for fast adaptation of deep networks. International conference on machine learning, pages 1126-1135. PMLR, 2017.
[9] J. Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine. D4rl: Datasets for deep data-driven reinforcement learning, 2020.
[10] H. Furuta, Y. Iwasawa, Y. Matsuo, and S. S. Gu. A system for morphology generalization via unified representation and behavior distillation. In The Eleventh International Conference on Learning Representations, 2022.
[11] A. Ghadirzadeh, X. Chen, P. Poklukar, C. Finn, M. Björkman, and D. Kragic. Bayesian meta-learning for few-shot policy adaptation across robotic platforms. In 2021 IEEE/RS/ International Conference on Intelligent Robots and Systems (IROS), pages 1274-1288 1280. IEEE, 2021.
[12] A. Gupta, L. Fan, S. Ganguli, and L. Fei-Fei. Metamorph: Learning universal controllers with transformers. In International Conference on Learning Representations, 2021.
[13] K. Hakhamaneshi, R. Zhao, A. Zhan, P. Abbeel, and M. Laskin. Hierarchical few-shot imitation with skill transition models. In International Conference on Learning Representations, 2021.
[14] N. Hansen, H. Su, and X. Wang. Td-mpe2: Scalable, robust world models for continuous control. In The Twelfth International Conference on Learning Representations, 2023.
[15] D. Hendrycks and K. Gimpel. Gaussian error linear units (gelus). arXiv preprint 2016.
[16] S. Hong, D. Yoon, and K.-E. Kim. Structure-aware transformer policy for inhomogeneous multi-task reinforcement learning. In International Conference on Learning Representations, 2021.
[17] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.

---

[18] W. Huang, I. Mordatch, and D. Pathak. One policy to control them all: Shared modular policies for agent-agnostic control. In International Conference on Machine Learning, pages 4455-4464. PMLR, 2020.
[19] D. Kim, Cho, S. Kim, C. Luo, and S. Hong. Chameleon: data-efficient generalist for dense visual prediction in the wild. arXiv preprini arXiv:2404. 18459, 2024.
[20] D. Kim, Kim, S. Cho, C. Luo, and S. Hong. Universal few-shot learning of dense prediction tasks with visual token matching. In The Eleventh International Conference Learning Representations, 2022.
[21] S. Kim, S. Shen, D. Thorsley, A. Gholami, W. Kwon, J. Hassoun, and K. Keutzer. Learned token pruning for transformers. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 784-794, 2022.
[22] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[23] V. Kurin, M. Igl. T. Rocktäschel, W. Bochmer, and S. Whiteson. My body is a cage: the role of morphology in graph-based incompatible control. In International Conference on Learning Representations, 2021.
[24] W. Liu, A. Rabinovich, and A. C. Berg. Parsenet: Looking wider to see better. arXiv preprint arXiv: 156.04006 2015.
[25] R. K. Mahabadi, S. Ruder, M. Dehghani, and J. Henderson. Parameter--fficient multi-task fine-tuning for transformers via shared hypernetworks. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 565-576, 2021.
[26] A. Majumdar, K. Yadav, S. Arnaud, J. Ma, C. Chen, S. Silwal, A. Jain, V.-P. Berges, T. Wu, J. Vakil, et al. Where are we in the search for an artificial visual cortex for embodied intelligence? Advances in Neural Information Processing Systems, 36:655-677, 2023.
[27] V. Sanh, L. Debut, J. Chaumond, and T. Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. In NeurIPS EMC2 Workshop, 2019.
[28] T. Schmied, M. Hofmarcher, F. Paischer, R. Pascanu, and S. Hochreiter. Learning to modulate pre-trained models in rl. Advances in Neural Information Processing Systems, 36, 2024.
[29] I. Schubert, J. Zhang, J. Bruce, S. Bechtle, E. Parisotto, M. Riedmiller, J. T. Springenberg, A. Byravan, L. Hasenclever, and N. Heess. generalist dynamics model for control. arXiv preprint ar 2335 2023.
[30] E. Todorov, T. Erez, and Y. Tassa. Mujoco: physies engine for model-based control. In 2012 IEEE/RS. international conference on intelligent robots and systems, pages 5026-5033, IEEE, 2012.
[31] S. Tunyasuvunakool, A. Muldal, Y. Doron, S. Liu, S. Bohez, J. Merel, T. Erez, T. Lillicrap, N. Heess, and Y. Tassa, dm iim_contool Software and tasks for continuous control. Software Impacts, 2020.
[32] A. Vaswani, N. Shazeer, N. Parmar, Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[33] Q Vuong, S. Levine, H. R. Walke, K. Pertsch, A. Singh, R. Doshi, C. Xu, J. Luo, L. Tan, D. Shah, et al. Open x-embodiment: Robotic learning datasets and rt-x models. In Towards Generalist Robots:
Learning Paradigms for Scalable Skill Acquisition @ CoRI.2023, 2023.
[34] T. Wang, R. Liao, J. Ba, and S. Fidler. Nervenet: Learning structured policy with graph neural networks. In International conference on learning representations, 2018.
12

---

[35] R. Xiong, Y. Yang. D. He, K. Zheng, S. Zheng, C. Xing, H. Zhang, Y. Lan, L. Wang, and T. Liu. On layer normalization the transformer architecture. In International Conference on Machine Learning, pages 10524-10533. PMIR, 2020.
[36] M. Xu, Y. Lu, Y. Shen, S. Zhang, D. Zhao, and C. Gan. Hyper-decision transformer for efficient online policy adaptation. The Eleventh International Conference on Learning Representations 2022.
[37] M. Xu, Y. Shen, S. Zhang, Y. Lu, D. Zhao, J. Tenenbaum, and C. Gan. Prompting decision transformer for few-shot policy generalization. In international conference on machine learning, pages 24631-2464, PMLR, 2022.
[38] D. Yarats, I. Kostrikov, and Fergus. Image augmentation all you need: Regularizing deep reinforcement learning from pixels. In International Conference on Learning Representations, 2021.
[39] Yoon, Kim, O. Dia, Kim, Y. Bengio, and S. Ahn. Bayesian model-agnostic meta-learning. Advances in neural information processing systems, 31, 2018.
[40] Y. Zhou, S. Sonawani, M. Phielipp. S. Stepputtis, and H. Amor. Modularity through attention: Efficient training and transfer of language-conditioned policies for robot manipulation. In 6th Annual Conference Robot Learning, 2022.
13

---

Appendix
This document provides the contents that are not included in the main text due to the page limit.
A Limitations and Broader Impacts
Limitations. Despite the promising results, the Meta-Controller framework has several limitations that warrant further investigation. One significant limitation is its reliance on simulated environments for training and evaluation, which may not fully capture the complexities and variabilities of real- world scenarios. This gap between simulation and reality could hinder the direct application of the framework to practical robotic tasks. Additionally, while the framework shows robust performance the few-shot learning setting, may still face challenges in environments with highly stochastic dynamics or in tasks that require extensive long-term planning. Another limitation is the assumption of a unified joint-level representation, which, although effective for the embodiments tested, may not generalize well to robots with significantly different morphologies or actuation mechanisms. Fur- thermore, the computational complexity of the transformer- baasedddee architecture could pose scalability issues for real-time applications, especially ethical implications of deploying such adaptable robotic systems need to be carefully considered to prevent potential misuse in sensitive areas.
ssoourc-coonsrrinee robotic platforms. Lastly, the
Broader Impacts. The proposed Meta-Controller framework for few-shot imitation learning has several significant broader impacts. Firstly, its ability generalize across various robot embodiments and tasks with minimal demonstrations can greatly enhance the adaptability and versatility of robotic systems in dynamic environments. This flexibility is crucial for deploying robots in real-world scenar- ios where they must quickly learn and adapt to new tasks without extensive retraining. Secondly, by leveraging modular approach that decouples embodiment-specifie and task-specific knowledge, the framework promotes efficient knowledge transfer. potentially reducing the computational resources and time required for training new robotic tasks. However, the deployment of such adaptive systems also raises concerns about their potential misuse. For instance, advanced robotic systems equipped with this technology could be employed in surveillance or military applications, leading ethical and privacy issues. Therefore, it is essential to implement safeguards and ethical guidelines to ensure the responsible of these advancements. Furthermore, the reliance on simulated environments for training and evaluation might limit the transferability of the results to real-world conditions, necessitating further research to bridge this gap.
B
More Details on Experiments
In this section, we describe the details of the tasks and embodiments in the used dataset and its collection processes.
B.1 More details on Datasets and Environment
Embodiments and Tasks in DeepMind Control (DMC) suite. The DeepMind Control (DMC) suite benchmark [31] is a collection of continuous control tasks implemented using the MuloCo physics engine (30. It provides variety of simulated environments to test and develop reinforcement learning algorithms. The environments in DMC are designed to be diverse and challenging, promoting the development of agents capable of handling wide range of tasks. To reduce the complexity of the problem and facilitate knowledge acquisition from diverse tasks and embodiments, we only 2D coordinate space. We then augment the dataset with number of tasks used in 334 14], including two tasks from newly added embodiments, walk and run) task of wolf, where we use the same reward function as the walk and run) task of walker, respectively. From a total of 38 tasks from 13 embodiments, we select an unseen embodiment with seen tasks (reacher-four wolf), a seen embodiment with an unseen task (walker), and an unseen embodiment with unseen tasks (hopper) as held-out evaluation tasks for comprehensive analysis. For the exact setup of the newly added embodiments and tasks, please refer to the github repository of NerveNet and TD-MPC2 We list all embodiments and tasks used in our experiments in Table
consider embodiments that operate
14

---

Table 4: List of all embodiments and tasks used in experiments.
Embodiment (E)
Task (7)
acrobot
swingup
catch
ball in cup
spin
balance
cartpole
swingup
cartpole two
poles
flip
flip backwards
jump
legs up
1ie down
cheetah
back
backvards
run front
stand back
stand front
hop
hopper hop backwards stand
spin
pendulum
swingup
pointmass easy
reacher hard
easy
reacher four
hard
reacher three hard
arabesque
backflip flip headstand
legs
walker
lie down run stand walk
walk backwards
run
wolf
walk
Dataset Collection. For each task, we train the DrQ-v2 agent (38 using an online replay buffer. The agent is trained with the hyperparameters specified in (38, generating 1000 to 2000 demonstration trajectories for each task. For the wolf embodiment that was not originally included in the experi- ments of the DrQ-v2, walker, After training, we filter out low-quality demonstrations with cumulative rewards smaller than 10 to ensure the reliability of the training data.
use the same hyperparameters those used for the corresponding tasks of
Data Processing. The states of various embodiments and tasks in DMC consist of two joint- level observations, such as the relative position and velocity of each joint, well as extrasensory observations are required to solve each task (e.g,, in reacher embodiment, the distance between its tip position and the goal position is given an extrasensory observation). The other per-joint attributes such as the movement or motion type can be extracted by the kinematic tree of each embodiment provided in XML format. To ensure compatibility in practical applications, we use
15

---

only the states defined in the default DMC setting. There are two different types of joints in the embodiments we consider: hinge joints and slide joints. Since these joints have distinct motion characteristics (angular motion and linear motion, respectively), we encode the states of each type separately to avoid semantic conflict. We standardize the relative position using a fixed axis (e.g. the z-axis in the xz plane) instead of using raw, unstandardized input. For joints with partially existing states (e.g., the free joint of the torso), we zero-pad the non-existen states. To handle extrasensory states provided by specific embodiment, we concatenate all extrasensory observations into single vector and project them into a single joint token gt E R To this end, we introduce embodiment- specific linear projection layers for the embodiments having extrasensory observations. Lastly, for each embodiment, we normalize the states by scaling them with min-max statistics calculated across the data for each embodiment.
B.2 More Details Implementation
Episode Sampling Strategy. In the episode sampling procedure in Eq. (0)). is crucial to ensure consistency between the query Q and the few-shot demonstration DD for effective learning from demonstrations. Since we use replay buffer as our meta-training dataset which encompasses demonstrations obtained by learning agents at diverse stages, naively applying a uniform sampling strategy on the entire replay buffer results in a high probability of selecting inconsistent queries and demonstrations. Instead, to ensure task consistency in each episode during episodic meta-learning (Section 3.3), we sample the conditioning demonstrations DD and the query data Q for Eq. (TO) from a temporal segment of each replay buffer B In other words, the demonstrations in D and Q obtained from expert agents at adjacent training epochs. In all experiments, segment size of 10 for episodic meta-learning.
temporal
Training Details. All models are trained for 200,000 iterations using the Adam optimizer 22 and
poly learning rate scheduler (24 with base learning rate of 2 x 10-4, After training. we fine-tune all models for 10,000 iterations with a fixed learning rate of 2 x 10-4, except for HDT [36), which requires a higher learning rate of 10-2, For meta-training, we train the model with 8 x RTX A6000
GPUs for approximately 25 hours, and we fine-tune the model on each task with 1 x RTX A6000 GPU for approximately hours.
As the baselines used in our experiments have not been demonstrated in our few-shot behavior cloning setting with unseen embodiments and tasks, we modify their base architecture or learning framework for fair comparison. When fine-tuning HDT, we first adapt the embodiment-speccin head parameters of HDT with the episodic meta-learning objective, as it does not generate appropriate adapter parameters with untrained head parameters of novel embodiment. We then obtain adapter parameters by forwarding every temporal chunk of the few-shot demonstration with the hyper-network and initialize adapter parameters by the mean of the outputs. Then, following the HDT procedure, we fine-tune only the adapter parameters using the BC objective. For fine-tuning the PDT+PEFT baseline, we apply PEFT techniques to the PDT model meta-trained without task-specifie parameters. For the few-shot adaptation of L2M, the learnable modulation pool and the corresponding low-rank projection matrices are fine-tuned.
Architectural Details. For transformer models, we employ Pre-LN transformer layer [35 which consists of self-attention layer followed by 2-layer MLP with 4x hidden dimension size. We use GELU [5] activation for the self-attention layers. To enhance the expressiveness of our matching-baat based policy network, we implement the matching module o in Eq (8) with multi-head
cross-attention (2), following Kim et. al. 20 and Kim et. al. [19] We list the hyper-parameter of Meta-Controller in Table 5
Implementation Framework. We implemented our model based on PyTorch Lightnnng ① which supports both Intel Gaudi-v2 (HABANA) and NVIDIA Al accelerators (CUDA). We provide the code for both systems on separate branches in the GitHub repository.
C Additional Results
In this section, we provide additional results and analysis on computational efficiency and robustness.
16

---

Table 5: Hyper-parameters of Meta-Controoler used in experiments.
Hyper-parameters
Value
Number of demonstrations used in each episode
4
Global Batch Size
64
Hidden dimension
512
Attention heads
4
Low-rank for structure encoder fs
Low-rank for motion encoder fm
Layerscale initialization
Training iteration
200,000
Learning rate warmup iterations
1000
Base learning rate
2 x 10
Table 6: 3-shot behavior cloning results on DeepMind Control suite.
Seen Fini.
Emb, (8)
hoppar
2olf
reacher-foe four
walker
Avg.
Task (7)
hop
hop bid.
stand
walk
run
hard
walk- bud.
MI-DT
7,9 d: 1.5
40.0
82
5.6 1
49.8 1: 9.9
32.2
10.1 1: 5.0
4,5 3.3.7
58.2 1: 6.7
25.2
PDT+PEFT
2.5 d. 1,2
42.6 7.
8.
39.0 10.8
37.9
8,3 1.
6.7 5.
60.0 10.7
25.7
MetaMorph
13.4
7.
51.8
2.
3.7
4.0.2.2
18.8
MTGv2
3.0
77
29.0
65.0 5.
25.1
44 19
2.8
19.4 -
27.9
Ours
3488 = 4.0
75.3
5.9
68.3 6.5
78.2 - 6.0
43.6 4.7
51.3 = 8.9
73.2 :
9.1
53.5 - 7.8
57.0
Table 7: 5-shot behavior cloning results on variations of embodiments in DeepMind Control suite.
Turget Env. (E,
hopper-stand stand
wolf-walk front-back log ratio
length
calf-thigh
Avg.
Emb. Variations
50%
2:1
1:2
1:2
MT-DT
7
23.0 # 8.2
2.
73.1 = 5.8
52.1 11.9
28.1
PDT+PEFT
3.2 1.2
14.4 + 7.
4.4 3.4
15.0 2.1 6.1
65.3 = 6.8
47.8
10.8
25.0
MetaMorph
5.8
18 1.7
39.9 + 9.5
5.7 4.7
29.0 - 6,2
60.3 = 7.8
7.
35.3
MTGv2
8.2
2.2
13.3 1:
4.6 1: 1.6
24.6 d. 3.9
73.8 5.5
4.0 2.1
21.4
75.3
73.2
51.7
C.1 Additional Results on 3-Shot Behavior Cloning
To investigate performance in lower-shot settings, evaluate model with 3-shot behavior cloning on tasks presented in table [ As shown in Table our model consistently outperforms baseline approaches, demonstrating robustness even with fewer demonstrations. This result highlights the model's ss adaptability to fewer examples while maintaining reliable generalization across different tasks and embodiments.
C.2 Additional Results Embodiment Variations
We further assess model performance on six additional embodiments by adjusting physical parameters. such as joint lengths (e.g., foot length of hopper) and ratios among joints (e.g., calf-thigh ratio of hopper, front back leg ratio of wolf). These modifications can make the tasks harder, as the original embodiments are optimized for specific tasks. In Table 7 we compare Meta-Controller with the nigh-performing baselines from Table Our model consistently outperforms all baselines in these challenging variants, demonstrating the robustness and adaptability of our approach variations in embodiment configurations.
17

---

backwards
walf,
53
88-
40-
42
Score
Score
20-
2000
<03E
1003 10003
2300
4000
6010
1006 11000
2303
4200
6030
5026
12000
2003 4300 6500
8036 10020
Rerations
Rerations
Iterations
Iterations
walker,
60-
50-
20-
53
Score
30-
20-
Reeations
Rerations
lterations
Iterations
Figure Learning curves of ours and PDT+PEFT in 5-shot settings.
TSNE visualization state embeddings
acribit
carpale carssile
Instabs duadiattion
Cheslah
reacher
rEROHOL 2r10
Figure 7: t-SNE visualization of embeddings of state encoder f. In the left figure, embeddings the same embodiment have the same color. In the right figure, embeddings of the same joint have the same color.
C.3 Additional Qualitative Results
We provide additional qualitative results about the behavior of the few-shot agents in the tasks used for evaluation. In Figure [1]18] we plot the rendered images of single evaluation rollout by the dmm_contof library [31 from the initial state (t 0) to the states unrolled by each agent, by skipping every ten timesteps in the visualization.
C.4 Visualization Learning Curves
We provide learning curves of Meta-Controller and PDT+PEFT in Figure Due the modular nature of our structure encoder, our model not only achieves better performance but also converges much more quickly than PDT+PEFT in every task.
C.5 Visualization Learned Embeddings
To gain insights into how our model represents different embodiments, we provide t-SNE visualiza- tion of embedding space of features obtained by structure encoder fa in Figure First, we observe that the embeddings are clustered by the joints of each embodiment, and the clusters corresponding the same embodiment are located nearby. Also, we note that the embeddings of slide joints (e.g.. cartpole, cartpole-two, cup, pointmass) and the embeddings of hinge joints (e.g., reacher, reacher-three, walker, acrobot, cheetah, pendulum) are separated in the right and the left regions. Thus, the state encoder captures both emboddment-speecfic and joint-specific knowledge, providing rich features to the policy network.
18

---

Table 8: Inference time of and VC-1
26
measured in single NVIDIA RTX 3090 GPU.
Model
Ours
VC-1 (VTT-L backbone)
Inference Time (ms)
17.8
26.8
Table 9:
Relative inference time of each module of ours, measured in single NVIDIA RTX 3090
GPU.
Module
State encoder (f)
Action encoder (g)
Action decoder (h)
Matching (r)
Inference Time (%)
31.54
31.83
34.08
2.55
C.6 Computational Complexity Analysis
Table presents the inference time comparisons between our model and VC-1 (26), which is a ViT- based foundation model designed for behavior cloning of continuous control tasks. Despite VC-1's high performance in visual tasks, its transformer-bast
architecture incurs notable computational costs. Our model achieves faster inference times, highlighting its efficiency relative VC-1. We also note that models like RT-2 [3]. which are designed for real-time robotic manipulation, typically involve architectures such as a 40-layer ViT and a large language model (LLM) with 3 billion parameters, requiring significantly higher computational resources than our approach.
Additionally, Table 2 provides breakdown of relative inference times for each module in model, demonstrating that the majority of computation time is allocated to the transformer-basee encoders and decoders. In this context, advancements in optimizing transformers, such sparse attention mechanisms [5]. model pruning [21]. and knowledge distillation [27]. can enhance inference speeds. Since these techniques are orthogonal ours, they can be naturally incorporated into our method for resource-constrained robotic platforms that require high speed inference.
C.7 Failure Case Analysis
We present visualizations of scenarios where the Meta-Controller performs less effectively and cumulative rewards over time for each scenario in Figures 8 and respectively. In these failure cases, we observed that agents struggle to obtain rewards until they reach specific posture. Once they achieve this posture (highlighted by the red box Figure B they begin solve the task effectively. This pattern is also reflected in Figure g where rewards remain near zero until certain timestep. after which they rise consistently. This result implies that encouraging the agent reach states similar to those in the demonstrations (e.g., via exploration strategies) would improve performance in challenging few-shot scenarios.
C.8 Robustness Analysis under Noise
To assess robustness in noisy environments, we introduce varying levels of noise to the transition dynamics and measure the resulting performance. Random noise sampled from 2/[-n, n) was added to the agent's action at each timestep, with three noise levels E [2%, 5%, 10%] of the action range. Figure plots the rewards of our model at each noise level compared to experts.
The results indicate that method maintains its performance increase, showing its robustness under stochastic environments. Interestingly, for tasks such as reacher-four, the performance increases with higher noise levels, likely due to the exploration effect induced by stochastic transitions. This robustness under stochastic dynamics indicates potential for the model's application
many tasks noise levels
scenarios where environmental variability common.
D Additional Ablation Studies
In this section, provide additional results ablation studies.
19

---

(a) hopper-hop
Expert
Ours
(b) hopper-stand
Expert
Ours
(c) walker-walk backwards
Expert
Figure 8: Failure cases of our model. As indicated by red boxes, the agent begins to solve tasks effectively after reaches specific posture. Cumulative Rewards Timestep
0.8
walker
Cumulative Rewards
0.6
100
200
300
500
Timestep
Figure 9: Cumulative rewards the failure cases in Figure
hopper, hop
hopper.
hopper,
69
marri
00
100
530
Noise Level
Ncise Level
Noise Level
Noise Level
wolf,,
reacher four,
reacher four, hard
walker walk backwards
Rewarrd
3%
12%
3%
10%
05
35%
25
Figure 10: Robustness analysis of our model in the presence of various noise levels in transition dynamics. Random noise ~ U[[nn, n) is added to each action, where n varies over 2%, 5%, and 10% of the action range. The shaded region represents the standard error.
20

---

Table 10: Additional ablation study the architectural components. Model variants evaluated in 5-shot setting.
Unseen Emb.
Soon Eimh.
f. fm a
hopper
wolf
reacher-
kalker
Avg.
hop
hap-bed.
stand
walk
run
easy
hard
xalk-bud.
K
X
X
6.4 1. 24 58.2 1. 8.4 7.7 4 1.1
32.4 1 8.7 27.4 6.4
14.6 7.5
4,9 1: 4.1
0.2 d: 1.7
19.0
X / / /
5.7127 44.7110.33
15.5.39
64.5199
53.118.5
9.317.0
0.0.10.88
48.41110.4
30.2
/ V
J
25.7 5.5 89.5
+ 1.4 42.5 9.3
83,9 6.1 60.7 + 3,4
53.7 8,9
53.2 + 10.7
71.6 6.0
60.1
49.116.1 87.2+1.6
82.5=4.9
91.71:511
67.3+311
56.11:888
10.8::10.6
84.3=5.7
71.1
Table 11: Ablation study on the adaptive parameters in the state encoder without the matching module in the policy network.
Unscen Emb.
Soen Emb.
P 0
(LE,T)
hopper
wolf
reacher-four
walker
Avg.
hop
hop- bud.
stand
walk
run
easy
hard
walk-bope
22.6.13.0
68.113.0
48. 185.7
71.714477
32.6139
13.7.66.5
10.3:15.
53.11192
40.0
50.0:15.8
91.610088
53.2499.8
78.3:8.4
61845.1
78.147.77
36.1::877
2.3::1.8
22.8:5.9
86.34116
83.4::4.2
79.7:6644
57.346.7
54.91995
24.8:82
73.4::5.0
60.3
D.1 Additional Ablation Studies on Architectural Components
Table I0 presents the additional ablation studies on key architectural components.structure encoder fa. motion encoder fm. action encoder 9, action decoder h and matching module o.. Consistent with the discussion in Section 5.3 we observe that removing the structure encoder fa (row 2) significantly reduces performance both seen and unseen embodiments. This indicates that the structure encoder captures essential morphology-related knowledge that enables cross-embodimeen generalization.
We also ablate the action encoder g and action decoder h (row 3), observing that removing them result in decreased adaptability and performance. The action encoder plays a key role in transforming raw action into latent representation, which effectively enhancing expressibility of the policy network and enables adaptation to various unseen tasks with non-convex relationships between states and actions. Additionally, by encoding actions along the temporal axis, the model can construct pool of transferrable action features related to local motor skills, which facilitates efficient transfer to unseen tasks that share modular skills but involve different skill combinations.
D.2 Additional Ablation studies on Adaptation Mechanism
To further analyze the adaptation mechanism of our model, we conduct additional ablation study by ablating the adaptive parameters in the state encoder without using the matching module in the policy network. On average, using both adaptive parameters (row 3) achieves the best performance. However, we note that the model variant without adaptive parameters in the motion encoder (row 2) achieves higher performance than using the parameters (row 3) in many tasks. We conjecture that this is due to over-fitting on the few-shot demonstrations, as the adaptive parameters in the motion encoder are both specific to the embodiment and task and more prone over-fitting. However, discussed with Table in Section such trends are not found in general (except for the easy task of seacher-fourz This indicates that in few-shot learning settings, the PEFT techniques must be employed together with a robust architecture such as matching to exploit its effectiveness maximally.
D.3 Additional Ablation Studies on Meta- Meta-Training Task Composition
To further understand how the composition of meta- training tasks affects performance, we conduct experiments with varying subsets of embodiments and tasks in Table 12 We select 3 combinations of training tasks, where we remove embodiments from the original TO embodiments. Then, we performed 5-shot behavior cloning experiments on the 8 tasks presented in table [
According Table T2 we observe that using all embodiments outperforms the baselines in most tasks, indicating that diverse set of embodiments makes the model robust to unseen embodiments. Our
21

---

Table 12:
Ablation study embodiments included in the meta-training data. We port the 5-shot score by removing four embodiments from the meta-training data, where choose three different subsets of the removing embodiments. Ei, E2. Es de- note (cartpole-two, cup, pendulum, ) (cartpole-two, pendulum, pointmass, reacher-three), and [cartpole-two, cheetah, cup, reacher-three), respectively.
Unsecn Emb.
Socn Emb.
Emhh (E)
hopper
9olf
reacher roarher-faur
xalker
Avg.
Task
hop
hop-bud.
stand
easy
hard
salk-bud.
&1
83.2
5.0
41.8 9.8
70.7
E2
85.2
81.6
83.9
= 34
5.1
78.8
wlo Eg
26.6 8.0 88.8
68.6
83.6
6.4
59.0 4: 29
38 4.141
81.8 1: 5.1
52.1
Use All Embodiments
49.11
87.2
82.5
499
91.7
5.1
67.3
1 3.1 56.1
8.8
50.8
10.6
84.3
5.7
71.1
findings also show that it is crucial to include embodiments with morphology and dynamics similar to the downstream tasks in the meta-training dataset. For example, removing the reacher- three task (as seen in row 2 and row 3) significantly drops the performance of the reacher- This result reveals that embodiments with similar dynamies or morphological features can facilitate more effective knowledge transfer during meta-testing, suggesting that the diversity of data greatly impacts performance.
task.
22

---

t=0
t=10
t=20
t=30
t=
t=
t=60
t=70
t=80
t=
Expert
M
FS-DT
MT-DT
PDT
PDT+PEFT
4
HDT
L2M
A
MetaMorph
4
MTGv2
Ours
Figure 11: Additional Qualitative Results hopper- hop task.
23

---

t=0
t=10
t=20
t=30
t=
t=
t=60
t=70
t=80
t=90
Expert
A
FS-DT
MT-DT
PDT+PEFT
I
HDT
L2M
N
MetaMorph
MTGv2
Ours
Figure 12: Additional Qualitative Results hopper-hop backwards task.
24

---

t=0
t=10
t=20
t=30
t=40
t=
t=60
t=70
t=80
t=90
Expert
FS-DT
C
MT-DT
PDT
PDT+PEFT
A
HDT
L2M
A
MetaMorph
4
h
MTGv2
Ours
Figure 13: Additional Qualitative Results hopper-st.ae stand task.
25

---

t=0
t=10
t=20
t=30
t=40
t=
t=60
t=70
t=80
t=
Expert
FS-DT
MT-DT
PDT+PEFT
HDT
L2M
MetaMorph
MTGv2
Figure 14; Additional Qualitative Results wolf-walk task.
26

---

t=0
t=10
t=20
t=30
t=40
t=50
t=60
t=70
t=80
t=90
Expert
FS-DT
MT-DT
PDT
PDT+PEFT
HDT
L2M
MetaMorph
MTGv2
X
Figure 15: Additional Qualitative Results wolf folf-runfe task.
27

---

t=0
t=10
t=20
t=30
t=40
t=
t=60
t=70
t=80
t=90
Expert
FS-DT
MT-DT
PDT+PEFT
HDT
L2M
MetaMorph
MTGv2
Ours
Figure 16: Additional Qualitative Results reacher four-easy task.
28

---

t=0
t=10
t=20
t=30
t=40
t=
t=60
t=70
t=80
t=90
Expert
FS-DT
MT-DT
PDT
PDT+PEFT
HDT
L2M
MetaMorph
MTGv2
Figure 17: Additional Qualitative Results reacher task.
29

---

t=0
t=10
t=20
t=30
t=
t=
t=60
t=70
t=80
t=90
Expert
FS-DT
MT-DT
PDT+PEFT
HDT
L2M
MetaMorph
MTGv2
Figure 18: Additional Qualitative Results walker-walk backwards task.
30

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No] or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the
relevant information is Not Available.
Please provide short (1-2 sentence) justificatin right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]" ", is perfectly acceptable to answer "[No] "provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). ). general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In both the abstract and the introduction, we claim our scope as few-shot behavior cloning to unseen embodiments and tasks and contribution the novel state encoder and a matching- based policy network, as well as experiments DMC benchmark.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
31

---

Justification:
We discuss the limitations of our work in Appendix A
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approxinations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer:
[NA]
Justification: Our paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide the details about the dataset, evaluation protocol, and implementa- tions in the experiments section 5 and the appendix.
Guidelines:
32

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome deseribe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification:
We have not cleared up the code in readable format yet.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
mublic/guides/CodeSubnn otteeeeuubmisiioooooooooc for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https c/iips.cc/public//uue
Codeeuumiiii wettsbbmmssiiioi ssionpolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
33

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: We provide the details about the dataset, evaluation protocol, and implementa- tions in the experiments section 5 and Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We provide the standard error in all experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split. initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (c.g. negative еттог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide the information on the computer resources needed to reproduce the experiments in Appendix B.2
Guidelines:
The answer NA means that the paper does not include experiments.
34

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https://neurips: httpp///ctt///tiiccsiieeeeeeeeee
Answer: [Yes]
Justification:
The research conducted with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the broader impacts of our work in Appendix A
Guidelines:
The answer NA means that there is no societal impact of the work performed.
NA No, they should explain why their work has no societal impact or why the paper does not address societal impact.
If the authors
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
35

---

The experiments are conducted in physies simulator, which does not have any societal impacts, to the best of our knowledge.
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We mentioned and cited the original owners of all assets we used in the paper.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi wwwwwe.hcde...m caree.comdaie com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not introduce any new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
36

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
We do not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification:
We do involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
37


Reviews:
[{"review_id": "cg44oWueZd", "reviewer": "Reviewer_TTwU", "summary": "This paper introduces Meta-Controller, a few-shot behavior cloning framework designed for adaptation to various embodiment and control tasks. The framework includes a transformer-based structure-motion state encoder that captures knowledge across different embodiments, and a matching-based policy network that generates adaptive policies. Experimental results demonstrate that the proposed method surpasses other baseline methods, and ablation studies confirm the effectiveness of each module within the framework.", "questions": "1、The ablation study in Table 2 is incomplete. Results should be included for scenarios where only the fs module is removed and where all three modules are removed.\n2、Since the author only provides results based on 4 different embodiment types. It raises the question of whether the policy would still be effective with variations in embodiment configurations, such as random changes in joint length within the same embodiment.\n3、Since the policy is learned from a fixed trajectory dataset, its sensitivity to noise perturbations during transitions is unclear. A robustness analysis would further validate the generalization ability of the proposed method.", "limitations": "The author thoroughly addresses the paper's limitations, which is commendable. However, some issues remain unresolved, such as the efficiency problems caused by the transformer-based architecture. This inefficiency contradicts the motivation of enabling adaptation to various tasks and embodiments in the real world. Similarly, the constraint of operating in a 2D space limits the method's practical applicability.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1、The structure encoder captures knowledge across different embodiments, enhancing the model's generalization ability across various embodiment types. Additionally, the motion encoder enables the model to comprehend the temporal dynamics of states, which simultaneously improves performance in controlling joints and achieving goals.\n2、The ablation study effectively demonstrates the contribution of each module within the framework.\n3、The architecture of each module is clearly illustrated and easy to understand.\n4、The appendix provides extensive details on training and evaluation, facilitating the reproduction of the experiment.", "weaknesses": "1、The paper uses only four different embodiments (12 in total) to demonstrate generalization ability, which is a relatively small sample size.\n2、The proposed method operates only in a 2D coordinate space, limiting its application potential in realistic 3D environments.\n3、The multiple transformer-based architecture constrains the possibility of real-time inference.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Q1.** The ablation study in Table 2 is incomplete.\n\n\nTable R.1 of the author rebuttal completes the ablation study as requested (see row 1 and 2). Consistent with the discussion in Section 5.3 of the main text, we observe that removing the structure encoder $f_s$​ significantly decreases performance on unseen embodiments and tasks. This result indicates that the structure encoder captures transferable and modular knowledge about various morphologies. Additionally, removing all three modules yields a fully supervised model that does not involve meta-learning. As expected, this model fails to adapt to few demonstrations and suffers from overfitting.\n\n\n\n\n> **Q2 & W1.** It raises the question of whether the policy would still be effective with variations in embodiment configurations.\n\n\nWe conducted experiments on six additional embodiments by manually changing joint lengths (e.g., foot length of hopper) or ratios among joints (e.g., calf-thigh ratio of hopper, front leg-back leg ratio of wolf). These modifications can make the tasks harder, as the original embodiments are optimized for specific tasks. In Table R.2 of the author rebuttal, we compare Meta-Controller with the high-performing baselines from Table 1 of the main text. Our model consistently outperforms all baselines in these challenging variants, demonstrating the robustness and adaptability of our approach to variations in embodiment configurations.\n\n\n\n\n> **Q3.** A robustness analysis would further validate the generalization ability of the proposed method.\n\n\nWe conducted an additional experiment by introducing noise to the transition dynamics. Random noise sampled from $\\mathcal{U}[-n, n]$ was added to the agent's action at each timestep, with three noise levels $n \\in$ $[2$%, $5$%, $10$%$]$ of the action range. Figure R.5 in the author rebuttal plots the rewards of our model at each noise level compared to experts. The results indicate that our method maintains its performance across many tasks as noise levels increase. This shows our model’s robustness under stochastic environments. Interestingly, for tasks such as reacher-four, the performance increases with higher noise levels, likely due to the exploration effect induced by stochastic transitions.\n\n\n\n\n> **W2.** The proposed method operates only in a 2D coordinate space, limiting its application potential in realistic 3D environments.\n\n\nWe clarify that our Meta-Controller, including the structure-motion state encoder and matching-based policy network, is designed to be applicable to any coordinate space. In principle, we can extend the input embedding layer of the state encoder to accommodate 3D coordinate inputs. \n\n\nThe primary challenge in demonstrating our method in 3D environments is the limited availability of diverse embodiments and tasks in existing reinforcement learning datasets. Since our goal is to achieve simultaneous generalization to unseen embodiments and tasks, we necessiate a meta-training dataset with diverse embodiments and tasks. This leads us to adopt the DMC dataset, which, to the best of our knowledge, the only dataset that meets the condition (other datasets are composed of either limited embodiments [1], limited tasks [2], or both) and is also widely used in the literature. Although DMC includes some 3D embodiments, their diversity is too limited to cover joint compositions in 3D coordinate space (only 4 unique 3D embodiments).\n\n\nTo demonstrate our method's scalability with diverse embodiments and tasks, we conducted additional experiments varying the size of the meta-training data. As shown in Table R.4 of the author rebuttal, our method's performance improves in general when adding more diverse embodiments and tasks in the meta-training dataset. This suggests the potential of our method in more realistic environments as more diverse datasets become available, which will be an important step forward.\n\n\nWhile we have not conducted direct experiments in 3D environments, we believe our key findings in 2D embodiments—namely, the generalization capability to unseen embodiments and tasks—will extend to 3D embodiments as well.\n\n\n\n\n> **W3.** The multiple transformer-based architecture constrains the possibility of real-time inference.\n\n\nWe clarify that our primary focus is not on real-time inference but rather on presenting a fundamental framework for achieving robust few-shot learning across various unseen embodiments and tasks, which, to our best knowledge, has not been addressed in previous work.\n\n\nRecent works often adopt transformer-based architectures much larger than ours, such as large language models (LLMs) [3, 5] and vision transformers (ViTs) [4, 5]. In Table R.5 of the author rebuttal, we compare the inference time of our model with VC-1 [4], which uses a ViT-L backbone. Our model achieves faster inference times than VC-1. Although we couldn't evaluate RT-2 [5] due to lack of official code, it involves a large network with at least a 40-layer ViT and a LLM with 3B parameters, requiring much higher computation costs. However, these works have demonstrated the feasibility of using transformer architectures in real-time applications. For example, RT-2 achieved real-time inference with their heavy network using cloud computing. Thus employing multiple transformer-based architecture would not be an essential limitation for real-time inference.\n\n\nIn resource-constrained robotic platforms, transformer-based architectures may hinder real-time inference, as discussed in Appendix A. However, advancements in optimizing transformers, such as sparse attention mechanisms, model pruning, and knowledge distillation, can enhance inference speeds. Since these approaches are orthogonal to our contribution, our method can naturally incorporate them for real-time applications with minimal requirements of the computing resource, although this is not the main focus of this paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "k3VSfRbWFk", "reviewer": "Reviewer_EmwS", "summary": "This paper introduces a framework called Meta-Controller for few-shot behavior cloning that can generalize to unseen robot embodiments and tasks in continuous control problems. The key contributions are:\n\n1. A joint-level input/output representation to unify state and action spaces across heterogeneous robot embodiments.\n\n2. A novel state encoder with two components:\n   - A structure encoder to capture morphological knowledge\n   - A motion encoder to capture dynamics knowledge\n   Both use adaptive parameters to specialize to specific embodiments and tasks.\n\n3. A matching-based policy network that leverages a few demonstrations to predict actions for new tasks.\n\n4. A training protocol involving episodic meta-learning followed by few-shot fine-tuning.\n\nThe authors evaluate Meta-Controller on various tasks from the DeepMind Control suite, demonstrating superior few-shot generalization performance compared to existing modular policy learning and few-shot imitation learning approaches. Key results show Meta-Controller outperforms baselines, especially on challenging tasks like the reacher-four embodiment. Ablation studies validate the importance of the structure encoder, motion encoder, and matching module.", "questions": "- In the case of robot arm manipulation, what are the benefits of utilizing the proposed structure-motion encoder over the direct application of end-effector-based control?\n- Could you clarify what is meant by the \"Embodiment-specific positional embedding\" as depicted in Figure 2? How do you implement it?\n- It appears that the action $m_j$ is conditioned solely on the feature of joint  $j$ ($z_j$). Why not consider the state features from all joints to simultaneously generate motions for each joint?\n- What necessitates the use of an action encoder and decoder? Why not simply retrieve actions based on their similarity?", "limitations": "See weaknesses.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "This paper has several strengths:\n\n- To the best of my knowledge, the proposed approach is novel. The Meta-Controller framework addresses an important challenge in robotics - simultaneous generalization to unseen embodiments and tasks with few-shot learning. This is a solid step beyond existing work that typically focuses on either embodiment generalization or task generalization, but not both. To solve the problem, this paper introduces a well-thought-out architecture that combines several innovative components:\n   - Joint-level I/O representation for handling heterogeneous embodiments\n   - Structure and motion encoders with adaptive parameters\n   - Matching-based policy network for few-shot adaptation\n\n- The evaluation is thorough. The authors conduct extensive experiments using the DeepMind Control suite, comparing their approach against multiple baselines from both modular policy learning and few-shot imitation learning domains. The paper also includes comprehensive ablation studies that validate the importance of each component in the architecture.\n\n- The performance is strong. Meta-Controller consistently outperforms existing methods, especially on challenging tasks like the reacher-four embodiment, demonstrating its effectiveness.\n\n- The paper is well-structured and clearly written, making it accessible despite the complexity of the topic. And the figures are nice and neat.", "weaknesses": "While the paper has many strengths, there are also some potential weaknesses or areas that could be improved:\n\n- The proposed method assumes the number of joints is equal to the action dimension. However, this assumption may not always hold, as many robots incorporate passive joints, such as in four-bar linkages. Please correct me if my understanding is wrong.\n\n- The proposed method appears to share conceptual connections to some retrieval-based techniques [1,2]. A comparative empirical evaluation with these related methods could provide helpful insights.\n\n- Limited real-world validation: As the experiments were conducted exclusively within simulated environments (the DeepMind Control suite), open questions remain regarding the approach's transferability to real-world robotic systems incorporating more complex dynamics and noise.  \n\n- Limited failure case discussion: Analyzing scenarios in which the Meta-Controller performs less effectively could offer perspectives for future enhancements. A more in-depth failure case analysis may strengthen the presentation.\n\n[1] Pari, Jyothish, et al. \"The surprising effectiveness of representation learning for visual imitation.\" arXiv preprint arXiv:2112.01511 (2021).\n\n[2] Sridhar, Kaustubh, et al. \"Memory-consistent neural networks for imitation learning.\" arXiv preprint arXiv:2310.06171 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Q1.** In the case of robot arm manipulation, what are the benefits of utilizing the proposed structure-motion encoder over the direct application of end-effector-based control?\n\nCompared to end-effector-based control, our method eliminates the need for separate low-level controllers manually tuned for each robot and simplifies the control process by learning a unified representation adaptable to various tasks and embodiments. By leveraging joint-level representations and the structure-motion state encoder, our framework seamlessly adapts to different robot morphologies and dynamics, ensuring robust performance across a wide range of embodiments. This is crucial for developing versatile and adaptive robotic systems.\n\n\n> **Q2.** Could you clarify what is meant by the \"Embodiment-specific positional embedding\" as depicted in Figure 2?\n\nAs explained in Section 3.2.1 (line 132-147), the \"Embodiment-specific positional embedding\" refers to a mechanism to incorporate specific information about the robot's physical configuration into the model. This embedding accounts for the unique positional relationships and characteristics of each joint within a given robot embodiment. Similar to positional encoding in transformers, it is implemented by adding a learnable vector to each joint's state token.\n\n\n> **Q3.** Why not consider the state features from all joints to simultaneously generate motions for each joint?\n\nThe architecture design of our state encoder allows the structure encoder to contextualize the joint representation considering all joints, while the motion encoder captures their temporal dynamics. By applying these two encoders in sequence, we can encode motion features that incorporate both spatial and temporal information, in a more computationally efficient way than using full spatio-temporal interactions. This approach is similar to axial attention techniques [6] used in video models, which are widely adopted for their efficiency and effectiveness in handling spatio-temporal data.\n\n\n> **Q4.** What necessitates the use of an action encoder and decoder? **W2.** The proposed method appears to share conceptual connections to some retrieval-based techniques.\n\nWhile matching is effective for few-shot learning, applying it directly to the raw action space imposes a strong constraint: the output action must be a convex combination of the support actions. To alleviate this constraint, we introduce an action encoder and decoder, allowing matching in the latent space rather than the raw label space. This enhances the expressiveness of the policy network, enabling adaptation to various unseen tasks with non-convex relationships between states and actions. Additionally, by encoding actions along the temporal axis, the model can construct a pool of transferrable action features related to local motor skills, which facilitates efficient transfer to unseen tasks that share modular skills but involve different skill combinations.\n\nWe note that retrieval-based techniques suggested by the reviewer are similar to our matching-based policy unless they interpolate (or hardly select) the raw actions. To validate matching in the latent space, we conducted an ablation study by removing action encoder and decoder from our framework. As shown in Table R.1 in the author rebuttal, removing the action encoder and decoder decreases performance, supporting their effectiveness.\n\n\n> **W1.** The proposed method assumes the number of joints is equal to the action dimension.\n\nWe clarify that our method can address the cases where the number of joints is not equal to the action dimension. As described in Section 3.1 (line 118-119), we assign zero values for free (non-actuable or passive) joints, and simply discard their tokens after encoding via the state encoder. This is because the states of passive joints provide useful information about the morphology of the embodiment during the encoding process, but there are no actions to predict for these joints. Note that most of the embodiments used in our experiment (e.g., hopper, walker, cheetah, acrobot, cartpole, wolf) contains passive joints.\n\n\n> **W3.** Limited real-world validation.\n\nWe clarify that our experiments were conducted exclusively within the DeepMind Control Suite (DMC) because, to our knowledge, it is the only benchmark currently available that includes both diverse embodiments and tasks.\n\nTo address concerns about the transferability of our approach to real-world robotic systems, we have extended our experiments to simulate more realistic conditions. Specifically, we introduced varying levels of noise to the control actions within the DMC environments, thereby making the transition dynamics both stochastic and noisy. As illustrated in Figure R.5 of the author rebuttal, the performance of our Meta-Controller does not significantly drops despite the presence of random noise. This result underscores the potential of our method to operate effectively under realistic conditions that involve complex dynamics and noise.\n\n\n> **W4.** Limited failure case discussion.\n\nTo address the reviewer’s concerns, we conducted additional analyses on the failure cases of the Meta-Controller. We present visualizations of scenarios where the Meta-Controller performs less effectively and cumulative rewards over time for each scenario in Figures R.3 and R.4 of the author rebuttal, respectively.\n\nIn these failure cases, we observed that agents struggle to obtain rewards until they reach a specific posture. Once they achieve this posture (highlighted by the red box in Figure R.3), they begin to solve the task effectively. This pattern is also reflected in Figure R.4, where the rewards remain near zero until a certain timestep, after which they rise consistently. This result implies that encouraging the agent to reach states similar to those in the demonstrations (e.g., via exploration strategies) would improve performance in challenging few-shot scenarios.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BD3IoJbS24", "reviewer": "Reviewer_iQ1f", "summary": "This paper aims to generalize to unseen embodiments and tasks by few-shot behavior cloning. It proposes a modular framework to capture both shared knowledge across all embodiments and embodiment-specific knowledge. It utilizes a matching-based method to enhance the robustness to overfitting. It shows superior performance in DMC simulations.", "questions": "* See weakness\n* What is the learning curve for the models compared with PDT+PEFT? Are the models more efficient since they are modular networks?\n* The experiments only compare the results of at least 5-shot. What about the performance when given fewer demonstrations?\n* The episode sampling strategy is confusing. The claim \"D and Q are obtained from adjacent training epochs\" limits the adaption of the models. \"a temporal segment size of 10 for episodic meta-learning\". Does it mean that the horizon of each few-shot demonstration is 10 and is it enough to provide task information? It is better to provide more details about few-shot demonstrations.", "limitations": "* The paper has provided several limitations including simulations, highly stochastic dynamics, long-term planning and computational complexity.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "* The frameworks proposed by the paper are reasonable. In order to generalize to different embodiments and share common knowledge, it designs embodiment-specific parameters and shared parameters for networks. \n* The paper demonstrates strong performance compared with the baselines.", "weaknesses": "* The models are complex and matching-based policies are time-consuming. The models consist of multiple different transformer blocks. The matching-based policies need to compute similarities with all pairs in demonstrations to predict one action.\n* The environments are too simple. The paper only focuses on the DeepMind Control suite and they are only state-based environments. What about the performance when the environments are image-based or manipulation tasks?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Q1.**  What is the learning curve for the models compared with PDT+PEFT?\n\nWe provide learning curves of Meta-Controller and PDT+PEFT in Figure R.2 of the author rebuttal. As the reviewer pointed out, due to the modular nature of our structure encoder, our model not only achieves better performance but also converges much more quickly than PDT+PEFT in every task. \n\n\n> **Q2.** Performance when given fewer demonstrations?\n\nWe present 3-shot behavior cloning results of baseline models and Meta-Controller in Table R.3 of the author rebuttal. We observe that our model outperforms all the baselines in most of the tasks, demonstrating its adaptability on unseen embodiments and tasks in a few-shot regime.\n\n\n> **Q3.** The episode sampling strategy is confusing.\n\nOur meta-training dataset consists of a replay buffer of an expert agent, collected throughout the agent’s learning period, which means the behavior or policy in the buffer can vary significantly. Initially, the expert agent's policy is sub-optimal and it improves over time. We observe that this leads to diverse policies for a single task. Our matching framework, however, requires consistency between input states and output actions in both support and query data.\n\nTo ensure this consistency, we sample episodes from similar training periods within the replay buffer. For instance, in the hopper-stand task with 1000 trajectories sorted by training period, we select a query trajectory (e.g., the 100th) and support trajectories within a neighboring range of size 10 (i.e., from the 95th to the 105th). The “temporal segment size 10” means this range. This ensures that the support and query data are temporally close, providing consistent and relevant task information.\n\n\n> **W1.** The models are complex and matching-based policies are time-consuming.\n\nWe clarify that transformer architectures are commonly adopted in the literature, and the complexity of our architecture would not be a significant weakness. Notably, recent works in the field use much larger transformer-based architectures than ours, such as large language models (LLMs) [3, 5] and vision transformers (ViTs) [4, 5]. In Table R.5 of the author rebuttal, we compare the inference time of our model and VC-1 [4], which uses a ViT-L backbone. Our model achieves faster inference times than VC-1. Although we couldn’t evaluate RT-2 [5] due to lack of official code, it involves a large network with at least a 40-layer ViT and a LLM with 3B parameters, requiring much higher computation costs compared to ours.\n\nRegarding the matching process, it is implemented with a single cross-attention layer, constituting only a small part of the overall computation. As shown in Table R.6 in the author rebuttal, most inference time is occupied by the encoders and decoders.\nIn resource-constrained robotic platforms, transformer-based architectures may hinder real-time inference, as discussed in Appendix A. However, advancements in optimizing transformers, such as sparse attention mechanisms, model pruning, and knowledge distillation, can enhance inference speeds. Since these techniques are orthogonal to our contribution, they can be naturally incorporated into our method for resource-constrained robotic platforms, although this is not the main focus of this paper.\n\n\n> **W2.** The environments are too simple, only focuses state-based environments. Performance of image-based or manipulation tasks?\n\nFirst, we clarify that our primary contribution is demonstrating simultaneous generalization to unseen embodiments and tasks using few-shot learning, which is a significant challenge in the field. This necessitated using the DeepMind Control Suite (DMC), the only dataset providing both diverse embodiments and tasks to our knowledge. Other datasets are limited in either range of embodiments [1] or tasks [2], or both. Although DMC might appear simple, it enables rigorous testing of our framework's core capabilities. Similar research, such as PromptDT and MetaMorph, used even simpler settings. To our knowledge, our setting represents one of the most challenging for few-shot out-of-distribution generalization in behavior cloning.\n\nSecond, our method is inherently designed for state-based environments, where \"state\" refers to proprioceptive sensors attached to each joint. Extending our method to image-based environments is beyond the scope of this paper. It is common in the literature to focus on state-based environments [2, 7], as many real-world robots are equipped with proprioceptive sensors.\n\nRegarding \"manipulation tasks,\" we interpret this to refer to 3D robotic arm manipulation tasks, which are also state-based environments. If our interpretation is incorrect, please let us know. Conducting experiments in these environments is feasible but challenging due to the limited availability of diverse and comprehensive 3D datasets. Most available datasets focus on specific joint compositions like robotic arms or quadrupeds, making it difficult to gather sufficient meta-training data to acquire transferable knowledge about 3D embodiments. This limitation reflects the current state of available resources rather than the capability of our approach.\n\nNonetheless, we recognize the importance of demonstrating the generalizability of our approach in more complex settings, and additional experiments in the author rebuttal may give some insights to this direction. Table R.2 shows that our method still performs well in more diversified embodiments. In Table R.5, we demonstrate that our method is robust to noise, highlighting its potential in noisy and stochastic real-world scenarios. Finally, as shown in Table R.4, our method's performance improves when we add more diverse embodiments and tasks in the meta-training dataset. This suggests the potential of our method in more complex scenarios, such as those with noisy transition dynamics or emerging diverse variants of embodiments, as more diverse datasets become available.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ralzfpb3n6", "reviewer": "Reviewer_dCFx", "summary": "This paper presents a method for few-shot imitation learning for new embodiment. Specifically, it presents a method that learns a state representation that decouples embodiment-specific and task-specific knowledge and a meta-learning framework that transfers between embodiments and tasks. Results suggest that the resulting policy improves over decision transformer-based or modular policy-based baselines.", "questions": "1. Given that state representation learning is decoupled from policy representation learning, it would be interesting to visualize the embedding learned by the state encoder and see how distinct it is between different morphologies.\n2. Does the performance of the model scale with model capacity and data? I.e. currently the model is pre-trained with “28 tasks from 9 embodiments … up to 2000 demonstration trajectories for each task and embodiment”. What if some of the tasks and embodiments are removed? Do some embodiments/tasks matter more than others?", "limitations": "The limitation section is present.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. A design choice (separating the state encoder from the matching-based policy network) that decouples state learning from task-specific knowledge learning. \n2. The proposed method demonstrates better generalization to unseen embodiments in DMC, outperforming existing baselines.", "weaknesses": "Only demonstrates results in DMC. The reviewer is not certain if this scales to real-world control environments, as the different embodiments operate at various scales, and the model needs to be able to be robust to environment dynamics (here the environment is static). The reviewer acknowledges that this limitation is also mentioned in the limitation section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Q1.** Given that state representation learning is decoupled from policy representation learning, it would be interesting to visualize the embedding learned by the state encoder and see how distinct it is between different morphologies.\n\n\nWe provide a t-SNE visualization of embedding space of features obtained by structure encoder in Figure R.1 of author rebuttal. First, we can observe that the embeddings are clustered by the joints of each embodiment, and the clusters corresponding the same embodiment are located nearby. Also, we note that the embeddings of slide joints (e.g., cartpole, cartpole_two, cup, pointmass) and the embeddings of hinge joints (e.g., reacher, reacher_three, walker, acrobot, cheetah, pendulum) are separated in the right and the left regions. Thus the state encoder captures both embodiment-specific and joint-specific knowledge to give rich features to the policy network.\n\n\n\n\n> **Q2.** Does the performance of the model scale with model capacity and data?\n\n\nTo address the reviewer’s question, we conducted additional experiments to validate the effect of data scale and how different compositions of meta-training dataset affects the performance of downstream behavior cloning. Specifically, we select 3 combinations of training tasks, where we remove 4 embodiments from the original 10 embodiments. Then, we performed 5-shot behavior cloning experiments on the 8 tasks presented in the main table. \n\n\nAccording to Table R.4 of the author rebuttal, we observe that using all embodiments outperforms the baselines in most tasks, indicating that a diverse set of embodiments makes the model robust to unseen embodiments. Our findings also show that it is crucial to include embodiments with morphology and dynamics similar to the downstream tasks in the meta-training dataset. For example, removing the reacher_three task (as seen in row 2 and row 3) significantly drops the performance of the reacher_four task. This result reveals that embodiments with similar dynamics or morphological features can facilitate more effective knowledge transfer during meta-testing, suggesting that the diversity of data greatly impacts performance.\n\n\n\n\n> **W1.** Only demonstrates results in DMC. The reviewer is not certain if this scales to real-world control environments, as the different embodiments operate at various scales, and the model needs to be able to be robust to environment dynamics (here the environment is static).\n\n\nOur experiments were conducted exclusively within the DeepMind Control Suite (DMC) because, to our knowledge, it is the only benchmark currently available that includes both diverse embodiments and tasks. Although our experiments lack real-world validation, we clarify that our primary contribution is the introduction of a fundamental framework for simultaneous generalization to unseen embodiments and tasks with few demonstrations, which we believe addresses a significant challenge in the field.\n\n\nTo address concerns about the transferability of our approach to real-world control environments, we have extended our experiments to simulate more realistic conditions. Specifically, we introduced varying levels of noise to the control actions within the DMC environments, thereby making the transition dynamics both stochastic and noisy. As illustrated in Figure R.5 of the author rebuttal, the performance of our Meta-Controller does not significantly drop despite the presence of random noise. This result underscores the potential of our method to operate effectively under realistic conditions that involve complex dynamics and noise.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We appreciate all the valuable comments provided by the reviewers. We will incorporate the additional results and clarifications made during the rebuttal into the camera-ready version of our paper.

We want to clarify that there was a typo regarding the details of embodiment and task. We use 30 tasks from 10 embodiments as training tasks, which is originally written as 28 tasks from 9 embodiments. The missing tasks are catch and spin from the “ball-in-cup” embodiment. We will update the main text and Appendix to fix this typo.

In the individual responses below, we address each reviewer's questions (Q1, Q2, ...) and weaknesses (W1, W2, ...) sequentially. For the tables and figures referenced in the rebuttal (numbered R.x), please refer to the attached PDF file. We summairze all references used in this rebuttal below.


## References


[1] Yu et al., "Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning", CoRL, 2020.


[2] Furuta et al., "A System for Morphology-Task Generalization via Unified Representation and Behavior Distillation", ICLR, 2022.


[3] Jiang et al., “VIMA: General Robot Manipulation with Multimodal Prompt”, ICML, 2023


[4] Majumdar et al., “Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?”, NeurIPS, 2023


[5] Brohan et al., "Rt-2: Vision-language-action models transfer web knowledge to robotic control" arXiv, 2023.


[6] Bertasius et al., “Is Space-Time Attention All You Need for Video Understanding?”, ICML, 2021.


[7] Chen et al. "Decision transformer: Reinforcement learning via sequence modeling." NeurIPS, 2021.

Meta-review: Robust discussion with serious concerns expressed about the feasibility of real time infernece. I am inclined to discount this objection for NeurIPS since this is primarily a models and algorithms and models venue. Other reviewers are positive about the method which leads to a recommendation to accept.
Decision: Accept (poster)