Universal embodied intelligence: learning from crowd, recognizing the world, and reinforced with experience
Authors: Luo Ji
Longfei Ma
Chang Zhou
Fei Wu
Hongxia Yang
Venue: Submitted to ICLR 2023 (2023)
Keywords: reinforcement learning
transformer
morphology
pretrain
finetune
generalization

Abstract: The interactive artificial intelligence in the motion control field is an interesting topic, especially when universal knowledge adaptive to multiple task and universal environments is wanted. Although there are increasing efforts on Reinforcement learning (RL) studies with the assistance of transformers, it might subject to the limitation of the offline training pipeline, in which the exploration and generalization ability is prohibited. Motivated by the cognitive and behavioral psychology, such agent should have the ability to learn from others, recognize the world, and practice itself based its own experience. In this study, we propose the framework of Online Decision MetaMorphFormer (ODM) which attempts to achieve the above learning modes, with a unified model architecture to both highlight its own body perception and produce action and observation predictions. ODM can be applied on any arbitrary agent with a multi-joint body, located in different environments, trained with different type of tasks. Large-scale pretrained dataset are used to warmup ODM while the targeted environment continues to reinforce the universal policy. Substantial interactive experiments as well as few-shot and zero-shot tests in unseen environments and never-experienced tasks verify ODM's performance, and generalization ability. Our study shed some lights on research of general artificial intelligence on the embodied and cognitive field studies. 

Under review as a conference paper at ICLR 2023
UNIVERSAL EMBODIED INTELLIGENCE: LEARNING FROM CROWD, RECOGNIZING THE WORLD, AND REIN- FORCED WITH EXPERIENCE
Anonymous authors Paper under double-blind review
ABSTRACT
The interactive artificial intelligence in the motion control field is an interesting topic, especially when universal knowledge adaptive to multiple task and univer- sal environments is wanted. Although there are increasing efforts Reinforce- ment learning (RL) studies with the assistance of transformers, it might subject the limitation of the offline training pipeline, in which the exploration and generalization ability is prohibited. Motivated by the cognitive and behavioral psychology, such agent should have the ability to learn from others, recognize the world, and practice itself based on its own experience. In this study, we propose the framework of Online Decision MetaMorphFFrmerr (ODM) which at- tempts to achieve the above learning modes, with a unified model architecture to both highlight its own body perception and produce action and observation pre- dictions. ODM can be applied on any arbitrary agent with a multi-joint body, located in different environments, trained with different type of tasks. Large-scale pretrained dataset are used to warmup ODM while the targeted environment con- tinues to reinforce the universal policy. Substantial online experiments as well few-shot and zero-shot tests in unseen environments and never-experienced tasks verify ODM's performance, and generalization ability. Our study shed some lights on research of general artificial intelligence the embodied and cogni- tive field studies. Code, result and video examples can be found on the website https: httpp:://baiimen htssss//baamxiiissaa ..r.ggittuuut giihtb.iii...
INTRODUCTION
Research of embodied intelligence focus on the learning of control policy given the agent with some morphology (joints, limbs, motion capabilities), while it has always been a topic whether the control policy should be more general or specific. As the improvement of large-scale data technology and cloud computing ability, the idea of artificial general intelligence (AGI) has received substantial interest (Reed et al.) 2022). Accordingly. a natural motivation is to develop a universal control policy for different morphological agents and easy adaptive to different scenes. It argued that such a smart agent could be able to identify its 'active self" by recognizing the egocentric, proprioceptive perception, react with exteroceptive observations and have the perception of world forward model (Hoffmmann & Pfeifer 2012). However, there is seldom such machine learning framework by so far although some previous studies have similar attempts in one or several aspects.
Reinforcement Learning(RL) learns the policy interactively based on the environment feedback therefore could be viewed as a general solution for our embodied control problem. Conventional RL could solve the single-task problem in an online paradigm, but is relatively difficult to imple- ment and slow in practice, and lack of generalization and adaptation ability. Offline RI facilitates the implementation but in cost of performance degradation. Inspired by recent progress of large model on language and vision fields, transformer-base
RL (Reed et al. 2022, Chen et al. 1021 Lee et al.] 2022 Janner et al. 2021 Zheng et al. 2022 Xu et al. (022)) has been proposed by trans- forming RL trajectories as a large time sequence model and train in the auto-regressive manner. Such methods provide an effective approach to train generalist agent for different tasks and envi- ronments, but usually have worse performance than classic RL, and fail to capture the morphology information. In contrast, MetaMorph (Gupta et al. 2022) chooses encode agent's body mor-

---

Under review as a conference paper at ICLR 2023
phology and performs online learning, therefore has good performance but lack of time dime-dependyncy consideration.
To have better solution of embodied intelligence, in which agent improve its skill by actual practice, learning from others (teachers, peers or even someone with worse skills), makes decision based the perception of "the world model' (Ha & Schmidhuber 2018: Wu et al. 2022). is reasonable to believe that an embodied intelligence agent should have the above three learning paradigm simultaneously. We propose such methodology by designing a morphology-timg
motivated from behavioral psychology
transformer-based RL architecture which is compatible with both offline and online learning. Offline training conducted on multi-task datasets which considers both learning from other agents and speculate the future system states. The online training allows the agent to improve its policy an on-policy way given single task.
Hamancid Enbrdimert
Supervbed
Fev-shee
Zerp-shot reforencin
Unseen tnomommetis
Learning
Figure 1: Application pipeline of ODM.
In this work, we propose framework called Online Decision Metamorphformer (ODM), which aims to study the general knowledge of embodied control across different body shapes, environments and tasks, as indicated in Figure [ The model architecture contains the universal backbone and the task-specific modules. The task task-specific modules capture the potential difference in agent body shapes, and the morphological difference is enhanced by a prompt based on characteristic of body shapes. We first pretrain this model with curriculum learning. by learning demonstrations from the easiest to the hardest task, from the expert to low-level players. The environment model prediction added as auxiliary loss. The same architecture can then be finetuned online given a specific task. During the test, we are able to test ODM with all training environments, transfer the policy to different body shapes, adaptive to unseen environments and accommodate with new types of tasks (e.g. from locomotion to reaching, target capturing or escaping from obstacles.).
Main contributions of this paper include:
We design a unified model architecture to encode time and morphology dependency simul- taneously which bridges sequential decision making with embodiment intelligence.
We propose training paradigm which mimic the process of natural intelligence emerging, including learning from others, boost with practices, and recognize the world.
We train and test our framework with agent in eight different body shapes, different envi- ronment terrain and different task types. These comprehensive analysis verifies the general knowledge of motion control learned by us.
RELATED WORKS
Among conventional RL methods, on-policy RL such Proximal Policy Optimiza- tion (PPO (Schulman et al., (017) is able to learn the policy therefore has good adaptive ability to environment, but is slow convergence and might have large trajectory variations. Off-policy RI.
Classic RL:
2

---

Under review as a conference paper at ICLR 2023
such DQN (Mnih et al.) 2015) improves the sampling efficiency but still require the data buffer updated dynamically. In contrast, offline RL (Fujimoto et al. 2019 Kumar et al. 2020; Kostrikov at. ) solve the problem similar with supervised learning, but might have degraded perfor- mance because of the distribution shift between offline dataset and online environment. In our work, aim reach the state-of-the-art performance for different embodied control tasks, therefore model architecture compatible with on-policy RI is proposed.
Transformer- RL: Among these efforts. Decision Transformer (DT) (Chen al. 2021 and Multi-game Decision Transformer (Lee et al. 2022) embodied the continuous state and action di- rectly and use a GPT-like casual transformer to solve the policy offline. Their action decision is conditioned on Return-to-Go (RTG), either arbitrarily set or estimated by model, since RTG is un-
known during inference. Instead, Trajectory Transformer (TT) (Janner et al.) (02T) discards RTG in the sequential modeling to avoid that approximation 1ttppiiimaiioo.o.ppp11111 (Xu et al. 2022) adds the task difference consideration into the model by using demonstrated trajectory as prompts. ODT (Zheng et al. 2022) first attempts to solve transformer-based RL in an online manner but mainly focus on supervised on actions instead of maximizing rewards. our work, we propose a similar model ar- chitecture but is able to conduct both offline learning and on-policy, actor-critic learning. The online learning employs PPO as the detailed policy update tools with the objective as reward maximization.
Morphology-basee RL: There are some other studies which focus on the agent's 's morphology infor- mation, including GNN-based RL which modeling agent joints as a kinematics tree graph (Huang et al., 2020), Amorpheus (Kurin et al., 202T) which encodes policy modular for each body joint, and MetaMorph (Gupta et al. (202) which intuitively use transformer to encode the body morphol- ogy sequence of joint properties, and train it by PPO. In our work, we have the morphology- aware encoder which is similar with MetaMorph and the same PPO update rule. However, compar- ing with Metamorph, we encode the morphology on not only state but also historical actions, and consider the historical contextual consideration.
3 PRELIMINARIES AND PROBLEM SETUP
We formulate a typical sequential decision making problem, in which on each time step f, an em- bodied agent conceives state s, R"* performs action a, E R"=, and receives scalar reward rs E RR. Reinforcement Learning (RL) can then be employed to produce the policy (a, (&)) which aims to maximize the expectation sum of discounted rewards. The actor-critic framework is a famous RL framework with the critic estimates the state value function V(s), which the actor determines the policy. Classical RL methodologies such as Proximal Policy Optimization (PPO) can be employed to solve the problem effectively, with detailed derivation in Appendix A
Here we redefine the aforementioned conventional RL notations in a more "embod- ied style', although still generalized enough for any arbitrary agent with multi-joint body. Inspired by the idea of Gupta et al. (2022), we differentiate the observation into the agent's proprioceptive observations, the agent's embodied joint-wise self-perception (e.g. angular, angular velocity of each joint), as well as the exteroceptive observation, which is the agent's global sensory (e.g. position, velocity). Given K-joint agent, we denote the proprioceptive observation by øaro E g xn in which each joint is embedded with n dimension observations. The exteroceptive observation is r r-dimensional which results in cant Rd and = fo o
Problem Setup:
Table 1: Comparison of conventional RL and our notations
State
Action
Conventonal
R
oore Moro C CRR
approach
n. M. Km
Connections
K
M.
Stepping forward from Gupta et al. (2022). we also define the action in the joint-dependent way; that is, assuming each joint has m degree of freedom (DoF) of movements (e.g. torque), the action is reshaped as E k To allow the room of different agent body shapes, we introduce binary masks which have the same shapes of nr and a and zero-pad the impossible observations or actions (e.g. DoF of humanoid's forearm should be smaller than its upper-arm due to their physical connection). Table [ visualizes the comparison between the conventional RL notations and our embodied version notations.
3

---

Under review as a conference paper at ICLR 2023
Attention-based Encoding: Given stacked time sequence vector E R with T the time length and as the embedding dimension, an time sequence encoder can be expressed as
Encr (a:) Attention(Q=x, K=r, V=x) R2'xe
according to the self-attention mechanism (Vaswani et al. 2017) with Q, K, V denoting query, key and value. Analogously, given stacked joint sequence vector RX*e with K as the number of joints, a morphology-awarr encoder instead learns the latent representation on this joint sequence
Enc (p) AttentionnQ=p. K=p, V=p) E Rxx
By pre-tokenizing either Ø or a into p. within the latent space with dimension e, their 'pose' latent variables can be encoded by Enc For each form of encoder, timestep or joint position info encoded by lookup embedding then adding to encoded vector. More details can be referred MetaMorph paper (Gupta al. 2022).
METHOD
4.1
MODEL ARCHITECTURE
Our ODM model structure contains unified backbone and task-specific modules. Within this pa- per's context, task might be related with different agent (potentially different body types), environ- ments and reward mechanism. Since the original system variables might have different dimensions, task-specific modules map them into a infforn-dimmesional latent space (e Eq 2 and reverse operations. The backbone has a two-directional, morphology- transformer structure, including morphology-awand encoders and a casual transformer. Architecture details are specified in Fig 2.
prediction
target
pertraining
oorphoiogical Prompt
Casual Transformer
finehuning
backbone
behavior imitation
world model prediction
BOS
8g
d1
on-policy
task--ppeeifc tokenizer
task-ip projector state valuo prodiction
masked
morphology aware
Figure 2: Model structure of ODM and its training paradigm.
Tokenizer: At each time t, observations and actions are first embedded into the latent space
of Embed, CRkK K.
Embed, (o E R af Embed, (a+) K.
Morphology-aware Encoder: Corresponding pose embedding vectors are obtained by traversing the agent's kinematic tree and encoding the morphology by Eg
of
Enc (of);
of MLP. E RR. of = Encar (a.)
Casual Transformer: To capture the morphology difference, we apply the prompt technique as in Xu et al. 2022), but embedding the morphology specifications instead of imitations
Prompt Embed K, m,
(5)
4

---

Under review as a conference paper at ICLR 2023
The casual transformer then translates the prompt and the imput sequence into the output sequence
output =Eney (Prompt, input)
(6)
input := (BOS, sy ag sy
(7)
output :- = ag: 8 0g
8 a5)
(8)
with forward casual time mask. Detailed structure is inherited from GPT2, decoder-only structure as in (Chen et al. 2021 Janner et al. 2021 Zheng et al. 2022).
Projector: The task-specific projectors map latent outputs back to the original spaces:
0₂ = Proja (at),
& = Proj, (8P): V Projy (8C)
(9)
Embed,, Embed,, Embed,, Embed,, Proj.. Proj.. Projy are all modeled as MLPs with LayerNorm and Relu between layers. More detailed configuration can be found in Appendix.
4.2 TRAINING PARADIGM
ODM has a two-phase training paradigm including pretraining and finetuning, as in Algorithm !
To mimic the learning process of human infant, we design curriculum-basee learning mechanism in which the training dataset transverses from the easiest to the most complicated one. During each epoch, the current dataset is trained in auto-regressive manner with two loss terms:
Pretraining:
L Limitation MSE(a,, af), L predicctoon
MSE &r, sf),
prerriin
TTp imitation
Th prodiction (10)
where MSE denotes the mean-square-error. demonstrations, while prediction encourages the agent to predict future observations
L imittation corresponds the imitation of action from
Finetuning: one extra predict head is activated to predict the state value Vi; this head as long as the very last prediction head of år are employed outputs of actor and critic:
V (st),, à π (st)
(11)
Actor and critic are then trained by PPO with more details introduced in Section A Keeping some extent of T.rrtrriin as auxiliary loss, this finetuning becomes a self-supervised plus model-based RI.
L finetume
1. PPO + 1221 peetrinn
(12)
Algorithm ODD
1: Initialize 0
2: Pretraining:
3:
set num of epoch 0
4: SWITCH between body shapes
5: activate grad of env-specific module of current env, freeze others
6: REPEAT learning from pioneers of different expert levels
7:
self-regressive training on p of mini-batch increment num of epoch save the model checkpoint
8:
9:
10: Finetuning:
load the model the target environment
12: REPEAT iterations:
for each actor do
14:
run current policy æ for 7 timesteps
15:
compute advantage estimates Åo, ...... ÅT end for update 0 with the surrogate L PPO mini-batel stop when converges
17:
18:
prodicctions (t. 0) is masked out, since it meaningless predict the very first, randomly initiated state.

---

Under review as a conference paper at ICLR 2023
5
EXPERIMENTS
5.1
EXPERIMENT CONFIGURATIONS
Bodies, Environments and Tasks: We practice with enormous agents, environments and tasks, to validate the general knowledge studied by ODM. These scenes include:
Body shape: including swimmer (3-joints, foot), reacher (1-joint and one-side fixed), hopper (1 foot), halfcheetah (2-foot), walker2d (2-foot), ant (4-foot), and humanoid on the gym-mujoco platform Fl walker (the agent called ragdoll has realistic humanoid body) on the unity platform (Juliani et al. 2018); and finally unimal, mujoco-based enviroment which contain 100 different morphological agents (Gupta et al. 2021).
Enviromment: flat terrain (FT), variable terrain (VT) or escaping from obstacles.
Task: pure locomotion, standing-up (humanoid), or target reaching (reacher, walker).
Baselines: We compare ODM with four baselines, each representing different learning paradigm:
Metamorph: a morphological encoding- bbaedd online learning method to learn a universal control policy (Gupta al. 2022).
DT: As with the expert action inference in Lee al. (2022) and deal with continuous space as in (Chen et al. 2021). We name it DJ in the following sections for abbreviation.
offline learning baseline, we implement the decision transformer
PPO: The classical on-policy RL algorithm Schulman et al. (2017). Code cloned from stable-baseline3 in which PPO is in the actor-critic style.
Random: The random control policy by sampling each action from uniform distribution from its bounds. This indicates the performance without any prior knowledge especially for the zero-shot case.
For purpose of pretraining, we collect offline data samples of hopper, halfcheetah, walker2d and ant from D4RL (Fu et al.) (2020), as sources of pioneer demonstra- tions. For these environments, D4RL, provide datasets sampled from agents of different skill levels, which corresponds to different learning pioneers in our framework, including expert, medium-experr, medium, medium-replac and random. We also train some baseline expert agents and using them to sample offline dataset walker and unimal. These dataset contains more than 25 million data sam- ples, with statistics details shown in Appendix, Table g Within each curriculum, we also rotate demonstrations from the above pioneers for training, indicated in Algorithm
Demonstrating pioneers:
5.2 EXPERIMENT RESULTS
Pretraining: Model is trained with datasets of hopper, halfcheetah, walker2d, ant, walker and uni- mal, from the easiest to the most complex. Figure 3 shows the loss plot. One can observe that the training loss successfully converges within each curriculum course; although its absolute value occasionally jumps different levels because of the environment (and the teacher) switching. Val- idation set accuracy is also improved with walker and unimal as exhibition examples in Figure 3.
Online Experiments: To make the online learning faster, we use 32 independent agents to sample the trajectory in parallel, with 1000 as the maximum episode steps. Experiment continues more than 1500 iterations after the performance converges. Figure 4 provides quick snapshot of online performances. Comparing with ODM without pretraining, returns ODM are higher at the very beginning, indicating knowledge from pretraining is helpful. As online learning continuous, the performance degrades slightly until finally grows again, and converges faster than the other two methods, although the entire training time (pretraining plus finetuning) longer.
During online testing, 100 independent episodes are sampled, and each is fed with unique random seed. We examine the averaged episode return and episode length our evaluation metries Ta- ble 2 shows the online testing performance. One can observe that our ODM outperforms or is at

---

Under review as a conference paper at ICLR 2023
Hopper
HalfCheetah
Walker
Welker2d
Unimal
3
2
5000
10000
15000
20000
Iterations
Figure 3: Time plot of pretraining performance. Orange: training loss. Green: validation MSE of walker; Blue: validation MSE of unimal.
Unimal
CEM (fiiean-peniiin
OCM (Cthorr ladaroci
Mit Micamogg
150
Figure 4: Comparison of averaged episode returns during online experiments. Left: unimal. Right: walker. Curves smoothed and values are rescaled for better visualization.
least similar with Metamorph's performance. It is worth noting that DT does not work for unimal, indicating the limitation pure offline method with changing agent body shapes.
Table 2: Averaged Episodic Performance online locomotion environments.
Metrie
Emw.
ODM
Metamorph
DT
PPO
Randons
waller
331 88 2800000
303 935:2270.8 16
252.742281.18
265 235:2332275
0.55
length
walker
133.29 : 55.88
128, 128.42$33.50
386.77
128 56.56-425 34.06
10.54::1.55
unimal
917.55 84
931.922333.21
347. 66:447
894
3941399--0033 50 34
321.98::1.44
Performaate
has substantill devitiins since walker forward process noise impiemeened.
Few-shot Experiments: We examine the policy transferability by providing several few-shot exper- iments. Pretrained ODM is loaded in several unseen tasks, which are listed in Table As few-shot test, online training only lasts for 500 steps before testing. ODM obtains the best performance except humanoid on flat terrain, indicating ODM has better adaptation ability than MetaMorph.
Zero-shot Experiments: Zero-shot experiments can be conducted by inferencing the model directly without any online finetuning. The unimal environment allows such experiment in which the flat terrain (FT) can be replaced by variable terrain (VT) or obstacles. Results are shown in Table 4 be observed that ODM reaches state-of- the-art performance for zero-shot tests, indicating that ODM has strong generalization ability by capturing general high-level knowledge from pretraining, without any prior experience.

---

Under review as a conference paper at ICLR 2023
Table 3: Performance in few-shot experiments.
Metamorph
PRO
Rankom
unimaal
obstacle
1011.86-F179J8
128715
932.34.5 19:45
2.0
unimal
VT
580.102-41.2 23
499 399.58
310,02432.955
487+8.
155.55+11697
143.01++11.31
13.331
0.14::2.000
relum
rescber
lurget reaching
-32.97=5.1
-33.28--4.58
24,28-4-.30
45.96---0.15
huntnnoid
FT
359 119.90--54444
360.874551 22
360.70-250.24
108.15
bumanodd
standup
76388.12 14388.124-906.00
78750041 +: 897701
75033,74 4405--55552 995.20
39921.774-4111
4-11-
uninal
obstacle
7NL.38
unimal
VT
780.23:1:89 02
764,68:1:77.73
$24, $24.10:2599.9
202.00888.77
length
swimmer
reacher
reaching
FT
1000
50
1000
90
1000
50*
1000
50"
humaneid
08.10:13.15
3.995
67.59
22.15:20117
humanosd
standup
1000
1000
1000
1000
The official reather enviromment has maximum spisode length limit of
Table 4: Performance in zero-shot experiments.
Metrii
Eay.
Task
GDM
Motamerph 52.1
DF
Random
unimal
obsnacl
1271.70
3.3
2.00144
return
unimal
VT
34.448
23.2-3.21
8.22
-48778
length
unimal
obstacle
VT
698.80--09.49
750.999286.25 25
756.90475.16
664-63---22225
885.13+88.54
228. 088.04255.7
542700338.70
822006665.8
Ablation Studies: To verify the effectiveness of each model component, we conduct the ablation tests for ODM with only online finetuning phase (wo pretrain) and with only pretraining (wo fine- tune); within the pretraining scope, we further examine ODM without the curriculum mechanism (wo curriculum) and morphology prompt (wo prompt). The DT method could be viewed as the ablation of both L and LPPO, so we do not list the ablation results of these two loss terms. We conduct the ablation study on unimal (all 3 tasks) as well as walker, with results shown in Table 5 Results shown that ODM is still the best on all these tasks, which indicating both learning from others' imitation and self-experiences are necessary for intelligence agents.
Table 5: Performance in ablation studies on unimal and walker.
Metric
Emv.
Tisk
ODM
wi preininn
we finetume
wo curricllmm
we promptt
unimal
PT
1197.22228.0
131122
403.12 84.55
484.44 73.43
453.11=00.13
unimal
obstacle
1611861119938
592.96 89
30.65:34911
78.34132.55 52
75.23:1:3.211
VI
580.10++-1.23
70.25331331
70.014:32.05
walker
FT
331.88 250.96
313.49 260
11264-12.23 72.23
109 1093441-6.119
111.82:-69976
unimal
FT
917.85
795.78117935
436,6622000 70.0
4332411 7000
436,66:70.0
obstuele
527.851153.25
$54.80.1.50 02
222.12317.0
230.761115.08
lengeh
unimal
VT
780.23-4:00002
526.20:334.61
309.75-15.41
205.51::33338
204. 14:114:30661
walker
FT
133,29-135.88
105:2911:4013
84.32-33.17
80.41:333.41
$2. 16:1:55443
Typical Visualizations: Generalist learning not only aims to improve the mathematical metrics, but also the motion reasonability from human's viewpoint. It is difficult for traditional RL to work on this issue which only solve the mathematical optimization problem. By jointly learning other agent's imitation and bridge with the agent's self-experiences, we assume ODM could obtain more universal intelligence about body control by solving many different types of problems. Here we provide some quick visualizations about generated motions of ODM, comparing with the original versions !
By examining the agent motion's 's rationality and smoothness, we first visualize motions of trained models on the walker environment. Since the walker agent has humanoid body (the 'ragdoll") such that reader could easily evaluate the motion reasonability based on real life experiences. Figure 5 exhibit key frames of videos on the same time points. In this experiment, we force the agent starts from exact the same state and remove the process noise. By comparing ODM (the bottom line) with PPO (the upper line), one can see the ODM behaves more like human while PPO keep swaying forwards and backwards and side to side, with unnatural movements such as lowering shoulders and twisting waist.
We compare the motion agility by visualizing on the unimal environment, in which the agent is encouraged walk toward arbitrary direction. Figure 6 comparing ODM with Metamorph. Meta- morph wastes most of time shaking foots, fluiding and gliding, therefore ODM walks longer distance than the Metamorph, within the same time interval
FFull version of videos can be found on the website ht /ba imaxi iiaaxsssi shi . thub. o FFguree grids could help reader recognize the comparison although the video is more obvious.
8

---

Under review as a conference paper at ICLR 2023
PPO
ODM
Figure 5: ODM improve motion fluency and coherence in the walker environment. Key frames are screened on Second 1, 2, 5, respectively. Video can be found on the website.
4
Metamorph
ODM
Figure 6: ODM improve motion agility of a typical unimal agent. Key frames are screened evenly from 30-second video. Video can be found on the website.
6 DISCUSSION
Our work can be viewed early attempt of embodied intelligence generalist accommodated for varied body shape, tasks and environment. One shortcoming of current approach is that ODM still has task-specific modules (tokenizers and projectors) for varied body shapes. By using some self-adaptive model structure (e.g. Hypernetwork) in these modules, it is possible to use one unified model represent the generalist agent. Another potential improvement is to add the value/return prediction into the sequence modeled by the casual transformer. That is, the agent is able to estimate "the value of its action' before the action is actually conducted, which is also known as 'metacog- nition'. The last interesting topic is the potential training conflict when training switch from offline to online. That might improved by some hyperparameter tuning (out of this paper's scope), e.g., some warmup schedule of PPO during finetuning; but could also be improved by different model architecture which could better accommodate knowledge learning from offline and online phases.
7 CONCLUSION
In this paper, motivated by the intelligence development process in the natural world, propose learning framework to learn a universal body control policy in arbitrary body shapes, environments and tasks. We combine ideas of learning from others, reinforcing with self-experiences, as well as the world model recognition. To achieve this, we design two-dimensional transformer structure which first encode the morphological information of agent states and actions at each time step, then encode the time sequential decision process to formulate the policy function. A two-phase training paradigm designed accordingly which the agent first learns from demonstrations pioneers on different skill levels, and from different tasks. After that, the agent interact with its own environment and further reinforce its skill by on policy RL. Online, few-shot and zero-shot experiments show that methodology is able learn some general knowledge for embodied motion control. We believe this work could shed some light on the embodied intelligence study when some kind of generalist intelligent is wanted.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Picter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 15084-15097. Cur- ran Associates, Inc., 2021. URL U neurips httpp/sscccpppppp/p////a pttp::/bb.bcc11100o00ppppppe.rpd
fi
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: datasets for deep data-driven reinforcement learning. CoRR, abs/2004.07219, 2020. URL https://arxiv.
brq/abs/2004. 07219
Scott Fujimoto, David Meger, and Doina Precup. Off-pooicy deep reinforcement learning without exploration. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 2062. PMLR, 09-15 Jun 2019. URL https::/pproceediin
.mlr.
/V2////99/0 press/ / f imot o 99 html
Agrim Gupta, Silvio Savarese, Surya Ganguli, and Li Fei-Fei. Embodied intelligence via learning and evolution. Nature Communications, 12, 2021. URL https://6di.org/10.111
Agrim Gupta, Linxi Fan, Surya Ganguli, and Li Fei-Fei. Metamorph: Learning universal controllers with transformers. In The Tenth International Conference on Learning Representations (ICLR), 2022.
David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Asso- ciates, Inc., 2018. URL
neuripseeeppppp httppps..cppppppe cc./paper/2218/iieee pdf
Matej Hoffmann and Rolf Pfeifer. The implications of embodiment for behavior and cognition: animal and robotic case studies. CoRR, abs/1202.0440, 2012. attp::/axxiv.orr
abs/1202. 0440
Wenlong Huang, Igor Mordatch, and Deepak Pathak. One policy to control them all: Shared modular policies for agent-agnostic control. In ICMI., 2020.
Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big se- quence modeling problem. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 1273-1286. Curran Associates, Inc., 2021. URL https: //pp:ceeeee////op/ppto ngs neurips. pdff
Arthur Juliani, Vincent-Pierre Berges, Esh Vekay, Yuan Gao, Hunter Henry, Marwan Mattar, and Danny Lange. Unity: A general platform for intelligent agents. CoRR, abs//809.02627, 2018.
URL http:://axii..tt// org/ abs/ /1////119.... 02627
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q- learning. 2021.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative gelearning for offline reinforcement learning. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.),
Advances in Neural Information Processing Systems, volume 33, pp. 1179 1191. Curran Asso- ciates, Inc., 2020. URL httpp:::/ppocceediiiiiii neuri ips. cc ccc/ppppr/222//11ee 6.2df (e02:205112.65533322122233883883
Vitaly Kurin, Maximilian Igl, Tim Rocktäschel, Wendelin Bochmer, and Shimon Whiteson. My body is cage: the role of morphology in graph-based incompatible control. In International Conference on Learning Representations, 2021. URLhttps: http://ppnneeeeeeeeeeeee net forum? id=N3zUDGN510)
10

---

Under review as a conference paper at ICLR 2023
Kuang- Huei Lee, Vincent-Pierre Berges, Esh Vekay, Yuan Gao, Hunter Henry, Marwan Mattar, and Danny Lange. Multi-game decision transformers. Arxiv, abs/2205.1524lv1, 2022. URL
btttps://arxe www./saaa...o wwwps////..5 0ttp//2222..111111 15241v1
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski. Stig Pe- tersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep rein- forcement learning. Nature, 518(7540):529- 533, February 2015. http://dx. doi. www.orgg110 10. 10..10334 1038 naturallee........ 12236
ISSN 00280836. URL.
Scott Reed, Konrad Zoina, Emilio Parisotto, Sergio Gomez Colmenarejo, Gabriel Barth-Mare Alexander Novikov, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg. Tom Ec- cles, Jake Bruce, Ali Razavi, Ashley Edwards, Nicolas Heess, Yutian Chen, Raia Hadsell, Oriol Vinyals, Mahyar Bordbar, and Nando de Freitas. A generalist agent. Transactions on Machine Learning Research, 11, 2022.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. In Yoshua Bengio and Yann LeCun (eds.). 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. URL http:
//aaxiv.org//aaa awtp//bs///ee a50.ea/./......... 02438
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1707.06344 2017. URL http::/erxiv.cr abs/
1707.06344
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of the 30th Inter- national Conference on Neural Information Processing Systems, NIPS 17, 2017.
Philipp Wu, Alejandro Escontrela, Danijar Hafner, Pieter Abbeel, and Ken Goldberg. Daydreamer: World models for physical robot learning. In CoRL, 2022.
Mengdi Xu, Yikang Shen, Shun Zhang, Yuchen Lu, Ding Zhao, Joshua Tenenbaum, and Chuang Gan. Prompting decision transformer for few-shot policy generalization. In Kamalika Chaud- huri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Pro- ceedings of the 39th International Conference on Machine Learning. volume 162 of Proceed- ings of Machine Learning Research, pp. 24631-2466 14645.. PMLR, 17-23 Jul 2022. URL https: //proceedings mlr www....res//vll...
xu22g. 1e222g.htt. html
Qinqing Zheng, Amy Zhang, and Aditya Grover. Online decision transformer. In Kamalika Chaudhuri, Stefanic Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Pro- ceedings of Machine Learning Research, pp. 27042 27059. PMLR, 17-23 Jul 2022. URL pttpss://proceedlngs.to mlr poress/ve 62/ /2//hhgg22ccchm


Reviews:
[{"review_id": "uJ2oEvGtPB", "reviewer": "Reviewer_t6eV", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an architecture and method for online training for robotic datasets ", "summary_of_the_review": "The paper proposes a mechanism to learn from multirobot learning and tests on sim environment in an online learning fashion.", "strength_and_weaknesses": "Strengths:\n- The proposal is complete and can be trained on various sim environments\n- Apparently able to learn from multi robot data\n\nWeaknesses:\n- Tested on sim only. Online learning is too slow to be practical in real.\n- The paper skips over a lot of prior work in online learning, relevant work section is pretty basic and boiler plate\n- Visual examination of frames is not exactly relevant. The dynamics of a human and humanoid may be different in sim to make difference joint configurations optimal", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Qpf23jyO5hJ", "reviewer": "Reviewer_UikL", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes Online Decision MetaMorphformer (ODM), a pipeline based on transformer structure to encode the morphology and temporal proprioceptive & exteroceptive information to learn from offline trajectories and finetune from environment interaction. This policy shows generalization in unseen settings with uneven terrain and obstacles, as well as few-shot finetuning. The state is modeled in terms of proprioceptive and exteroceptive observations and the actions are torques for each DOF of the joint. The trajectory is passed through a casual transformer to predict the subsequent next state and action per timestep. This is pre-trained to minimize MSE between the states and action predictions with respect to a dataset, sorted from the easiest to the hardest task for curriculum learning.8 different body shapes, interactive environments, and few-shot & zero-shot tasks are considered to learn and evaluate the policy. The proposed approach (ODM) is compared to MetaMorph, DT, PPO, and random baseline.\n\n", "summary_of_the_review": "Overall, the paper presents an interesting integration of supervised pre-training and online RL fine-tuning over morphologically different agent trajectories and demonstrates few-shot and zero-shot transfer to unseen shapes and environments. While the ideas for state-action representation, transformer-based modeling, and two-phase training paradigm existed in common knowledge, the paper presents an interesting evaluation with pure offline (DT) and online (MetaMorph) variants to address the same problem. It opens a broader research question to the community on how to reduce the conflict between the learning signal between two training phases across differing morphological shapes and environments. ", "strength_and_weaknesses": "## Strengths\nThe paper presents an integrated pipeline of pre-training from offline trajectory dataset & finetuning with RL for different body shapes. The state-action representation based on Gupta et al 2022, handles proprioceptive and exteroceptive observations and continuous-valued torques for each DoF in every joint of the agent. While existing papers have used such a general framework before, this work demonstrates how it can pre-train with supervised learning and fine-tune with RL. \nThe evaluation highlights how a (1) pure online method like MetaMorph could take some time to improve the returns as compared to pretrained ODM in unimal and walker, (2) a pure offline method like DT can not adapt to changing body shape to unimal.   \n\n\n## Weakness\nThe pretraining with supervision and finetuning with RL is quite common and may work in certain environments, and the work showcases this sensitivity in Fig 4. However, the work does not introduce any specific inductive bias in modeling or loss to handle the potentially conflicting learning signal in offline pretraining and online RL. The concern is that the shown results for combined pretraining + finetuning are currently highly sensitive to training tricks and schedules used and might not generalize.  \n\nHow does the two-phase training for ODM compare to MetaMorph in terms of the wall clock time? The claim that ODM starts better in Fig. 4 is unclear as the pretraining time is unknown.  \n\nThe section on limitations or concrete future directions is missing.  ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "OCJeQTDOHg9", "reviewer": "Reviewer_Vvnb", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a transformer architecture that can deal with multiple environments, tasks, and state-action spaces for a locomotion environment. Specifically, the transformer takes a description of morphology as a prompt and draws multi-head attention over a sequence of state-action pairs. For training, the paper proposes to imitate experts from simple tasks to complex tasks through a hand-designed curriculum and fine-tune via on-policy RL (PPO). The result shows that pre-training helps, and the proposed architecture can generalize to unseen tasks in few-shot and zero-shot settings. ", "summary_of_the_review": "While this paper makes an attempt to solve an important problem, the presentation of the paper needs to be improved, and the results are not convincing enough to be presented at ICLR. ", "strength_and_weaknesses": "[Strength]\n* Proposes a potentially interesting architecture. \n\n[Weakness]\n* The presentation of this paper needs to be improved significantly. (see the next section).\n* The improvement over the baselines (especially MetaMorph) is not statistically significant (Error bars highly overlap). \n* Many important experiments are missing. For example, there is experiment showing generalization to new morphologies, which is important for supporting the main claim of the paper. There is no ablation for the use of curriculum in the pre-training phase, the use of morphology prompt, and the use of each loss term (e.g., reconstruction loss). The paper's claims are not well-supported without these experiments.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "3qX16j703lJ", "reviewer": "Reviewer_o4Ch", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces Online Decision MetaMorphFormer (ODM) to learn a universal body control policy in arbitrary body shapes, environments and tasks.\nThe paper proposes a time-morphology transformer architecture which take advantage of both time and morphology dependency for a general purpose policy. Also, motivated by cognitive and behavioral psychology, this paper combines learning from others based on curriculum learning with on-policy RL. The experiments show that ODM is adaptive in the real world and also learns generalized knowledge, such as motions with rationality, smoothness, and agility.", "summary_of_the_review": "This paper introduces new learning architecture called Online Decision MetaMorphFormer (ODM) to learn a universal body control policy in arbitrary body shapes, environments and tasks.\nObjectives of the paper is clear, and experiments show better performance.\nWith a time-morphology transformer, earned motions have more rationality, smoothness, and agility as well\n", "strength_and_weaknesses": "The strength of this paper is as follows.\n1 new idea of a time-morphology transformer, ODM,  that takes advantage of both time and morphology dependency for a general purpose policy.\n2 new idea of learning from others and on-policy RL using curriculum learning and PPO\n3 generalization performance for different environments\nAlso, this paper shows improved learned motions rationality, smoothness, and agility to support claims.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject