Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes an architecture and method for online training for robotic datasets 
strength_and_weaknesses: Strengths:
- The proposal is complete and can be trained on various sim environments
- Apparently able to learn from multi robot data

Weaknesses:
- Tested on sim only. Online learning is too slow to be practical in real.
- The paper skips over a lot of prior work in online learning, relevant work section is pretty basic and boiler plate
- Visual examination of frames is not exactly relevant. The dynamics of a human and humanoid may be different in sim to make difference joint configurations optimal
clarity,_quality,_novelty_and_reproducibility: Clarity:
- 2/5

Quality
3/5

Novelty
1/5

Reproducibility
3/5
summary_of_the_review: The paper proposes a mechanism to learn from multirobot learning and tests on sim environment in an online learning fashion.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes Online Decision MetaMorphformer (ODM), a pipeline based on transformer structure to encode the morphology and temporal proprioceptive & exteroceptive information to learn from offline trajectories and finetune from environment interaction. This policy shows generalization in unseen settings with uneven terrain and obstacles, as well as few-shot finetuning. The state is modeled in terms of proprioceptive and exteroceptive observations and the actions are torques for each DOF of the joint. The trajectory is passed through a casual transformer to predict the subsequent next state and action per timestep. This is pre-trained to minimize MSE between the states and action predictions with respect to a dataset, sorted from the easiest to the hardest task for curriculum learning.8 different body shapes, interactive environments, and few-shot & zero-shot tasks are considered to learn and evaluate the policy. The proposed approach (ODM) is compared to MetaMorph, DT, PPO, and random baseline.


strength_and_weaknesses: ## Strengths
The paper presents an integrated pipeline of pre-training from offline trajectory dataset & finetuning with RL for different body shapes. The state-action representation based on Gupta et al 2022, handles proprioceptive and exteroceptive observations and continuous-valued torques for each DoF in every joint of the agent. While existing papers have used such a general framework before, this work demonstrates how it can pre-train with supervised learning and fine-tune with RL. 
The evaluation highlights how a (1) pure online method like MetaMorph could take some time to improve the returns as compared to pretrained ODM in unimal and walker, (2) a pure offline method like DT can not adapt to changing body shape to unimal.   


## Weakness
The pretraining with supervision and finetuning with RL is quite common and may work in certain environments, and the work showcases this sensitivity in Fig 4. However, the work does not introduce any specific inductive bias in modeling or loss to handle the potentially conflicting learning signal in offline pretraining and online RL. The concern is that the shown results for combined pretraining + finetuning are currently highly sensitive to training tricks and schedules used and might not generalize.  

How does the two-phase training for ODM compare to MetaMorph in terms of the wall clock time? The claim that ODM starts better in Fig. 4 is unclear as the pretraining time is unknown.  

The section on limitations or concrete future directions is missing.  
clarity,_quality,_novelty_and_reproducibility: The paper is quite well structured but needs refinement. The code, hyperparameter, and environment details for reproducibility are provided in a hyperlink https://baimaxishi.github.io/ and the appendix.   


### Clarification questions:
- Writing in section 3.1 is a little unclear, in terms of what aspects are differently modeled than Gupta et al 2022.   
- Line 4 in Algorithm 1: switch between 6 ~~environments~~ body shapes?
- Fig 2 is unclear and has typos. How can the same encoder/decoder (denoted by the same color) be used for state and action representation, when they have different dimensional inputs? Eq 9 and 10 are not evident in the figure.
 ~~pertaining~~ pretraining, ~~atenion~~ attention.
- Fig 3, what is the x-axis? 
  
#### References 
----
Agrim Gupta, Silvio Savarese, Surya Ganguli, and Li Fei-Fei. Embodied intelligence via learning and evolution. Nature Communications, 12, 2021. URL https://doi.org/10.1038/ s41467-021-25874-z.
summary_of_the_review: Overall, the paper presents an interesting integration of supervised pre-training and online RL fine-tuning over morphologically different agent trajectories and demonstrates few-shot and zero-shot transfer to unseen shapes and environments. While the ideas for state-action representation, transformer-based modeling, and two-phase training paradigm existed in common knowledge, the paper presents an interesting evaluation with pure offline (DT) and online (MetaMorph) variants to address the same problem. It opens a broader research question to the community on how to reduce the conflict between the learning signal between two training phases across differing morphological shapes and environments.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces Online Decision MetaMorphFormer (ODM) to learn a universal body control policy in arbitrary body shapes, environments and tasks.
The paper proposes a time-morphology transformer architecture which take advantage of both time and morphology dependency for a general purpose policy. Also, motivated by cognitive and behavioral psychology, this paper combines learning from others based on curriculum learning with on-policy RL. The experiments show that ODM is adaptive in the real world and also learns generalized knowledge, such as motions with rationality, smoothness, and agility.
strength_and_weaknesses: The strength of this paper is as follows.
1 new idea of a time-morphology transformer, ODM,  that takes advantage of both time and morphology dependency for a general purpose policy.
2 new idea of learning from others and on-policy RL using curriculum learning and PPO
3 generalization performance for different environments
Also, this paper shows improved learned motions rationality, smoothness, and agility to support claims.

clarity,_quality,_novelty_and_reproducibility: 
Objectives of the paper is well described with clarity. Formulas and architectures are helpful to understand the proposed architecture. This paper proposes new learning architecture and has enough experiments and comparisons with baselines.  
This paper has no codes available.
summary_of_the_review: This paper introduces new learning architecture called Online Decision MetaMorphFormer (ODM) to learn a universal body control policy in arbitrary body shapes, environments and tasks.
Objectives of the paper is clear, and experiments show better performance.
With a time-morphology transformer, earned motions have more rationality, smoothness, and agility as well
review_title: null

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: This paper proposes a transformer architecture that can deal with multiple environments, tasks, and state-action spaces for a locomotion environment. Specifically, the transformer takes a description of morphology as a prompt and draws multi-head attention over a sequence of state-action pairs. For training, the paper proposes to imitate experts from simple tasks to complex tasks through a hand-designed curriculum and fine-tune via on-policy RL (PPO). The result shows that pre-training helps, and the proposed architecture can generalize to unseen tasks in few-shot and zero-shot settings. 
strength_and_weaknesses: [Strength]
* Proposes a potentially interesting architecture. 

[Weakness]
* The presentation of this paper needs to be improved significantly. (see the next section).
* The improvement over the baselines (especially MetaMorph) is not statistically significant (Error bars highly overlap). 
* Many important experiments are missing. For example, there is experiment showing generalization to new morphologies, which is important for supporting the main claim of the paper. There is no ablation for the use of curriculum in the pre-training phase, the use of morphology prompt, and the use of each loss term (e.g., reconstruction loss). The paper's claims are not well-supported without these experiments.
clarity,_quality,_novelty_and_reproducibility: * Clarity: This paper needs a major revision. There are many grammatical errors, unclear notations, and unclear explanations. Some examples are the following.  
  * Why is it called Morphology-aware "Auto-Encoder" when there is no reconstruction loss? 
  * What does $+$ symbol mean in Equation 15?
  * $L^{prediction}$ seems like a reconstruction loss. Why would it encourage the agent to learn a "world model"?
  * X-axis is not defined in the plots.
  * Error bars are missing in the plots. 

* Quality: Many important experiments are missing as pointed out above. 

* Novelty: Although the particular architecture is new, the idea of learning a universal architecture that can handle multiple morphologies and tasks through supervised learning (e.g., GATO) and reinforcement learning (e.g., MetaMorph) has been explored.

* Reproducibility: The paper described hyperparameters in the appendix but not the code. 
summary_of_the_review: While this paper makes an attempt to solve an important problem, the presentation of the paper needs to be improved, and the results are not convincing enough to be presented at ICLR.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Summary:

This paper proposes a transformer architecture (ODM) that can deal with multiple environments, tasks, and state-action spaces for a locomotion environment. Specifically, the transformer takes a description of morphology as a prompt and draws multi-head attention over a sequence of state-action pairs. For training, the paper proposes to imitate experts from simple tasks to complex tasks through a hand-designed curriculum and fine-tune via on-policy RL (PPO). The result shows that pre-training helps, and the proposed architecture can generalize to unseen tasks in few-shot and zero-shot settings.

Strengths:

The reviewers appreciated the novelty of the algorithm for handling multiple morphologies.  The author response addressed some misunderstandings on the application of past works to multiple morphologies (connections to MetaMorph and Gato).  

Weaknesses:

The paper retains a poor attention to presentation detail, and contains both simple typos (casual/causal) and loose technical writing (reviewers Vvnb, UikL).  These errors impede communication with the reader, and significantly reduce the potential contribution of the work.  
Reviewer Vvnb commented "The improvement over the baselines (especially MetaMorph) is not statistically significant (Error bars highly overlap)."  This a big issue, and speaks to a lack of clarity in communication, despite the author's response to this comment.

The experimental results are also troublesome in several ways.  The tables bold differences that do not appear to be statistically significant.  The use of PPO seems to be a weak baseline in comparison to SAC.  Even more troubling is the fact that random seeds are only mentioned in testing and not in training (presumably only one seed).  There are deep flaws either in the descriptions of the experiments in the paper, or in the experiments themselves.
justification_for_why_not_higher_score: Two reviewers indicate reject and two indicate a weak accept.  Reviewer VVnb is senior and provided an insightful review.  After examination of the paper, the area chair believes that the paper in its current form does not present its ideas with adequate clarity for publication.
Decision: Reject