Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes the Motion Diffusion Model (MDM) for multi-modality human motion generation, such as text-to-motion, action-to-motion, and unconditioned generation, etc. MDM is transformer-based and adopts an existing diffusion method that predicts the signal itself at each timestep instead of the noise so that geometric losses could be used. MDM achieves impressive results on several motion generation tasks, and requires lower computation resources at the same time.
strength_and_weaknesses: Pros
- A simple and general framework, along with some good-quality generation results. "simple" is for the model simplicity, and "general" is for the ability to enable different forms of condition (text, action, editing...) 
- Lightweight compared to other multi-modality motion generation works.
- Easy to read and understand.
- Rich demos.

Cons

The necessity of the diffusion model is not clearly presented, except for adding geometric losses. The motivation for using diffusion models which predict the signal itself instead of the noise still needs a discussion. If the reason is that geometric losses could be added, given that we supervise each MDM with the GT $x_0$, it seems that there is no need to stack MDM for the diffusing process. 

- What will the result be when $T=1 or  2$ both for training and testing, compared to $T=1000$?  If there is just a small difference, it seems that the good performance is contributed to the transfomer encoder and CLIP, which has been widely used in many works.

- Another concern is about the low diversity caused by predicting the signal at each step, which seems opposite to the original goal of diffusion models.
 
- Have the authors tried to predict $\epsilon$ and not use geometric losses in MDM? 

Ablation experiments on diffusion models are appreciated and needed since they could help readers understand the motivation for introducing the diffusion model better. 


clarity,_quality,_novelty_and_reproducibility: The writing is clear and easy to understand, but the motivation for using diffusion model (predicts signal) is not clear yet.

The quality is good shown in the demos. However, some ablation experiments are missing (stated above), which is important.

The proposed method is somewhat novel and the code will be published. 
summary_of_the_review: Overall, I lean to accept this paper at this time, since it is a simple and general framework with promising results in several motion generation tasks. However, I'm still confused about the necessity of introducing diffusion here, especially in predicting the signal itself. My rating may change depending on the experiment results I mentioned above.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper presents a diffusion model for synthesizing human motion parameters. The proposed model consists of a diffusion model adapted to the human motion data format, plus task-specific geometric losses. The proposed model is applied to multiple scenarios such as text-to-motion, action-to-motion, motion editing, and unconstrained synthesis. Extensive quantitative and qualitative evaluations show the promising performance gain achieved by the proposed model.
strength_and_weaknesses: Paper strengths:
- The paper presents a novel method, which is among the first to apply diffusion models to human motion synthesis. The method is well-thought, containing multiple non-trivial design choices to make the method work well on human motion data.
- The paper is well-written and easy to follow. The demo is high-quality and well-made.
- The experimental evaluations are thorough and convincingly demonstrate the effectiveness of the proposed model.

Paper weaknesses:
- Intuitively, the synthesized human motion should be varying-length. However, this does not seem to be addressed in the paper. I assume the clip length was fixed and sequences were temporally stretched/squeezed to meet the specified length. It would be great if additional discussions can be added.
- It would be great to see a more detailed analysis of each particular geometric loss term and the influence of its weighting.
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written and has relatively high quality. The presented work appears original, although its arxiv version can be easily located via a search engine, I tend to believe the submission is not a duplicate work if the author lists are the same.
summary_of_the_review: Overall I think this is a strong paper, which presents the first steps of applying diffusion models to human motion synthesis. The presented method is technically sound and contains multiple insightful details. The paper's evaluation is comprehensive and its demonstration is well-executed. My concerns are minor and can be easily addressed. Therefore, my initial recommendation is to accept the submission.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper introduces a denoising diffusion model for both conditional and unconditional human motion (pose sequence) generation. Except for a couple of concurrent arXiv papers, this is the first time diffusion models have been applied to full-body human motion. The paper proposes a transformer-based model that can be conditioned on a text or action embedding in order to synthesize specific motions. The proposed approach makes several design decisions that make the diffusion model amenable to human motion including predicting clean samples rather than noise at each step and applying additional losses to improve motion quality. The model is evaluated on a large variety of tasks including text/action-conditioned generation, unconditioned generation, in-betweening, and motion editing, showing strong performance both qualitatively and quantitatively.
strength_and_weaknesses: 
Strengths:
* The adaptation of the diffusion framework is done in a principled way, with reasonable design choices suitable for human motion modeling. For example, predicting the final clean motion from the network instead of the noise at each step is a subtle but important choice, which allows using geometry penalties during training and easy editing through overwriting at test time. Also using classifier-free guidance to control the influence of conditioning. Though no component is particularly novel (i.e. they are adapted from other domains), their combination gives a strong first pass at diffusion for human motion.
* The approach is novel in the sense that it is the first diffusion model for full body motion (besides concurrent arXiv papers which show diffusion being used in a more limited capacity). 
* The model is demonstrated on a wide range of tasks (text-conditioned, action-conditioned, unconditioned, in-filling, editing). This large task variation really shows the capabilities of diffusion for human motion which may push the community to creatively continue in this direction. 
* Quantitively, the model is competitive or better than several relevant baselines across the evaluated tasks. Qualitatively, the results are quite convincing. Overall, the evaluation is quite thorough.
* Code to be published.

Weaknesses:
* A couple components of the architecture were not clearly justified. First, the use of a transformer rather than the usual U-Net. The intro says the transformer “better fits the temporal and non-spatial nature of motion data” but a U-Net baseline is not compared against in experiments.  Second, why is CLIP is used rather than some other language-only model (as in TEMOS) to get the text embedding? Is the “visual” nature of the CLIP latent space actually helpful for human motion?
* Sec 3 could use a bit more detailed intro to diffusion models for those not familiar, especially since this model is new to human motion.  For example, before Eq (2) it is stated that the model predicts the clean samples rather than the noise, but doesn’t explain that these are equivalent formulations.
* Eq. 7 combines the clean motion predictions from the network for classifier-free guidance. This is a bit different from the original formulation of Ho & Salimans, which combines the noise prediction from the model. Are these formulations equivalent? (perhaps this doesn’t matter in practice).
* For the user study in Sec 4.1. it’s not clear what the users were asked to evaluate. Is it the motion quality? Is it how well the motion matches the text? Ideally both should be evaluated.
* Sampling from the diffusion model is quite slow (~1min for a sequence) as discussed in the conclusion. However, there has been much recent work on speeding up sampling lately, so this can be improved in future work.

Questions, comments, and clarifications (not considered in rating):
* How many parameters are in the architecture? Is the transformer considered “lightweight” because it is faster or smaller than U-Net?
* Intro: saying “non-spatial nature of motion data” is a bit confusing. Joints are positions which are by definition “spatial”. Maybe the authors meant the data is less structured, but even so joints are part of the kinematic tree structure.
* Related work: Gu et al [Stochastic Trajectory Prediction via Motion Indeterminacy Diffusion, CVPR 2022] use a similar diffusion transformer for pedestrian trajectory prediction.
* In practice, how long can generated sequences be before motion quality starts to degrade? Or can it truly handle “arbitrary” lengths?
* If both joint positions and rotations are available in the pose, which are actually used as the output?
* Since there are so many evalution metrics, it would be good to define them in the appendix for easy access.
* It would be interesting to see how classifier guidance works for text and action-to-motion instead of having to re-train separately for each task. 

clarity,_quality,_novelty_and_reproducibility: The paper is generally well written despite a few missing details (mentioned above), and I think the quality of the work is high considering the principled design taken for the proposed diffusion model and extensive experiments. As mentioned above, the work is novel in how it combines previous diffusion components for human motion and I think it will inspire many followup works.

Between the paper details and code (promised to be released), the paper is reproducible.

summary_of_the_review: Overall, the paper clearly introduces and thoroughly evaluates a strong diffusion model for human motion. Despite some small weaknesses, I think the qualitative results and flexibility of the model make it very useful for the community. Therefore, I recommend accept.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a diffusion based motion modeling method that uses transformers a backbone for the diffusion process. The authors train a general conditional motion generation method using classifier free guidance which allows them to use the same model for conditioned and unconditioned generation. In experiments, the authors perform favorably against other motion generation methods in the tasks for unconditional generation, text-conditioned generation, action-conditioned generation, in-betweening, and motion editing.
strength_and_weaknesses: Strengths:
+ New diffusion architecture that combines transformers and diffusion
+ Quantitatively and Qualitatively outperforms baselines in most of the evaluation tasks.
+ Nice visualizations of the diffusion process and results


Weaknesses:

- Comparison between simply U-Net and the proposed backbone.
Given that the architectural novelty is the use of transformers instead of U-Net, I suggest the authors provide an ablation comparing the two design choices. The ablation should compare performance, inference time, and maybe any analysis showing differences of one versus the other. I feel like this is something critical that is missing from the paper, and would be good to have.


- Inference time?
Given that diffusion models are known for being slow, it would be good to provide the time it takes for inference in comparison to other models. This would show how much room for improvement there is in this aspect of the method, and the tradeoffs of using the proposed method versus the performance gain over the baselines.
clarity,_quality,_novelty_and_reproducibility: The paper is clear, and the presentation and results are of high quality. The novelty of the paper seems to be there in mixing transformer backbones with a diffusion process. I don't think there is enough detail to reproduce the work. Things such as number of attention heads, hidden dimension, and other details of the transformer backbone are missing.
summary_of_the_review: All-in-all, this is a nice paper with good results and nice presentation. However, I am slightly learning towards acceptance due to the concerns presented in the weakness section, and lack of detail for reproducibility. Nevertheless, I am looking forward to the author's response and am willing to increase my score depending on it.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper presents a motion diffusion model(MDM) for generating human motion animation given an arbitrary condition or no condition. MDM is a transformer-based approach with light-weight architecture. It adopts an existing diffusion method that predicts the signal itself at each time step instead of the noise so that geometric losses could be used. The model is evaluated on a large variety of tasks both qualitatively and quantitatively.

The strengths include the simple and general framework, the comprehensive evaluation, the promising results, and the well-written content. The main weakness and concerns are some missing details, motivation of using diffusion model, and ablation study. After rebuttal, the authors addressed most of the concerns along with a revision.
justification_for_why_not_higher_score: Although the authors adequately addressed the concerns brought up in the review along with a revision incorporating some components of feedback, they should be sure to incorporate all the remaining feedback in the final version of the paper.
justification_for_why_not_lower_score: All reviewers are positive on the paper. Considering it's among the first to apply diffusion models to full-body human motion synthesis and will inspire followup work, I recommend accept.
Decision: Accept: notable-top-25%