Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors propose NeMF (neural motion fields), an implicit neural representation for human motion. Similar to NeRF, they demonstrate that they can well encode a mapping function from time-step `t` to poses. They also learn a generative model by conditioning this on a latent variable `z` to encode different styles/kinds of motion.

They demonstrate that NeMF can well encode a single motion sequence of varying lengths. Their generative samples yield good quality motions, as demonstrated quantitatively and qualitatively compared to previous methods. They also perform a few downstream tasks such as motion in-betweening, spare keyframe in-betweening, and re-navigating tasks by formulating them as a latent space optimization task (find `z` that best approximates the inputs).

strengths_and_weaknesses: Strengths:
- The problem setup and formulation makes sense. The authors demonstrate good performance in both single motion encoding and the generative model.
- The implicit neural representation has several advantages (as shown by temporal sub-sampling and good motion reconstruction and synthesis results).
- The authors have done several evaluations on different tasks. The supplemental material show convincing qualitative visualizations of the model.

Weakness:

I think the paper suffers a bit from a lack of details and clarity in writing (detailed below).

Otherwise, I think the paper's contributions would be a good addition to the community.

questions: I’m curious to know more about latent-space optimization and convergence. Is the optimization stable, and does it converge when not using custom initialization? Also, for the re-navigating task, is the model able to optimize and generate complex paths (say writing out the word NEURIPS). Some more details on the optimization setup would be helpful.

It’s interesting to note that NeMF has significantly better foot-skate numbers when compared to HuMoR even though HuMoR explicitly models ground contacts. Could the authors comment more on this phenomenon? Is it due to diverging motion?

I think some places in the text are missing adequate explanations and justifications (especially in the methods and experimental section). Some places are:
- Before Eq 2 maybe a line or two saying you are only interested in predicting the joint angles (and not the full X_t as defined previously).
- Eq 7 and 8, it’s unclear why you move back to X and X_t notation for Eq 7. It’s also not clear in Eq 8 what lower-case x is, it’s not defined previously/after the Eq.
- Details/setup of how the AIST++ motion in-betweening was done is missing.
- Typos: Eq 1 should have an angular velocity term (last term)

limitations: Yes, the authors have included both limitations and broader impact sections in the main paper.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes an implicit neural model for kinematic motions. Compared to the seminal NeRF model, which learns a continuous function over spatial coordinates, the proposed NeMF framework learns a continuous function over temporal coordinates. Meanwhile, NeMF outputs a vector of the kinematic pose compared to the color and occupancy information in NeRF. The paper also proposes to extend NeMF beyond only encoding a single motion sequence - following the recent trend on the generalization of implicit neural models. This is achieved by adopting a VAE framework to make NeMF a generative model controlled by a latent code input. The model is trained following the standard VAE training objective including the reconstruction loss and KL loss. The proposed approach is evaluated on several motion generation task (i.e. motion in-betweening and re-navigation) using the AMASS dataset, and compared with several prior methods including HuMoR and HM-VAE. Results have shown that NeMF have achieved better performance on motion reconstruction and motion synthesis (Tab. 3, 4, and 5).
strengths_and_weaknesses: ### Strengths
- [Original] The proposed framework does look novel - by adapting the impactful NeRF model to the domain of kinematic motions. It is certainly interesting to see how these models work not just on static scene reconstruction but also on reconstructing motions.
- [Quality] The method is technically sound. The key idea, as mentioned above, of extending NeRF to the temporal domain, is quite simple and well executed.
- [Clarity] The presentation is overall clear. The core idea as well as the technical details are well presented and easy to follow. Math notations have been used properly.

### Weaknesses
- [Clarity/Contribution] From the current presentation, it is unclear how the proposed work relates to the work of [20]. How does this paper add over [20], and why is [20] not discussed in the related work section?
- [Quality] Although the theoretical framework looks nice, I'm still a little bit skeptical on how the latent space optimization would work practically. It seems that this heavily depends on how smooth the latent space of $z$ is. If the space learned from VAE is not sufficiently smooth, the optimization (e.g. for Eq. 9 and 10) will always be trapped in useless local minimum. Would this be an issue at all? It would be helpful to somehow show the landscape by visualizing the pose transformation as one traverses in the latent space.
- [Clarity] The detail of the generative task for Sec. 4.2.2 and Sec. 4.2.4 is missing. What is the generative task here, i.e. input and output? Is it trying to predict the pose at arbitrary $t$ given some keyframes from an unseen (test) sequence, so the approach is to perform latent space optimization for $z$ and then run forward pass on the evaluated $t$s?

### Justification of Rating
Overall this looks like a solid submission with decent novelty and contribution. I just have some concerns on the comparison with prior work as well as some technical details as listed below.
questions: ### Questions
- In general I'm wondering how the latent code handles translation in time. For example, in the current framework, the exact motion clip (e.g. with a time interval [0, T]) after translation (e.g. to interval [-T/2, +T/2]) will have a completely different latent code. This seems a bit awkward and somewhat redundant. I wonder whether there has been any observed pattern on the latent code for translation in time, also, whether one should eventually decouple the translation dimension from the latent code.
- [Eq. 5] Should there also be a loss term for global root translation? The supplementary Sec. 1.1 mentions a loss function for the global motion predictor (i.e. $\mathcal{L}$ in Eq. 2). Is that also part of $\mathcal{L}_{rec}$ here? If not, how is the global motion predictor loss applied in the framework?
- [Tab. 3] How come some of the numbers in "Ours" is exact to those in Tab. 2 (e.g. 5.988 MRE, 2.870 MPE, 6.157 MOE, and 29.692 MTE) while some are different (e.g. 6.508 versus 2.073 FID, 2.118 versus 2.774 Diversity, and 0.566 versus 0.573 FS)?

### Suggestions
- [L171] "In our experiments, $z_l$ and $z_g$ are initialized using the SLERP data to facilitate convergence." -> Detail is unclear. Is SLERP first used to perform interpolation in the joint angle space, and then the joint angles are passed through the encoder to obtain the latent code?
- [Fig. 2] I wonder whether the absolute velocity values are the best metric to look at for this motion reconstruction evaluation. Would it be better to evaluate on the error (of both position and velocity) rather than the absolute value? The issue may be that it is not possible to get the ground-truth position and velocity at frame rates higher than 30 FPS. Then would it be reasonable to train on 15 FPS and evaluate on 30 FPS then just for this experiment?
- [L273] "Compared to HuMoR and our model, it is **obvious** that HM-VAE" models an over-smoothed latent space, thus filtering high-frequency details in both the reconstructed and synthesized motions, ..." -> Not so obvious. Following the weakness point for [20] above, it is unclear how [20] works and why it "models an over-smoothed latent space" while HuMoR and "Ours" don't.
- [L274] "..., making their results worse in both the reconstruction and synthesis task." -> Not precisely. HM-VAE seems to outperform HuMoR sometimes, e.g. on MRE, MPE, MOE?
- [High level] So far the paper still utilizes a low dimensional vector at the output for obtaining human pose and shape. How does it compare to making the model "fully implicit", i.e. on not just the temporal but also the spatial domain (x, y, z) as well, like in the original NeRF and other implicit shape models? I guess as the first step, it may just be easier to rely on a forward function to generate the mesh from the pose vector, which is why the authors go in this route first. Yet a "full implicit" model will make the framework looks more elegant and potentially better integrated with the efforts on the NeRF side.
- [High level] The paper has shown results on an array of "offline" tasks, where the motion of the full sequence can be pre-solved through a latent code and fixed thereafter. Meanwhile, I would imagine an "online/interactive" variant of the NeMF framework would be useful, especially in many gaming applications (e.g. real-time character control, etc.). Are there ways to adapt the current framework to an interactive setup? Perhaps one can run the latent space optimization and motion generation in a continuous loop by streaming motion generation in short time segments (e.g. 1 sec)? If so, it would be nice to add some mention to such extensions.

limitations: Limitations and potential negative societal impact are addressed in Sec 5.1 and 5.2 respectively.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: In this paper, the authors propose a novel representation of human motion as neural motion fields. Inspired by the success of neural fields in other areas, this work learns an implicit function to represent motions. This representation can help to sample diverse and natural motions and facilitate various downstream applications. The experiments demonstrates the ability of their method.
strengths_and_weaknesses: Strengths:
+The authors proposed an INR-based model to learn motions. The model is continuous and can generate sequences of arbitrary frame rates in principal.
+Non-autoregressive generation. The function formulation allows the generation of poses at specific time steps directly without first generating frames before them while preserving the smoothness of the full sequence.
+The paper gives extensive experiments on the efficacy and applications of its proposed model.

Weaknesses:
-The formulation of the VAE is very similar to HM-VAE. The introduction of INR is novel and very interesting, but it seems the combination of the two is simple.
-The formulation only uses a latent z to decide the motion type, so for each task the optimization process only finds a deterministic z, which gives a single generated sequence.
-In Figure. 5 the keyframes given for the HM-VAE example are different from the others. Some typos remain in the paper (Section 4.1.1 for example). 

questions: Please refer to weakness.
limitations: Please refer to weakness.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors propose to formulate human motion as a generative implicit representation. The authors demonstrate the effectiveness of such approach. The authors showcase the model in various tasks.
strengths_and_weaknesses: Strength:
* Overall, I love everything in the paper. Things are presented in a clear and coherent way, making things easy to follow. 
* The baselines and experiment comparisons are all very solid. The authors did a great job explaining the baselines and comparing with them.
* The authors did an excellent job explore using the model in various scenarios. This is very informative and showcases the robustness of the model. 

Weakness:
* Important related work missing: Occupancy Flow: 4D Reconstruction by Learning Particle Dynamics, ICCV 2019. The part where the authors talk about " Inspired by their success, our key insight is to interpret time as the parameter of a motion function and learn the landscape of the spatio-temporal kinematic manifold as an implicit motion field parameterized by temporal coordinates." This is exactly what the highly-cited prior work has done. Though the prior work differs in motion representation, it should be discussed.
* And a nitpick for the previous point's quote is "implicit motion field" should be "implicit motion representation". 
questions: I don't have any specific technical question. 
limitations: The authors do a great job in this aspect.
review_title: null

Meta-review: metareview: All reviewers are in favor of accepting the submission post rebuttal.  The AC agrees.   Please further revise the paper based on the reviews, in particular the very constructive comments from reviewer wNce.
confidence: Certain
Decision: Accept