NeMF: Neural Motion Fields for Kinematic Animation
Authors: Chengan He
Jun Saito
James Zachary
Holly Rushmeier
Yi Zhou
Venue: NeurIPS 2022 Accept (2022)
Keywords: Motion Modeling
Implicit Neural Representations
Neural Fields

Abstract: We present an implicit neural representation to learn the spatio-temporal space of kinematic motions. Unlike previous work that represents motion as discrete sequential samples, we propose to express the vast motion space as a continuous function over time, hence the name Neural Motion Fields (NeMF). Specifically, we use a neural network to learn this function for miscellaneous sets of motions, which is designed to be a generative model conditioned on a temporal coordinate $t$ and a random vector $z$ for controlling the style. The model is then trained as a Variational Autoencoder (VAE) with motion encoders to sample the latent space. We train our model with a diverse human motion dataset and quadruped dataset to prove its versatility, and finally deploy it as a generic motion prior to solve task-agnostic problems and show its superiority in different motion generation and editing applications, such as motion interpolation, in-betweening, and re-navigating. More details can be found on our project page: https://cs.yale.edu/homes/che/projects/nemf/.

NeMF: Neural Motion Fields for Kinematic Animation
Chengan He Yale University chengan he@yale. edu
Jun Saito Adobe Research
James Zachary
Adohe Research
jsai to@aooee com com
zachary@adobe.com com
Holly Rushmeier
Yi Zhou
Yale University
Adobe Research
Sally.rushmmier@@yalae edu
yizho@adobe. com
Abstract
We present an implicit neural representation to learn the spatio-temporal space of kinematic motions. Unlike previous work that represents motion discrete sequential samples, we propose to express the vast motion space as a continuous function over time, hence the name Neural Motion Fields (NeMF). Specifically, we use neural network to learn this function for miscellaneous sets of motions, which is designed to be generative model conditioned and a random vector z for controlling the style. The model is then trained as a Variational Autoencoder (VAE) with motion encoders sample the latent space. We train our model with a diverse human motion dataset and quadruped dataset prove its versatility, and finally deploy task-agnostic problems and show its superiority in different motion generation and editing applications, such as motion interpolation, in mn-eetwennnng, and re-navigating. More details can be found on our project page: https: /cs, yale. edu / snte/rojjectae /nemf/.
temporal coordinate
a generic motion prior solve
Introduction
Motion synthesis and editing is a core problem animation and game production, as well as in emerging applications like artificial agents. Traditional algorithms have limited capability of automat- ically producing convincing motions with high diversity and complexity. With the recent availability of large-scale motion capture data, more interest has shifted to deep learning-based methods. People have adapted various deep learning technologies such as Recurrent Neural Networks [34], Rein- forcement Learning [25, 35] and Normalizing Flows [10] for skeletal motion generation. However, those methods are designed as an auto regressvee process that depends on its own past values and some stochastic terms. With this constraint, those methods have to sequentially predict the motion at discrete time steps. Consequently, they cannot just directly infer the motion at certain frames since they must predict the past motion first. Moreover, with this temporally asymmetric design, is always hard for those methods to incorporate the control or editing of future frames and, thus, is difficult apply them to tasks like motion in-betweening.
Inspired by the recent success in neural radiance fields (NeRF) for novel view synthesis [32], we introduce Neural Motion Fields (NeMF) to model the spatio-temppral space of kinematic motions. Given that a motion sequence consists different poses at different time steps, we represent a motion sequence as a continuous function f & + f()) which parameterizes the entire sequence by the temporal coordinate This function can be approximated by a multilayer perceptron network (MLP) whose parameters optimized by minimizing the reconstruction loss between the generated and ground truth motion.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

f(t:
f(t; x2)
Motion Prior
11
12
13
4
ts
Temporal Field
Figure 1: Overview of generative NeMF, which consists of motion prior and a continuous temporal field. By sampling the motion prior, NeMF is able to infer variety of motions with different styles. By sampling the temporal field, NeMF is able generate motion arbitrary frames.
However, the NeMF function defined above overfits one motion sequence. To extend general motion synthesis, as illustrated in Figure 1, we further introduce a random vector & as the conditioning variable to the function as ft; z), which now encodes the mapping from the normal distribution to the manifold spanned by all plausible motions. By varying we expect f to output motion with different styles, and by varying t, we obtain the pose arbitrary time. We train this generative NeMF model in the form of Variational Autoencoder (VAE), where motion sequences are fed to convolutional encoders first obtain the sampling of z, and then z is passed through an MLP decoder for the reconstruction of the whole motion. To the best of our knowledge, our paper presents first continuous and generative implicit motion representation that can sample different types of motion an arbitrary time step.
This neural motion representation can serve generic motion prior solve task-agnostic problems. Following Li et al. [21] who trained hierarchical human motion prior and applied it in different tasks, we demonstrate utilizing a trained NeMF in various applications, including motion interpolation, motion in un-betwennigg and motion re-navigating. We formulate those tasks optimization problems to find the latent variable z that minimize the target energies and restore the complete plausible motion sequences. Although we did not train this NeMF model for any of these tasks specifically, it achieves performance equivalent or superior to task-specific alternatives in our experiments.
In summary, our contributions are as follows:
We propose NeMF that represents the continuous motion field as function over time and design a VAE architecture to train generative NeME
We validate that model can reconstruct and synthesize motion with superior quality and diversity than statc-of-the-an neural motion priors.
We demonstrate NeMF in various offline motion editing and creation tasks.
2
Related Work
To clarify our design choices, we categorize the kinematic motion modeling into two types: time series models and space-time models. The former approach views motions the kinematic pose evolving over time, which is often the choice for interactive applications where the future depends on unknown factors such as real-time user inputs. The latter is our approach, which is suitable for offline applications where we know what to expect for the overall motion within some time range.
2.1 Time Series Models
Time series models are often formulated an auto- auto-regressive model that predicts the future based on current and past observations. The predictions are fed into the model again to make further predictions recursively. GPDM [44] modeled such dynamics of human locomotion over time with Gaussian process. Recently people studied various deep learning-hased auto-regressive architectures, including Recurrent Neural Networks [5, 29, 49], Reinforcement Learning Networks, Neural ODE [16]. Transformers [1,
22, 36] and other attention models [28]. Many works demonstrated success in

---

real-time interactive kinematic character controls [13, 40-42, 47]. The auto- regressive approach is also suitable for embracing the uncertainty of future predictions with generative models, for example, in the form of VAE as in HuMoR [38] and Motion VAE [25], and in the form of Normalizing Flows as in MoGlow [10].
In applications, Harvey et al. [9] used LSTM [11] with positional encoding to achieve the state-of- the-art result for the motion in-betweening task. But it limited to a fixed in-betweening setting where some consecutive past frames need to be given as the seed. One of the key applications of our model is also motion in-betweening, but our method has no constraint on the setting and can generate the missing motion both between motion clips and between sparse keyframes.
2.2 Space-Time Time Models
Different from time series models, another approach is directly model the spatio-temporal kinematic state itself. Such models are often used in combination with a time series model, as in [20] where the dynamic control policy is learned over the GPLVM- spatial kinematic prior. One of our inspirations, Motion Fields [19], can be seen as a variant of this approach where the time-series dynamics are learned over the spatial kinematic model represented directly by the data and a hand- crafted distance function. More recently, space-time neural network models overcame the scalability issue in previous methods due to the availability of large-scale datasets [12, 14, 15, 45, 48]. Zhou al. [51] demonstrated long-term motion in- in-betweening by learning the space-time kinematic prior without supervision using Generative Adversarial Networks (GAN) [7]. Kaufmann et al. [18] also proposed a convolutional network for motion in in-betweening. Our other inspiration, HM-VAE [21], learned generic motion prior for multiple downstream applications. However, their reliance on a temporally convolutional decoder fixes the output frame rate, thus limiting its application to temporal sub-sampling. As a space-time model, our work NeMF has the advantage of being able to edit any frames in the motion sequence while maintaining fidelity and the source style compared to the aforementioned works.
2.3 Implicit Neural Representations
Implicit neural representations explosively gained popularity with the success of SIREN [39] and NeRF [32], which achieved state-of-the-ard results in many tasks including novel view synthesis, geometry reconstruction, and solving differential equations. Among different tasks, the core idea of neural implicit representations is to build continuous function that maps the spatial or temporal coordinates to any signal at that location while maintaining high-free -reg-frreuency details. Inspired by their success, our key insight is to interpret time as the parameter of a motion function and learn the landscape of the opatio-temporal kinematics manifold as an implicit motion representation parameterized by temporal coordinates. Similar ideas have been proposed in modeling time-varying 3D geometries [33] and dynamic scenes [24, 37], and we extend it to the animation domain.
3 Formulation
In this section, we will first give the definition and notation of our motion representation, then present the formulation of NeMF for a single motion sequence, followed by an extended version of generative NeMF for the entire motion space.
3.1 Motion Representation
Similar to Zhou et al. [51] and Li et al. [21]. we divide the motion into two parts: local motion, which contains the pose of the skeleton relative to the root at time and global motion, which is the global translation of the root joint. Following Fussell et al. [6]. we represent the local motion at time t as x3
matrix X, composed of joint positions x" e Jx3 velocities x E Jx3 rotations xf E / in 6D rotation form [50], and angular velocities x E R /33;
X =x
X x xJ) E
(1)
where J is the number of joints. Since all other quantities can be computed from joint rotations, we focus on predicting joint rotations x in the formulation. We then factor out the root orientation r R from local motion by multiplying the inverse of the root transform to each quantity in X,-
3

---

In this way, all poses will be transformed to a local space with the same facing direction, thus making them easier to learn for our network. For the global motion, based on previous works [21, 51], we use neural network to predict the velocity of the root joint it E R as well as its height r E IR from X, We provide details of our global motion predictor in the supplemental.
3.2 Neural Motion Fields
Unlike most other works that represent motion as a discrete sequential process, we represent motion continuous vector field of kinematic poses in the temporal domain. Hence we define this motion field as a function that maps temporal coordinates / to joint rotations and root orientations:
f rx].r()
The function f can be approximated by neural network with the parameters 0 that can be optimized by minimizing the reconstruction loss between the generated and ground truth motion, where the ground truth poses be considered discrete samples of f at integer time steps.
Similar to NeRF [32], we train an MLP with positional encoding of to given motion sequence. Considering sequence with T frames, we first convert the 6D rotations x; and rī to rotation matrices R, and R₃ with the Gram-Schmite Schmidt-like process described in [50], and then compute the geodesic distance to measure the rotational difference:
Tr (R,,
Tr (RO(Ro) -1
Loot
arccos
Cori
arccos
(3)
We then evaluate the L1 loss on local joint positions obtained from forward kinematies (FK) [43]. which regularizes the skeletal topology to obtain better results [34]:
Cpss
Σ
The reconstruction loss is finally expressed as weighted sum of the above terms with weighting factors Ama, Auti and Apas:
Lrs Aro Cori + >pos
(5)
and we optimize the network parameters to minimize this loss function.
3.3 Generative Neural Motion Fields
Although the neural motion field defined above can represent a motion sequence in a compact way, cannot generate variety of motions. To change to different motion sequence, the entire network needs to be re-trained from scratch, which severely limits its application. Therefore, in this section, we extend NeMF generative model which can represent the entire motion space instead of specific motion sequence.
First of all, introduce conditioning variable to the input of f. thus parameterizing the entire spatio-temporal kinematics space as a function:
f -1,4, 2) + (x,,,?),
(6)
where defines the spatial location on the manifold and t. controls the temporal evolution of the sequence.
We then propose a VAE to formulate Equation as a latent variable model with Gaussian distribution. Based our motion representation, the VAE contains two separate convolutional motion encoders learn and parameterize the posterior distribution of latent variables z; and zg. which control the local motion and root orientation respectively:
do (z:) X) (z Mo, (X), oo (X)), (02 | (r)) = NZ,5 Hos (r"), a (r")),
where X and r" are the concatenation of all X, and rf within the same sequence.
The combination of z and Zg forms the final representation of the latent variable 2, which are then passed through the MI.P decoder to produce joint rotations x" and root orientations r for each time step f, thus, defining the output probability distribution (x".. r Æ1, #g).

---

In addition to the probabilistic perspective, our model can also be interpreted as learning a motion prior from imput sequences, where each latent variable on the prior corresponds a natural motion sequence. The MLP decoder then approximates the mapping function between the motion prior and pose space, thus allowing the navigation in the pose space to have a consistent motion style with a fixed z.
During training, we consider the modified variational lower bound:
1x (xxxxx,,x r²) 2 aa A8g Poo",o r² Z1; [21 =n)] DKI (zi X) || p(=:))
(8)
- -DKL (2g [r") p(zz)),
where the expectation term measures the reconstruction error of the decoder, and the KL divergence DKL regularizes the encoders' oulputs to be near p(zi) and p(zz), which NV I) our scenario. Thus, we formulate the loss function L = Lree + AKLCKL with the weight λKL to approximate Equation 8, and the network parameters (01, 22 8) are optimized during training to minimize the loss.
3.4 Applications
After training the generative NeMF, we can deploy it as generic motion prior to solve different tasks via latent space optimization.
Motion In-betweening is long-standing animation creation problem of generating motion in the interval between two clips or sets of keyframes. Our insight is, given a sparse set of observations, we can search in the latent space of NcMF to approximate the entire underlying motion thanks to our continuous representation. Thus, we define the energy function as the reconstruction loss on given frames T. which measures the differences on joint rotations, root orientations, joint positions and root translations:
2 arg min
E DvoeCoor + Xon + >xpos + Auans Luans
(9)
21:2g
where Euass weighted by Atras evaluates the L1 loss on root joint positions rt predicted from our standalone global motion predictor. To facilitate convergence, SLERP is first used to perform interpolation in the joint angle space, and then the interpolated inputs are passed through encoders to obtain the initialization of 21 and Zg.
Motion Re-navigating is a task where we redirect a reference motion with a new trajectory while preserving its style. More formally, the goal is to generate motion that 1) looks similar to the given exemplar; and 2) follows the given trajectory as close as possible. To formulate the energy function, first borrow ideas from time-series analysis by introducing the soft dynamic time warping metric (LaDtw) [4] to measure the similarity between joint positions in the canonical frame [38]. Then, we project the predicted root joints onto the ground plane and compute the Lı loss between them and the given 2D trajectory. Last but not least, an additional regularization term is introduced to ensure that the generated and reference motion have similar angle between their forward direction f, and the tangent direction t, on the trajectory, where the similarity is determined by LDTTW as well. In total, the energy function we try minimize can be expressed as:
21 2
arg min Σ CoTT (x,, x!) Aura proj 21.5g
pPro ||
Aungle te. tz), (10)
where Asim, Amrjj and Aangle are three weighting factors introduced to balance different terms, In our experiments, zi are initialized from the encoder's 's output, while my are optimized from scratch.
4 Experiments
We train our model on AMASS dataset [27] for most of the experiments. After processing, we have roughly 20 hours of human motion sequences at 30 fps for training and testing. We additionally train our model for the reconstruction experiments on quadruped motion dataset [47], which contains 30 minutes of dog motion capture at 60 fps. Details provided in the supplemental.
5

---

4.1 Sanity Test
4.1.1 Motion Reconstruction
We first perform sanity test to validate our single-motion NeMF model described in Section 3.2, where we test the reconstruction capability of NeMF with different lengths of motions.
For AMASS data, we pick 16 different motion sequences for each length and report the reconstruction errors in Table 1. For evaluation metrics, we report mean rotation enror (MRE, 9) and mean position error (MPE, cm) for each joint, which measure the geodesic distance between joint rotations and Euclidean distance between root-aligned joint positions. For the root joint, we further report its orientation error (MOE, "), which measures the geodesic distance on the root orientation. From the results shown in Table 1, we observe that simple MLP with positional encoding is able to to achieve very low reconstruction error (lower than 4mm positional error and 0.7" rotational error) for sequence lengths varying from 32 to 512 frames.
We further train our NeMF model on very long (4, 336 frames and 73s) quadruped mo- tion sequence and visualize the result in the supplemental. Our predicted motion is visu- ally almost identical to the ground truth.
Table 1: Mean reconstruction errors of single-motion NeMF for motion of different lengths. Mean rotation error (9), mean position error (cm), and mean orien- tation error (9) are reported.
Sequence Lengths
4.1.2 Temporal Sampling
Metries
32
64
128
256
512
MRE
0.610
0.482
0.381
0.369
0.379
Unlike other motion models, NeMF theoret- ically guaranteed to generate smooth motion in arbitrary frame rates. We find that the di- mension of the Fourier features generated by positional encoding plays critical role the generalization here.
MPE
0.314
0.249
0.213
0.192
0.170
MOE
0.465
0.340
0.344
0.321
0.300
Since the maximal frequency of the Fourier fea- tures is determined by a hyperparameter as illustrated in [32], we uniformly sample T. from 1 to 21 and train NeMF with these different se- tups. For each trained model, we respectively generate 30 and 60 fps motions and report their mean per-joint velocity as the smoothness met- ric in Figure 2. Although large value of L does not harm the results with the training frame rate, such a model struggles to generalize to dif- ferent frame rates. However, since positional encoding is crucial to achieve satisfactory re- construction results, we set = 7 throughout experiments to balance the trade-off. In the supplemental video, we show that the generated motion remains smooth even sampled at 240 fps.
55.
08.0
65.0
Figure 2: Mean per-joint velocity (cm/s) evalu- ated on the same motion with different L.
4.2 Generative NeMF
4.2.1
Evaluation Metrics
We evaluate the reconstruction capability of our generative NeMF model through direct network inference and motion synthesis capability through latent space sampling. To better describe motion reconstruction, we further measure the Euclidean distance on the root joint translation (MTE, cm) and global joint displacement on the final step (FDE, cm) [45]. For estimating the versatile motion quality, we introduce three metries, namely Fréchet Inception Distance (FID), diversity (Diversity), and foot skating (FS), following related papers [8, 25, 36, 47]. Generally speaking, a lower FID suggests a more natural result, a higher diversity indicates more various result, and foot skating shows the accumulated drift of foot joints during contact. Please see supplemental for details.

---

4.2.2 Ablation Study
We first take look at the effect of the positional encoding function. By removing positional encoding. the input temporal coordinate will be passed directly to the MLP to infer pose parameters. Results in the first row of Table 2 show that the model without positional encoding produces higher errors in most metrics we evaluate, thus demonstrating the necessity positional encoding in terms of the fidelity of the results.
We further experiment on using single motion encoder instead of two separated ones to process both local motion and root orientation. In this simpler design, the root orientation and local motion are entangled in the same latent space. From the results we reported in the second row of Table though the simpler version produces a lower reconstruction error on the root orientation, it has worse performance on joint rotations, joint positions and root translations.
Table 2: Ablation study.
Motion Reconstruction
Motion Synthesis
MRE
MPE J
MOE I
MTE 1
FID
Diversity t
FS
No Positional Encoding
6.413
2.952
6.027
33.524
2.146
2.783
0.612
Single Motion Encoder
8.661
4.503
5.982
35.391
2.185
2.806
0.818
Full Model
5.988
2.870
6.157
29.692
2.073
2.774
0.573
4.2.3 Latent Space Sampling
In our model, we can navigate in the latent space to control the motion styles and synthesize novel motion. To examine the smoothness of the latent space and see whether our model can blend different styles of motion at the sequence level, we linearly interpolate from two existing motion sequences and infer the novel ones. Previous works like [46] demonstrate motion interpolation between similar motion patterns, such as from walking to zombie-style walking. However, we show in the supplemental video that our model can blend the high-level perceptual style between much harder cases like jumping and punching-
Since we disentangle the latent space for local motion and root orientation, we also experiment to combine 21 and 2g from different motions. Through this, can create interesting editing results like canceling the spinning motion of a pirouette jump (see supplemental).
4.2.4 Comparison with Other Generative Models
We compare our method with other deep learning- based motion priors motion reconstruction and synthesis in Table We select HuMoR [38] and HM-VAE [21] to represent the "time series model" and "space-time model" respectively.
Table 3: Comparison of NeMF with other generative motion models.
Motion Reconstruction
Motion Synthesis
MRE
MPE 4
MOE 1
MTE $
FDE 1
FID
Diversity t
FS
HuMoR [38]
13.008
9.071
17.097
23.882
62.756
8.687
1.741
0.904
HM- [21]
10.258
7.686
13.054
90.924
137.218
7.998
2.002
0.690
Ours
5.988
2.870
6.157
29.692
42.985
6.508
2.118
0.566
Although HuMoR can generate convincing motions, we observe two shortcomings of HuMoR: 1) in motion reconstruction, HuMoR's results will gradually diverge due to its auto-regressive prediction. From the guantitative results in Table 3 and qualitative results in Figure 3, divergence can be observed both the large FDE and global translation difference respectively. 2) motion synthesis, HuMoR tends to favor common motions like walking when inferring long sequences, thus yielding relatively high FID and low diversity. As for our method, we can get rid of these artifacts since we handle the entire sequence at once instead of predicting the motion frame by frame.
7

---

HM- VAB [21]
HuMoR [38]
Ours
Figure 3:
Comparison of motion reconstruction with other generative motion models. Predicted
motions are overlapped on top of the ground truth motion (yellow).
Compared to HuMoR and our model, HM- VAE models an over-smoothed latent space, thus filtering out high-frequency details in both the reconstructed and synthesized motions. In short, our model outperforms these state-of-the-ard methods in most of the metrics reported in Table 3, as well as achieving the closest match of the ground truth motion visualized in Figure
4.3 In-hetweening Tasks
In this task, we compare our method with traditional and deep learning-bass motion in in-betweening methods. For quantitative evaluation, tion errors cannot fully reflect the quality of motion for highly-varied plausible solutions.
the FID and foot skating metries only, since reconstruc-
Table 4: Motion clips in-betweening.
Table 5: Sparse keyframe in in-betweening.
FID
FID
Length (frames)
10
20
30
Length (frames)
5
10
15
20
SLERP
0.027
0.170
0.455
SLERP
0.032
0.305
0.713
1.191
Inertialization [2]
0.025
0.184
0.496
HM-VAE [2]]
2.300
2.370
2.434
2.423
RMI [9]
0.264
0.362
0.609
Ours
0.085
0.302
0.612
0.879
HM-VAE [21]
0.137
0.387
0.675
Foot Skating
Ours
0.024
0.141
0.365
SLERP HM-VAE [21]
0.868
1.224
1.278
1.193
Foot Skating
0.892
0.837
0.841
0.751
SLERP
1.193
1.200
1.023
Ours
0.719
0.826
0.837
0.804
Inertialization [2]
1.237
1.234
1,151
RMI [9]
1.629
1.833
1.6413
HM-VAE [21]
0.862
0.835
0.832
Ours
0.646
0.697
0.660
Motion Clips In-befweening. In this task, we aim at generating motion in the interval from 10 frames (0.33s) to 30 frames (1s) between two clips. We compare with traditional local interpolation methods including SLERP and Inertialization [2]. and deep learning- methods including Robust Motion In-betweening (RMI) [9] and HM-VAE [21]. In the results reported in Table 4, our method outperforms all other alternatives quantitatively. We further experiment on real dancing footage by randomly picking several pairs of videos from +IST+ [23] with their corresponding SMPL [26] parameters. We set the transition length to 30 frames and optimize for latent variable to produce the gap-filling motion. As demonstrated in Figure 4, our model is capable of producing natural one-second transition between these real dancing footage.
Sparse Keyframe In-betweening. In this task, set of sparse keyframe skeleton poses given every 5, 10, 15 or 20 frames. We compare our method with SLERP and RAE, and report the quantitative results in Table 5. Neither RMI nor Inertialization is applicable here since they both require multiple frames at the beginning.
From Table 5, SLERP has the best FID score for interval of 5 frames, but becomes worse with increased interval length. Compared to SLERP and HM-VAF, ours has much better quantitative and qualitative results, especially for long intervals (Figure 5). In the onionskin images, though SLERP's

---

Figure 4: Generating the transition (cyan) between +IST+ motion clips (yellow).
result looks similar to ours, result has more natural high-freguh figh-freueency details. HM- MAEE fails to reconstruct the poses at keyframes and generates over-smoothed results. Our result, although different from the ground truth, is visually plausible in the supplemental video.
SLERP
HM-VAB [21]
Ours
Ground Truth
Figure 5: Comparison of sparse keyframe in-betweening. The translucent in-between poses are generated from the opaque reconstructed keyframes given every 20 frames.
4.4 Re-navigating Tasks
In this task, we experiment on redirecting the reference motion to different synthetic trajectories. Qualitative results reported in Figure 6. The original motion style is well preserved in the results, and the character has a natural orientation while walking along the trajectory.
Reference motion
Straight line
Sinusoidal curve
Figure 6: Motion re-navigating from reference walking motion (left) to straight line (middle) and sinusoidal curve (right).
Conclusion
We propose implicit neural motion representation that defines continuous motion field over style and time. We design it to be generative model for the whole motion space with a learned prior. We design and test different network architectures and use the trained generative model as a motion prior for solving different tasks like motion interpolation, motion in- in-betweening and motion re-ne re-navigating.
5.1 Limitation and Future Work
In this paper we train the generative NeMF model in the form VAE, so has the limitation of not always giving satisfactory results when sampling the latent space. A promising direction is to design other generative architectures such as GAN, Normalizing Flows, and Diffusion Models for NeMF. Besides, though several latent variables may exist that satisfy the optimization constraints, our deterministic optimization process can only find one possible the final result, thus limiting probabilistic motion synthesis like MoGlow [10].

---

In the future, since the optimization problems we formulated are analogous to the latent space opti- mization problems of StyleGAN [17], how transfer the algorithm of those successful applications designed for StyleGAN to our NeMF setup is an interesting direction to explore.
So far our method focuses motion modeling and cannot adapt to outputs with different body shapes, On the other hand, some works [0, 31] leverage implicit representations to model skinned articulated objects with given poses. Therefore, it would be interesting combine our work with them to enable the animation and rendering of specific character with novel motions.
To keep model aggossg,,,,,c, choose to solve motion tasks by optimizing z. But for practical applications, task-specific inference models may be preferred for performance consideration. To have an ead-to-end inference model for real-time applications, a possible solution is to design and train the encoder to directly predict z with different input settings.
5.2 Broader Impact
Our model can be applied for kinematic motion creation in animation and game production. Due to the limitation of diversity in the training data, our model is not guaranteed to generate feasible but uncommon motions. We cannot guarantee that our model will avoid generating motions that could be perceived as offensive to some viewers.
Acknowledgments and Disclosure of Funding
This work was supported in part by NSF Grant No. IIS-2007283. Many thanks to Ruben Villegas for helpful discussions. Special thanks to Jiaman Li for sharing with us the pre-trained HM- M--VEE model.
References
[1] Emre Aksan, Manuel Kaufmann, Peng Cao, and Otmar Hilliges. spatio-temporal transformer for 3d human motion prediction. 2021 International Conference on Vision (3DV), pages 565-574. IEEE, 2021. 2
[2] David Bollo. Inertialization: Higf-Peeformance animation transitions in 'gears of war'. Proc. of GDC 2018, 2018. 8
[3] Chen, Yufeng Zheng, Michael Black, Otmar Hilliges, and Andreas Geiger. Snarf: Differentiable forward skinning for animating non-rigid neural implicit shapes. In International Conference on Computer
Vision (ICCV), 2021. 10
Marco Cuturi and Mathieu Blondel. Soft-dtw: A differentiable loss function for time-series. In Proceed- ings of the 34th International Conference on Machine Learning Volume 70, ICML'17, page 894-903.
JMLR.org. 2017. 5
[5] Katerina Fragkiadaki, Sergey Levine, Panna Felsen, and Jitendra Malik. Recurrent network models for human dynamics. In Proceedings of the IEEE international conference on computer vision, pages
4346-4354, 2015.
[6] Levi Fussell, Kevin Bergamin, and Daniel Holden. Supertrack: Motion tracking for physically simulated characters using supervised learning. ACM Trons. Graph., 40(6), dec 2021. 3
[7] Ian Goodfellow. Jean Pouget-Abadie, Mehdi Mirza. Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems, vol- ume 27. Curran Associates, Inc., 2014. 3
[8] Chuan Guo, Xinxin Zuo. Sen Wang, Shihao Zou, Qingyao Sun, Annan Deng, Minglun Gong, and Li Cheng. Action2motion: Conditioned generation of 3d human motions. In Proceedings of the 28th ACM International Conference on Multimedia (MM 220), 2020. 6
[9] Félix G. Harvey, Mike Yurick, Derek Nowrouzezahrai,, and Christopher Pal. Robust motion in-betweening.
ACM Trans. Graph., 39(4), jul 2020. 3,8
[10] Gustav Fje Henter, Simon Alexanderson, and Jonas Beskow. Moglow: Probabilistic and controllable motion synthesis using normalising flows. ACM Trans. Graph., 39(6), nov 2020. 1,3,9
[11] Sepp Hochreiter and Jürgen Schmidhuber, Long short-term memory. Neural Comput., 9(8): 1735-1775878 nov 1997. 3
[12] Daniel Holden, Oussama Kanoun, Maksym Perepichka, and Tiberiu Popa. Learned motion matching. ACM Trans. Graph., 39(4):53:1-53112 July 2020. 3
10

---

[13] Daniel Holden, Taku Komura, and Jun Saito. Phase-functioned neu-ral networks for character control. ACM Trans. Graph. 36, 2017. 3
[14] Daniel Holden, Jun Saito, and Taku Komura. A deep learning framework for character motion synthesis and editing. ACM Trans. Gruph., 35(4):1-11, July 2016. 3
[15] Daniel Holden, Jun Saito, Taku Komura, and Thomas Joyce. Learning motion manifolds with convolutional autoencnders. In SIGGRAPH Asia 2015 Technical Briefs, number Article 18 in SA "15, pages 1-4, New York, NY, USA, November 2015. Association for Computing Machinery. 3
[16] Boyan Jiang, Yinda Zhang. Xingkui Wei, Xiangyang Xue, and Yanwei Fu. Learning compositional representation for 4d captures with neural ode. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5340-5350, 2021. 2
[17] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). June 2019. 10
[18] Manuel Kaufmann, Emre Aksan, Jie Song, Fabrizio Pece, Remo Ziegler, and Otmar Hilliges. Convolutional autoencoders for human motion infilling. In 2020 International Conference on 3D Vision (3DV), pages 918-927. IEEE, 2020. 3
[19] Yongjaon Lee, Kevin Wampler, Gilbert Bernstein, Jovan Popović, and Zoran Popović. Motion fields for interactive character locomotion. In ACM SIGGRAPHI Asia 2010 papers, number Article 138 in SIGGRAPH ASIA '10, pages 1-8, New York, NY, USA, December 2010. Association for Computing Machinery.
[20] Sergey Levine, Jack M. Wang, Alexis Haraux, Zoran Popović, and Vladlen Koltun. Continuous character control with low-dimensional embeddings. ACM Trans. Graph., (1), 2012.
[21] Jiaman Li, Ruben Villegas, Duygu Ceylan, Jimei Yang, Zhengfei Kuang, Hao Li, and Yajie Zhao. Task- generic hicrarchical human motion prior using vaes. 2021. 2, 3, 4, 8,9
[22] Jiaman Li, Yihang Yin, Hang Chu, Yi Zhou, Tingwu Wang. Sanja Fidler, and Hao Li. Learning to generate diverse dance motions with transformer. arXiv preprint arXiv:2008. 08/71, 2020. 2
[23] Ruilong Li, Shan Yang. David A. Ross, and Angjoo Kanazawa. Ai choreographer: Music conditioned 3d dance generation with aist++, 2021. 8, 13
[24] Tianye Li, Mira Slaveheva, M. Zollhöfer, S. Green, Christoph Lassner, Changil Kim, Tanner Schmidt,
S. Lovegrove, M. Goesele, R. Newcombe, and Z. Lv. Neural 3d video synthesis from multi-view vidco.
2022. 3
[25] Hung Yu Ling, Fabio Zinno, George Cheng, and Michiel Van De Panne. Character controllers using motion VAEs. ACM Trans. Graph., 39(4):40:1 12, July 2020, 1,3,6 6
[26] Matthew Loper, Naureen Mahmood, Javier Romero. Gerard Mols-Mol,, and Michael J. Black. SMPL: A skinned pealii-person linear model. ACM Trans. Graphics (Proc. SIGGRAPH Asia), 34(6):28811224811 October 2015. 8
[27] Naureen Mahmood, Nima Ghorbani, Nikolaus F. Troje, Gerard Pons-Moll, and Michael J. Black. AMASS: Archive of motion capture as surface shapes. In International Conference on Computer Vision, pages 5442-5451, October 2019. S, 13
[28] Wei Mao, Miaomiao Liu, and Mathieu Salzmann. History repeats itself: Human motion prediction via motion attention. In Earopean Conference on Computer Vision, pages 474-489. Springer, 2020. 2
[29] Julieta Martinez, Michael J Black, and Javier Romero. On human motion prediction using recurrent neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2891-2900, 2017. 2
[30] Marko Mihajlovic, Shunsuke Saito, Aayush Bansal, Michael Zollhoefer, and Siyu Tang. COAP: Com- positional articulated occupancy of people. In Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), June 2022. 10
[31] Marko Mihajlovic, Yan Zhang. Michael Black, and Siyu Tang. LEAP: Learning articulated occupancy of people. Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), June 2021. 10
[32] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren
Ng. Nerf: Representing scenes neural radiance fields for view synthesis. In ECCV, 2020. 1, 3.4.4, 6
[33] Michael Niemeyer, Lars Mescheder, Michael Occhslc, and Andreas Geiger. Occupancy flow; 4d recon- struction by learning particle dynamics. In Proc. of the IEEE International Conf. on Computer Vision (ICCV), 2019. 3
[34] Dario Pavllo, David Grangier, and Michael Auli. Quaternet: A quaternion based recurrent model for human motion. In British Machine Vision Conference (BMVC), 2018. 1,4

---

[35] Xue Bin Peng. Angjoo Kanazawa, Jitendra Malik, Pieter Abbeel, and Sergey Levine. Sív: Reinforvement learning of physical skills from videos. ACM Transactions On Graphics (TOG), 3766))1114 2018. 1
[36] Mathis Petrovich, Michael J. Black, and Gül Varol. Action-conditioned 3D human motion synthesis with transformer VAE. In International Conference Computer Vision (ICCV), pages 10985-10995, October
2021. 2, 6
[37]
Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pages 10318- 2021.
[38] Davis Rempe, Tolga Birdal, Aaron Hertzmann, Jimei Yang, Srinath Sridhar, and Lconidas J. Guibas. Humor: 3d human motion model for robust pose estimation. In International Conference Computer Vision (ICCV), 2021. 7, 8
[39]
Vincent Sitzmann, Julien N.P. Martel, Alexander W. Bergman, David B. Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. In Proc. NewrlPS, 2020. 3
[40] Sebastian Starke, He Zhang, Taku Komura, and Jun Saito. Neural state machine for character-scene interactions. ACM Trans. Gruph., 38(6):-11, November 2019. 3
[41] Sebastian Starke, Yiwei Zhao, Taku Komura, and Kazi Zaman, Local motion phases for learning multi- contact character movements. ACM Trans. Graph., 39(4):54:1554:11 July 2020.
[42] Sebastian Starke, Yiwei Zhao, Fabio Zinno, and Taku Komura. Neural animation layering for synthesizing martial arts movements, ACM Trans. Graph., 40(4);1-16, July 2021.
[43] Ruben Villegas, Jimei Yang. Duygu Ceylan, and Honglak Lee. Neural kinematic networks for unsupervised motion retargetting. In The IEEE Conference on Computer Vision and Pottern Recognition (CVPR), June 2018. 4
[44] Jack M Wang, David J Fleet, and Aaron Hertzmann. Gaussian process dynamical models for human motion. JEEE Trans. Pattern Anal. Mach. Intell., 30(2):883 298, February 2008.
[45] Ye Yuan and Kris Kitani. Dlow: Diversifying latent flows for diverse human motion prediction. In European Conference on Computer Vision, pages 346-364. Springer, 2020. 3,6
[46] M Ersin Yumer and Niloy J Mitra. Spectral style transfer for human motion between independent actions.
ACM Transactions Graphics (TOG), 35(4):1-8, 2016. 7
[47] He Zhang, Sebastian Starke, Taku Komura, and Jun Saito. Mode-adaptive neural networks for quadruped motion control. ACM Trans. Graph., 37(4), jul 2018. 3, 5,5,6 6, 13
[48] Siwei Zhang, Yan Zhang, Federica Bogo, Mare Pollefeys, and Siyu Tang. Learning motion priors for 4d human body capture in 3d scenes. In Proceedings of the IEEE/CVF International Conference on Computer
Vision, pages 11343 11353, 2021. 3
[49] Yan Zhang, Michael Black, and Siyu Tang. are more than our joints: Predicting how 3d bodies move. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 3372-3382, 2021. 2
[50] Zhou, Connelly Barnes, Jingwan Lu, Jimei Yang, and Hao Li. On continuity rotation representa- tions in neural networks. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 5745-5753, 2019. 3,4
[51] Zhou, Jingwan Lu, Connelly Barnes, Jimei Yang, Sitao Xiang, et al. Generative tweening: Long-term inbetweening of 3d human motions. arXiv preprint arXiv:2005. 2020. 3,4
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 5.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Section 5.2.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
12

---

(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [No]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See supplemental and the beginning of Section 4.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [No] We fixed random seeds to throughout our experiments.
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes] See supplemental.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] Our model is trained on the AMASS [27] dataset and quadruped dataset from [47], we also use AIST+ [23] to test motion in-betweening.
(b) Did you mention the license of the assets? [Yes] See supplemental.
(c) Did you include any assets either in the supplemental material URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you'l 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "o_cP7tmM_xC", "reviewer": "Reviewer_wNce", "summary": "This paper proposes an implicit neural model for kinematic motions. Compared to the seminal NeRF model, which learns a continuous function over spatial coordinates, the proposed NeMF framework learns a continuous function over temporal coordinates. Meanwhile, NeMF outputs a vector of the kinematic pose compared to the color and occupancy information in NeRF. The paper also proposes to extend NeMF beyond only encoding a single motion sequence - following the recent trend on the generalization of implicit neural models. This is achieved by adopting a VAE framework to make NeMF a generative model controlled by a latent code input. The model is trained following the standard VAE training objective including the reconstruction loss and KL loss. The proposed approach is evaluated on several motion generation task (i.e. motion in-betweening and re-navigation) using the AMASS dataset, and compared with several prior methods including HuMoR and HM-VAE. Results have shown that NeMF have achieved better performance on motion reconstruction and motion synthesis (Tab. 3, 4, and 5).", "questions": "### Questions\n- In general I'm wondering how the latent code handles translation in time. For example, in the current framework, the exact motion clip (e.g. with a time interval [0, T]) after translation (e.g. to interval [-T/2, +T/2]) will have a completely different latent code. This seems a bit awkward and somewhat redundant. I wonder whether there has been any observed pattern on the latent code for translation in time, also, whether one should eventually decouple the translation dimension from the latent code.\n- [Eq. 5] Should there also be a loss term for global root translation? The supplementary Sec. 1.1 mentions a loss function for the global motion predictor (i.e. $\\mathcal{L}$ in Eq. 2). Is that also part of $\\mathcal{L}_{rec}$ here? If not, how is the global motion predictor loss applied in the framework?\n- [Tab. 3] How come some of the numbers in \"Ours\" is exact to those in Tab. 2 (e.g. 5.988 MRE, 2.870 MPE, 6.157 MOE, and 29.692 MTE) while some are different (e.g. 6.508 versus 2.073 FID, 2.118 versus 2.774 Diversity, and 0.566 versus 0.573 FS)?\n\n### Suggestions\n- [L171] \"In our experiments, $z_l$ and $z_g$ are initialized using the SLERP data to facilitate convergence.\" -> Detail is unclear. Is SLERP first used to perform interpolation in the joint angle space, and then the joint angles are passed through the encoder to obtain the latent code?\n- [Fig. 2] I wonder whether the absolute velocity values are the best metric to look at for this motion reconstruction evaluation. Would it be better to evaluate on the error (of both position and velocity) rather than the absolute value? The issue may be that it is not possible to get the ground-truth position and velocity at frame rates higher than 30 FPS. Then would it be reasonable to train on 15 FPS and evaluate on 30 FPS then just for this experiment?\n- [L273] \"Compared to HuMoR and our model, it is **obvious** that HM-VAE\" models an over-smoothed latent space, thus filtering high-frequency details in both the reconstructed and synthesized motions, ...\" -> Not so obvious. Following the weakness point for [20] above, it is unclear how [20] works and why it \"models an over-smoothed latent space\" while HuMoR and \"Ours\" don't.\n- [L274] \"..., making their results worse in both the reconstruction and synthesis task.\" -> Not precisely. HM-VAE seems to outperform HuMoR sometimes, e.g. on MRE, MPE, MOE?\n- [High level] So far the paper still utilizes a low dimensional vector at the output for obtaining human pose and shape. How does it compare to making the model \"fully implicit\", i.e. on not just the temporal but also the spatial domain (x, y, z) as well, like in the original NeRF and other implicit shape models? I guess as the first step, it may just be easier to rely on a forward function to generate the mesh from the pose vector, which is why the authors go in this route first. Yet a \"full implicit\" model will make the framework looks more elegant and potentially better integrated with the efforts on the NeRF side.\n- [High level] The paper has shown results on an array of \"offline\" tasks, where the motion of the full sequence can be pre-solved through a latent code and fixed thereafter. Meanwhile, I would imagine an \"online/interactive\" variant of the NeMF framework would be useful, especially in many gaming applications (e.g. real-time character control, etc.). Are there ways to adapt the current framework to an interactive setup? Perhaps one can run the latent space optimization and motion generation in a continuous loop by streaming motion generation in short time segments (e.g. 1 sec)? If so, it would be nice to add some mention to such extensions.\n", "limitations": "Limitations and potential negative societal impact are addressed in Sec 5.1 and 5.2 respectively.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n- [Original] The proposed framework does look novel - by adapting the impactful NeRF model to the domain of kinematic motions. It is certainly interesting to see how these models work not just on static scene reconstruction but also on reconstructing motions.\n- [Quality] The method is technically sound. The key idea, as mentioned above, of extending NeRF to the temporal domain, is quite simple and well executed.\n- [Clarity] The presentation is overall clear. The core idea as well as the technical details are well presented and easy to follow. Math notations have been used properly.\n\n### Weaknesses\n- [Clarity/Contribution] From the current presentation, it is unclear how the proposed work relates to the work of [20]. How does this paper add over [20], and why is [20] not discussed in the related work section?\n- [Quality] Although the theoretical framework looks nice, I'm still a little bit skeptical on how the latent space optimization would work practically. It seems that this heavily depends on how smooth the latent space of $z$ is. If the space learned from VAE is not sufficiently smooth, the optimization (e.g. for Eq. 9 and 10) will always be trapped in useless local minimum. Would this be an issue at all? It would be helpful to somehow show the landscape by visualizing the pose transformation as one traverses in the latent space.\n- [Clarity] The detail of the generative task for Sec. 4.2.2 and Sec. 4.2.4 is missing. What is the generative task here, i.e. input and output? Is it trying to predict the pose at arbitrary $t$ given some keyframes from an unseen (test) sequence, so the approach is to perform latent space optimization for $z$ and then run forward pass on the evaluated $t$s?\n\n### Justification of Rating\nOverall this looks like a solid submission with decent novelty and contribution. I just have some concerns on the comparison with prior work as well as some technical details as listed below.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "gmSrNx4dCVz", "reviewer": "Reviewer_umUH", "summary": "The authors propose NeMF (neural motion fields), an implicit neural representation for human motion. Similar to NeRF, they demonstrate that they can well encode a mapping function from time-step `t` to poses. They also learn a generative model by conditioning this on a latent variable `z` to encode different styles/kinds of motion.\n\nThey demonstrate that NeMF can well encode a single motion sequence of varying lengths. Their generative samples yield good quality motions, as demonstrated quantitatively and qualitatively compared to previous methods. They also perform a few downstream tasks such as motion in-betweening, spare keyframe in-betweening, and re-navigating tasks by formulating them as a latent space optimization task (find `z` that best approximates the inputs).\n", "questions": "I’m curious to know more about latent-space optimization and convergence. Is the optimization stable, and does it converge when not using custom initialization? Also, for the re-navigating task, is the model able to optimize and generate complex paths (say writing out the word NEURIPS). Some more details on the optimization setup would be helpful.\n\nIt’s interesting to note that NeMF has significantly better foot-skate numbers when compared to HuMoR even though HuMoR explicitly models ground contacts. Could the authors comment more on this phenomenon? Is it due to diverging motion?\n\nI think some places in the text are missing adequate explanations and justifications (especially in the methods and experimental section). Some places are:\n- Before Eq 2 maybe a line or two saying you are only interested in predicting the joint angles (and not the full X_t as defined previously).\n- Eq 7 and 8, it’s unclear why you move back to X and X_t notation for Eq 7. It’s also not clear in Eq 8 what lower-case x is, it’s not defined previously/after the Eq.\n- Details/setup of how the AIST++ motion in-betweening was done is missing.\n- Typos: Eq 1 should have an angular velocity term (last term)\n", "limitations": "Yes, the authors have included both limitations and broader impact sections in the main paper. \n", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The problem setup and formulation makes sense. The authors demonstrate good performance in both single motion encoding and the generative model.\n- The implicit neural representation has several advantages (as shown by temporal sub-sampling and good motion reconstruction and synthesis results).\n- The authors have done several evaluations on different tasks. The supplemental material show convincing qualitative visualizations of the model.\n\nWeakness:\n\nI think the paper suffers a bit from a lack of details and clarity in writing (detailed below).\n\nOtherwise, I think the paper's contributions would be a good addition to the community.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "CST3UgWnGTf", "reviewer": "Reviewer_Qiic", "summary": "The authors propose to formulate human motion as a generative implicit representation. The authors demonstrate the effectiveness of such approach. The authors showcase the model in various tasks.", "questions": "I don't have any specific technical question. ", "limitations": "The authors do a great job in this aspect.", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n* Overall, I love everything in the paper. Things are presented in a clear and coherent way, making things easy to follow. \n* The baselines and experiment comparisons are all very solid. The authors did a great job explaining the baselines and comparing with them.\n* The authors did an excellent job explore using the model in various scenarios. This is very informative and showcases the robustness of the model. \n\nWeakness:\n* Important related work missing: Occupancy Flow: 4D Reconstruction by Learning Particle Dynamics, ICCV 2019. The part where the authors talk about \" Inspired by their success, our key insight is to interpret time as the parameter of a motion function and learn the landscape of the spatio-temporal kinematic manifold as an implicit motion field parameterized by temporal coordinates.\" This is exactly what the highly-cited prior work has done. Though the prior work differs in motion representation, it should be discussed.\n* And a nitpick for the previous point's quote is \"implicit motion field\" should be \"implicit motion representation\". ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "3manAblHBxU", "reviewer": "Reviewer_1jgD", "summary": "In this paper, the authors propose a novel representation of human motion as neural motion fields. Inspired by the success of neural fields in other areas, this work learns an implicit function to represent motions. This representation can help to sample diverse and natural motions and facilitate various downstream applications. The experiments demonstrates the ability of their method.", "questions": "Please refer to weakness.", "limitations": "Please refer to weakness.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n+The authors proposed an INR-based model to learn motions. The model is continuous and can generate sequences of arbitrary frame rates in principal.\n+Non-autoregressive generation. The function formulation allows the generation of poses at specific time steps directly without first generating frames before them while preserving the smoothness of the full sequence.\n+The paper gives extensive experiments on the efficacy and applications of its proposed model.\n\nWeaknesses:\n-The formulation of the VAE is very similar to HM-VAE. The introduction of INR is novel and very interesting, but it seems the combination of the two is simple.\n-The formulation only uses a latent z to decide the motion type, so for each task the optimization process only finds a deterministic z, which gives a single generated sequence.\n-In Figure. 5 the keyframes given for the HM-VAE example are different from the others. Some typos remain in the paper (Section 4.1.1 for example). \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept