NeRM: Learning Neural Representations for High-Framerate Human Motion Synthesis
Authors: Dong Wei
Huaijiang Sun
Bin Li
Xiaoning Sun
Shengxiang Hu
Weiqing Li
Jianfeng Lu
Venue: ICLR 2024 poster (2024)
Keywords: Human motion synthesis
implicit neural representation
diffusion model
any-framerate training
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: Directly exploiting the variety of framerates available in the dataset for high-framerate motion generation.

Abstract: Generating realistic human motions with high framerate is an underexplored task, due to the varied framerates of training data, huge memory burden brought by high framerates and slow sampling speed of generative models. Recent advances make a compromise for training by downsampling high-framerate details away and discarding low-framerate samples, which suffer from severe information loss and restricted-framerate generation. In this paper, we found that the recent emerging paradigm of Implicit Neural Representations (INRs) that encode a signal into a continuous function can effectively tackle this challenging problem. To this end, we introduce NeRM, a generative model capable of taking advantage of varied-size data and capturing variational distribution of motions for high-framerate motion synthesis. By optimizing latent representation and a auto-decoder conditioned on temporal coordinates, NeRM learns continuous motion fields of sampled motion clips that ingeniously avoid explicit modeling of raw varied-size motions. This expressive latent representation is then used to learn a diffusion model that enables both unconditional and conditional generation of human motions. We demonstrate that our approach achieves competitive results with state-of-the-art methods, and can generate arbitrary framerate motions. Additionally, we show that NeRM is not only memory-friendly, but also highly efficient even when generating high-framerate motions.

Published as a conference paper at ICLR 2024
NERM: LEARNING NEURAL REPRESENTATIONS FOR HIGH- FRAMERATE HUMAN MOTION SYNTHESIS
Dong Wei', Huaijiang Sun'; Bin Li", Xiaoning Sun', Shengxiang Hu', Weiqing Jianfeng Lu²
1 School of Computer Science and Engineering, Nanjing University of Science and Technology Tianjin AiForward Science and Technology Co., Ltd.
1 (csdwei, caunuaijjinne sunxiacning hushengxiang, www.aanng..wwwiiiie.eee wi.weiging, wwwfferjust.cdu.cuoommmm
2 1 www.aea..oow....ooommmmomm.
ABSTRACT
Generating realistic human motions with high framerate is an underexplored task, due to the varied framerates of training data, huge memory burden brought by high framerates and slow sampling speed of generative models. Recent advances make a compromise for training by downsampling high-frameran details away and discarding low-framerate samples, which suffer from severe information loss and estricted-framerea generation. In this paper, we found that the recent emerg- ing paradigm of Implicit Neural Representations (INRs) that encode signal into a continuous function can effectively tackle this challenging problem. To this end, we introduce NeRM, generative model capable of taking advantage of varied- size data and capturing variational distribution of motions for high- high-framerate mo- tion synthesis. By optimizing latent representation and an auto-decoder condi- tioned on temporal coordinates, NeRM learns neural representations for sampled motion clips that ingeniously avoid explicit modeling of raw varied-size motions. This expressive latent representation is then used to learn a diffusion model that enables both unconditional and conditional generation of human motions. We show that NeRM not only achieves competitive results with state-of-the-arr meth- ods, but also be capable of generating arbitrary-framerate motions. Moreover, it can remain memory- memory-frienll yet highly efficient even when generating high- framerate motions.
INTRODUCTION
The technology of human motion synthesis has been widely applied in various scenarios (Koppula Saxena 2013 Van Welbergen al., (2010) [Sun al. (202) of filming, robotics and gaming ani- mation. In this task, generating high-quality motions with high framerate holds promising value for practical use (Feng et al.) 2023; Wang et al. 2019). For example, the gaming system can showcase high-framerate animation
nigh-performaneee devices to enable smooth and visually immersive gaming experience for players, while adaptively reducing the framerate on low- low-performance devices.
Existing methods (Petrovich al. 2021 Zhang et al. 20022 Tevet et al. 2023 Cai et al. 2021 Wang et al. 2022) have neither considered, nor been able to handle le such demands, which is primarily ly due to the following reasons: (1) High-framerate motion generation would impose substantial memory overloads that is unacceptable for current generative models, whose slow sampling speed further complicates this issue. (2) The raw motion data usually exhibits variability in framerates, such as text-to-motion dataset HumanML.3D (Guo er al. (202)), wherein the framerate of each subset varies from 20 fps to 250 fps, making challenging for direct use during training. methods employ a typical pre- pre-prccessing step to construct dataset with fixed and target framerate. Motions exceeding the target framerate are downsampled, removing high-freeueccy details; and motions with inadequate framerate are excluded, discarding low- low-frequency structural information. However, such naïve process is inherently suboptimal, models and can only generate motions with fixed yet relatively low framerate. In practice, as human eyes are very sensitive to high-frequency details, desired to embrace the natural diversity
a result, all these
restricts the capabilities of data-driven
Corresponding author.

---

Published as a conference paper at ICLR 2024
220tps
6n1gs
400ps
20pps
130p
framerate 201ps
framorate 200p maintained
framonato 200ps discarded
all
downsampling
M
201p8
201Gp
206ps
206ps
Standard training: varied-framedd pre-processing
Ours
Figure/ I: Motions are captured under different sampling rates. To realize uniform training on them well ensuring acceptable memory burden, existing models have to downsample sequences to fixed, target framerate (such as 20 fps), and remove samples with that even lower. Our design can handle sequences their native framerates, making full use of available annotated motion resources.
of motion framerates and process them at their native framerates, so that preserve and utilize the subtle nuances in original motions, thereby improving visual sensation for viewers.
Meanwhile, Implicit Neural Representations (INRs) (Mildenhall et al. 2021 Watson et al. 2023
have emerged as a new paradigm for representing continuous signals without explicit modeling, and gained popularity in various domains. For instance, Park et al. (2019) encodes geometry into a neural network that maps input 3D coordinates to corresponding SDF values. Accordingly, as opposed to discrete grid-wise signal values, INRs amortize the signal values of arbitrary coordinates into a compact neural representation, which eliminates the need for large memory allocation that would be proportional to the coordinate dimension and resolution. to be highly effective at modeling complex and dense signals, such as images (Skorokhodov et al., 2021, videos (Yu al. 2022) and 3D scenes (Mildenhall et al. 2021)
this respect, INRs have shown
Inspired by this, we propose NeRM, variational implicit neural representation- generative model for high-framerate human motion synthesis. NeRM represents motion sequence as a con- tinuous function f: r~> [(t) that is parametrized by Multi-Layer Perceptron (MLP) conditioned on temporal coordinates. To mitigate excessive overfitting of motions, we introduce latent code which can be thought of encoding of a sampled motion clip (a short segment of motion), as conditional variable to the function ft, z). When keeping the latent code z constant and varying the coordinate t, NeRM allows to efficiently generate motions at arbitrary framerates and time steps; and decouple high- gigheframerate synthesis from prohibitive memory requirements. We find these latent codes by interpreting them as variational distribution with optimized parameters (i.e., mean and covariance) in the representation space. This simple process naturally supports any-framerate training and showcases significant departure from previous works (He al.. 2022 Mao et al., 2021 Wei et al., 2024) that require an explicit and sophisticated encoder to infer distributions of motions. After optimizing the representative motion latent codes, motivated by the success of stable diffusion (Rombach et al.] (022). we learn to model the distribution of varied- framerate motions in latent space. Various conditions available to the model at training (e.g. action labels, text prompts) can be used to generate the realistic motions conforming to those conditions.
Additionally, acquiring qugl-quality human motion sequences with well annotations expensive and limited (Chen et al. (2023). In this case, only temporal coordinate information is insufficient for the network to characterize continuous motion fields. We draw inspiration from Vector Quantized (VQ) codebook in 3D scenes (Yin et al. (2022), and inject such prior descriptors into the coordinated- based network. With the help of the codebook, our approach has the potential to enrich the feature representation of each temporal coordinate and then enhance the quality of generated motions.
We conduct comprehensive experiments on various datasets, including HumanML33 (Guo et al. 2022). KIT (Plappert et al. 2016). HumanAct12 (Guo al.) (020) and UESTC (Ji et al. 2018). Numerical results demonstrate NeRM to be extremely competitive with of baselines. Furthermore, NeRM appreciates the following intriguing properties:
2

---

Published as a conference paper at ICLR 2024
Arbitrary framerate training: is capable of training on mixed-frament framerate datasets without pre- processing, like downsampling and discarding (Figure
High-framerate motion generation: can synthesize high-frau frgmerramerate motions (Figure 4b) of ~ 120 fps without suffering from increasing architecture size and massive memory resources (Appendix)
High inference speed: achieve x20x speed up in compared to Tevet et al. (2023) (Appendix)
Temporal sub-sampling: enable the generation of poses at specific time steps directly without first generating frames before, while preserving the smoothnesss of the entire sequence (Figure
Flexible conditional generation: allow for unconditional generation (Figure 5 as well as condi- tional on action labels (Table 3 text descriptions (Table 1
2 RELATED WORK
Denoising Diffusion Probabilistic Models (DDPMs). DDPMs (Sohl-Dickstein et al, 2015 Ho et al. (2000) have witnessed significant progress in the image synthesis domains, such as Imagen (Saharia et al., 2022) and DAILL-E (Ramesh et al. 2022). Inspired by their works, most recent methods have adapted this advanced generative model to motion generation tasks. MotionDiffuse (Zhang et al. 2022) designs an effective DDPM-based architecture for controllable text-driven mo- tion synthesis, allowing for motion manipulation in body parts. MDM (Tevet et al. 2023) makes predictions of the sample rather than the noise, facilitating the use of established geometric losses. As sampling of these approaches in raw motion space computationally expensive, MLD (Chen ct al. 2023) employs VAE with transformer backbone to map motions into latent space, where the diffusion model is trained. However, current motion diffusion models interpret the human motion as a sequence of poses, overlooking the inherent continuous temporal dynamics. They are also con- strained train exclusively on motions with fixed yet relatively low framerate, which hinders their ability to generate motions at higher framerates. In this paper, we combines the merits of diffusion model and implicit neural representation produce high-quality motions free from size limitations.
Implicit Neural Representations (INRs). INRs have shown extraordinary capability in modeling 3D shape neural representations (Park et al. 2019 Mescheder et al. 2019). In particular, NeRF (Mildenball et al. (021) uses multi-layer perceptron network to render 3D-consiteent images with texture details. Following the success of NeRF, INRs have proven to be a powerful tool in many tasks such as view synthesis (Bautista et al. 2022 Watson et al. (023), image generation (Dupont et al.) 2023 Chai et al. 2022) and video encoding (Chen et al. 2021). Most recently. concurrent works
(He et al.) 2022 Cervantes al. 2022 Wang al.) (2023) have been proposed to learn implicit neu- ral fields to represent human motions. NeMF et al.] (022) adopts a VAE framework to make it generative model controlled by latent code. However, this approach requires re-training the neural network to overfit a new signal, which is computationally costly. Moreover, NeMF can only find one possible latent code to satisfy conditional constraints, leading deterministic motion genera- tion like NeMo /Wang et al.) 2023). Such problems have been addressed by (Cervantes et al. 2022). which generates novel motion sequences for a target action class by fitting conditional Gaussian Mixture Model (GMM). Unfortunately, this specific design can be only used for action- conditiona motion generation, and is not versatile enough for other conditions (e.g. texts). In addition, these methods still do not support urbitrary-frameran training. In our approach, we draw upon the inno- vations in coordinate-based functions to sample motion clips at different framerates, and introduce the diffusion model to capture variational distribution of latents, thereby enhancing the capability of diverse motion sampling with flexible framerates and multimodal conditions.
3 METHOD
An overview of our NeRM is described in Figure 2 Our goal is to learn a generative model to synthesize human motion given an arbitrary condition. Let X (xr Im ) denote collection of training samples, where each sample x' [T is motion sequence of T human poses.
We decompose the task of learning a generative model in two stages. The first stage is responsible for capturing the motion of different framerates and durations, thereby learning latent code 2; for
3

---

Published as a conference paper at ICLR 2024
amooo Dpacoad Posscoo
Codebook Coordinate
Attemtio Moduies
Dencise
Framenate embedking
samptre contert v see an
Codebbokk
Re- parmistrttzttinn
Condition calngery
Seperated
Motion
Mi
(0.1)
Roomsdractedioss
Ciep
KJ, Joss
Live
CizE
Figure 2: sto-stage pipeline of NeRM. In the first stage (left), we sample clips random framerates from full-size motion sequences for training. A latent code z is introduced as the encoding of these motion clips, which directly optimized by the parameters of posterior normal distribution instead of from motion encoder. The latent, together with the rodebook-enhance tu and the framerate 8, are then fed into the decoder fe to produce motion clips. The second stage (right) uses the latent codes input to our diffusion model and can be guided by various conditions.
normalized coordinates
each x' E X, while the second stage learns the distribution of the latent variables obtained from the first stage. This decouples the modeling of the complex distribution from varied-size motions.
3.1 MULTI- VARIATIONAL IMPLICIT NEURAL REPRESENTATION
In standard motion synthesis, all training samples typically required have consistent fixed framerate (see Figure D Our key idea is to naturally exploit the variety of motion framerates available in the dataset, learning from poses that are discarded, to enable high-framerate generation.
Variational INRs. Unlike most works that represent motion as a discrete sequential process (Wei al., 2003), we represent motion continuous field of human poses the temporal domain so that the inherent continuous temporal dynamics can be preserved. Formally, we denote a human pose of sequence at timestep as x; E Jx that represented by either joint rotations or positions, where J is the number of joints and D is the dimension of the joint representation.
For each sequence i, we define the continuous motion field as a function f((t) at that maps a temporal coordinate to a pose. By minimizing the reconstruction loss between ground truth {ai]i and the generated motion { we can obtain an overfitting neural field fa(-) for sequence i, thereby generating poses arbitrary time steps by sampling the field. However, such modeling requires to be re-trained from scratch for a new motion sequence. Therefore, we introduce a latent encoding of sequence thus parameterizing the whole training set X as a function fo : (t, 22,) + ff, where fo is a decoder shared among all sequences (i E {1, 2,..... ... ..n,,, n.), and z; is shared among all time steps (t E {1,2,-. ... .T)) This formulation allows us swap different latents for producing different motion sequences.
code 2;
Nevertheless, the latent space distribution pl is continuous since each z; is optimized inde- pendently to reconstruct a single sample via an over-parameterized decoder fo- This renders the interpolation between different points in the latent space meaningless. Inspired by VAE (Kingma & Welling 2013). we propose a variational INR to leverage the continuous nature of the latent space distribution. Specifically, we treat each latent code z; normal distribution, where the mean μ4; and covariance matrix Σ; are optimized. During training, we use the re- parameterization trick to sample an instance from this distribution. As such, the generated outputs exhibit smooth transitions between the corresponding and zj. The motion field can thus be formulated as
fe (t, z.) + 2/. s.t. 2; ( E.).
(1)
We represent human motions arbitrary framerates s, with different durations I, continuous functions over normalized temporal coordinates, and sample a random motion clip according to the center e and the number of poses in clip. This enables any framerate training of NeRM.
Contimuous-frameria training, To learn from multi-fe falteiifa motions in training sets, we design approach by exploiting the temporal consistency in motions generate motion clips, which
4

---

Published as a conference paper at ICLR 2024
is similar to patches in image. Images continuous in 2D space, and motion sequences continuous only in temporal dimension, but we can not simply handle this difference by ignor- ing one dimension. The essential difference between image and motion lies in that, image size is determined only by resolution, while motion size is determined by both framerate and duration. For convenience, we assume maximum duration of Lma seconds for the motion continuous
Recl Tenpand
and treat each motion function defined on bounded, normalized coordinate domain ranging from -1 1. This implies that each second corresponds to an interval of 2/lmaz- The decoder Sa always generates motion clips of fixed number of poses m, but each clip implic- itly corresponds to short segment, cen- tered at v E [-1,1], of the longer (entire) motion. Denoting the framerate of the i-th entire motion as sf fps, we have that the in normalized coordi-
Eoerdi Binkl
Coontrater Bape
Figure 3: Illustrative description of the random clip sampling from the entire motion at framerate accord- ing to the center v and the clip size m.
clip size is 2__
Imer
nates (see Figure 33). During training,
first sample clips from motions, and then pass them to the fixed-framerate decoder se- Therefore, Eqn. can be reformulated as:
fo (fe 2i, s²) M ldssp. s.t. ~ N(p. E,)
where t., E _ denotes normalized temporal coordinates of motion clip with poses, which is extracted from all coordinates t" = [-1, -s'-man -s'-lmant4
a'mmann sm-amax 1] E Rl
st-Imar 3"-lmar
according the clip center v. and á d'p denotes the corresponding human poses. Compared to current motion generative models (Tevet et al.) [023] Chen et al.) (023), our approach replaces the explicit input sequences with clip-dependent coordinates to allow for variable-framerate training.
Codebook-enhanced representation. As observed in Tancik et al. (2020), the Fourier em- bedding of coordinates endows INRs with the capability to learn high-frequency variations, ad- dressing the problem know as "spectral bias" (Rahaman et al., 2019]. Specifically, we employ (t) (cos(2tb) t), sin (2 cos (os(2tbnt)) sin(Trrt] as our embedding function, where by is sampled from an isotropic distribution. However, we empirically find that such embedding insufficient for the decoder fo to characterize human motion fields effectively.
To overcome this limitation, inspired by CoCo-NeRF (Yin al. 2022) that utilizes codebook in- Coordinate formation for 3D geometry neural representations, we propose to deploy Codebook-Coorddnatee At-
tention (CCA) modulation to enrich the Fourier features of each coordinate. The detailed design of CCA can be found in Appendix. Given a pre- mre-trained codebook containing N codes E {e, N with e; Rd where d is the dimension of codes, we define Q {q:} ) /== the learnable query vec- tors, which are used to query the motion-relevant prototypes from the codebook via a cross-attention mechanism. The feature representations of these prototypes are then iteratively boosted by passing several self-attention layers. Subsequently, the prototype information gradually incorporated into each (t) through gross-attention modules. This workflow establishes connection between the codebook prior and coordinate embedding, thereby improving its feature representation.
Loss objective. For the motion sequence i, NeRM calculates the reconstruction loss over all time steps of the generated motion clip and the ground truth, and uses the Kullback-Leibler (KI.) diver- gence to encourage the generated latent space approach target distribution:
L = Ciee + AKI = 1| =2 + AKI.DR (N Es)lp(())),
(3)
where we regularize the posterior to match a prior p(z) that is set as NNO0 I). Diffusion processes (which we show later) converge towards Gaussian distributions so modeling data to approximate this distribution results in faster and more stable training. We also add ΛKL to control the strength of regularization. The optimization problem for the model parameters can then be defined as following:
(1t E,))=11, = arg min L. Me
g = arg min
Σ
min Li P.
(4)
5

---

Given a set of latent codes Z {z; ~ NNu E!) optimized in (4). we aim at learning De- noising Diffusion Probabilistic Models (DDPMs) (Ho et al. 2020 to capture the distribution p( Z) their latent space. In general, DDPMs with discrete timesteps have a fixed Markovian forward process g(=x|=*-1) where g(zu) denotes the data distribution and g(=x) is defined to approximate the standard normal distribution, where the subscript represents the time step. DDPMs then learn to Published as a conference paper at ICLR 2024
where the first term represents the clip-wise parameters for each sequence, while the second term represents the decoder parameters that are shared across the entire dataset, with multiple framerates.
Progressive training. At the beginning of to Iearn globally coherent motions, like conventional motion synthesis methods. The clip temporal unchanged. Due the lower computational cost this time,
training, first train at fixed framerate
coordinate f,, and the framerate we are able to train the model for an extended number of iterations associated with this setup. Sub- sequently, we involves multiple framerates. As the model has already learned crucial representations from fxee-framerrae training, fewer iterations are required to achieve convergence.
3.2 CONDITIONAL MOTION LATENT DIFFUSION MODEL
reverse the forward process P (zk) with learnable parameters é. We train our latents Pz(Z) by learning to denoise 2ą to Zg for all timesteps ls (Ho et al. 2020), which can be formulated as:
min Z,, [lle Eslvaaz + VI âgε, k)]
where k denotes the timestep sampled from uniform distribution, Or denotes noise magnitude parameter with a fixed scheduling, E (0, I) is the noise, and és denotes the learned denoising model. We build a transformer-basee denoising model with long skip connections for Eg. whose superiority has already been proved by time series data (Bao et al. 2023).
Flexible conditions. Our diffusion model in latent space also supports conditional motion genera- tion by capturing distribution of p( (C), such as text and action. Given paired data {2 Z, c C), only the denoising model C need to augmented with conditioning variable e form co (z, k c), sharing common motion variational INRs. To address various the domain encoder is employed for condition embedding, e.g., we employ the frozen text encoder of CLIP (Radford et al. 2021 to map text prompt, and build the learnable embedding (Petrovich et al., 2021) for each action category. We incorporate these embedded conditions into a transformer-basee Eg by concatenation.
Sampling. At the inference time, we start by sampling žк (0, and iteratively apply eg to denoise zk. We also perform classifier-free guidance (Ho & Salimans 2022) with conditional zero-mask instead of
variable to increase diversity and prevent overfitting. In practice, the condition with a certain probability at each training iteration. When sampling, we have:
<o(zk, k, c) = (z k, c) + r)) k; 0),
(6)
where r is the guidance scale. After reversing the diffusion Markov Chain obtain zo, we feed as input to the variational INR decoder So and reconstructs plausible human motions.
4 EXPERIMENTS
We evaluate NeRM on tasks of: (1) text-to-motion, (2) action-to-motion and (3) unconditional mo- tion generation. We provide dataset introduction, evaluation metrics, results and visualizations. We present a new motion quality metric clip-FID that measures FID over varied-framerate motions, and quantifies the performance of our distinctive high-framerate property. More qualitative results, ablation studies and implementation details are in Appendix.
4.1 TEXT-TO-MOTION
Datasets, Given free-form texts, we conduct text-to-motion experiments on KIT Motion-Language dataset (Plappert et al. 2016) and HumanML3D (Guo et al.) 2022). (1) The former one is composed of 3,91 human motion sequences with 6,353 free-form texts as descriptions. Usually, prior works Tevet al. 2023) downsample the 100 fps motion data into 12.5 fps for training and testing; we, however, maintain its original framerate. (2) The latter one is a recently proposed dataset that inte- grates AMASS (Mahmood et al. 2019) and HumanAct12 (Guo al. 2020) form 14,616 motion

---

Published as a conference paper at ICLR 2024
Table I: Results of conventional text-to-motion synthesis HumanMI.31 and KIT dataset. All methods use the real motion length from the ground truth for guidance. The right arrow + means results are better when closer to that of real motion. means unavailable results. Bold indicates best results; underline indicates second best; + indicates 95% confidence interval.
HumunML33
Cao al. 2022
KIT Plappent al. 2016
Method
HID L
R-Procision
Muhtindda Dist
Doversity
MM
PID
R-Precisios
Multinodil Dist
Divessity
MM
(Tup-3)
(Tup-3)
600
0.75777-0 00
20774 -334
9,500 mG
0.011 AM
0,779 900
2.788h-c1
11.08 mm
1L22
Abule & Mocency
2019
12,02 B6G
0, 0,886
5,200++-05
7.076
6.545 003
0,083+-00
9.077 5.147
6.5
0.552-000
5.0124.224
8,332 mig
5.200=-ur
0.5314-017
4.08g
9,563=0m
12M
Crun et
2044
1
0.7
9,18
2 CM0
2.770
0.089g--00
3,401
10.91
1.482
at
2023
2.521
MDM
Tevel ol 2015
0.544
0.611
5,58844-22
9,530
2. 2,790
0,497
0,289
9.191
10.85
1-507
Phy
0.433
MLD
Ches ét al. 2012
0.47a
0, 0,772m
3,196
U.724
2. 2-113
0.404
0,734
3.2014--32
10.90
2.192
NcRM tdised framerate train)
0.489 o1a
0.7774-000
3.18
9.092 mi
2:330 on1
nos
0.7277-005
3.803
10.50* mis
1.501 9-3
NrRM (nzative framerate train)
0.389
0.779
3.178
9.547
2.105
0.472
0.736
3.180
10.94
1,785
sequences with 44,970 textual annotations. It totally involves 19 sub-datasets, whose framerates vary from 20 fps to 250 fps. Existing motion generative models downsample them into 20 fps, and represent each frame by concatenating root velocities, joint positions. joint velocities, joint rotations and foot contact binary labels. We keep this representation, but on their original framerates.
Evaluation metries. Following Guo et al. (2022). we evaluate our model by calculating: (1) Frechet Inception Distance (FID), the similarity between the distribution of generations and real motions; (2) R-Precision and Multimodal-Diss the degree which the generated motions adhere to the given texts; (3) Diversity, the variance within the generated distribution; (4) Multimodality (MM), the gen- eration diversity within single text prompt. In addition, the aforementioned metrics are designed for evaluating global structure. but ignore the realism of details. We thus propose a new metric dubbed (5) elip-FID to evaluate the quality of nigh-frammrrte generative details. It randomly ex- tracts small clips from high-framerate real/generated motions, and calculates FID over ground truth and generated motion clips, sampled at random center u and clip size m. As clip-FID preserves the target framerates without downsampling, we find it more sensitive to local details, enabling a more accurate assessment of artifacts such as foot sliding. Detailed introductions are in Appendix.
Conventional motion generation. We compare our NeRM to text-to-motion baselines JL.2P (Ahuja Morency 2019). Hier (Ghosh et al. (02T), T2M (Guo et al.) [022), and more recent motion diffusion models MDM (Tevet et al. 2023). MoFusion (Dabral et al. 2023). PhysDiff (Yuan et al. 2023), and MLD (Chen et al. 2023). In Table I we show our performance of 20 fps generation on HumanML3D and 12.5 fps on KIT, which are aligned with the generations of existing baselines. Our method outperforms these baselines in terms of FID, R-Precision, Multimodal Dist and Diversity on both datasets, which proves that utilizing the native framerates of raw data is much beneficial for generation. This advantage exactly stems from our capability of handling varied-framerate data that other models are unable to process. It is worth mentioning that our method achieves better results on HumanML3D dataset than on KIT dataset because the former one has more abundant motion data.
Notably, since NeRM is constructed with two stages (INR diffusion), we draw our attention to the comparison with MLD (Chen et al., 2023) that also generates motions with a two-stage (VAE + diffusion) design. Our model outperforms MLD on both datasets. This is owing to the advantage of continuous motion fields in Additionally, compared to transformer backbone in VAE decoder of MLD, NeRM achieves fast inference speed due to its simple MLP decoder of INR, which can be found in Appendix.
framework, rather than explicitly modeling motion inputs.
Fairness discussion. To remove our superiority brought by raw training data, we also train NeRM on pre-processed fixed-framerate datasets for fair comparison with baselines. From Table I NeRM fixed-framerie train) obtains comparable results to others, which can be attributed to our powerful latent diffusion framework and codebook-enhancee representation. Also, it significantly surpassed by our native-framerae training, which further confirms the effectiveness of using raw data.
High-framerate motion generation. Results in Table 2 show our unique property of generating high-quality motions at different high framerates. However, for basclines, they can neither undertake the memory burden and sampling time brought by high-framerate motions, nor effectively train on
7

---

Published as a conference paper at ICLR 2024
(o) Conventional generation
(c) Generation at any time steps
Text: Person walks then takes lorge their left and then goes back onto the some path they were
Ficed-infervel genanction
(0)
Ours
Full la batk onlo the
All cornect
MLD
Sa Holl lost slaaid and
Thr Deg which S artiioct
aadias Bank
alde Eeewards
The TIgS too correctly
PIEL lost fedeeaician Junn ragli tben groundd led sdacts
MLD
Ours
Text: This person shuffles right then walks forward and circles back.
(b) Highhframerate details
Figure 4: (a) Generating 20 fps motion. The failure in MLD is removed by NeRM. (b) Generating 100 fps motion, where figa-frrmerate details are highlighted with purple, pink, yellow and green sequentially. NeRM directly generates them, showing delicate, realistic and reasonable movement details: while MLD can only realize low-framerate generation, so we attach it with interpolation to force it into high-fran nigh-framerate, which appears foot sliding artifacts. (c) Temporal sub-sampling: NeRM can realize any-time generation, while current models (e.g. ML.)) cannot.
varied-framerte framerate raw data. Therefore, for fair comparison under high-framerate setting, we conduct upsampling by spherical linear interpolation (Shoemake 1985) on their low-framerate generations towards the target framerate, and calculate the corresponding clip-FID values. NeRM yields much better performance all the target framerates, indicated by clip-FID metric that provide special evaluation on generated high-framerate details.
Visualizations. We further investigate the ability of NeRM by the qualitative analysis. Figure 4h illustrates the visu- alizations of conventional 20 fps gener- ation on sample from HumanML3D between MLD and ours. The failure of MLD does not happen to our model, which proves that our full usage of raw data and continuous human motion fields are effective for high-quality motion generation. To demonstrate the high-framerate genera- tion ability NeRM, we also exhibit their visualized comparison over high-framerate performances in Figure 4b. We set the target framerate to 100 fps and amplify the details on human feet. We ob- serve that our model yields smooth and natural generations, while the baseline can only generate the fixed 20 fps and then conduct motion interpolation towards 100 fps. This operation will lead to typical kind of low-quality phenomenon, i.c., foot sliding, which results in incoherent and less high-performance devices. Remarkably, another unique property of our NeRM is to enable the generation of pose at specific time step directly without first generating frames before. while maintaining the smoothness of the entire sequence. Existing state-of-th the--rt text-to-motion methods have to sequentially generate the motion at fixed time intervals, which lim- its their application to temporal sub-sampling. As shown Figure 4b. NeRM is able infer any frame by sampling the temporal field t.
Table 2: Evaluation of generated motions at different framerate (fps) on HumanML3D dataset using clip-FID.
Method
20
40
60
100
120
T2M
1.067
2.831
6.442
9.182
11.264
MDM
0.544
1.882
4.605
5.966
8.401
MLD
0.473
1.465
3.816
5.138
7.878
NeRM
0.389
0.493
0.680
0.903
1.315
life-like human behaviors
4.2 ACTION-TO-MOTION
Datasets & Metrics. Action-to mottnntt-motion aims generating relevant motion sequences given input action label. We experiment on UESTC (Ji et al., 2018) and HumanAct12 (Guo et al.] (020). The former one is composed of 40 action classes with 25K samples, while the latter contains 12 classes
8

---

Published as a conference paper at ICLR 2024
Table 3: Quantitative results of action-to-motion synthesis UESTC and HumanAct| dataset.
UESIC
et
2018
Hummnce12
Guo et al. 2020
Method
FID
FID
Accuracy
Diversity -
FIDras
Accuracy
Diversity
Real
2,92
2,70
0.988
$3.31* 520
16.16
0.020
0,997
6,850 100
2.450
ACTOR Petrovich et 2021
20,5
20. 20.0342.00
0
31,96
14,82
0,905 0.120
0.840 100
2.530
MDM
Tevel el 2025
9.9gh1 as
12.8 12.81 a6
0.95g
$3.02
14.26 12
0.100 no
0.990 000
2.520*.000
MLD
12
15.79
0.95g
$3.52
14
13.57=00
0,077 oue
0.964
10002
0.831 100
2.624
INR-MEL
Cervanies al. 2022
9.5 9.55+-00
15.00=.00
0
31.50
14.68=00
0.114
0.970
6. 786
2,507
NeRM (Ours)
11
0.956
33 10.20
14.11
0.106
0
6.806 0022
2.492
with 1,191 samples. As these two datasets do not provide high-framerate data, we remain their original framerates during training, and evaluate NeRM using the set of metrics in (Guo et al. 2020), including FID, Accuracy, Diversity and Multimodality. This task indicates that our NeRM can accept other multi-modal signals as conditions.
Results. We compare our NeRM to baseline ACTOR (Petrovich et al. 2021), INR-MLP (Cervantes) et al., 2022), MDM (Tevet et al. 2023) and MLD (Chen et al. 2023), where ACTOR and MLD are under transformer-basee by Tevet et al.] (2023), we perform 20 evaluations, each consisting of 1000 samples, and present the average along with a 95% confident interval. Quantitative comparison is reported in Table 3 where NeRM achieves competitive results on both datasets. It is worth mentioning that INR-MLP (Cervantes et al. 2022) is also built upon implicit neural representation and consist of MLP layers, similar ours. However, INR-MLP can only accomplish the action-conditioned generation task and cannot apply to other tasks due to its specific design. Moreover, it focuses on variable-length motion generation and does not consider crbitrary-framerde training sets and ligh-rraerrae generation. NeRM outperforms INR-MLP in terms of Accuracy and Diversity, demonstrating the high capability of latent diffusion model and the enhanced representation of sadebook-coordingg attention.
VAE setting; MDM and MLD under diffusion setting. As provided
4.3 UNCONDITIONAL GENERATION
Datasets & Metrics. Finally, we use part of AMASS (Mahmood al. 2019 dataset evaluate the generations of NeRM. This dataset can be also regarded as the motion part of HumanMLLD (Guo) al. 2022), discarding its text part. We train on its native varied framerates, with FID, clip-FID and Diversity for motion quality and diversity.
Results. We compare NeRM with baseline ACTOR (Petrovich et al. 2021). MDM (Tevet et al.) (2023), NeMF He et al. 2022) and MLD (Chen et al., 2023). For AC- TOR, we use the 6-layers transformer VAF from it, and
ACTIOR
2M
4.04
CA2
then follow TEMOS (Petrovich et al.) 2022) to make it class-agnostic, as suggested by MLD. The other three methods naturally support this task. In Figure 5 we show that NeRM achieves the best performance against baselines over conventional generation in terms of mo- tion quality, and competitive results in terms of diversity. NeRM also outperforms INR-based method NeMF due to flexible multi-framerate training and powerful generative ability of the diffusion model.
7:m
1.22
HD
Dinraaly
Opt1
Figure 5: Comparison of unconditional motion generation on part of AMASS dataset with state-of-the-art methods.
CONCLUSION
In this work, we present NeRM, novel neural representation for human motions, which involves learning a latent representation and then capturing its distribution with diffusion models. Our key insight is that by directly learning a continuous motion field over temporal coordinates without explicit modeling, we can train neural network jetworkjointly on processed low-frameratt motions to learn global structure and on clips from the raw varied-frameratt framerate motions to learn details. This lifts the fixed-frameria framerate requirement of previous motion generative models that treat motions consecutive frame sequence and discard frequencyy details. With such representation, we show that NeRM can efficiently generate high-quality motions at high framerates with flexible conditions controlled.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENT
This work was supported in part by the National Natural Science Foundation of China (NO. 62176125, 61772272).
REFERENCES
Chaitanya Ahuja and Louis-Philippe Morency. Language2pose: Natural language grounded pose forecasting. In International Conference on 3D Vision, 2019.
Simon Alexanderson, Rajmund Nagy, Jonas Beskow, and Gustav Eje Henter. Listen, denoise, action! audio-driven motion synthesis with diffusion models. ACM Transactions Graphics (TOG), 42 (4):1-20, 2023.
Maor Ashkenazi, Zohar Rimon, Ron Vainshtein, Shir Levi, Elad Richardson, Pinchas Mintz, and Eran Treister. Nemn learnin neural representations for neural networks. International Conference on Learning Representations, 2023.
Fan Bao, Shen Nie, Kaiwen Xue, Yue Cao, Chongxuan Li, Hang Su, and Jun Zhu. All are worth words: A vit backbone for diffusion models. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, 2023.
Miguel Angel Bautista, Pengsheng Guo, Samira Abnar, Walter Talbott, Alexander Toshev, Zhuoyuan Chen, Laurent Dinh, Shuangfei Zhai, Hanlin Goh, Daniel Ulbricht, al. Gaudi: A neural architect for immersive 3d scene generation. Proceedings of the Advances in Neural Infor- mation Processing Systems, 2022.
Yujun Cai, Yiwei Wang, Yiheng Zhu, Tat-Jen Cham, Jianfei Cai, Junsong Yuan, Jun Liu, Chuanxia Zheng, Sijie Yan, Henghui Ding, et al. A unified 3d human motion synthesis model via conditional variational auto-encoder. Proceedings of IEEE/CVF International Conference on Computer Vision, pp. 11645-1 1665511655, 2021.
Pablo Cervantes, Yusuke Sekikawa, Ikuro Sato, and Koichi Shinoda. Implicit neural representations for variable length human motion generation. In Proceedings of the European Conference on Computer Vision, 2022.
Lucy Chai, Michael Gharbi, Eli Shechtman, Phillip Isola, and Richard Zhang. Any-resolution train- ing for high-resolution image synthesis. Proceedings of the European Conference on Computer Vision, 2022.
Hao Chen, Bo He, Hanyu Wang, Yixuan Ren, Ser Nam Lim, and Abhinav Shrivastava. Nerv: Neural representations for videos. Proceedings of the Advances Neural Information Processing Systems, 2021.
Xin Chen, Biao Jiang, Wen Liu, Zilong Huang, Bin Fu, Tao Chen, and Gang Yu. Executing your commands motion diffusion in latent space. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
Rishabh Dabral, Muhammad Hamza Mughal, Vladislav Golyanik, and Christian Theobalt. Mofu- sion: A framework for denoising- diig--ddifsioo-baseee motion synthesis. Proceedings of the IEEE/CVF Conference Computer Vision and Patten Recognition, 2023.
Emilien Dupont, Hyunjik Kim, SM Eslami, Danilo Rezende, and Dan Rosenbaum. From data to functa: Your data point is a function and you can treat like one. In Proceedings of the International Conference Machine Learning, 2023.
Brandon Y Feng. Hadi Alzayer, Michael Rubinstein, William T Freeman, and Jia-Bin Huang. 3d motion magnification: Visualizing subtle motions with time varying radiance fields. Proceed- ings of the IEEE/CVF International Conference Computer Vision, 2023.
Anindita Ghosh, Noshaba Cheema, Cennet Oguz, Christian Theobalt, and Philipp Slusallek. Syn- thesis of compositional animations from textual descriptions. In Proceedings of the IEEE/CVF International Conference Computer Vision, 2021.
10

---

Published as a conference paper at ICLR 2024
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Proceedings of the Advances in Neural Information Processing Systems, 27, 2014.
Chuan Guo, Xinxin Zuo, Sen Wang, Shihao Zou, Qingyao Sun, Annan Deng, Minglun Gong, and Li Cheng. Action2motion: Conditioned generation of 3d human motions. In Proceedings of the 28th ACM International Conference on Multimedia, 2020.
Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang, Wei Ji, Xingyu Li, and Li Cheng. Generating diverse and natural 3d human motions from text. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
Chengan He, Jun Saito, James Zachary, Holly Rushmeier, and Yi Zhou. Nemf: Neural motion fields for kinematic animation. Proceedings of the Advances in Neural Information Processing Systems, 2022.
Jonathan Ho and Tim Salimans. arXiv:2207.12550 12598, 2022.
Classifier-free
diffusion guidance.
arXiv preprint
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Proceed- ings of the Advances in Neural Information Processing Systems, 2020.
Yanli Ji, Feixiang Xu, Yang Yang, Fumin Shen, Heng Tao Shen, and Wei-Shi Zheng. A large-scale rgb-d database for arbitrary-view human action recognition. Proceedings of the 26th ACM international Conference Multimedia, 2018.
Korrawe Karunratanakul, Konpat Preechakul, Supasorn Suwajanakom,, and Siyu Tang. Guided motion diffusion for controllable human motion synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 215121111222 2162, 2023.
Diederik P Kingma and Max Welling, Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 2013.
Hema Swetha Koppula and Ashutosh Saxena. Anticipating human activities for reactive robotic response. In International Conference on Intelligent Robots and Systems, 2013.
Jogendra Nath Kundu, Maharshi Gor, and R Venkatesh Babu. Bihmp-gan: Bidirectional 3d human motion prediction gan. In Proceedings of the AAAI Conference on Artificial Intelligence, pp- 2019.
Maosen Li, Siheng Chen, Yangheng Zhao, Zhang, Yanfeng Wang, and Qi Tian. Dynamic mul- tiscale graph neural networks for 3d skeleton based human motion prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
Naureen Mahmood, Nima Ghorbani, Nikolaus F Troje, Gerard Pons-Moll, and Michael J Black.
Amass: Archive of motion capture as surface shapes. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, 2019.
Wei Mao, Miaomiao Liu, Mathieu Salzmann, and Hongdong Li. Learning trajectory dependencies for human motion prediction. In Proceedings of the IEEF/CVF International Conference on Computer Vision, 2019.
Wei Mao, Miaomiao Liu, and Mathieu Salzmann. Generating smooth pose sequences for diverse hu- man motion prediction. In Proceedings of the IEEF/CVF International Conference on Computer Vision, 2021.
Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Oc- cupancy networks: Learning 3d reconstruction in function space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
Marko Mihajlovic, Shunsuke Saito, Aayush Bansal, Michael Zollhoefer, and Siyu Tang. Coap: Compositional articulated occupancy of people. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 13201-13210, 2022.

---

Published as a conference paper at ICLR 2024
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, (5()):99-106, 2021.
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
Mathis Petrovich, Michael Black, and Güll Varol. Action-conditioned 3d human motion synthesis with transformer vac. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.
Mathis Petrovich, Michael J Black, and Gül Varol. Temos: Generating diverse human motions from textual descriptions. In Proceedings of the European Conference on Computer Vision, 2022.
Matthias Plappert, Christian Mandery, and Tamim Asfour. The kit motion-language dataset. Big data, 4(4):236-222, 2016.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In Proceedings of the International Conference on Machine Learning. 2021.
Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In Proceedings of the International Conference Machine Learning, 2019.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen, Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 1(2):3, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, 2022.
Olaf Ronneherger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In Proceedings of the Medical Image Computing and Computer-Assisted Intervention, 2015.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Proceedings of the Advances in Neural Information Processing Systems, 2022.
Ken Shoemake. Animating rotation with quaternion curves. In SIGGRAPH, pp. 245-254, 1985.
Ivan Skorokhodov, Savva Ignatyev, and Mohamed Elhoseiny. Adversarial generation of continuous images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, 2021.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. Proceedings of the International Conference on Machine Learning. 2015.
Xiaoning Sun, Huaijiang Sun, Bin L., Dong Wei, Weiqing Li, and Jianfeng Lu. Defeenet: Con- secutive 3d human motion prediction with deviation feedback. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 5527-5536, 2023.
Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Kridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Proceedings of the Advances in Neural Information Processing Systems, 2020.
Guy Tevet, Sigal Raab, Brian Gordon, Yonatan Shafir, Daniel Cohen-Or, and Amit Bermano. Human motion diffusion model. In International Conference on Learning Representations, 2023.
12

---

Published as a conference paper at ICLR 2024
Jonathan Tseng, Rodrigo Castellon, and Karen Liu. Edge: Editable dance generation from music. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Proceedings of the Advances in neural information processing systems, 2017.
Herwin Van Welbergen, Ben JH Van Basten, Arjan Egges, Zs M Ruttkay, and Mark H Overmars. Real time animation of virtual humans: trade-off between naturalness and control. In Computer Graphics Forum, 2010.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. Proceedings of the Advances in Neural Information Processing Systems, 2017.
Jingbo Wang, Yu Rong, Jingyuan Liu, Sijie Yan, Dahua Lin, and Bo Dai. Towards diverse and natural scene-aware 3d human motion synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20460-20469, 2022.
Kuan-Chieh Wang, Zhenzhen Weng, Maria Xenochristou, João Pedro Araújo, Jeffrey Gu, Karen Liu, and Serena Yeung. Nemo: Learning 3d neural motion fields from multiple video instances of the same action. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2023.
Lin Wang, Yo-Sung Ho, Kuk-Jin Yoon, et al. Event-based high dynamic range image and very high frame rate video generation using conditional generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
Daniel Watson, William Chan, Ricardo Martin Buriala,,,,,,,, Jonathan Ho, Andrea Tagliasacchi, and Mo- hammad Norouzi. Novel view synthesis with diffusion models. International Conference on Learning Representations, 2023.
Dong Wei, Huaijiang Sun, Bin Li, Jianfeng Lu, Weiqing Li, Xiaoning Sun, and Shengxiang Hu. Human joint kinematics ceffusiion-rfinemmee for stochastic motion prediction. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 61 10-61118 2023.
Dong Wei, Huaijiang Sun, Bin Li, Jianfeng Lu, Weiqing Li, Xiaoning Sun, and Shengxiang Hu. Enhanced fine the-grained motion diffusion for text-driven human motion synthesis. In Proceedings of the AAAI Conference on Artificial Intelligence, 2024.
Fukun Yin, Wen Liu, Zilong Huang, Pei Cheng, Tao Chen, and Gang Yu. Coordinates are not lonely-codebook prior helps implicit neural 3d representations. Proceedings of the Advances in Neural Information Processing Systems, 2022.
Sihyun Yu, Jihoon Tack, Sangwoo Mo, Hyunsu Kim, Junho Kim, Jung-Wer Woo Ha, and Jinwoo Shin. Generating videos with dynamics-aware implicit generative adversarial networks. In International Conference on Learning Representations, 2022.
Ye Yuan, Jiaming Song, Umar Iqbal, Arash Vahdat, and Jan Kautz. Physdiff: thysics-guided human motion diffusion model. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023.
Jianrong Zhang, Yangsong Zhang, Xiaodong Cun, Yong Zhang, Hongwei Zhao, Hongtao Lu, Xi Shen, and Ying Shan. Generating human motion from textual descriptions with discrete repre- sentations. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recog- nition, 2023a.
Mingyuan Zhang, Zhongang Cai, Liang Pan, Fangzhou Hong, Xinying Guo, Lei Yang, and Ziwei Liu. Motiondiffuse: Text-driven human motion generation with diffusion model. arXiv preprint arXiv:2208. 15001, 2022.
Mingyuan Zhang, Huirong Li, Zhongang Cai, Jiawei Ren, Lei Yang, and Ziwei Liu. Finemogen: Fine-grained spatio-temporal motion generation and editing. In Proceedings of the Advances in Neural Information Processing Systems, 2023b.
13

---

Published as a conference paper at ICLR 2024
A OTHER RELATED WORK
Human Motion Synthesis. Human motion synthesis aims at generating natural motions consistent with any signal that describes the motion, such action category (Guo al., 2020 Petrovich al. 2021), text (Guo et al., 2022 Tevet et al. 2023), music seng et al. 2023 Alexanderson et al. 2023) and historical pose sequences (Mao et al. 2019 Li et al. 2020). With these guidance, earlier works (Ahuja & Morency 2019 Ghosh et al. (222)) focus on learning a shared latent space for motion and conditions deterministically. limiting one-to-one mapping from condition to motion. Some recent
works have put more emphasis on the promotion of diversity, and learned to model the distribution of motions based on the development of deep generative models, like Variational AutoEncoders (VAEs)
(Kingma& & Welling 2013) and Generative Adversarial Networks (GANs) (Goodfellow et al. 2014).
BiHMP-GAN (Kundu et al. 2019), conditioned on a given starting sequence, uses the discriminator of GANs to regress the random vector for multiple probable predictions. (Cai et al. 2021] presents a VAE-based unified framework for generalized motion synthesis that covers motion prediction, completion, interpolation and recovery. [Wang al. (2022) involves the modeling of human-scene interaction, path planning and body movement, to implement motion generation in the given scene environment with target action sequence. ACTOR (Petrovich et al. 202I) and TEMOS (Petrovich) et al. 2022) suggest employing a VAE to map action labels and texts into variational distribution with transformer structure (Vaswani et al., 2017]. respectively.
However, all the aforementioned methods primarily focus on fixed-size motion modeling, and do not take into consideration the aspect motion framerate. As a result, higher-framerate details and lower-framerate global structures in datasets are disregarded. Moreover, these methods are con- strained to generating motions of fixed sizes. In contrast, our proposed method addresses these limitations by incorporating available varied framerates into considerations and enabling the gener- ation of motions at arbitrary framerates.
Motion Representation. Motion representation is very crucial for the subsequent synthesis task. There existing several motion representation on different datasets, such as (1) the classical SMPL- based motion parameters (Petrovich et al., 202T). (2) the redundant hand crafted motion features (Guo et al.] 2022). and (3) straight-forward joint positions (Mao et al. (019). We opt for the first two representations in this paper. Particularly, The first one is widely used in motion capture, and the second one is mainly used in character animation. Following (Tevet et al. 2023] Chen et al. 20233), we employ the SPML parameters in action-to-motion task for a fair comparison, and redundant hand-crafted features in -ox-tto-motion and unconditional tasks.
B
MORE DETAILS
B.I IMPLEMENTATION
Our NeRM are decomposed into two stages, including INR and latent diffusion. For INR, the hidden layer size is fixed 1,024. We use the pre-trained codebook of (Zhang al., 2023a), which is trained by VQ- VQEE (Van Den Oord et al. 2017) on the HumanML3D dataset. The codebook size set to 512 512. The number of learnable query embeddings of codebook-coordinal attention is 256, and the dimension of each embedding is 128. We employ a frozen CLIP- model as our text encoder for text descriptions, and a learnable embedding for action categories. The shape of latent codes is set to 256, which is then injected into condition by concatenation for diffusion training and inference. Our models are trained with the AdamW optimizer using a fixed learning rate of 10-4. Our batch size is set to 4,09 during the INR training stage and 64 during the diffusion training stage separately. Since INR requires learning a latent code for each training sample, we set its batch size large for efficient training. Besides, INR model was trained for 20,000 epochs and diffusion model was trained for 3,000 epochs. The number of diffusion steps is 1,000 during training while 50 during inference. The corresponding variances 3, in diffusion are scaled linearly from 8.5 x 10-1 to 0.012. We train our models under Pytorch on NVIDIA GeForce RTX 3090.
B.2 AUTO-DECODING OF LATENT CODE
Unlike traditional auto-encoder whose latent code is produced by the encoder, we draw inspiration from DeepSDF Park et al. (2019) in 3D shapes and auto-decoder to learn the latent code
14

---

Published as a conference paper at ICLR 2024
Q
Cross-Attr
Self-Atth
Self -Attn
Self AAtn
Query Embedding:
Normalized
Codebook enhanced Nermalized Coorcinate
Codebook
Temponal
Gress-Atte
Figure 6: Detailed network architecture of
Attention (CCA).
without an encoder. This encoder-less design avoids explicit modeling of raw motions, making arbitrary-framerate training feasible. To construct distributions where one can sample representa- tion generate
high-quality motion, we gain insight from Variational Auto-Encoder (VAE) that utilizes an encoder to infer a distribution from which representations of motions can be sam- pled, and employs decoder reconstruct the data from the representation. As our design does not contain an encoder, we model each motion by sampling from a learnable distribution with op- timized parameters, i.e., mean μ and covariance E, leading to learnable parameters {pr, } J for all training dataset. When training, we sample a latent representation z; from a distribution (H4 E,,, We then combine 2; with temporal coordinate encodings 7(), and input them into a shared MLP decoder fo. The parameters (4i, E; and 0 are optimized simultaneously by minimizing the reconstruction loss between the generated and ground truth.
B.3 SYNTHESIZING LONG SEQUENCES
Ideally, NeRM is possible to generate motions with arbitrary framerates and durations by setting appropriate temporal coordinates. However, either significant increase of requires significant memory resources. Thus, we employ an iterative-synthesis approach to generate non-overlapping motion clips and assemble them into longer motions. By leveraging clip-based multi-framerate training conditioned on temporal coordinates, we learn continuous motion fields and ensure smooth transitions between motion segments. Notably, the framerate range we support is still limited by the original training data. The model is unlikely to learn motion patterns that exceed the highest framerate present in the training data.
C DETAILS OF EVALUATION METRICS
C.I EVALUATION METRICS
Frechet Inception Distance (FID). FID is widely used for overall generative quality evaluation. FID is calculated by extracting features from 1,000 generated motions and real motions in test set. Instead of the inception neural network image domain, we extract deep representation of the motion with the evaluator network Guo et al. (2022), as suggested by Tevet et al. (2023).
R-Precision (Top-3). This metric used for text-motion matching measurement. For each gen- eration, its corresponding GT description and randomly selected 31 mismatched descriptions are gathered pool. Then the Euclidean distance between the motion feature and each text feature in the pool is calculated, where accuracy of Top are picked. The GT description falling into the Top 3 candidates represents a successful retrieval.
MultiModality Dist. Like R- R-Precision, this metric also measures text-motion similarity. We calcu- late the average Euclidean distance between the motion feature of each generation and text feature of the corresponding description.
Diversity. Diversity measures generative variance across all motions. We randomly select mo- tions from all generations, and put each of them in either of the two subsets (.1...., gD. and .gf...., 9D. }, with the same size as D = 300, where q, indicates a motion feature vector. Diver- sity can be expressed as:
D.
Diversity =
Σ 114 - d.1l2.
(7)
D.
15

---

Published as a conference paper at ICLR 2024
Ground 20fps
AIV
Pretrained Evaluator
Baissing Methods
FID
Resize 20rps
Interpolation
Extract motion clip while keeping framerate
Pretrained Evaluator
dip-FID
Our Method
Row Mation Datoset
Ground Truth: Multi-Framerate
Figure 7: Comparisons of conventional FID our clip-FID. FID evaluates global structure, but downsamples all human motions to common 20fps which ignores high-fraue framerreee details. In trast, clip-FID takes motion clips instcad, thereby keeping the original framerates. We employ both metrics to validate the effectiveness of our method.
Table 4: Mean reconstruction errors of MLD and NeRM for motion of different framerates.
Framerates (fps)
Method Metric
20
40
60
100
120
MLD MPJPE
0.027
0.062
0.113
0.184
0.228
NeRM / MPJPE
0.016
0.019
0.013
0.019
0.011
MLD/ MRE
0.074
0.105
0.194
0.254
0.387
NeRM / MRE
0.041
0.036
0.034
0.035
0.038
MultiModality (MM). MM measures the generated motions diversify within each condition (text or action). We randomly pick S text descriptions from all descriptions. Then the motions generated by the s-th description are randomly picked and put into one of the two subsets: {9a.1..... D. or .g..:....,g L,D. J. with subset size D. = 10. MultiModality can be expressed as:
5
D
MM Sx
1 d',il2.
(8)
Accuracy. We employ the pre-trained action recognition model classify 1,000 generate motions. The obtained overall recognition accuracy demonstrates the correlation of the motion and its action label.
C.2 MOTION CLIP-BASED FID (CLIP-FID).
All the metries above are designed for low-framerate data, and unable to be trained on original high-framene data due to prohibitive memory requirements. Thus, we here present a new met- ric clip-FID that is aimed better capturing the realism of details high framerate by avoiding downsampling. As depicted in Figure 7 we show the comparison of standard FID and clip-FID, where FID evaluates global structure (the coarse information the entire motion is preserved), but downsamples all human motions to common 20fps, ignoring high-framerate details; clip-FID takes motion clips extracted/cropped from motions (the detailed information of the motion clip is preserved), keeping the original framerates. Note that we randomly sample clips (a short segment of motion) of size m from real motions at framerate and center v to generate the corresponding clip. A lower value implies better high-framerate details.
16

---

Published as a conference paper at ICLR 2024
Table 5: Ablation study effectiveness of time encoding.
Motion Reconstruction (MRE)
Motion Synthesis (clip-FID)
Simple
Codebook
40
60
100
120
40
60
100
120
x
0.134
0.141
0.164
0.094
0.091
0.471
0.803
1.070
1.769
2.944
x
0.053
0.049
0.039
0.057
0.043
0.397
0.519
0.701
0.142
1.717
v
0.041
0.036
0.034
0.035
0.038
0.389
0.493
0.680
0.903
1.315
Table 6: Ablation study effectiveness of Variational INRs.
Motion Reconstruction (MRE)
Motion Synthesis (clip-FID)
Variational INRs
20
40
60
100
120
20
40
60
100
120
x
0.032
0.030
0.031
0.036
0.027
1,280
2.924
7.012
10.482
14.654
0.041
0.036
0.034
0.035
0.038
0.389
0.493
0.680
0.903
1.315
D NETWORK ARCHITECTURE
Coordinate Attention (CCA). Inspired by CoCo-NeRF (Yin al. 2022). we apply CCA modulation to enrich the Fourier features of each coordinate. The detailed architecture is illus- trated in Figure [ Specifically, we employ one cross-attention block to learn dependency between learnable query embeddings beddings are fed into self-attention blocks with three layers to improve their feature representations further and obtain the final motion-relevant prototypes Q. Finally, we conduct once cross-attention operation between ② and Fourier embeddings of each coordinate (tt. All attention modules are based on transformer (Vaswani et al. 2017) with head Attention mechanism, Layer Normalization, Feed-Forward Network and GELU activation.
{g: T-d and codebook prototypes & = {e: Then, the em-
MLP Decoder. Similar to other neural representations (Ashkenazi al. 2023), the NeRM decoder fe is constructed using a simple neural network architecture. It consists of a 9-layer MLP with ReLU activations and layer normalization. The hidden layer size remains constant across the network. We also incorporate residual connections within each layer to improve gradient flow. In contrast to previous approaches (Chen al., 2023 Tevet al. 2023) that rely on transformer backbone, our choice of a simple decoder offers benefits in terms of inference speed for generating new motions.
Latent Denoiser. Different from the UNet-based architecture (Ronneberger et al. 2015 in latent diffusion model that designed for image synthesis, our latent denoiser 6g is built VIT backbone with long skip connections (Bao al. 2023). which is more appropriate for time series data.
E ADDITIONAL. EXPERIMENTS
Our NeRM consists of an INR model fo and latent diffusion model cs. We conduct additional experiments to evaluate the effectiveness of each component, as well as reconstruction capability. We also report time costs inference and GPU memory fulti-framerate training dataset.
E. RECONSTRUCTION CAPABILITY
We first compare the reconstruction capability of MLD (Chen et al., 2023) and our NeRM with dif- ferent framerates. Note that MLD learns latent representation via Variational AutoEncoder (VAE) based on transformer encoder. We randomly select 100 different motion sequences with their orig- inal framerate HumanML33D dataset, and report the average reconstruction errors in Table Since MLD cannot process multi-framest framerate dataset, we use spherical linear interpolation to generate higher-fi framerate higher-framerate motions for MLD. For evaluation metrics, make use of the Mean Per Joint Posi- tion Error (MPJPE) (Mao et al., 2019). commonly used for image- 3D human pose estimation. We also employ Mean Redundant Error (MRE) that measures the Fuclidean distance between two
17

---

Published as a conference paper at ICLR 2024
Table 7: Ablation study on decoder architecture.
Motion Reconstruction (MRE)
Motion Synthesis (clip-FID)
MLP
Transformer
20
40
60
100
120
20
40
60
100
120
/
0.037
0.032
0.037
0.035
0.037
0.381
0.487
0.607
0.782
1.199
X
0.041
0.036
0.034
0.035
0.038
0.389
0.493
0.680
0.903
1.315
Table 8: Ablation study on the dimension d and weight parameter Aez of latent representation.
128
256
512
MRE@ MREs
0.0459
0,0412
0.0407
AgL.
le-5
le-4
le-3
MRE@ 206ps
0.0410
0.0412
0.0459
poses represented by redundant hand-crafted features. From the results in Table we find that our NeRM with simple MLP can achieve lower reconstruction error with 20 fps than MLD with com- plicated transformer backbone. When the framerate increases, the performance MLD deteriorates significantly while NeRM still maintains very low reconstruction error.
E.2 ABLATION STUDY
In this section, we provide ablation studies on the HumanML3D dataset to evaluate the effectiveness of network design.
Effectiveness of Time Encoding. Time encoding plays a critical role in the generalization. In Table [] we evaluate the influence of different encoding manners, including No Time Encoding, Simple Time Encoding, and Satebooo-Coordinate Encoding. Here, "Simple" means that we only Fourier features to encode t like traditional NeRF (Mildenhall et al. (00T)); and "Codebook" is our design of NeRM. From the table, we find that the model without time encoding achieves poor performance in both terms of motion reconstruction and motion synthesis. This can be attributed to "spectral bias" (Rahaman et al. 2019). In other words, INRs with simple MLP layers cannot learn geghffreeuency variations from motion data. Our codebook-based representation reaches the best performance, which confirms that the codebook is beneficial for feature representation of temporal coordinates.
Effectiveness of Variational INRs. We investigate the influence of variational INR by comparing a variational INR with variatiool version. In this case, the latent code z of -or-variational INR is obtained by optimizing the following:
arg min @
2l for i 1.2.....7
where dii fo (I 8). Comparison results are shown in Table 6 We observe that the reconstruc- tion errors non-variational INRs smaller than variational ones, as non-variational INRs may overfit the motion sequence by powerful neural network. However, the realism of non-variational INRs is significantly improved by the variational ones. This suggests that the variational approach strongly regularizes the latent space and enhances the capability of sampling new motions.
Decoder Architecture. Furthermore, we compare the performance of MLP-based and transformer- based models. Table 7 shows that the self-attention mechanism can slightly improve the ability of motion synthesis. However, the latent representation learned by diffusion model fed to the decoder of our variational INR, which implies that the complexity of the network greatly impacts the generation speed of new motions. Therefore, we pick MLP layers as our decoder architecture.
Effectiveness of latent representation. The latent representation is crucial variable in our NeRM. It acts as a bridge between the variational INR model fo and the diffusion model cg. In our context, the quality of latent representation distribution. N (M, Σ) of z. Therefore, we investigate the impact of different dimensions of z and weight A the KI. loss for motion reconstruction in Table g When the dimension is set to
be influenced by the dimension and the variational
18

---

Published as a conference paper at ICLR 2024
Table 9: Ablation study effectiveness of time normalization.
Time Normalization
MRE@
FID
Diversity
X
0.118
0.958
9.892
0.041
0.389
9.547
1.0-
Methoss
Time
FID
dip-FD
13M
0.052
1.067
8.443
MOM
$2.17>
0.564
4.605
0.6
MLD
0.366
0.473
3816
N+RM
F10
0.6
0.4
02
Time
Figure 8: Average inference time (seconds) for generating motion sequence. The circle size is proportional to the value of clip-FID. Bigger circle indicates worse performance of high-framerate details.
256, the motion reconstruction loss becomes significantly small, and higher dimension can slightly improve the reconstruction ability. For ÅKT, a higher weight results in smoother latent space but increases the reconstruction error. We set λ to a small value (i.e., 0.0001) like Chen et al. (2023).
Effectiveness of time normalization. In this section, we explore the effect of time normaliza- tion. Table D shows the results on HumanML.3D dataset with 20 fps. Notably, 'x' indicates that directly use the true temporal coordinates, i.e., {1, 2.... T), instead of normalized temporal coordinates fe... From the table, we find that time normalization plays a vital role in time encoding of implicit neural representation.
E.3 INFERENCE TIME
In Figure 8
we report average inference time on per sequence. As T2M (Guo et al., 2022) built under VAEs, it uses the least time for generation; but under diffusion setting, are the most time- saving. To be specific, due to the latent diffusion design, MLD (Chen et al., (023) and our NeRM much faster than MDM (Tevet al. 2023). We even faster than MLD in that we use simple MLP decoder of INR, rather than the mansformer-based decoder in MLD. We yield the best FID for best global motion quality, and significant superiority in clip-FID (the target framerate 60 fps), indicated by the smallest circle size. All of these experiments are conducted on NVIDIA GerForce RTX 3090.
to
E.4 MEMORY BURDEN
By padding zeros, current text-to-motion generative models are able to train motions with the same framerate (20 fps) and different duration 2 < ( 9.8 (seconds). When exploiting native framerates of motions, one possible solution is to padding zeros according to the maximum framerates (250 fps). However, this operation cannot capture accurate temporal dependency as Exed-framerate training. Another problem is that the dimensionality of padded motions needs to be much larger than what state-of-the-art diffusion models can be trained on. For example, MDM (Tevet et al. 2023) trains the diffusion model on motions where the maximum number of poses is 196 (9.8s x 20/ps), while we use motions of maximum size 2450 (9.8s x p50 This imposes a substantial burden on approach it from the perspective of implicit neural representation,
GPU memory. Alternatively, which enables us to process clips (a short segment of motion) with size m (m < 2450), therein significantly reducing the memory burden. In addition, find representative latent code for each
19

---

Published as a conference paper at ICLR 2024
A short fragment from the sequence generated by
Indtogrdiiins vileo lae ciskt II Rapppeas to bodlwardss
Text: A man walking forward slowly, and then stops.
Counecily Jamp
Figure 9: High-framerate generation (120 fps). We visualize short fragment from the whole gen- eration, with purple, pink, yellow, green, and grey sequentially indicating high-framerate motion changing. The entire generation can be found in our video.
Conventional generation
High-framerate generation
88%
76%
6.2%
-13%
.4%%
42%
Question (a)
Question (b)
Question (a)
Question (b)
NeRM beats MDM
NeRM beats MLD
NeRM beats GT
Figure 10: User study HumanML3D dataset.
motion and learn the distribution in this low-dimensional latent space efficiently, which decouples the modeling the distribution from varied-size human motions.
E.5 VISUALIZATION
more examples, where our NeRM constantly outperforms MLD (Chen et al. 2023) in both terms of basic motion quality and high-framerate quality. Note that, as MLD cannot directly generate high-framerate mo- tions, we use SLERP to interpolate them towards the target framerate (120 fps). We also select short fragment from our video with full discussion shown in Figure Additionally, for clearer observation on artifacts of baseline, we slow down the video by 6 times, so the motions may appear be slower.
A video is contained in supplement. We provide 120 fps generation
E.6
USER STUDY
Human eyes are the ultimate evaluation for human motion performance. We asked 17 people over question (a) "Which of the two motions more realistic" and (b) "Which the two motions is more consistent with the given texts". In each two-motion pair, provide one motion generated by NeRM and the other by baselines MDM Tevet et al., 2023). MLD (Chen et al.) 2023) and GT from HumanMM3D. We randomly pick 8 cases for each question. We evaluate conventional 20 fps generation and high-framerate generation (cach of 120, 100, 60 and 40 fps has 2 cases). Shown in Figure 10 our NeRM gains more preference than baselines, and even comparable to GT motions.
20

---

Published as a conference paper at ICLR 2024
F LIMITATIONS AND FUTURE WORK
Although our method has achieved promising performance for high-framerate motion synthesis, still has the following challenges. (1) While our method currently supports generation with external conditions such as action labels or text prompts, it has limitations in incorporating fine-grained internal conditions such as keyframes or trajectory.
promising direction for future research is to design more comprehensive framework (as exemplified by Karunratanakul et al. (2023): [Zhang al. (2023b)) that can simultaneously consider both external and internal conditions. By integrating these factors, we can achieve more precise and nuanced motion generation. (2) The quality of motion generated using INRs highly dependent on the dataset. If the dataset lacks high- high-framerate data, the performance of generating high- fraheerreeeete motions may not be optimal. Additionally, the model unlikely to learn motion patterns that exceed the highest framerate present in the training data. (3) While our method demonstrates fast inference time, the training process be relatively slow, particularly when dealing with the dataset containing numerous samples. This is because our method learns latent code for each training sample. (4) Our method is designed for motion modeling and can not adapt to outputs with varying body shapes. Some recent works Mihajjovic et al. (2022) have employed INRs to model skinned articulated objects with specifie poses. It would be interesting explore the integration of our method with them.
21


Reviews:
[{"review_id": "1vbhyMyqFR", "reviewer": "Reviewer_VxZF", "summary": "The paper addresses the challenge of generating realistic human motions at high framerates, a task made difficult by inconsistent training data framerates, memory constraints, and the slow performance of generative models. Current solutions downsample high-framerate details or discard low-framerate samples, leading to information loss. The authors propose NeRM, a generative model utilizing Implicit Neural Representations (INRs) to harness varied-size data for high-framerate motion synthesis without explicitly modeling raw motions. NeRM not only outperforms other methods but also efficiently produces motions at any desired framerate while remaining memory-efficient.", "questions": "1. Two-stage training gives good performance. It would be good to give more details on training the auto-encoding of the latent code.\n\n2. In order to avoid retraining of INR, some other methods choose to use latent code to modulate the weights of INR. Is there any insight for the choice in the paper? \n\n3. How do you determine the number of codes in the codebook?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper proposes to use a novel variational INR to generate arbitrary framerate motion. The experimental results support this claim that NeRM outperforms other methods in different framerate generation. \n\n2. It enables to generalize INR to the new data without retraining by introducing the latent code to the INR input. \n\n3. The presentation is overall clear. The core idea as well as the technical details are well presented and easy to follow. \n\n4. The performance outperforms other baselines.", "weaknesses": "1.  As for the generation part, the part of the input for INR is z, it seems that the model heavily depends on the quality of the latent representation z. \n\n2. The idea of using time INR has been use in NeMF: Neural Motion Fields for Kinematic Animation, which may limit the novelty contribution of this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jjdQx27eSh", "reviewer": "Reviewer_9SXt", "summary": "This paper proposes a neural representation, i.e., NeRM, for representing continuous human motions. NeRM directly learns a continuous motion field over temporal coordinates without explicit modeling, making the training with varied-framerate motions and high-framerate motion generation possible. The authors leverage the proposed representation to (un-)conditional motion generation with diffusion models, showing the efficiency and effectiveness of high-quality motion generation.", "questions": "1. How can we determine if the feet sliding shown in Figure 4 is a result of training with a low framerate?\n\n2. Does the use of variational INR, i.e., normalizing the latent code $\\mathbf{z}_i$ to a normal distribution with KL loss, affect the preservation of high-framerate details? Such normalization typically leads to a smoother representation space?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The primary contribution of this work is the proposed neural continuous motion representation, NeRM, which I find both interesting and innovative. It effectively represents continuous motion sequences at any framerate level. This representation addresses limitations observed in previous works, such as MLD, which necessitates the input motion sequence to have the same framerate and fails to capture high-frequency details. The experiments conducted on various conditional and unconditional motion generation tasks are thorough and robust. The paper is well-organized and clearly presented.", "weaknesses": "My main concern revolves around the motivation of representing and directly generating high-framerate human motions. As discussed in the introduction, the authors present two key points: (1) high-framerate motion generation is inefficient, and (2) training with a fixed framerate cannot adequately utilize the dataset. However, I believe that training with fixed, low-framerate data might suffice to produce high-framerate results through interpolation. Hence, there might be no imperative need to use per-frame human poses during training.\nThe related discussion in the introduction appears to be not highly convincing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rWxhRXi8AI", "reviewer": "Reviewer_KnYZ", "summary": "This paper focuses on the human motion synthesize for frame rate scenarios. It recognizes the limitations of previous efforts of generating high-frame-rate human motion sequences. The key idea is to fuse the data of different framerates into training by normalizing the time positions into relative and centerized time indices with a continuous mapping from time position to the pose configurations. A progressive training is leveraged to relieve the pressure of learning motion patterns under different frame rates by bootstrapping from the fixed frame rate. The generation of motion sequences is done by a conditional diffusion model where the latent code is the motion code from encoders and a codebook-based attention module.", "questions": "Please see my questions above.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The proposed method can accept training data in arbitrary frame rate while supporting human motion generation in high frame rate.\n- The proposed method is flexible to support human motion generation of different schemes, such as unconditional, or conditional to different constraints, such as action label or text descriptions.\n- Leveraging the latent code for diffusion model and integrating the codebook-based attention module, the method is designed in a whole to support different modalities for conditions and good expressiveness.", "weaknesses": "I put both my recognized weaknesses and the questions I would like authors’ response to here. \n\n- Question: Eq 2 is confusing to me. Normalizing the clip length from the standard seconds to the relative length has been commonly adopted. In Eq 2, given both the relative time position and the frame rate, the input information to the generation function f_\\theta is exactly the same as what previous works input. Could the authors elaborate more about their differences here?\n- Question: with different frame rates for the target motion sequence, is the number of time steps in diffusion, i.e. k in Eq 5, the same? I understand that they are two different “time steps” but I still would like to get a sense that whether the diffusion model is able to capture the motion of different frame rates or it is only to recover the pose in a single time step, static.\n- In my understanding the claim that “NeRM can generate motions with arbitrary framerates s and durations l by setting appropriate temporal coordinates.” may be inaccurate. Technically, by the design of the method, the claim can be partially correct. But given the training data captured under certain frame rates, without considering the motion speed and sensitiveness of capture sensor etc, the model is unlikely to learn motion patterns in frame rates exceeding the highest frame rate contained in the training data.\n- Question: is there any reason that a close baseline Nemf (He et al 2022) is not included in the benchmarking comparisons?\n- The method is constructed with multiple components and requires settings of many introduced hyper-parameters, such as the normalizing of time positions, the codebook-coordinate attention and etc. The authors may need to provide corresponding ablation studies to support the effectiveness of these modules and the help readers understand the resources of performance gains more clearly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KaJUBoveeo", "reviewer": "Reviewer_WpJV", "summary": "The paper introduces NeRM, a generative model for high-framerate human motion synthesis using Implicit Neural Representations (INRs).\nNeRM can handle varied-size data and capture the variational distribution of motions for high-framerate motion synthesis.", "questions": "It would be beneficial to include more detailed explanations and insights into the proposed clip-FID metric for evaluating the quality of high-framerate generative details. How does clip-FID preserve target framerates without downsampling and how does it capture local details and artifacts such as foot sliding?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper provides a clear and concise description of the problem statement, methodology, and evaluation metrics. The paper addresses the underexplored task of generating realistic human motions with high framerates. By leveraging the advantages of INRs and diffusion models, NeRM offers a memory-friendly and efficient solution for high-framerate motion synthesis.", "weaknesses": "* The paper could provide more detailed explanations and insights into the limitations and challenges of using Implicit Neural Representations (INRs) for high-framerate motion synthesis. This would help readers understand the potential trade-offs and constraints associated with the proposed approach.\n\n* The paper could benefit from a more extensive discussion on the generalizability of NeRM to different datasets and motion types. It would be valuable to explore the performance of NeRM on diverse motion datasets and evaluate its ability to handle complex and varied motion patterns\n\n* Some more motion synthesis literatures can be included in this paper, such as:\n[a] A unified 3d human motion synthesis model via conditional variational auto-encoder\n[b] Towards diverse and natural scene-aware 3d human motion synthesis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)