Towards Efficient and Diverse Generative Model for Unconditional Human Motion Synthesis

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-06 08:12:53

Abstract: Recent generative methods have revolutionized the way of human motion synthesis, such as Variational Autoencoders (VAEs), Generative Adversarial Networks (GANs), and Denoising Diffusion Probabilistic Models (DMs). These methods have gained significant attention in human motion fields. However, there are still challenges in unconditionally generating highly diverse human motions from a given distribution. To enhance the diversity of synthesized human motions, previous methods usually employ deep neural networks (DNNs) to train a transport map that transforms Gaussian noise distribution into real human motion distribution. According to Figalli's regularity theory, the optimal transport map computed by DNNs frequently exhibits discontinuities. This is due to the inherent limitation of DNNs in representing only continuous maps. Consequently, the generated human motions tend to heavily concentrate on densely populated regions of the data distribution, resulting in mode collapse or mode mixture. To address the issues, we propose an efficient method called MOOT for unconditional human motion synthesis. First, we utilize a reconstruction network based on GRU and transformer to map human motions to latent space. Next, we employ convex optimization to map the noise distribution to the latent space distribution of human motions through the Optimal Transport (OT) map. Then, we combine the extended OT map with the generator of reconstruction network to generate new human motions. Thereby overcoming the issues of mode collapse and mode mixture. MOOT generates a latent code distribution that is well-behaved and highly structured, providing a strong motion prior for various applications in the field of human motion. Through qualitative and quantitative experiments, MOOT achieves state-of-the-art results surpassing the latest methods, validating its superiority in unconditional human motion generation.

Main paper: 
 introduction: 
 Human motion synthesis task aims to generate diverse and coherent human motion sequences that satisfy specific cues or spatiotemporal constraints. This field has wide-ranging applications, such as human motion understanding [31, 39] , autonomous driving [19, 20, 41, 42] , and animation [12, 13] . Unconditionally generating diverse human motions from a given data distribution still presents a significant challenge, especially when the human motion datasets are diverse, unstructured, and unlabeled. Although current human motion generation methods have shown promising ability to produce human motions, fully capturing the potential diversity from a given distribution for unconditional human motion synthesis remains a challenge.Recent advancements in deep generative models, including Variational Autoencoders (VAEs) [6, 18, 21, 35] , Generative Adversarial Networks (GANs) [8, 17, 23] , and Denoising Diffusion Probabilistic Models (DMs) [1, 32] , emerge as the dominant approaches for capturing the data distribution cross all possible human motions. Specifically, VAEs leverage an encoder network to map the data distribution into a Gaussian latent distribution. This representation of latent space is mapped back to the data distribution using a decoder network. Although significant progress has been made, VAE-based methods assume Gaussian distribution for the latent space of the data distribution. This assumption often leads to the generation of blurry human motion joints, thereby reducing the accuracy of the generated human motions. As shown in the bottom of Figure . 1, we visualize the results of the end pose of different motion sequences, most human motions using VAE tend to converge to the same pattern. Another popular approaches are the GAN-based methods, an unconditional generator is trained to generate real-like motions from random noise, while a discriminator is employed to measure the distinction between the generated samples and real ones. Training GANs can be challenging and sensitive to hyperparameters. Moreover, GANs tend to sample human motions from the major patterns while ignoring the minor patterns. DMs-based methods define a diffusion process, in which Gaussian noise is gradually added to the input data using a Markov chain. As DMs-based methods rely on multiple iterations of denoising diffusion steps, each requiring a substantial amount of computation, the overall inference time can be prohibitively long for large-scale datasets or complex models.The purpose of previous human motion generative methods is to calculate a transport map using DNNs. This transport map transforms Gaussian white noise into the distribution of human motions. By pushing the initial noise distribution forward, the transport map generates a new distribution that approximates real human motions. The similarity between these two distributions determines the generalization ability of the designed method [3] . As pointed out by previous works [22, 33] , DNNs can only represent continuous mappings. The transport map may be discontinuous when the human motion data has multiple modes [22] . This phenomenon would incur the mode collapse or mode mixture, in which the generated human motion samples heavily concentrate around the most dense areas of the dataset while neglecting other less frequent or sparse regions of the data distribution during sampling. The sampled human motions often have less diversity and look unrealistic. The problem still exists even when the data distribution has a single mode. These phenomena show that the mode collapse or mode mixture cannot be resolved in existing human motion generative methods.To overcome the limitations, we propose a novel method called MOOT, for unconditional human motion synthesis that leverages optimal transport mapping. Our motivation is to address the mode collapse problem and mode mixture, thereby enhancing the diversity while ensuring the accuracy of generated human motion sequences. Previous generative methods for human motion generation mainly consist of two steps, i.e., manifold embedding and probability distribution transport. Manifold embedding aims to find the encoding and decoding maps between the data manifold embedded in the human motions and the latent space. Probability distribution transport aims to transport a given noise distribution to the data distribution. In this work, the proposed MOOT separates the process into two stages to enhance the diversity of unconditional human motion synthesis. Specifically, in the first stage, a human motion reconstruction network is designed to learn the manifold embedding of human motions. The manifold embedding is implemented by Gated Recurrent Units (GRU) and transformer [30] , which aims to capture temporal smooth and spatial relations between human motions. For the second stage, we employ the optimal transport (OT) mapping to achieve the probability distribution transport and generate diverse human motions. According to Brenier's theorem [5] , the optimal transport map can be denoted by the gradient map of the Brenier potential, which is continuous in the computing process. The OT mapping intends to map the noise distribution into the latent space of human motions, which can avoid the mode collapse and mode mixture, thereby enhancing the diversity of generated human motions. Figure . 1 briefly describe the human motion synthesis process of the proposed method and vanilla VAE. The proposed MOOT can generate more diverse human motion sequences.In summary, our main contributions are as follows:• We propose a novel method called MOOT for unconditional human motion synthesis, which aims to avoid mode collapse and mode mixture based on the theory of optimal transportation mapping. • We employ the encoder of reconstruction network, based on GRU and transformer, to encode human motions into latent space. Then we utilize the Brenier potential between the noise distribution and latent space of human motions to represent the optimal transport map. The extended OT map combines the generator to generate new human motions. • The proposed method is evaluated on widely-used human motion datasets in the comprehensive experiments. The obtained results demonstrate the effectiveness of the proposed method over the state-of-the-art approaches for unconditional human motion generation task. 
related work 2.1 human motion generation: 
 The emergence of neural networks has a significant impact on the field of human motion synthesis. Previously, the focus was primarily on specific human motion generation, conditioned on some limiting factors, such as music [16] , text [38] , and action [25] . Numerous research papers have explored these areas. These works share a common spirit with our research, and here we will elaborate on them further. For example, Li et al. [16] proposed the task of generating a long sequence of realistic 3D dance motions that are well correlated with the input music. Petrovich et al. [25] learned an action-conditioned latent representation by training a VAE network. They sample from the learned latent space and query a series of positional encodings to synthesize motion sequences conditioned on actions. Degardin et al. [7] fused the architectures of GANs and GCNs to synthesize the kinetics of human motions. Zhang et al. [38] incorporated Denoising Diffusion Probabilistic Models (DDPM) [10] into motion generation, and propose to guide the generation pipeline with input texts softly, which increases the diversity of the generation results.Another line of work focuses on unconditional human motion synthesis [26] . Holden et al. [11] presented a pioneer work in deep unconditional human motion synthesis. They propose a motion synthesis and editing method based on a deep learning framework, which can automatically learn an embedding of human motion in a non-linear manifold. Recent MDM [32] utilized a classifier-free diffusion-based generative model for the human motion domain, and reported a trading-off between diversity and fidelity of human motions under an unconditional human motion generation. Modi [26] employs the style of StyleGAN to synthesize human motions, they utilize a mapping network to map noise into this latent space to enhance the diversity.Previous generative methods for human motion generation map the noise Gaussian distribution to the latent space distribution through a deep neural network. These methods usually suffer from mode collapse or mode mixture. In this work, we propose a new perspective human motion generation method to enhance diversity of the generated human motion sequences.Given the inherent indeterminacy of future human motions, the generative models like variational autoencoders (VAEs) [34, 36, 40] , generative adversarial networks (GANs) [2, 23] and denoising diffusion probabilistic model (DDPM) [1, 29] are appropriate methods for diverse human motion prediction due to their capability of generating a diverse set of valid solutions. Generally, VAEs [34] utilize the encoder-decoder mechanism to handle the task by capturing the probability distribution of human motions. VAEs-based methods assume that the latent distribution of human motions follows a Gaussian mixture distribution. This assumption often leads to blurry human motion joints, resulting in poor performance in terms of detail and accuracy in the generated human motions. GANs [8] can promote the quality of generated human motions in a degree. While being a powerful tool in generating realistically looking samples, GANs suffer from the mode collapse problem. The major modes (with higher likelihood) corresponding to a particular human motion pattern will more likely generate samples, minor patterns (those with lower likelihood) will almost not generate any samples. Recent promising diffusion model [10] has been proposed as an alternative generative approach for human motion generation task. For example, Tevet et al. [32] modes of conditioning, and different generation tasks. Ye et al. [37] attempted to introduce physics simulation, which is based on a pre-trained physical model through reinforcement learning. However, their approach is computationally intensive and low-efficiency inference process. Recent studies by [33] realized that the mode collapse and mode mixture are caused by the discontinuous functions with continuous DNNs. To alleviate this, Nagarajan et al. [22] proposed to utilize gradient-based regularization to address these issues. Khayatkhoei et al. [15] propose the utilization of multiple GANs to alleviate mode collapse and mode mixture. Another approach that has shown success in overcoming mode collapse is the normalized diversification method. Pan et al. [24] proposed UniGAN with a Normalizing Flowbased generator and uniformity regularization. Song et al. [28] utilized a new training objective that additionally optimized over the generated samples. However, if the target data distribution has multiple modes, the transport map of previous generative works is discontinuous. DNNs can only represent continuous mappings, the intrinsic conflict causes mode collapse. In this work, we propose to employ optimal transport to avoid mode collapse for unconditional human motion synthesis. 
preliminaries: 
 Optimal Transport Problem Let 𝜇 ∼ P (𝑋 ) and 𝜈 ∼ P (𝑌 ) be two sets of probability measures defined on 𝑋 and 𝑌 . 𝑋 ⊂ R 𝑑 and 𝑌 ⊂ R 𝑑 are two subset of 𝑑-dimensional Euclidean space R 𝑑 . The density functions are: 𝜇 (𝑥) = 𝑓 (𝑥)𝑑𝑥, 𝜈 (𝑦) = 𝑔(𝑦)𝑑𝑦. The total measures are equal, 𝜇 (𝑥) = 𝜈 (𝑦). The optimal transport problem can be attributed to Monge's problem, as shown in Figure . 2. Given a cost function 𝑐 (𝑥, 𝑦): 𝑋 × 𝑌 → [0, +∞], which indicates the cost of moving each unit mass from the source 𝑥 ∈ 𝑋 to the target 𝑦 ∈ 𝑌 , the Monge's problem seeks the most efficient 𝜇-measurable map 𝑇 : 𝑋 → 𝑌 byEQUATIONwhere 𝑇 # 𝜇 denotes the push-forward measure induced by 𝑇 . The map 𝑇 is measure-preserving: if for any measurable set 𝐵 ∼ 𝑌 , the set 𝑇 -1 (𝐵) is 𝜇-measurable and 𝜇 (𝑇 -1 (𝐵)) = 𝜈 (𝐵). The measurepreserving condition is denoted as 𝑇 # 𝜇 = 𝜈. A minimum 𝑇 * to this problem is called an OT map. Intuitively, Monge's problem finds a (2)However, it has two drawbacks, 𝜇-mass cannot be split leading to hard constraints, its transport map may not exist. To overcome the above shortcomings, Kantorovich [15] where 𝑐 𝑖 𝑗 = 𝑐 (𝑥 𝑖 , 𝑦 𝑗 ) is the cost that transports one unit mass from 𝑥 𝑖 to 𝑦 𝑖 , and𝜋 (𝜇, 𝜈) = {𝑃 | 𝑚 𝑖=1 𝑝 𝑖 𝑗 = 𝜈 𝑗 , 𝑛 𝑗=1 𝑝 𝑖 𝑗 = 𝜇 𝑖 , 𝑝 𝑖 𝑗 ≥ 0} is the coupling measure.Brenier's Approach For quadratic Euclidean distance cost, the existence, uniqueness, and intrinsic structure of the optimal transportation map were proven by Brenier [5] .Briener Potential Theorem Suppose 𝑋 and 𝑌 are the Euclidean space R 𝑑 and the transportation cost is the quadratic Euclidean distance 𝑐 (𝑥, 𝑦) = 1/2∥𝑥 -𝑦 ∥ 2 . Furthermore 𝜇 is absolutely continuous, 𝜇 and 𝜈 have finite second order moments,∫ 𝑋 |𝑥 | 2 𝑑𝜇 (𝑥) + ∫ 𝑌 |𝑦| 2 𝑑𝜈 (𝑦) < ∞,then there exists a convex function 𝑢 : 𝑋 → R, the so-called Briener potential, its gradient map ∇𝑢 gives the solution to the Monge's problem,EQUATIONThe Brenier potential is unique upto a constant, hence the optimal transportation map is unique. 𝑢 is called a Brenier solution.Legendre Transform Given a function 𝜑 : R 𝑛 → R, its Legendre Transform is defined as follows:𝜑 * (𝑦) := sup 𝑥 (⟨𝑥, 𝑦⟩ -𝜑 (𝑥)) .(5) 
method: 
 In this section, we first briefly introduce the overview of our task from an optimal transport perspective. Then we describe the reconstruction process of human motions. Finally, we explain how we compute the optimal transport map, including the matching learning and the process of generating new samples.An overview of the MOOT method is illustrated in Figure . 1. Our key idea is to synthesize diverse and natural human motions unconditionally by training a human motion generation model. The MOOT mainly involved in two stages, one is the human motion reconstruction (Section 4.2), the other is optimal mapping and generation module (Section 4.3). The human motion reconstruction network, implemented by the GRU and transformer network, which intends to learn the manifold embedding of human motions, the detail is described in Figure . 3. The optimal mapping and generation module aims to calculate the optimal transportation map 𝑇 that aligns the noise distribution with the latent space distribution of human motions, then generate new human motions by combining the generator of reconstruction network, the detail is described in Figure . 4. Given a random noise, the proposed method can generate diverse and natural human motion sequences through the learned optimal transport mapping.The human motion reconstruction network aims to learn the manifold embedding of human motions, the detail is described in with parameter 𝜙 intends to capture the smoothness property of human motions, and then encode the human motions into latent space 𝑍 . The formula for using the encoder to map the human motions into latent space is computed as follows:𝐴𝑡𝑡𝑒𝑛𝑡𝑖𝑜𝑛(𝑄, 𝐾, 𝑉 ) = 𝑠𝑜 𝑓 𝑡𝑚𝑎𝑥 ( 𝑄𝐾 𝑇 √︁ 𝑑 𝑘)𝑉 , ℎ𝑒𝑎𝑑 𝑖 = 𝐴𝑡𝑡𝑒𝑛𝑡𝑖𝑜𝑛(𝑸 (𝑖 ) , 𝑲 (𝑖 ) , 𝑽 (𝑖 ) ), Ẽ𝑡 = 𝐶𝑜𝑛𝑐𝑎𝑡 (ℎ𝑒𝑎𝑑 1 , . . . , ℎ𝑒𝑎𝑑 𝐻 )𝑾 (𝑂 ) ,EQUATIONwhere𝑊 (𝑂 ) denotes the concatenation weight matrix, the attention is computed by dot products of the query 𝑸 with all keys 𝑲 , divide each by √︁ 𝑑 𝑘 , and apply a softmax function to obtain the weights on the values 𝑽 . In addition, the architecture of the generator is the same as the encoder. The generator aims to map the latent space 𝑍 back to the reconstructed human motion sequence X . To train the reconstruction network, the mean-square-error (MSE) loss function is utilized to measure the difference between the reconstructed and the original motion sequences. Specifically, we constrain the perjoint 𝐿 2 distance on human joints. 𝑋 and X are the input and the reconstructed motion sequences, respectively. The reconstruction loss is defined as follows:EQUATIONwhere 𝐴 and 𝑇 denote the number of joints and the generated sequence length, respectively. The calculated results contain the spatial and temporal information of human motion sequences, and map the human motions into latent space 𝑍 . The human motion reconstruction process is the first stage of MOOT, which learns the manifold embedding of human motions.After learning the process of human motion reconstruction, previous generative methods aimed to increase the diversity of human motions by utilizing a generator based on DNNs. This generator transformed a known continuous distribution, such as Gaussian white noise, into the real distribution of human motions. However, the sampled human motions from the noise tend to concentrate primarily on the most dense areas of the dataset, resulting in mode collapse and mixture. To address this issue, we introduce optimal transport mapping to generate a wider range of diverse human motions, which consists of two stages, i.e., optimal matching modeling and optimal sample generation. Optimal Matching Modeling. Optimal matching modeling aims to match the noise distribution with the latent space of human motions, which involves learning the optimal transport map (OT map) 𝑇 , as shown in Figure . 4. In this work, the noise distribution 𝜇 is continuous that defined on a convex domain Ω ⊂ R 𝑑 , the latent space domain of human motions is a discrete set, ∫ Ω 𝑐 (𝑚,𝑇 (𝑚))𝑑𝜇 (𝑚). When the cost function is the 𝐿 2 distance 𝑐 (𝑚, 𝑧) = 1/2∥𝑚 -𝑧 ∥ 2 , Brenier's theorem claims that the optimal transport map is given by the gradient map of a piece-wise (PL) convex function, the so-called Brenier potential 𝑢 ℎ : Ω → R, 𝑢 ℎ (𝑚) := max 𝑛 𝑖=1 {𝜋 ℎ,𝑖 (𝑚)}, where 𝜋 ℎ,𝑖 (𝑚) = ⟨𝑚, 𝑧 𝑖 ⟩ + ℎ 𝑖 is the hyperplane corresponding to 𝑧 𝑖 ∈ 𝑍 . The vector ℎ is the unique optimizer of the following convex energy under the condition that 𝑖 ℎ 𝑖 = 0,𝑍 = {𝑧 1 , 𝑧 2 , • • • , 𝑧 𝑁 }, 𝑧 𝑖 ∈ R 𝑑 ,EQUATIONwhere 𝑤 𝑖 (𝜂) is the 𝜇-volume of 𝑊 𝑖 (𝜂). The convex energy 𝐸 (ℎ) can be optimized simply by gradient descend method with ∇𝐸 (ℎ) = (𝑤 𝑖 (ℎ) -𝜈 𝑖 ) 𝑇 . The key is to compute the 𝜇-volume 𝑤 𝑖 (ℎ) of each cell 𝑊 𝑖 (ℎ). We draw 𝑀 random samples from noise distribution, we can find 𝑊 𝑖 in which 𝑚 𝑗 ∈ 𝑊 𝑖 by 𝑖 = arg max 𝑖 {⟨𝑚 𝑗 , 𝑧 𝑖 ⟩ + ℎ 𝑖 }, 𝑖 = 1, 2, . . . , 𝑀. When 𝑀 is large enough, ŵ𝑖 (ℎ) converges to 𝑤 𝑖 (ℎ). The gradient of the energy is approximated as ∇𝐸 (ℎ) ≈ ( ŵ𝑖 (ℎ) -𝜈 𝑖 ) 𝑇 . We use adam algorithm to minimize the energy. Sampling of 𝑚 is independent of each other and finding the cell that 𝑚 is located only involves matrix multiplication and sorting. In this way, each Sample 𝑀 uniformly from noise distribution {𝑚 𝑗 } 𝑀 𝑗=1 .Calculate ∇𝐸 (ℎ) = ( ŵ𝑖 (ℎ) -𝜈 𝑖 ) 𝑇 .∇𝐸 (ℎ) = ∇𝐸 (ℎ) -𝑚𝑒𝑎𝑛(∇𝐸 (ℎ)).Update 𝐸 (ℎ) by adam algorithm. 7: until Converge 8: Optimal transport map 𝑇 (•) ← ∇(max 𝑖 ⟨•, 𝑧 𝑖 ⟩ + ℎ 𝑖 ). 9: repeat 10:Sample 𝑚 ∼ 𝜇, Find the smallest 𝑑 + 1 vertex around 𝑚 as {𝑐 𝑖 0 , 𝑐 𝑖 1 , . . . , 𝑐 𝑖 𝑑 }.Generate latent code T (𝑚) = 𝑑 𝑘=0 𝜆 𝑘 𝑧 𝑖 𝑘 with 𝑑 𝑘=0 𝜆 𝑘 = 1. 12: until Generate new samples 𝑝. random sample from the noise distribution is aligned with the latent code distribution.Optimal Sample Generation. The above process describes the matching modeling process that aims to align latent space with the noise distribution, but this process does not generate new samples. Therefore, we extend the optimal transport map 𝑇 to a piecewise linear mapping T . The projection in the source domain induces a cell decomposition of Ω, of which each cell is of 𝜇-volume 𝜈 𝑖 and is mapped to the corresponding 𝑧 𝑖 . By representing the cells of 𝜇-volume centers as 𝑐 𝑖 := ∫ 𝑊 𝑖 (ℎ) 𝑚𝑑𝜇 (𝑚), then the point-wise map 𝑐 𝑖 ↦ → 𝑧 𝑖 . The cell decomposition induces a triangulation of the centers 𝐶 = {𝑐 𝑖 } : 𝑖 𝑓 𝑊 𝑖 ∩ 𝑊 𝑗 ≠ ∅, then 𝑐 𝑖 is connected with 𝑐 𝑗 to form an edge[𝑐 𝑖 , 𝑐 𝑗 ]. Similarly, if 𝑊 𝑖 0 ∩𝑊 𝑖 1 • • • ∩𝑊 𝑖 𝑘 ≠ ∅ then there is a 𝑘-dimensional simplex [𝑐 𝑖 0 , 𝑐 𝑖 1 , . . . , 𝑐 𝑖 𝑘 ].All these simplices form a triangulation of 𝐶 (a simplicial complex), denoted as T (𝐶). We can triangulate 𝑍 in the same way to obtain the triangulation T (𝑍 ). Once a random sample 𝑚 is drawn from the distribution 𝜇, we can find the simplex 𝜎 in𝑇 (𝐶) containing 𝑚. Assume the simplex 𝜎 has 𝑑 + 1 vertices {𝑐 𝑖 0 , 𝑐 𝑖 1 , . . . , 𝑐 𝑖 𝑑 }, the bary-centric coordinates of 𝑚 in 𝜎 is defined as 𝑚 = 𝑑 𝑘=0 𝜆 𝑘 𝑐 𝑖 𝑘 , and 𝑑 𝑘=0 𝜆 𝑘 = 1 with all 𝜆 𝑘 non-negative. Then the generated latent code of 𝑚 under this piece-wise linear map is given by T (𝑚) = 𝑑 𝑘=0 𝜆 𝑘 𝑧 𝑖 𝑘 , then the 𝑚 is mapped to be T (𝑚). 𝑧 𝑖 is used to construct the simplicial complex T (𝑍 ) in the support of the target distribution, we can guarantee that no mode is lost. The algorithm for optimal sample generation is shown in Algorithm. 1.In summary, MOOT can synthesize diverse and natural human motion sequences through the above-mentioned two stages. The human motion reconstruction network learns a structured wellstructured laent space of human motions through an encoder. The optimal matching aims to match the noise distribution with the latent space distributions. To generate new samples, we extend the OT map to point-wise mapping T , and then combining the generator of the reconstruction network to generate human motions. Finally, MOOT can fully capture the potential diversity from a given distribution for unconditional human motion synthesis. 
experimental design: 
 In this section, the experimental design will be described, encompassing the popular human motion datasets, parameter settings, evaluation metrics, and baseline methods.The experiments are conducted on four widely used motion capture datasets, i.e., Human3.6M (H3.6M) [14] , HumanEva-I [27] , HumanAct12 [9] , and GRAB [4] . H3.6M consists of 11 subjects and 3.6 million frames at 50 Hz. The proposed method trains a model on five subjects (S1, S5, S6, S7, and S8) and is then tested on two subjects (S9 and S11). HumanEva-I contains 3 subjects recorded at 60 Hz, each of which performs 5 action categories. The pose is represented by 15 joints. HumanAct12 contains 12 subjects in which 12 categories of actions with per-sequence annotation are provided. The sequences with less than 35 frames are removed, which results in 727 training and 197 testing sequences. Subjects P1 to P10 are used for training, P11 and P12 are used for testing. GRAB consists of 10 subjects interacting with 51 different objects, performing 29 different actions. We use 8 subjects (S1-S6, S9, S10) for training and the remaining 2 subjects (S7, S8) for testing.For the parameter settings of the experiments, the batch size is set to 128, and the number of used human joints is 16. The number of multi-head in transformer is set to 8. The number of vertices 𝑑 to compute the bary-centric is set to 5. In this work, 𝜇 = 𝜈 = 1 𝑁 . The sampled 𝑀 and 𝑁 are set to 4990 and 1000 × 128, respectively. The proposed method is implemented based on the PyTorch framework in Python 3.6. To guarantee the convergence of the proposed method, the Adam optimizer is adopted to train our model. The learning rate is initially set to 10 -2 with a 0.98 decay every 10 epochs. The proposed method is trained for 500 epochs.In addition, for a fair comparison, our method employs the following evaluation metrics: APD, ADE, FDE, Multi-Modal ADE, and FDE metrics (MMADE and MMFDE). APD metric aims to evaluate the diversity of the results, while the other four metric aim to evaluate the accuracy. In particular, lower is better for all metrics except the APD metric.In this work, we compare the proposed method with state-of-the-art methods using the VAEs, GANs and DMs for unconditional human motion synthesis, including Dlow [36] , MT-VAE [34] , MOJO [40] , HP-GAN [2] , GSPS [21] , BeLFusion [1] , MotionDiff [32] , MDM [29] and ACTOR [25] . 
results and analysis: 
 The section endeavors to provide a comprehensive analysis of the experimental results for accuracy and diversity, including the quantitative comparison results between the state-of-the-art methods and the proposed method, ablation studies, and qualitative analysis. Ablation studies are conducted to justify the influence of introduced optimal matching module in MOOT on diversity and accuracy, as shown in Table 3 . Note that MoDi [26] presents a StyleGAN-based style generative model. They utilized 3D convolutions network as the backbone, and employed a mapping network that mapped noise into latent space, which is implemented by the MLP with several DNNs linear layers. To demonstrate the efficacy of the proposed MOOT method, we first utilize the same backbone network as MoDi, and tests different mapping styles to assess their impact on diversity and accuracy metrics. From the results, it is observed that the mapping styles have the greatest impact on the diversity metric of the results. Secondly, we test the difference of different backbone networks for the results, and using GRU and transformer as the encoder and generator network. These results show slightly improvement on the accuracy metrics. These findings again indicate that the influence of MOOT in enhancing the diversity while assuring the accuracy of unconditional human motion synthesis.In this section, we show the qualitative results of the proposed method on the Human3.6M and HumanEva-I datasets. As depicted in Figure. 5 and Figure . 6. All human motion sequences are unconditionally generated from random noise. Specifically, we first sample a random vector from noise distribution, and map it into a well-structured latent space using the introduced optimal matching, then generate diverse human motions combining the generator. These qualitative results demonstrate that MOOT can generate diverse and coherent human motion sequences. 
conclusion: 
 This paper introduces a novel method called MOOT for unconditional human motion synthesis. MOOT aims to address the issues of mode collapse and mode mixture in generated human motion sequences through the optimal transport mapping. Firstly, the MOOT method employs a human motion reconstruction network to learn the manifold embedding of human motions. This module combines GRU and transformer networks as the encoder and generator backbone network to capture temporal smoothness and spatial relationships among human motions. Secondly, MOOT utilizes the optimal transport mapping to align the noise distribution with the latent space distribution of human motions. The optimal transport mapping is computed using gradient mapping based on Brenier's theorem, which helps alleviate the issue of discontinuity caused by DNNs. Effective exploration of the Brenier potential and optimal transport mapping is achieved through optimization algorithms, thereby overcoming the problems of mode collapse and mode mixture that exist in current unconditional human motion generation methods. Finally, extensive experimental results demonstrate the efficacy of the proposed MOOT method in the task of unconditional human motion synthesis. 
 
 Appendix:  


Decision: The four reviewers' decisions are split (A, BA, BR, WR). The method using optimal transport is novel in this area, and its effectiveness has been verified by experiments. However, the concern is that optimal transport has been used in other areas, and the evaluation metric is rather biased. But still, I lean toward accepting this paper.

Reviews: review: paper_summary: paper_summary: The paper presents MOOT, a novel model for synthesizing diverse and high-quality human motion sequences without conditions. It addresses the limitations of existing methods like VAEs, GANs, and DMs in generating diverse motions. MOOT achieves this through a two-stage process, employing an encoder-generator framework combining GRU and Transformer networks for structured latent space learning, and an optimal transport map, extended to a piecewise linear mapping, to align a noise distribution with the learned latent space. This approach effectively enhances diversity in generated motions.
strength_weakness: strength_weakness: Enhanced Diversity and Quality: MOOT is designed specifically to generate human motion sequences that are both diverse and coherent. By aligning a noise distribution with a well-structured latent space of human motions, it ensures the synthesized motions are not only realistic but also span a broad range of possible movements, thus improving upon the limitations of previous methods.

Structural and Temporal Coherence: The employment of a human motion reconstruction network, which integrates GRU and Transformer networks, captures both the temporal continuity and spatial relationships in human motion sequences. This leads to more realistic motion synthesis with natural transitions and spatial plausibility.

Technical Rigor: The paper demonstrates a solid theoretical foundation with the use of Brenier's theorem for computing the optimal transport mapping, which provides a continuous solution, helping to overcome the discontinuity issues associated with Deep Neural Networks (DNNs) in approximating transport maps.

Comprehensive Evaluation: The experimental design includes rigorous testing on multiple benchmark datasets such as Human3.6M, HumanEva-I, HumanAct12, and GRAB. The study encompasses a range of parameters, evaluation metrics, and baseline comparisons, validating MOOT's superior performance in generating diverse human motion sequences.

Practical Relevance: The ability to generate diverse and natural human motion sequences has direct implications for several application areas, including animation, robotics, autonomous systems, and human-computer interaction, highlighting the practical significance of the research.
limitations: limitations: 1. This article is a lot like human motion prediction, where a lot of the experimental setups, and comparison methods are derived from motion prediction. But the title of the author's paper is motion generation. Please explain and clarify thes.
2.Use the last frame of the observation sequence for motion synthesis, why not use more frames? Why use only the last frame? Can we not use this frame?
3.I think that the author did not explain the differentiation between the two tasks of motion synthesis as well as motion prediction, and there is a logical confusion in the article.
4. There are many grammatical problems, line 100, incomplete in the sentence, line 204, the word 'describe' should be' describes',and so on.
5. The explanation is not clear in Figure 1. What do the different colored dots in Figure 1 mean? What do the lines indicate? It is suggested that the author should elaborate in CAPTION.
paper_summary: paper_summary: This paper is about the problem of unconditional human motion generation from historical observation. The authors focus on an important pain point of this problem "the generated human motions tend to heavily concentrate on densely populated regions of the data distribution", and attribute this problem to the inherent limitation of DNNs in representing only continuous maps. Under this motivation, the authors propose a new method based on optimal transport in the latent space, to transfer the continuous latent distribution to a multimodal discrete distribution. The authors show in the experiments that by doing this, the models can generate diverse motion sequences while keeping the diversity.
strength_weakness: strength_weakness: The work is very well motivated and well presented. It has been observed in the community that the unconditional motion generation models tend to collapse to stationary sequences, especially the poses that appear most frequently in the data set. Motivated from this point, this work present a novel approach to tackle this limitation, with a solid theoretical proof and comprehensive experiment results. I do like this paper. Although I raised many limitations later, this doesn't say a criticism of the article, but I hope these suggestions can make the article better.
limitations: limitations: - As a work on generative AI, it is less convincing to merely show the quantitative results in the table. Instead, I would prefer to see more qualitative demos, i.e. the generated sequences in a video. The images (Fig. 5 and Fig. 6) showing the generated poses, which I understand that they follow the previous work, are not sufficient in today’s research on generative AI.  
- Missing reference
  - the theory of optimal transport is not developed in this article, so the Section 3 should be proper referenced, such as [1]. Specifically, the “Kantorovich Relaxation” dates back to the 1940s, it is not a recent development
  - a highly related work [2] should be involved and discussed, which also aims for boost generation quality by optimizing the sampling space
- Typos
  - line 36, Denoising Diffusion Probabilistic Models usually refers to DDPMs, or Diffusion Models, not DMs
- Since this work optimize a OT mapping  to a discrete set (or a multimodal Dirac distribution), I'm curious how this method compares with learning a VQ-VAE model with soft assignment (i.e. the Gumbel-Softmax VQ). A pretrained VQ model can be easily adapted to this task with a auxiliary autoregressive prior model (i.e. GPT), which was shown in a related text-to-motion generation work [3]
- As discussed in [4], the performance on low ADE and hight APD may due to that the model tends to generate one very accurate future sequence (low ADE), and many low quality and unreal sequences (high APD). So it is better to also show other metrics, such as the median value of ADE/FDE in [4] and the FID and action prediction accuracy in [4,5]

[1] Peyré, Gabriel, and Marco Cuturi. ""Computational optimal transport: With applications to data science." Foundations and Trends in Machine Learning, 2019  
[2] Dang, Lingwei et al. "Diverse Human Motion Prediction via Gumbel-Softmax Sampling from an Auxiliary Space." ACM MM, 2022 
[3] Zhang, Jianrong et al. "T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representations." CVPR, 2023  
[4] Bie, Xiaoyu et al. "HiT-DVAE: Human Motion Generation via Hierarchical Transformer Dynamical VAE", arXiv, 2022  
[5] Petrovich, Mathis, Michael J. Black, and Gül Varol. "Action-conditioned 3d human motion synthesis with transformer vae." ICCV, 2021
paper_summary: paper_summary: This work presents an alternative way of solving the unconditional human pose generation problem, where the task is to generate a sequence of 3D poses of human motion such that they cover the data distribution, as assumed by a generative modeling objective. The authors propose an approach that is based on semi-discrete Optimal Transport (OT), where instead of having a Neural Network directly approximate the transport map between the prior (noise) distribution and the latent space, an OT map is first calculated based on the Brenier potential to map the prior distribution into the latent space, and then a neural network generates the corresponding motion sequence. In order to embed the motion sequence in latent space, an AutoEncoder based on GRU and Transformer is proposed. Several experiments validate the proposed approach, followed by ablation studies that motivate the design choices.
strength_weakness: strength_weakness: - The experimental results are extremely impressive. MOOT seems to outperform all other human motion generation models, showing that the proposed model does a solid job of capturing the data distribution. This is even more impressive when considering the different type of competitor models that the authors compare with.

- Overall, the paper is very well-written and self-contained. The explanations of the semi-discrete OT problem and how it solves the mapping from the prior distribution to the latent space are clear and help with the presentation of the work.
limitations: limitations: - The biggest weakness of the paper is the novelty, particularly the lack of reference to [1]. Essentially, the proposed model is identical to [1], who proposed using semi-discrete OT in an AE architecture to improve generation and tackle the issue of singularity sets in end-to-end NN-based generative models. Both the problem motivation, solution, algorithm for the calculation of the map (from the Brenier potential), and the extension of the transport map so that sample generation can be performed. Furthermore, the encoder-decoder architecture uses typical NNs used for sequential data (GRU + Transformer) and follows the usual L2 loss to learn the latent space. Thus, the proposed work simply applies these techniques to 3D human poses. A major point of concern is that the use of the semi-discrete OT map for generating data is presented as essentially the main contribution, while it has already been presented in [1].

Minor:
- The critiques of generative models in lines 130-156 are only somewhat true. While VAEs are used with Gaussian priors in practice, this is not a necessary choice. The prior for Variational Inference can be chosen based on the scenario. There is work that also proposes a mixture of priors with learnable components [2]. Furthermore, DDMs are naturally slow under the Markov assumption, which is upheld in the vanilla formulation, but there are several follow-up works that speed up the inference [3]. Nevertheless, the authors compare with prominent generative solutions in their task, so these details would simply make the presentation of the paper more technically correct.

- The metrics used for the evaluation have to be discussed in more detail. In the current version, it is difficult to understand what the results tell us besides that MOOT outperforms all methods. In would be helpful for the reader to understand what MOOT improves the most and understand why, by looking at the gap of a particular evaluation metric.

[1] An, Dongsheng, et al. "AE-OT: A new generative model based on extended semi-discrete optimal transport." ICLR 2020

[2] Tomczak, Jakub, and Max Welling. "VAE with a VampPrior." International conference on artificial intelligence and statistics. PMLR, 2018.

[3] Song, Jiaming, Chenlin Meng, and Stefano Ermon. "Denoising Diffusion Implicit Models." International Conference on Learning Representations. 2020.
paper_summary: paper_summary: The paper presents a new method for unconditional motion generation. The method is composed of 2 parts. First a GRU and Transformer based VAE to learn to reconstruct motion and obtain a good latent representation of the human motion. the second part is a optimal mapping algorithm to map noise to the VAE latent encoding using the theory of optimal transportation mapping with the goal of generating better motion and avoiding common pitfalls of unconditional generative models (e.g. converging toward a mean pose). The proposed method outperforms the current state of the art by a good margin.
strength_weakness: strength_weakness: The extensive experiments show clear improvements over the state of the art.

The idea of replacing the network used to map noise to latent encoding by a simple algorithm based on the theory of optimal transportation mapping is interesting and could solve common issues of generative model while perhaps reducing complexity.

Extensive preliminaries that help understanding the rest of the paper.
limitations: limitations: For experiments how do the authors deal with the fact that they compare against some conditional method? e.g ACTOR is conditioned on class while MotionDiff is condictioned on past motion. What steps were taken to ensure a fair comparison ?

Regarding quantitative results. How are accuracy metrics (ADE, FDE) computed for the proposed method ? Since it is unconditional how are the generated samples compared to the samples of the ground truth that contains several classes with diverse motions?

Why isn't MotionDiff used in the comparisons on GRAB and HumanAct-12 ? It had the best results on HumanEva-I.

There is no qualitative comparison with the state of the art. Figures 5 and 6 only show samples from the proposed method. It would have been good to be able to compare the quality of the results with other methods. Also videos would have been appreciated to better evaluate the quality of the generated motions.

A study of the complexity of the model or of the inference time would have been interesting as the proposed approach diverge from existing methods and could have given additional argument in favor of the proposed method.

Some insight on the possible applications of unconditional generation would have been interesting.

The Optimal Matching Modeling part could be a bit clearer. Maybe put all longer equations outside the text ?

line 518 "This generator transformed a known continuous distribution, such as Gaussian white noise, into the real distribution of human motions" Is it actually the case? I don't think it output the real distribution but only something resembling it.

line 546 "𝑧𝑖 ∈ R𝑑 , which is a Dirac measure, 𝜈 =..." Is v the dirac measure or is it zi ?