# A Supervised Variational Auto-encoder for Human Motion Generation Using Convolutional Neural Networks

Ali Ismail-Fawaz, Maxime Devanne, Stefano Berretti, Jonathan Weber, Germain Forestier

<figure><img src="image_1.png" /></figure>

## To cite this version:

Ali Ismail-Fawaz, Maxime Devanne, Stefano Berretti, Jonathan Weber, Germain Forestier. A Supervised Variational Auto-encoder for Human Motion Generation Using Convolutional Neural Networks. International Conference on Pattern Recognition and Artificial Intelligence (ICPRAI), Jul 2024, Jeju Island, South Korea. pp.166 - 181, 10.1007/978-981-97-8705-0_11. hal-04946867

HAL Id: hal-04946867

https://hal.science/hal-04946867v1

Submitted on 13 Feb 2025

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# A Supervised Variational Auto-Encoder for Human Motion Generation using Convolutional Neural Networks

Ali Ismail-Fawaz¹, Maxime Devanne¹, Stefano Berretti², Jonathan Weber¹, and Germain Forestier¹,³

¹ IRIMAS, Université de Haute-Alsace, France

{ali-el-hadi.ismail-fawaz,maxime.devanne,jonathan.weber,germain.forestier}@uha.fr

² MICC, University of Florence, Italy - stefano.berretti@unifi.it

³ DSAI, Monash University, Australia - germain.forestier@monash.edu

**Abstract.** Human motion generation is an important research domain addressed by a significant amount of work in the recent years, with the availability of new datasets captured either from sensors or cameras. Most of the existing approaches are based on the Variational Auto-Encoder (VAE) architecture using Recurrent Neural Networks (RNN) or Transformers. In this paper, we propose to handle human motion sequences as Multivariate Time Series (MTS), and construct a VAE based on Convolutional Neural Networks (CNN). Furthermore, the proposed architecture uses an action classification task to add the conditioning aspect to the generative model. Our proposed Supervised VAE (SVAE) achieves competitive results on the HumanAct12 dataset, both in terms of quality and diversity of the generated sequences.

## 1 INTRODUCTION

Collecting human motion datasets can be expensive as well as time consuming. Some approaches to extract human motion data are based on 2D video representations or 3D skeleton based sequences extracted using either MoCap sensors [12] or Kinect cameras [11]. On the one hand, MoCap sensors present a more accurate pose extraction, but they are expensive and require subjects' cooperation to wear a multiple sensor suit. On the other hand, Kinect cameras can also be expensive and time consuming to extract some data. For these reasons, the number of examples comprised by these datasets is limited in most of the cases. Eventually, this hinders the generalization of action recognition models. To solve this problem, data augmentation methods can be used. Following this idea, generative models are a possible way for augmenting the data.

More than one approach does exist to handle a generative model such as Generative Adversarial Networks (GANs) and Variational Auto-Encoders (VAEs) [13]. Generative models tend to learn a distribution in order to randomly sample a new human motion sequence. VAEs [9] do that by learning the best Gaussian
<figure><img src="image_3.png" /><figcaption>**Fig. 1.** Summary of our proposed Supervised VAE (SVAE) architecture. The input skeleton sequences are fed to the encoder and a random sample is taken from the learned Gaussian distribution in the latent space. This random sample is then fed to a classifier for the action recognition task as well as to the decoder to reconstruct the sequence.</figcaption></figure>

distribution in the latent space. The usage of VAE models has become very common in the recent years for human motion generation [1, 4, 10, 13, 19].

Human motion samples are usually represented as a sequence containing a 3D skeleton pose with a specific number of joints for each frame. We consider human motion sequences as Multivariate Time Series (MTS). A time series is a sequence of values changing according to time flow. An MTS is a time series that represents the change of multiple observed variables with respect to time. In the case of human motion sequences, we argue that the samples can be represented as an MTS with *number_of_joints* × *dimension* variables.

In the last decade, the task of Time Series Classification (TSC) has been addressed by a significant amount of work [17]. A comparative study [8] has shown that the usage of Convolutional Neural Networks (CNN) is very effective for this task. This is due to their ability to extract time dependent features in the time series sample. More recently, Ismail-Fawaz et al. [7] showed that by constructing hand-crafted filters to detect specific patterns, the model can generalize better. For the latter reason, we propose evaluating the usage of CNNs for the task of human motion generation. To the best of our knowledge, the usage of CNNs for this task has not been explored before. In this work, our model is based on the Fully Convolutional Network architecture (FCN) proposed in [17].

We address the problem of generation by constructing a generative model trained on human motion sequences captured by using Kinect cameras and collected in the HumanAct12 dataset [4]. Several works have been proposed in the literature addressing the human motion generation problem in an auto-regressive
way [1,4]. Some other works have tackled this problem by generating the whole sequence at the same time [10,13]. In this paper, we address the task by considering the generation of the whole sequence instead of an auto-regressive approach. This is done for two reasons: first, to avoid a long inference time for the generation part of the evaluation; second, the auto-regressive approach requires the generation of the posterior skeleton based on a prior one, which can lead to error propagation in the sequence generation.

In order to control the action label of the generated samples during evaluation, a conditioning aspect should be added to the model. We developed an approach for incorporating the conditioning technique in the CNN based VAE model. This approach adds a classification task to the optimization problem. By doing this, we propose the **Supervised Variational Auto-Encoder (SVAE)** for skeleton-based human motion generation. A summary of our proposed SVAE model is given in Figure 1. We evaluated the usage of this model compared to the state-of-the-art, and show that our CNN-based generative model is competitive.

In summary, our contributions are:

- A novel variant of a Variational Auto-Encoder, called **Supervised VAE (SVAE)** is proposed, which is obtained by adding a classifier;

- Representing a human motion sequence as a Multivariate Time Series by using a CNN based architecture;

- Extensive experiments showing that SVAE is competitive to the state-of-the-art.

# 2 BACKGROUND AND RELATED WORK

## 2.1 Variational Auto-Encoder (VAE)

Variational Auto-Encoders were first introduced in [9], as an adaptation of the original Auto-Encoders. This adaptation transformed the latent space of the Auto-Encoder from a discrete space into a continuous one. Originally, the Auto-Encoders consisted of an Encoder and a Decoder. The Encoder performs the down-sampling operation to construct the latent space and the Decoder performs the up-sampling operation in order to reconstruct the input sample. Auto-Encoders can be used for generation by randomly selecting a point in the latent space and feeding it to the Decoder. The limitation of Auto-Encoders for generation is the fact that the latent space is non-regularized. This means that when the latent space of an Auto-Encoder is given, the randomly selected point can have no meaning. This is due to the fact that the latent space is not a continuous one.

In order to solve the problem of a non-regularized latent space, VAEs propose the usage of a continuous latent space. This is done by approximating a Gaussian distribution using the Encoder, and feeding the Decoder with a randomly sampled vector from this distribution.

The model's parameters are optimized using the back propagation algorithm by minimizing the sum of two losses: (1) the reconstruction loss, and (2) the
Kullback-Leibler (KL) divergence. The reconstruction loss is usually the Mean Squared Error such as in the Auto-Encoder. The KL loss is the additional loss to the Auto-Encoder's mechanism. Its purpose is to regularize the latent space and minimize the distance between the distribution given by the Encoder and the normal Gaussian distribution (zero mean and unit variance).

## 2.2 Generative Models for Human Motion

**Video Human Motion Generation.** Before the availability of 3D human motion datasets, some work tackled the motion generation problem on the 2D scale. For instance, the work in [2] proposed a two-stage Generative Adversarial Network (GAN) in order to generate 2D videos of human motion. The model proposed in [2] is composed of two stages: the first stage generates a human skeleton starting from random noise, and the second stage transforms the generated skeleton to an image. This is done on multiple frames in order to generate a video. In [15], the authors proposed a video generation model, MoCoGAN, a GAN that conditions on specific content to control the generated motion. MoCoGAN learns two different spaces, the first representing the content and the other the frame sequence. These two models, MocoGAN and Two-Stage GAN, were adapted for 3D human sequences in [4].

**3D Skeleton Human Motion Generation.** With the availability of 3D human motion datasets, the authors in [4] argued the limitations of existing datasets and proposed a new publicly available one with human motion sequences. This dataset, HumanAct12, is adapted from the PHSPD dataset [20,21] that uses one polarization camera as well as three Kinect v2 cameras. Details about the dataset are presented in Section 4.1, and a visualization of some samples is given in Figure 2.

The authors in [4] also proposed a VAE model to tackle the generation problem of skeleton-based human motion sequences. This model, Action2Motion, is an auto-regressive VAE that consists of two Encoders, a prior and a posterior Encoder, and one Decoder. The model approximates a latent representation of the prior and posterior time frames, while minimizing the KL loss between their distributions. By doing this, the regularization of the two encoders is ensured. It is important to note that the generative model proposed in [4] is a conditional VAE. The conditioning is given over the action label as well as the time stamp value in order to differentiate between the prior and posterior pose frames.

Other works tackled the conditioning VAE modeling. For instance, the authors of [1] proposed a conditional VAE model but instead of adding the label in a raw manner, the model learns a latent representation of this condition. In other words, the model is made of two VAEs, CS-VAE and LCP-VAE. One encodes the condition, i.e., the prior knowledge, and the other encodes the future sequence, i.e., the posterior knowledge. The model proposed in [1] can be applied in two ways, either (1) using the action label as prior knowledge or (2) using a part of the training sequence as prior knowledge. The authors showed that the
<figure><img src="image_4.png" /><figcaption>**Fig. 2.** Three different samples from the HumanAct12 dataset [4], corresponding to the Boxing, Walk and Call human actions. Each part of the body is shown in a different color, the left leg in *green*, the right leg in *red*, the left arm in *blue*, the right arm in *purple*, and the body spine including the head in *orange*.</figcaption></figure>

second approach performs better. By using a real human motion sequence (prior) to generate the rest (posterior), the diversity in the generated samples increases. This is due to the fact that for each input sequence, more than one generation can be produced. Recently, Transformers showed a big impact on translation models [16] and image recognition [3]. For this reason, the authors in [13] used the Transformers architecture to tackle the human motion generation problem. The authors proposed ACTOR, a Transformer-based VAE model to generate 3D human motion sequences. The authors in [10] proposed a Quantized VAE in order to generate 3D human motion sequences. In this model, a Generative Pre-trained (GPT) like model is used in the latent space after being quantized in order to predict latent indices. More recently, the authors in [19] proposed a VAE that encodes the input human motion sequence in two streams at the same time (UM-CVAE), resulting in action agnostic and action aware representations. With the performance of Denoising Diffusion Probabilistic Models (DDPM) [6] for image generation, the authors in [18] proposed a DDPM for human motion generation: MotionDiffuse. This diffusion model uses an Attention mechanism architecture for the Denoising model with residual connections, instead of the U-Net as in the original model [6].

Most of the previous works used VAE models but without convolutions. In what follows, we propose an approach that tackles the human motion generation task with one dimensional convolutions. We add a conditioning technique in order to obtain a latent space composed of mini clusters for each action label. This conditioning technique is based on a classification task. A detailed view is presented in the following for the proposed SVAE model.
# 3 Proposed method

In this section, we introduce the original VAE model and its adaptation that we call Supervised VAE (SVAE) with the associated classification task. A Fully Convolutional Network (FCN) architecture [17] is used as a backbone for the two models. For the down-sampling part (Encoder), one dimensional convolutions are used. For the up-sampling part of the models (Decoder), transposed one dimensional convolutions are used. Transpose convolutions are based on a repetition of the input values followed by a one dimensional convolution. In what follows, we present the standard VAE and the *Supervised* VAE, both based on a CNN architecture. These two models are illustrated in Figure 3.

## 3.1 Standard Variation Auto-Encoder (VAE)

For this architecture, the VAE originally proposed in [9] is adapted for time series data in the specific case of human motion sequences. Two losses are minimized by the optimizer: the reconstruction loss and the Kullback-Leibler (KL) divergence. For the reconstruction loss, we adapted the standard Mean Squared Error loss by not averaging over the time dimension:

$$ \mathcal{L}_{rec}(\mathbf{S}, \hat{\mathbf{S}}) = \frac{1}{J * D} \sum_{t=0}^{L-1} \sum_{i=0}^{J-1} \sum_{j=0}^{D-1} (\mathbf{S}_{t,i,j} - \hat{\mathbf{S}}_{t,i,j})^2, \quad (1) $$

with $\mathbf{S}$ being the real skeleton, and $\hat{\mathbf{S}}$ the reconstructed one. $L$ is the length of the sequence, $J$ the number of joints in a skeleton, and $D$ the dimension of each joint. Removing the average over the time dimension aims to prevent the model from finding the average sequence for each optimization step during the training phase. This is due to the huge reduction in dimensionality of the optimization problem, which leads to underfitting.

The KL loss is calculated to approximate the Gaussian distribution in the latent space to a normal standard one with a zero mean and a unit standard deviation. The closed form of the KL loss is:

$$ \mathcal{L}_{KL}(\mu, \sigma^2) = -\frac{1}{2} \sum_{i=0}^{D_{LS}-1} (1 + \log \sigma_i^2 - \mu_i^2 - \sigma_i^2), \quad (2) $$

with $D_{LS}$ being the dimension of the latent space.

## 3.2 Supervised Variational Auto-Encoder (SVAE)

In order to have the conditioning advantage in the VAE, we propose to add a classification task in the latent space. The classifier (MLP based) is placed in the latent space of the SVAE model. Both the classifier and the Decoder are fed with a randomly sampled vector from the Gaussian distribution given by the Encoder. For this model, given the additional task of classifying the input
<figure><img src="image_5.png" /><figcaption>Fig. 3. The Variation Auto-Encoder (VAE) model for the generation of human motion. The architecture uses the FCN as a backbone in the Encoder and Decoder. For the Decoder, one dimensional convolutions are replaced by the transpose convolutions in one dimension. Each convolution block, standard (green) or transpose (orange), is followed by a batch normalization operation and a ReLU activation function. The Fully Connected layers (cyan) are added for the SVAE model.</figcaption></figure>

samples, a Cross-Entropy (CE) loss is used in addition to the reconstruction and KL losses. This loss minimizes the uncertainty of the approximated class distribution:

$$ \mathcal{L}_{CE}(\mathbf{y}, \hat{\mathbf{y}}) = - \sum_{c=0}^{C-1} y_c \cdot \log_2(\hat{y}_c), \qquad (3) $$

with $\mathbf{y}$ and $\hat{\mathbf{y}}$ being the true and predicted class distributions, respectively, and $C$ the number of classes.

Each loss used to optimize the model's parameters is assigned a weight. The motivation of using a weighted version of the model comes from the original $\beta$-
VAE [5]. In the original β-VAE [5] model, the total loss of the VAE is controlled by a hyperparameter β as follows:

$$ \mathcal{L}_{total} = \mathcal{L}_{rec} + \beta \mathcal{L}_{KL}. \quad (4) $$

The β parameter controls if, during training, the VAE focuses more on the quality of reconstruction or on the quality of disentanglement in the latent space. Higher values of β result in learning more disentangled latent feature, hence in a better latent representation of the data. Conversely, lower values of β result in focusing more on obtaining a better quality of the reconstruction. In our experiments, we associated for each loss used in our SVAE model a weight as seen in (5). For deciding the best performing values of these weights, we performed a study as presented in the experimental section:

$$ \mathcal{L}_{total} = \lambda_{rec} \cdot \mathcal{L}_{rec} + \lambda_{KL} \cdot \mathcal{L}_{KL} + \lambda_{CE} \cdot \mathcal{L}_{CE}. \quad (5) $$

It is important to note that, in the case of the original VAE architecture, the weight of the CE loss λ$_{CE}$ is set to zero.

# 4 Experimental Setup

## 4.1 Dataset and Preprocessing

For evaluating our proposed model, we rely on the open source HumanAct12⁴ dataset proposed in [4]. This dataset comprises 1,191 human motion sequences of 12 different actions. Each frame in the sequence represents a 3D skeleton with 24 joints. In order to avoid the problem of sequences of different length, a resampling algorithm is employed on the dataset. The target length is set to the average length found in the input sequences, i.e., 75. The resampling algorithm used in this work is based on the Fourier Transform, and is implemented using the Scipy python module<fnref n="5" />. Before training, we normalize the skeleton sequences using a Min-Max normalization. This normalization is applied to each dimension separately. Suppose S represents all the skeleton sequences available in a dataset with four dimensions, i.e., the number of samples, the sequence length, the number of joints, and dimension of each joint, respectively. Then, the normalization is computed as follows:

$$ S[;, ;, ;, d] = \frac{S[;, ;, ;, d] - \min(S[;, ;, ;, d])}{\max(S[;, ;, ;, d]) - \min(S[;, ;, ;, d])}, \quad (6) $$

for d : 1 → 3 (3D skeleton). In the case of a train/test split, the test set is normalized using the min and max values from each dimension of the train set.

<footnote n="4">⁴ https://drive.google.com/drive/folders/1hGNkxI9jPb
dFueUHfj8H8zC2f7DTb8NG</footnote>

<footnote n="5">⁵ https://docs.scipy.org/doc/scipy/reference/generated/scipy.signal.resample.html</footnote>
## 4.2 Implementation Details

For the Encoder and Decoder architecture used in our model (Figure 3), the following parameters are used:

- Number of filters per convolution layer: 128;

- Kernel size of the filters: 40, 20 and 10 for the three layers in the Encoder, respectively (in reverse order for the Decoder);

- Dimension of the latent space: 16.

The classifier architecture in the SVAE model comprises a fully connected layer with 8 units, followed by a dropout of 50%, and two fully connected layers with C units each, being C the number of classes. The generative models are trained using the default setup of the Adam optimizer with a learning rate decay. The best model is used for evaluation, by monitoring the training loss. The generative models are trained with a batch size of 32 for 2000 epochs. We follow [4] and use a GRU based classifier for the feature extraction part of the evaluation metrics. The code of this work is available on: https://github.com/MSD-IRIMAS/SVAE-4-HMG

## 4.3 Evaluation Metrics

The performance of different generative models are compared using different metrics for evaluating the quality and diversity of the generated samples. In what follows, we summarize the three metrics used in our experiments.

**Fréchet Inception Distance (FID).** The FID metric evaluates the quality of the generated samples. It computes the minimum distance needed to move one distribution to another. Note that the last layer (classification layer) is removed from the pre-trained classifier in order to extract only the latent space. The mean and covariance of this latent space is computed for the real and generated samples to obtain two Gaussian distributions. The FID metric is then computed between both distributions. The classifier is pre-trained only on real samples. The closed form of the FID is given by:

$$ 
\begin{aligned}
FID(\mu_{real}, \text{Cov}_{real}, \mu_{gen}, \text{Cov}_{gen}) = & \sum_{i=0}^{d-1} (\mu_{real,i} - \mu_{gen,i})^2 \\
& + \text{trace}(\text{Cov}_{real} + \text{Cov}_{gen} - 2 * \sqrt{\text{Cov}_{real} \odot \text{Cov}_{gen}}),
\end{aligned} 
\quad (7) $$

with ⊙ being the matrix dot product, and d the latent space dimension of the pre-trained action recognition classifier.

**Diversity.** This metric measures the diversity in the generated samples. Similar to the FID, the same pre-trained classifier is necessary for this metric as well. The
diversity computes the Euclidean distance between random pairs of randomly selected groups of samples as follows:

$$Diversity(\mathbf{X}_{gen}) = \frac{1}{S_d} \sum_{i=0}^{S_d-1} \sqrt{\sum_{j=0}^{d-1} (\mathbf{V}_{i,j} - \mathbf{V}'_{i,j})^2}, \quad (8)$$

with $\mathbf{V}$ and $\mathbf{V}'$ being two randomly selected groups of samples from $\mathbf{X}_{gen}$ of size $S_d$ each. The dimension of the latent space of the pre-trained classifier is $d$ as in Section 4.3. In order to remove the variance in the random selection of the $S_d$ samples, this evaluation is done multiple times on the same set of $X_{gen}$, and the results are averaged over the different experiments.

It is important to note that for the FID and Diversity metrics, no train/test split is required. For the FID metric, this is due to the fact that it compares the distribution of the real samples and the generated ones. For this reason, the placement of evaluation leakage does not exist in this case. For the Diversity metric, no split is required because the metric only uses generated samples for evaluation. For these reasons, the action recognition classifier, used as feature extractor for the FID and Diversity metrics, is trained on all the dataset samples.

For the FID and Diversity metrics, all of the presented results are averaged over 125 experiments (five initializations for both the generative model and the classifier, and five generations per model).

**Accuracy.** The accuracy metric is the evaluation of a pre-trained classifier on a test set; in this case, the classification task is an action recognition one. In order to evaluate the quality of the generated samples using the accuracy metric, a train test split is adapted. This split is not available with the HumanAct12 dataset, so we randomly generated four different splits. The dataset includes 12 subjects so that each split provides 9 training subjects and 3 testing ones. Using cross subject train/test split avoids evaluation on a subject seen by the generative model that could result in a biased evaluation.

Supposing that subject indices vary from 0 → 11, we used the following random splits:

- Split 0: Subjects 2, 5 and 8 for testing.

- Split 1: Subjects 11, 10 and 0 for testing.

- Split 2: Subjects 7, 3 and 4 for testing.

- Split 3: Subjects 9, 6 and 1 for testing.

Using these splits, a baseline is first set by training a classifier on real samples and evaluating its accuracy on the real test set. Second, a generative model is trained on the same real training set. Third, a second classifier is trained on generated samples (same labels distribution as real training set). The evaluation of the second classifier on the real test set is compared to the one of the first classifier. The closer the accuracy of the second classifier is to the baseline one, the higher the quality of the generated samples.
<figure><img src="image_6.png" /><figcaption>**Fig. 4.** Latent space representation in a 2D projection (using PCA) of the SVAE model on the training samples. Latent points corresponding to the same action label are separated and show a Gaussian distribution.</figcaption></figure>

Another way to evaluate the accuracy metric is by applying data augmentation [14]. For this evaluation, the same train/test splits were used as described above. The only difference is that in this experiment, the classifier is trained on both real and generated samples. The data augmentation technique proposed in this work adjusts the labels distribution in order to have a uniform one.

# 5 Experimental Results

In this study, the SVAE model uses the values of loss functions weights 0.4995, 0.001, 0.4995 for the reconstruction, KL and CE losses, respectively. To find these best values, a grid search was applied over a set of different weights combinations, monitoring evaluation metrics. The best combination found by this study is to give equal weight to the classification and reconstruction loss, while minimizing the weight of the KL loss. A 2D representation of the latent space of the training samples using the best SVAE model is visualized in Figure 4. This representation was obtained using the Principal Component Analysis (PCA) algorithm. The ability of the SVAE model to separate between multiple classes in the latent space enables a better control of the generation conditioned on the label.

## 5.1 Comparing with existing Approaches

In order to compare the performance of our proposed architecture with state-of-the-art models on the HumanAct12 dataset, we present the results in Table 1. For this comparison, we included only our best SVAE model. In order to minimize the bias in the metric evaluation, the generated samples follow the same labels distribution as the training set. The results show that our proposed SVAE model achieves better quality (FID) than the Action2Motion model and is not very far from the state-of-the-art model. The diversity of the SVAE is not that different than the one on the real samples. This means that the model was able to preserve diversity enough to have a good quality.
<table><caption>**Table 1.** Comparing our proposed model, Supervised Variational Auto-Encoder (SVAE), with other approaches from the literature using the FID and Diversity metrics. For each approach, the average<sup>+standard deviation</sup> of the FID and Diversity are presented. For the line including the Real samples, a random split into two sets of samples is used in order to calculate the FID metric.</caption><thead><tr><th>Model</th><th>FID ↓</th><th>Diversity ↑</th></tr></thead><tbody><tr><td>Real</td><td>00.030<sup>+0.005</sup></td><td>06.860<sup>+0.070</sup></td></tr><tr><td>Two-stage GAN [4]</td><td>10.480<sup>+0.089</sup></td><td>05.960<sup>+0.049</sup></td></tr><tr><td>Act-MoCoGAN [4]</td><td>05.610<sup>+0.113</sup></td><td>06.752<sup>+0.071</sup></td></tr><tr><td>Action2Motion [4]</td><td>02.458<sup>+0.079</sup></td><td>07.032<sup>+0.038</sup></td></tr><tr><td>ACTOR [13]</td><td>00.120<sup>+0.000</sup></td><td>06.840<sup>+0.030</sup></td></tr><tr><td>PoseGPT [10]</td><td>00.080<sup>+*.***</sup></td><td>06.850<sup>+*.***</sup></td></tr><tr><td>UM-CVAE [19]</td><td>00.090<sup>+0.000</sup></td><td>06.810<sup>+0.020</sup></td></tr><tr><td>MotionDiffuse [18]</td><td>00.070<sup>+0.000</sup></td><td>06.850<sup>+0.020</sup></td></tr><tr><td>SVAE (ours)</td><td>00.560<sup>+0.170</sup></td><td>06.640<sup>+0.100</sup></td></tr></tbody></table>

Results show that the SVAE model is able to generate high-quality human motion sequences (see generated examples in Figure 5). In other words, the convolution filters are able to extract temporal features in the input sequence, which concludes that human motion skeleton sequences can be represented as Multivariate Time Series (MTS).

## 5.2 Fixing labels distribution with data augmentation

Applying data augmentation is a straightforward way to evaluate the quality of the generated samples. We approach this method by fixing the labels distribution in the training set. In other words, for each train set, we generate new human motion sequences using our SVAE model. We augment each class in order to obtain a uniform labels distribution, while the most populated class remains untouched. We also trained the model on the generated samples that follows the same number of samples and labels distribution as the train set. Results are presented in Table 2. On the one hand, the results are similar (not significantly different) than when training on real samples. Nevertheless, the results are not significantly worse than when training on real samples, which shows that the generated samples have a sufficient quality. On the other hand, the usage of data augmentation to fix the labels distribution boosts the performance.

## 6 Conclusions

In this paper, we presented a new architecture for tackling the human motion generation task. Many works have approached this task using auto-regressive or full generation techniques using Recurrent Networks, Fully Connected, Transformers, etc. We proposed to recast the problem into Multivariate Time Series generation by constructing a new VAE based architecture that includes convolutions. This architecture, the Supervised VAE (SVAE), replaces the conditioning
<table><caption>**Table 2.** Test accuracy values on four different train/test splits: (first row) training on real samples; (second row) training on generated samples (following the same distribution as the real samples); (third row) training on augmented samples (real+generated to have a uniform labels distribution).</caption><thead><tr><th>Train On</th><th>Split 0</th><th>Split 1</th><th>Split 2</th><th>Split 3</th></tr></thead><tbody><tr><td>real</td><td><b>87.77</b><sup>+1.55</sup></td><td><b>76.47</b><sup>+2.23</sup></td><td>87.78<sup>+1.40</sup></td><td>77.43<sup>+2.54</sup></td></tr><tr><td>generated</td><td>72.47<sup>+2.91</sup></td><td>64.39<sup>+2.42</sup></td><td>77.37<sup>+4.32</sup></td><td>69.75<sup>+7.31</sup></td></tr><tr><td>augmented</td><td>86.07<sup>+2.10</sup></td><td>73.51<sup>+1.32</sup></td><td><b>89.34</b><sup>+1.77</sup></td><td><b>78.71</b><sup>+2.53</sup></td></tr></tbody></table>

technique typically used in the literature by adding a supervision task in the latent space. Even though this paper focuses on human motion generation, the proposed architecture is generic and could be used on other time series data. Results compared to real samples in case of quality and diversity show that the model is able to learn and extract generic features. The quality of the generated samples were then evaluated using data augmentation that uniforms the labels distribution in the training set. Nevertheless, our proposed architecture can present a large number of trainable parameters even though they are fast to be optimized, thanks to convolutions. For this reason, we aim in the future to reduce the number of trainable parameters in our architecture, while preserving the quality of the results, additionally to evaluating SVAE on other datasets.

# 7 ACKNOWLEDGMENTS

This work was supported by the ANR DELEGATION project (grant ANR-21-CE23-0014) of the French Agence Nationale de la Recherche. The authors would like to acknowledge the High Performance Computing Center of the University of Strasbourg for supporting this work by providing scientific support and access to computing resources. Part of the computing resources were funded by the Equipex Equip@Meso project (Programme Investissements d'Avenir) and the CPER Alsacalcul/Big Data. The authors would also like to thank the creators and providers of the HumanAct12 dataset.

# References

1. S. Aliakbarian, F. Saleh, L. Petersson, S. Gould, and M. Salzmann. Contextually plausible and diverse 3d human motion prediction. In *IEEE/CVF Int. Conf. on Computer Vision (ICCV)*, pages 11333–11342, 2021.

2. H. Cai, C. Bai, Y.-W. Tai, and C.-K. Tang. Deep video generation, prediction and completion of human action sequences. In *European Conf. on Computer Vision (ECCV)*, pages 366–382, 2018.

3. A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. *arXiv preprint arXiv:2010.11929*, 2020.
4. C. Guo, X. Zuo, S. Wang, S. Zou, Q. Sun, A. Deng, M. Gong, and L. Cheng. Action2motion: Conditioned generation of 3d human motions. In *ACM Int. Conf. on Multimedia*, pages 2021–2029, 2020.

5. I. Higgins, L. Matthey, A. Pal, C. Burgess, X. Glorot, M. Botvinick, S. Mohamed, and A. Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. In *International conference on learning representations*, 2017.

6. J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. *Advances in Neural Information Processing Systems (Neurips)*, 33:6840–6851, 2020.

7. A. Ismail-Fawaz, M. Devanne, J. Weber, and G. Forestier. Deep learning for time series classification using new hand-crafted convolution filters. In *IEEE Int. Conf. on Big Data (BigData)*, pages 1–8. IEEE, 2022.

8. H. Ismail Fawaz, G. Forestier, J. Weber, L. Idoumghar, and P.-A. Muller. Deep learning for time series classification: a review. *Data Mining and Knowledge Discovery*, 33(4):917–963, 2019.

9. D. P. Kingma and M. Welling. Auto-encoding variational bayes. *arXiv preprint arXiv:1312.6114*, 2013.

10. T. Lucas, F. Baradel, P. Weinzaepfel, and G. Rogez. Posegpt: quantization-based 3d human motion generation and forecasting. In *European Conf. on Computer Vision (ECCV)*, pages 417–435. Springer, 2022.

11. R. Lun and W. Zhao. A survey of applications and human motion recognition with microsoft kinect. *International Journal of Pattern Recognition and Artificial Intelligence*, 29(05):1555008, 2015.

12. N. Mahmood, N. Ghorbani, N. F. Troje, G. Pons-Moll, and M. J. Black. Amass: Archive of motion capture as surface shapes. In *IEEE/CVF Int. Conf. on Computer Vision (ICCV)*, pages 5442–5451, 2019.

13. M. Petrovich, M. J. Black, and G. Varol. Action-conditioned 3d human motion synthesis with transformervae. In *IEEE/CVF Int. Conf. on Computer Vision (ICCV)*, pages 10985–10995, 2021.

14. G. Pialla, M. Devanne, J. Weber, L. Idoumghar, and G. Forestier. Data augmentation for time series classification with deep learning models. *Advanced Analytics and Learning on Temporal Data (AALTD)*, 2022.

15. S. Tulyakov, M.-Y. Liu, X. Yang, and J. Kautz. Mocogan: Decomposing motion and content for video generation. In *IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)*, pages 1526–1535, 2018.

16. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. *Advances in Neural Information Processing Systems (Neurips)*, 30, 2017.

17. Z. Wang, W. Yan, and T. Oates. Time series classification from scratch with deep neural networks: A strong baseline. In *Int. Joint Conf. on Neural Networks (IJCNN)*, pages 1578–1585. IEEE, 2017.

18. M. Zhang, Z. Cai, L. Pan, F. Hong, X. Guo, L. Yang, and Z. Liu. Motiondif-fuse: Text-driven human motion generation with diffusion model. *arXiv preprint arXiv:2208.15001*, 2022.

19. C. Zhong, L. Hu, Z. Zhang, and S. Xia. Learning uncoupled-modulationvae for 3d action-conditioned human motion synthesis. In *European Conference on Computer Vision (ECCV)*, pages 716–732. Springer, 2022.

20. S. Zou, X. Zuo, Y. Qian, S. Wang, C. Guo, C. Xu, M. Gong, and L. Cheng. Polarization human shape and pose dataset. *arXiv preprint arXiv:2004.14899*, 2020.

21. S. Zou, X. Zuo, Y. Qian, S. Wang, C. Xu, M. Gong, and L. Cheng. 3d human shape reconstruction from a polarization image. In *European Conf. on Computer Vision (ECCV)*, pages 351–368. Springer, 2020.
<figure><img src="image_7.png" /><figcaption>**Fig. 5.** Generated samples produced by the proposed SVAE models conditioned on four different actions: Warm Up, Drink, Lift Dumbbell and Sit. For each action, we present two different generated examples to showcase the diversity of the proposed SVAE model.</figcaption></figure>