# Establishing a unified evaluation framework for human motion generation: A comparative analysis of metrics

Ali Ismail-Fawaz, Maxime Devanne, Stefano Berretti, Jonathan Weber, Germain Forestier

## ► To cite this version:

Ali Ismail-Fawaz, Maxime Devanne, Stefano Berretti, Jonathan Weber, Germain Forestier. Establishing a unified evaluation framework for human motion generation: A comparative analysis of metrics. Computer Vision and Image Understanding, 2025, 254, pp.104337. 10.1016/j.cviu.2025.104337. hal-05142916

HAL Id: hal-05142916

https://hal.science/hal-05142916v1

Submitted on 3 Jul 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
<figure><img src="image_2.png" /></figure>

# Establishing a unified evaluation framework for human motion generation: A comparative analysis of metrics

<figure><img src="image_3.png" /></figure>

Check for
updates

Ali Ismail-Fawaz<sup>a,\*</sup>, Maxime Devanne<sup>a</sup>, Stefano Berretti<sup>b</sup>, Jonathan Weber<sup>a</sup>,
Germain Forestier<sup>a,c</sup>

## ARTICLE INFO

MSC:

41A05

41A10

65D05

65D17

Keywords:

Human motion generation

Evaluation of generative models

Generative models metrics

Human motion data

## ABSTRACT

The development of generative artificial intelligence for human motion generation has expanded rapidly, necessitating a unified evaluation framework. This paper presents a detailed review of eight evaluation metrics for human motion generation, highlighting their unique features and shortcomings. We propose standardized practices through a unified evaluation setup to facilitate consistent model comparisons. Additionally, we introduce a novel metric that assesses diversity in temporal distortion by analyzing warping diversity, thereby enhancing the evaluation of temporal data. We also conduct experimental analyses of three generative models using two publicly available datasets, offering insights into the interpretation of each metric in specific case scenarios. Our goal is to offer a clear, user-friendly evaluation framework for newcomers, complemented by publicly accessible code: https://github.com/MSD-IRIMAS/Evaluating-HMG.

## 1. Introduction

Evaluating generative models is one of the most challenging tasks to achieve (Naeem et al., 2020). This kind of challenge is largely absent in discriminative models, where evaluation primarily involves comparison with ground truth data. However, for generative models, evaluation involves quantifying the validity between real samples and those generated by the model. A common method for evaluating generative models is through human judgment metrics, such as Mean Opinion Scores (MOS) (Streijl et al., 2016). However, this type of evaluation assumes a uniform perception among users regarding what constitutes ideal and realistic generation, which is often not the case. For this reason, generative models require quantitative evaluation based on measures of validity between real and generated samples. This similarity is quantified on two dimensions: fidelity and diversity. On the one hand, fidelity is the measure of similarity between real and generated spaces on the marginal distribution scale. On the other hand, diversity is the measure of how varied a set of samples is, indicating the extent to which the diversity of the generated set in generative models aligns with the diversity of the real set.

Generative models are used for almost every type of data ranging from images (Ho et al., 2020) to time series data (Lee et al., 2023;

Ismail-Fawaz et al., 2023c). In this work, we investigate human motion generation (Guo et al., 2020), a subject that has attracted substantial interest in the research community. This surge in interest is attributable to the availability of datasets generated by advanced technologies like Kinect cameras and motion capture systems (Lun and Zhao, 2015; Asteriadis et al., 2013; Mahmood et al., 2019). With these technologies, a variety of 3D human motion skeleton-based datasets have been developed. The availability of these datasets, coupled with their importance in applications ranging from the cinematic universe to the medical field, has led the research community to take a keen interest in generating 3D skeleton-based motion samples. An early work that addressed this new approach of handling a human motion generation model is Action2Motion (Guo et al., 2020). It is a recurrent based deep learning model with a Conditional Variational Auto-Encoder (CVAE) (Kingma and Welling, 2013; Sohn et al., 2015) architecture where, for each frame, the model learns the parameters of a CVAE. The authors in Ali-akbarian et al. (2021) proposed CS-VAE, a novel CVAE model that utilizes the first part of the sequence as the condition to encode, which is then used to generate the rest of the sequence. With the rise of self-attention and Transformer models (Vaswani et al., 2017), a non-recurrent CVAE with a Transformer backbone architecture, called

<footnote><sup>a</sup> IRIMAS Université de Haute-Alsace, Mulhouse, France</footnote>

<footnote><sup>b</sup> MICC, University of Florence, Florence, Italy</footnote>

<footnote><sup>c</sup> DSAI Monash University, Melbourne, Australia</footnote>

<footnote>* Corresponding author.</footnote>

<footnote>E-mail address: ali-el-hadi.ismail-fawaz@uha.fr (A. Ismail-Fawaz).</footnote>

<footnote>URL: http://www.hadifawaz1999.github.io (A. Ismail-Fawaz).</footnote>
ACTOR, is proposed in Petrovich et al. (2021). Further, with the significant improve of performance in language models thanks to Generative Pre-trained Transformers (GPTs) (Radford et al., 2018), the authors in Lucas et al. (2022) proposed PoseGPT, a Quantized CVAE (Van Den Oord et al., 2017), with a GPT used on the quantized latent space to predict the next token sequentially. More recently, the authors in Zhong et al. (2022) proposed an uncoupled CVAE (UM-CVAE) model with two encoders, one being action agnostic and the other action aware. By considering how much Convolutional Neural Networks (CNNs) dominate the classification field for time series (Ismail-Fawaz et al., 2022, 2023a, 2024a, 2023b), the authors in Ismail-Fawaz et al. (2024b) showcase their contribution to human motion generation by proposing a CNN based VAE, trained in a supervised manner to classify the latent space (SVAE). Finally, Denoising Diffusion Probabilistic Models (DDPM) have changed the course of image generation (Ho et al., 2020; Rombach et al., 2022; Bansal et al., 2024), and started to give effect to human motion generation as well. For instance, in Zhang et al. (2024) the first DDPM dedicated for 3D skeleton based human motion generation was proposed that outperformed the state-of-the-art. In Yuan et al. (2023), the human motion DDPM was adjusted so that it can take into consideration the physical aspects of the human skeleton.

An emerging area of research closely related to human motion generation is text-to-motion retrieval, which focuses on retrieving the most relevant motions from a database based on a textual query. This field provides valuable insights into the joint understanding of human motion and natural language, as demonstrated in recent works such as Messina et al. (2023). Although this paper primarily focuses on generative models, the advancements in text-to-motion retrieval offer complementary perspectives on how motion data can be effectively utilized and understood.

While the generative models presented above each offers unique strengths, such as the temporal modeling capabilities of recurrent networks or the global attention of transformer-based models, they also come with trade-offs. For example, recurrent models like Attention2Motion (Guo et al., 2020) are effective in modeling sequential data but can struggle with long-term dependencies, while transformer models such as ACTOR (Petrovich et al., 2021) excel in capturing global relationships but require significantly more computational resources. Similarly, diffusion models (Zhang et al., 2024) have set new benchmarks in accuracy but often demand extensive datasets and longer training times. Importantly, the goal of this paper is not to compare generative models directly, but rather to provide a comprehensive and flexible evaluation framework. This allows for fair comparisons between models, regardless of the specific architecture used, and helps address the various dimensions of motion generation, such as temporal consistency, realism, and diversity.

All the related works on human motion generation mentioned above use some common evaluation approach, while some propose their own new metric as well. The primary challenge lies in defining the metrics for both fidelity and diversity, given that there is no singular optimal solution, no metric to rule them all. For this reason, several approaches were proposed to address this issue, and novel metrics were defined as well. Evaluating existing metrics alone is not a viable solution. Consequently, numerous diverse metrics have emerged, each tailored to measure specific aspects of fidelity or diversity. However, the generation setups of all these previous works are not always detailed and well explained. Moreover, each paper can have a different generation framework posterior to training, which can make the comparison between models somehow problematic.

We thus believe, that it is important to unify the evaluation process. In this work, we first summarize the evaluation metrics from the literature with a unified evaluation framework in order to have a comparison that is as fair as possible. We believe that this work can be a good starting point for newcomers in the field. It is important to note, that through our experimental setup and analysis, we showcase that it is difficult to find the best model that wins over all metrics, as a

slight change in the architecture and hyper parameters can drastically change the metric values resulting in a different interpretation. Instead, we consider a detailed experiment using three variants of a CVAE model trained on the same dataset, highlighting the analysis of each metric.

One special criteria for human motion data, as well as any sequential data, is the temporal dependency between its features (Dau et al., 2019). A common information that has been of interest to many researchers in the field of time series is temporal distortion. Temporal distortion is the measure of time shift, frequency change and warping between a pair of time series samples. This is directly applied in the case of human motion sequences given they are multivariate time series (Bagnall et al., 2018). The metrics utilized in this study are not limited to human motion generation but are applicable to various generative models across different domains. In a case where a generative model is not able to generate the same type of temporal distortions as in the real set of samples, the existing metrics do not capture this lack of diversity given they are computed on a latent representation independent of the temporal aspect of the sequences. For this reason, additionally to presenting the existing metrics in the literature, in this work we propose a novel metric which is specific for temporal data used in the application of human motion generation. This metric, that we called *Warping Path Diversity (WPD)*, evaluates the diversity of temporal distortions in both the real and generated space. In particular, if it is important to generate sequences with different temporal distortion such as different frequencies, then the generative model scores a high value on the *WPD* metric.

In this work, we did not rely on data augmentation as an evaluation metric. This is due to the fact that if we suppose there are not enough data to train a supervised model so we rely on augmented samples provided by a generative model, then the question is: how were we able to train a generative model if there are not enough data? Leading into a paradox problematic, we decided not to experiment with this kind of metric, instead rely on similarity between real and generated distributions through different approaches. A brief summary of all the metrics used in this work is presented in Fig. 1. In this figure, metrics are divided into two categories: fidelity (in **blue**) and diversity (in **green**). Subsequently, the metrics are then organized in sub-categories

<figure><img src="image_4.png" /><figcaption>**Fig. 1.** Summary of all the evaluation metrics for human motion generation used in this work. First, the metrics are divided into two groups: the fidelity metrics in **blue**, and diversity metrics in **green**. Second, the metrics are divided into sub-groups indicating the measure is based on which criteria. The fidelity metrics are: (1) Fréchet Inception Distance (FID), (2) Accuracy On Generated (AOG), and (3) *density*. The diversity metrics are: (1) Average Pair Distance (APD), (2) Average per Class Pair Distance (ACPD), (3) *coverage*, (4) Mean Maximum Similarity (MMS), and (5) the proposed Warping Path Diversity (WPD). (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)</figcaption></figure>
<figure><img src="image_5.png" /><figcaption>Fig. 2. Two steps are followed prior to calculating evaluation measures. In the first step, a model is trained on some supervised task using the real set of data (in blue). This model consists of a feature extractor encoder (in green) and a final layer for the supervised task. In the second step, the pre-trained encoder's latent representation of the real (in blue) is extracted, and samples are generated (in red). The metrics are then computed over this latent representation. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)</figcaption></figure>

depending on which approach they utilize for the evaluation, this includes: accuracy based, distribution based, distance based, neighbor based, neighbor/distance based and warping based.

The contributions of this work are summarized as follows:

* We review most of the frequently used metrics in the literature of human motion generation with their mathematical setup and their interpretation;

* We propose a novel metric specific to human motion generation called Warping Path Diversity (WPD) to measure diversity in terms of temporal distortion;

* We define a unified experimental setup in order to fairly evaluate and compare generative models;

* We support this work with a user-friendly publicly available code to evaluate new generative models on all of the considered metrics: https://github.com/MSD-IRIMAS/Evaluating-HMG.

It is important to note that the primary goal of this paper is not to assess or benchmark the latest generative models but to propose a general and robust evaluation framework for human motion generation, with metrics that can be applied across various generative models rather than focusing on model performance comparison.

While this paper focuses on the evaluation of skeleton-based human motion generation, the proposed framework is designed to be general and flexible. It can be easily adapted to evaluate generative models that use mesh-based representations of human motion, such as the method introduced by Xue and Seo (2024). This extension would allow for a broader comparison of models that utilize more complex avatar representations, facilitating comprehensive evaluation across different motion generation approaches.

## 2. Generative models metrics

The generative models evaluation metrics are divided into two sub-categories called fidelity and diversity. The fidelity metrics measure the extent to which we can trust the generated samples to accurately replicate the characteristics of the real distribution. The higher the fidelity of generated examples, the more challenging it is to differentiate between real and generated samples, resulting in a more reliable generative model. The diversity metrics quantify the extent to which the generated samples differ from each other. A greater diversity within a generated set indicates that the generative model's capacity to produce samples is not confined to a particular segment of the real distribution.

In what follows we first define essential terms to comprehend the rest of this work. Second, we review and detail main metrics employed in the literature for both fidelity and diversity.

### 2.1. Definitions

To understand the metrics that follow, it is necessary to establish some definitions:

* A human motion sequence is referred to as $x \in \mathbb{R}^{D \times T}$, where T corresponds to the length of the sequence and D corresponds to the number of characteristics representing a human skeleton.

* A set of N real samples is referred to as $\mathcal{X} = \{x_i\}_{i=1}^N$ and follows a distribution $P_R$;

* A set of M generated samples is referred to as $\hat{\mathcal{X}} = \{\hat{x}_j\}_{j=1}^M$ and follows a distribution $P_G$;

* A pre-trained deep learning model $\mathcal{G} \circ \mathcal{F}(\cdot)$ is made of a feature extractor $\mathcal{F}$ and a last layer $\mathcal{G}$ achieving the desired task (e.g. classification).

In order to compute most of the metrics, the deep learning model $\mathcal{G} \circ \mathcal{F}$ is trained to solve a specific task on the real set of data. This task is usually classification, hence $\mathcal{G}$ represents a softmax layer. This feature extractor is subsequently employed to project the real data $\mathcal{X}$ and the generated data $\hat{\mathcal{X}}$ into a latent representation. Thus, this allows the calculation of metrics within this latent space. This procedure is illustrated in Fig. 2, showcasing two steps: (1) the training a deep learning model to solve a task on the real data; and (2) the use of the feature extractor part (discarding the last task layer) to encode real and generated samples into a latent space. For the rest of this work, we refer to the latent space of $\mathcal{X}$ and $\hat{\mathcal{X}}$ as V and $\hat{V}$ respectively, such as:

$$V = \mathcal{F}(\mathcal{X}) \text{ and } \hat{V} = \mathcal{F}(\hat{\mathcal{X}}), \qquad (1)$$

where both V and $\hat{V}$ are now two-dimensional matrices, with the same number of examples of both $\mathcal{X}$ and $\hat{\mathcal{X}}$ respectively, and each dimension f, with f being the dimensionality of the latent representation of $\mathcal{X}$ and $\hat{\mathcal{X}}$ produced by $\mathcal{F}$.

For every metric detailed in this section, two versions will be computed: one is calculated using the generated samples (coupled with real samples depending on the metric), and one is calculated on the real samples. The reason for this setup is to have a reference value of the metric over $P_R$. This is done by randomly splitting V into two subsets $V_1$ and $V_2$, and calculating the metrics as if $V_1$ is the latent space of the real samples and $V_2$ the latent space of the generated samples.

In what follows, we denote by $\mathcal{N}(\mu, \Sigma)$ a multidimensional Gaussian distribution of mean vector $\mu$ and covariance matrix $\Sigma$.
<figure><img src="image_6.png" /><figcaption>Fig. 3. On the left, an example computation of the amount of energy needed (FID) to transform the standard Gaussian distribution (in blue) to another Gaussian distribution with higher values of μ and σ² (in red). On the right, the amount of energy (FID) needed for this transformation, gradually increasing with the increase of μ and σ², can be observed. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)</figcaption></figure>

## 2.2. Fidelity metrics

The fidelity metrics in the literature are either distribution based, neighbor based or accuracy based.

### 2.2.1. Fréchet Inception Distance (FID)

First proposed in 2017 by Heusel et al. (2017), the FID metric is still the most used measure to evaluate generative models. It is an improvement of a previous measure, the Inception Score (IS) (Salimans et al., 2016), which mainly evaluates the classification of a pre-trained Inception image classifier on the generated samples. The Inception model is pre-trained on real samples, which means the IS measure takes into consideration, indirectly, the real distribution $P_R$. However, it never quantifies the difference between both $P_R$ and $P_G$. For this reason, the authors in Heusel et al. (2017) proposed a novel evaluation metric called FID, which is based on calculating the Fréchet Distance (Fréchet, 1906) between two Gaussian distributions in Inception's latent space of both $\mathcal{X}$ and $\hat{\mathcal{X}}$.

The Fréchet Distance (FD) (Fréchet, 1906) measures a distance between continuous finite curves. To understand what FD measures, a famous example goes as follows: Imagine a person and their dog, each wanting to traverse a different finite curved path. The speed of the person and the dog can vary but they are not allowed to go backward on the path. The FD between these two curves is the length of a leash, small enough so that both the person and the dog can traverse the whole finite curve. In the case of two probability distributions (Fréchet, 1957), the FD is calculated between the Cumulative Distribution Functions (CDFs) of both distributions. For the case of multidimensional Gaussian distributions (Dowson and Landau, 1982) $P_1 \sim \mathcal{N}(\mu_1, \Sigma_1)$ and $P_2 \sim \mathcal{N}(\mu_2, \Sigma_2)$, both of dimension $f$, the FD is calculated as follows:

$$FD(P_1, P_2)^2 = \text{trace}(\Sigma_1 + \Sigma_2 - 2(\Sigma_1 \cdot \Sigma_2)^{1/2}) + \sum_{i=1}^{f} (\mu_{1,i} - \mu_{2,i})^2, \quad (2)$$

where the values of FD (or FID) range from 0 to $+\infty$.

**Setup for generative models.** First, the mean vectors $\mu_f$ and $\hat{\mu}_f$ of both V and $\hat{V}$, respectively, are estimated empirically, as well as their covariance matrices $\Sigma_f$ and $\hat{\Sigma}_f$. Second, the FD is calculated using Eq. (2). For consistency with the literature, we refer in the rest of this work for this metric as the FID instead of FD, even though the Inception network is not used in the case of human motion.

**Interpretation.** The FID can be considered as the amount of energy (or effort) that is needed to transform one Gaussian distribution to another one. This can be seen in Fig. 3, where the probability density function of two Gaussian distributions is plotted (left side). The amount of energy (FID) needed to transform the $\mathcal{N}(\mu = 0.0, \sigma^2 = 1.0)$ to $\mathcal{N}(\mu = 1.2, \sigma^2 = 1.8)$ increases when the mean and variance of the first distribution are increased (right side). Given that no amount of energy is needed to transform a distribution to itself, the starting point of the

plot in Fig. 3 (right side) is 0.0. This last interpretation is consistent with the original definition of the FID, as it is a distance.

Many works in the literature mention that a lower value of $FID_{gen}$ implies a high-fidelity of generated samples: However, this can be miss-interpreted in some cases. For instance, if we suppose a generative model learned to copy-and-paste the real samples, this would results into a perfect alignment between $P_R$ and $P_G$. This would produce an $FID_{gen}$ of 0, but without generating samples that bring any new value. For this reason, we propose that the comparison two generative models over fidelity using the FID metric should follow this Theorem:

**Theorem 1.** a generative model $Gen_1$ is considered more fidelitous than another model $Gen_2$ on the FID metric if $FID_{gen1} < FID_{gen2}$ while respecting the following constraint:

$$\forall \epsilon > 0, \quad FID_{gen} = FID_{real} + \epsilon.$$

This means that the FID metric evaluates how closely the generated data approximates the distribution of the real data, not implying that the generation model has direct access to real data. The goal is for the generated data to statistically resemble the real data distribution as much as possible, rather than replicating it exactly.

More details on this setup are given in Section 4.5.

### 2.2.2. Accuracy On Generated (AOG)

Generative models can be trained with a condition by incorporating information about the relevant characteristics associated with each sample. In the case of a labeled dataset, this information can be in the form of a discrete label (class information), a continuous value associated to a sample, or even be a text description of the sample. Usually, conditioning the generative model offers a better control on the generated sample. For instance, the generation of a human motion sequence could be conditioned with the action the person should perform (e.g., “running”, “jumping”, etc.).

A possible approach to evaluate the conditioning capability of the generative model, is to look at the score of a classifier pre-trained on solving the task over real samples $\mathcal{X}$, assuming the generated set $\hat{\mathcal{X}}$ is an unseen set. The classifier used here is $\mathcal{G} \circ \mathcal{F}$ defined in Section 2.1. In this study, we refer to this metric as the Accuracy On Generated (AOG), and it is formulated as follows:

$$AOG(\hat{\mathcal{X}}, \mathcal{G} \circ \mathcal{F}) = \frac{1}{M} \sum_{i=1}^{M} 1\{\mathcal{G} \circ \mathcal{F}(\hat{\mathcal{X}}_i) == \hat{\mathcal{Y}}_i\}, \quad (3)$$

where $\hat{\mathcal{Y}}$ represents the set of labels employed to condition the generation process, serving as ground truth labels that $\mathcal{G} \circ \mathcal{F}$ is expected to predict. Additionally, 1 denotes the indicator function defined as:

$$1\{\text{condition}\} = \begin{cases} 1 & \text{if condition is True} \\ 0 & \text{if condition is False} \end{cases} \quad (4)$$

where the values of AOG in Eq. (3) range from 0 to 1.

<figure><img src="image_7.png" /></figure>
<figure><img src="image_8.png" /><figcaption>Fig. 4. This example showcases the computation of the AOG metric for two generative models. Given a real space of data, where the samples are spread over three sets of classes (red triangles, blue squares, and green circles), the generative model should be able to conditionally generate new samples. This condition is simply to which of the possible classes the generated example should belong to. From the example, two generative models should be able to generate six samples, each conditioned on a class. The set of conditions used is Ŷ (the ground truth). Posterior to generating, a pre-trained model classifies the generated samples. Hence, the AOG metric corresponds to the accuracy of classification between the predicted class provided by the pre-trained model and the ground truth. In this example, Model1 obtains the optimal AOG value of 100%, while Model2 struggles to correctly condition its generation (AOG of 50%). (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)</figcaption></figure>

**Setup for generative models.** The AOG metric is calculated between the ground truth labels Ŷ and the predictions of GoF(.) using Eq. (3).

**Interpretation.** The value of the AOG metric and the generative model's conditioning capability are directly proportional. However, a generative model with a very low value of AOG may indicate that the generative model is capable of generating samples from the same label and not the whole possible set of labels. Additionally, a generative model with a 100% AOG, would not certainly mean that the generated samples are fidelitous with respect to $P_R$. For instance if the generated samples contains a significant amount of residual noise, but enough amount to not affect the classifier's performance, then a high value of the AOG metric leads to wrong conclusions about the generation's fidelity. A visualization example on the AOG's behavior is given in Fig. 4.

### 2.2.3. Density and precision

Relying on one metric to assess the performance of generative models is not always feasible, especially that it is not an easy challenge to evaluate both fidelity and diversity in a single metric. This was argued in Sajjadi et al. (2018) that two generative models can have the same value of the FID metric, but still differ in terms of fidelity and diversity, qualitatively. Hence, the authors in Sajjadi et al. (2018) proposed two metrics, precision and recall to quantitatively evaluate fidelity and diversity, respectively. This was followed by an improved version of these two metric in Kynkäänniemi et al. (2019) by avoiding the hypothesis of a uniformly dense latent space and estimating density functions using the k-nearest neighbor algorithm. In particular, precision represents the portion of samples generated by $P_G$ that can be sampled from $P_R$ as well. The formulation of the improved - precision (referred to as precision in the rest of this work) is as follows:

$$ precision = \frac{1}{M} \sum_{j=1}^{M} 1(\hat{V}_j \in \text{manifold}(\mathbf{V}_1, \dots, \mathbf{V}_N)), \quad (5) $$

where manifold($\{a_1, a_2, \dots, a_n\}$) = $\bigcup_{i=1}^n B(a_i, NND_k(a_i))$, $B(c, r)$ is a sphere in $\mathbb{R}^{\text{dimension}(a_i)}$ of center c and radius r, and $NND_k(a_i)$ is the distance from $a_i$ to its $k_{th}$ nearest neighbor in the set $\{a_j\}_{j=1, j \neq i}^N$. The values of precision range from 0 to 1.

In Naeem et al. (2020), the authors identified some limitations of both precision and recall metrics. They proposed two novel metrics,

density and coverage for fidelity and diversity, respectively. In this section, we discuss the fidelity aspect only, and in Section 2.3.3 the diversity aspect for recall and coverage. Two main limitations of the precision metric were highlighted in Naeem et al. (2020). Firstly, in the simplest case scenario where both real and generated samples follow the same distribution ($P_R$ and $P_G$ are identical), the authors observed that no closed formulation can be found for the expected value of precision. Secondly, in the case where the real set of data $\mathcal{X}$ contains some outliers that can lead to the generation of outliers in $\hat{\mathcal{X}}$, the interpretation of the precision to be considered good can be mislead.

The mathematical formulation of the density metric is as follows:

$$ density = \frac{1}{k.M} \sum_{j=1}^{M} \sum_{i=1}^{N} 1(\hat{V}_j \in B(V_i, NND_k(V_i))), \quad (6) $$

where the values of density range from 0 to $\frac{N}{k}$.

The authors in Naeem et al. (2020) proved that in the case of identical distributions $P_R$ and $P_G$, the expected value of density is 1. This showcases that, with enough samples and a high number of neighbors, the convergence of density to the value of 1 is proportional to the fidelity of the generated samples. In what follows, we present a constructed example on a real human motion dataset to showcase the second benefit of density over precision, the detection of outlier in the real distribution.

An illustration of the outlier scenario is presented in Fig. 5 on a synthetically constructed dataset. Fig. 5 showcases on the left the latent space representation of five real samples and four generated samples. Among the five real samples, an outlier is present and highlighted in red in both the latent space (left) and series space (right). It can be seen from the figure that some generated samples are correctly placed in the distribution of the four real samples, and two other generated are closer to the fifth real outlier. The precision metric, in this scenario, fails to identify the existence of an outlier in the real distribution, resulting in high precision of 1 usually interpreted as a high fidelity score. However, the density metric is able to identify that two generated samples are only present in one neighbor circle, belonging to the outlier sample, hence producing a non-perfect value $1.25 \neq 1$.

**Setup for generative models.** First, we find the distance of each sample in $F(\mathcal{X})$ to its $k_{th}$ nearest neighbor in $F(\mathcal{X})$. Second, we calculate both the precision and density metrics using Eqs. (5) and (6).
<figure><img src="image_9.png" /><figcaption>Fig. 5. This example showcases the computation of both the density and precision metrics over a synthetic dataset. On the left, we present the latent representation of the real set of data (blue points), the real outlier (red point), the generated examples around the outlier (black points), and generated examples around non-outliers (in green). The circles in blue represent the neighborhood area of each latent point of a real sample, for both outlier and non-outlier real samples. On the right, we present the original series space of the data with the same associated colors. It can be seen that the density metric overcomes the existence of an outlier and does not produce a perfect measure, 1.25 ≠ 1. However the precision does not detect the outlier and produces a perfect measure of 1. This example highlights how the density metric concludes that the generated samples are not fully fidelitous to the real distribution because of the generations around the real outlier that a generative model should overcome. The number of neighbors used for both metrics is set to 2. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)</figcaption></figure>

<figure><img src="image_10.png" /></figure>

**Interpretation.** On the one hand, the *precision* metric counts the number of generated samples existing in at least one neighbor sphere of any real samples. On the other hand, *density* counts the number of neighbor spheres containing each of the generated samples. While both metrics quantify how fidelitous generated samples are to real samples, the *density* metric quantifies this aspect individually between all pairs of real and generated samples. In contrast, the *precision* metric falls short when considering the union of all neighboring spheres, as it does not assess the bias of some generated samples due to the existence of a real outlier.

## 2.3. Diversity metrics

Evaluating *fidelity* measures only does not result in a full reliability on generated samples. For this reason diversity measures are required. In what follows, we present the *diversity* metrics used in the literature, ranging from distance based approaches to neighbor based approaches.

### 2.3.1. Average Pair Distance (APD)

Originally proposed in 2018 in Zhang et al. (2018) to compute distances between images and adapted for human motion generative models' evaluation in 2020 in Guo et al. (2020), the Average Pair Distance (APD) has ever since been the go-to metric for evaluating diversity of generative models. The APD metric can evaluate the diversity of any set of data, not necessarily generated ones. This is done through calculating the average Euclidean distance between randomly selected pairs of samples from the same dataset. This random selection is repeated R times over $S_{apd}$ number of pairs for each experiment. The final value of the APD metric is the average outcome of the repeated experiments. The APD metric calculated on the generated set of samples, for one random selection $r \in \{1, 2, \dots, R\}$ is formulated as follows:

$$APD_r(\hat{X}) = \frac{1}{S_{apd}} \sum_{i=1}^{S_{apd}} \sqrt{\sum_{j=1}^{f} (S_{i,j} - S'_{i,j})^2}, \quad (7)$$

where $S$ and $S'$ are two randomly selected subsets of $\hat{V} = F(\hat{X})$, i.e. $S, S' \subset \hat{V}$.

The APD metric is then calculated by averaging over the R random experiments:

$$APD(\hat{X}) = \frac{1}{R} \sum_{r=1}^{R} APD_r(\hat{X}), \quad (8)$$

where $APD_r(\hat{X})$ is calculated using Eq. (7). An illustration of the APD metric is represented in Fig. 6 showcasing the same procedure followed to calculate both $APD_{real}$ and $APD_{gen}$.

**Setup for generative models.** We calculate the APD metric using Eq. (7) and (8).

**Interpretation.** The APD metric evaluates if a generative model is able to overcome the model collapse phenomena that leads to generating the same outcome over and over. If the expected outcome of the generative model is to be as diverse as possible, then the APD metric should be as high as possible. However, an issue may occur with this interpretation: What happens in the case where $APD_{gen} > APD_{real}$? For this reason, as interpreted in Guo et al. (2020), we present the following theorem:

**Theorem 2.** a generative model $Gen_1$ is considered more diverse than another model $Gen_2$ if and only if:

$$APD_{gen1} > APD_{gen2} \text{ and } \forall \epsilon > 0, |APD_{gen1} - APD_{real}| < \epsilon.$$

This means that $Gen_1$ can only be considered more diverse than $Gen_2$ if its diversity score ($APD_{gen1}$) is not only higher than that of $Gen_2$, but also sufficiently close to the diversity of the real data ($APD_{real}$). This ensures that a model with artificially high diversity is not considered better if it deviates significantly from the true diversity of the real data. Proving Theorem 2 can be done by using a counter example to break its negation. For instance, if the real data has a diversity of $APD_{real} = 5$, and the generative model is not trained, but randomly initialized and used to generated samples, then the diversity of generated samples $APD_{gen}$ can be higher than 5, while the generations are simply random, and without any correlation with $P_R$ exists.

### 2.3.2. Average per Class Pair Distance (ACPD)

Similar to the APD, the Average per Class Pair Distance (ACPD) evaluates the diversity of the generated samples, with the same interpretation as for the APD. The two metrics differ in terms of which scale
<figure><img src="image_11.png" /><figcaption>Fig. 6. This example showcases the computation of the APD metric over both real samples (in blue) and generated samples (in red). For each of the latent representations of real and generated samples, two sets of randomly chosen values are constructed of size $S_{apd}$ each. The APD metric is then calculated between these two sets of random selected values S and S'. This random selection is repeated R times, the final APD value is the average of all the R computed APD values. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)</figcaption></figure>

is diversity evaluated. For instance, the APD metric evaluates on the scale of $P_R$, but it never evaluates the diversity on the scale of each sub-cluster in $P_R$. For this reason, the ACPD metric calculates the average APD on each sub-cluster in $P_R$. The mathematical formulation of the ACPD is presented in the following:

$$ACPD(\hat{\mathbf{X}}) = \frac{1}{C \cdot S_{apd}} \sum_{i=1}^{S_{apd}} \sqrt{\sum_{j=1}^{f} (S_{c,i,j} - S'_{c,i,j})^2}, \quad (9)$$

where C is the total number of classes in $P_R$, $S_c$, $S'_c$ are randomly selected subsets from $\hat{\mathbf{V}}[\hat{\mathbf{Y}} == c]$, and $\hat{\mathbf{Y}}$ are the labels used to generate $\hat{\mathbf{X}}$. In other words, contrary to APD, the ACPD metric randomly selects the subsets S and S' from the latent representation $\hat{\mathbf{V}}$ of $\hat{\mathbf{X}}$ that were generated using the label c, hence the notation $S_c$ and $S'_c$.

Similarly to APD, given the randomness factor in ACPD, the experiment is repeated R times to calculate $ACPD_r, r \in \{1, 2, \dots, R\}$, and finally averaged as:

$$ACPD(\hat{\mathbf{X}}) = \frac{1}{R} \sum_{r=1}^{R} ACPD_r(\hat{\mathbf{X}}). \quad (10)$$

It is important to note that this metric is restricted only to labeled datasets.

**Setup for generative models.** We calculate ACPD using Eqs. (9) and (10).

**Interpretation.** The ACPD metric evaluates the diversity of the generated samples per sub-cluster in $P_R$, hence evaluating if a model is able to overcome the generation of samples from only one cluster in $P_R$. This last problem would lead to generating high diversity in one cluster (class) and less for others. This is because machine learning often encounters imbalanced labeled data, and the ACPD metric helps to assess whether a generative model can effectively address this issue.

Similarly to the APD metric, ACPD is calculated on both real and generated samples producing $ACPD_{real}$ and $ACPD_{gen}$, where a generative model is considered class diverse when $ACPD_{gen}$ is close to $ACPD_{real}$.

### 2.3.3. Coverage and recall

As mentioned in Section 2.2.3, the authors in Naeem et al. (2020) proposed two metrics to replace the improved precision and recall metrics (Kynkäänniemi et al., 2019; Sajjadi et al., 2018). The recall metric represents the number of real examples sampled using $P_R$ that can be sampled using $P_G$. This is done by counting the number of real samples that exist in at least one neighborhood of a generated sample. The recall metric is formulated as:

$$recall = \frac{1}{N} \sum_{i=1}^{N} 1(\mathbf{V}_i \in \text{manifold}(\hat{\mathbf{V}}_1, \hat{\mathbf{V}}_2, \dots, \hat{\mathbf{V}}_M)), \quad (11)$$

where manifold(.) and 1(.) follow the same definition detailed in Section 2.2.3. The recall metric is bounded between 0 and 1.

Some limitations of the recall metric have been identified by authors of Naeem et al. (2020), and can be summarized as:

1. Basing the neighborhood areas on the generated samples can produce misleading interpretation given the higher chance of having an outlier sampled by $P_G$ compared to an outlier sampled by $P_R$.

2. No closed form solution does exist for the expected value of the recall metric in the case of $P_R$ and $P_G$ being identical distributions.

For this reason, authors of Naeem et al. (2020) proposed the coverage metric relying on neighborhood areas around the real samples $\mathcal{X}$. The coverage metric counts the number of real samples that include at least one generated sample from $\mathcal{X}$. The coverage metric is formulated as:

$$coverage = \frac{1}{N} \sum_{i=1}^{N} 1(\exists j \text{ s.t. } \hat{\mathbf{V}}_j \in B(\mathbf{V}_i, NND_k(\mathbf{V}_i))), \quad (12)$$

where $B(\cdot, \cdot)$ and $NND_k(\cdot)$ follow the same definitions detailed in Section 2.2.3. The coverage metric is bounded between 0 and 1.

The coverage metric overcomes the first limitation of recall by avoiding to use neighborhoods of generated samples. A simple example of how recall fails to detect the over-estimated neighborhood of generated samples and how coverage overcomes it, is illustrated in Fig. 7. We utilize for this illustration the same synthetic example as in Fig. 5. On the one hand, it can be seen that the recall metric results in a perfect diversity (left scatter plot). This is due to the over-estimated neighborhood produced because of the outlier generated samples. On the other hand, the coverage metric is able to distinguish between good and outlier generated samples by simply relying on the neighborhood of the real samples.

The coverage metric overcomes as well the second limitation of recall, as proven in Naeem et al. (2020). The closed form solution of the expected value of coverage in the case of identical distributions $P_R$ and $P_G$ is simplified to:

$$E[coverage] = 1 - \frac{(N - 1) \dots (N - k)}{(M + N - 1) \dots (M + N - k)}, \quad (13)$$

which reduces to, in the case where both N and M are high enough:

$$E[coverage] = 1 - \frac{1}{2^k}. \quad (14)$$

**Setup for generative models.** First, we find the distance of each sample in $\hat{\mathbf{V}}$ to its $k_{th}$ nearest neighbor in $\hat{\mathbf{V}}$ and calculate the recall metric using Eq. (11). Second, we find the distance of each sample in $\mathbf{V}$ to its $k_{th}$ nearest-neighbor in $\mathbf{V}$ and calculate the coverage metric using Eq. (12).
<figure><img src="image_12.png" /><figcaption>Fig. 7. Computation of both the coverage and recall metrics over a synthetic dataset. In the left side and right side of the figure (same plot with different interpretation), we present the latent space representation of real samples (in blue) and generated samples (in green and in black). The points in green represent generated samples that are similar to real samples in blue. The points in black represent outliers in the generated space. In the middle part of the figure, we show the original series space of the data with the same associated colors, where it highlights that the outlier generations (series in black) are not similar to the original space of series in blue unlike the reliable generated samples in green. The left plot of the latent space contains the neighbor areas around the generated samples, as the recall metric requires, and the right plot of the latent space contains the neighbor areas around the real samples as the coverage metric requires. It can be seen that the coverage metric overcomes the existence of outlier generations and produces a non-perfect measure of 0.5 < 1 - 1/4 in contrary to the recall metric that produces a perfect measure of 1. This example highlights how the coverage metric concludes that the diversity in the generated space is not as good as the real space because of the existence of generated outliers. The number of neighbors used for both metrics is set to 2. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)</figcaption></figure>

**Interpretation.** On the one hand, the *recall* metric produces the portion of real samples that falls inside at least one neighborhood of the generated samples. On the other hand, the *coverage* metric produces the complete opposite, which is the proportion of the real samples that contains at least one generated sample in their neighborhood. While both metrics evaluate the diversity aspect of a generative model, the *coverage* metric is a less risky approach relying on real neighborhoods.

Given the *coverage* metric has an expected value of $1 - \frac{1}{2^k}$ in the case of identical $P_R$ and $P_G$, then the interpretation of a high diversity depends on the choice of the number of neighbors $k$.

### 2.3.4. Mean Maximum Similarity (MMS)

Originally proposed in Cervantes et al. (2022), the Mean Maximum Similarity (MMS) is an evaluation metric measuring novelty in the generated set. For instance, a generative model can simply learn to generate almost exactly the same data used for training, leading to as much diversity as the real set of data. However, this can be misleading, as such generative model is not solving the task at hand. For this reason, the authors in Cervantes et al. (2022) proposed a new metric to quantify novelty, which we interpret in this work as diversity.

The MMS quantifies novelty/diversity by averaging the distances of each of the generated samples to its real nearest samples. It is given by:

$$MMS = \frac{1}{M} \sum_{j=1}^{M} \sqrt{\sum_{d=1}^{f} (\hat{V}_{j,d} - V_{NN_j,d})^2}, \quad (15)$$

where $V_{NN_j}$ (from the real set) is the nearest neighbor to $\hat{V}_j$ (from the generated set). A visual representation of the MMS metric is shown in Fig. 8.

**Setup for generative models.** First, for every sample in $\hat{V}$, the distance to its nearest neighbor in $V$ is calculated and averaged to produce $MMS_{gen}$. Second, for every sample in $V$, the distance to its second nearest neighbor in $V$ is calculated and averaged to produce $MMS_{real}$. We do not utilize the $V_1$ and $V_2$ sets for the MMS metric, as we are following the setup of calculating $MMS_{real}$ in the original paper (Cervantes et al., 2022).

**Interpretation.** The authors in Cervantes et al. (2022) interpreted their proposed MMS metric as $MMS_{gen}$ should always be higher than $MMS_{real}$ to indicate high-novelty. However, a limit for this metric is

observed when the model is generating random samples far from the real set; in this case, the novelty is simply an over estimation of the generated population. For this reason, this metric alone is not sufficient for an easy interpretation of a generative model's performance. For this metric, as proposed in the original paper (Cervantes et al., 2022), it is important to note that $MMS_{real}$ is calculated between the real set of samples and itself entirely, and not between two sub-sets as for all other metrics.

### 2.4. Overview

In the following, we provide an overview of how each metric has been used in prior studies. The FID measure has become ubiquitous in generative AI research (Heusel et al., 2017), extending far beyond human motion generation and is now considered the “most common” metric for assessing the fidelity of generated data in a wide range of tasks. On the other hand, APD and ACPD were first introduced for human motion generation in Action2Motion (Guo et al., 2020) and have since been used primarily in studies that build upon or incrementally improve this approach, such as ACTOR (Petrovich et al., 2021) and PoseGPT (Lucas et al., 2022). MMS, by contrast, has only been applied in the original paper that proposed it (Cervantes et al., 2022) as it is still very recent. In this work, we introduce the AOG as an alternative “accuracy-based” metric, aiming to avoid the reliance on data augmentation methods, which we discussed in the introduction. Finally, while Density and Coverage have been well-established in image generation (Naeem et al., 2020), to our knowledge, they have not yet been applied to human motion generation. We propose their introduction to the field, given their relevance for evaluating the diversity and quality of generated motions.

Foot skating, where a character's feet slide unnaturally, is a common artifact in human motion generation. Although not a major metric, it has been quantified in certain works, such as Karunratanakul et al. (2023). While, in this paper, we do not focus on foot skating, it remains an important factor that can be considered in future work for a more comprehensive evaluation of motion quality.

Similar metrics have been explored in speech gesture generation studies, which offer valuable insights into generative model evaluation. For instance, Yoon et al. (2020) explored auto-encoding based latent space learning and objective metric validation in the context of gesture generation from text, audio, and speaker identity. Additionally, Kucherenko* et al. (2024) analyzed the correlation between
<figure><img src="image_13.png" /><figcaption>**Fig. 8.** This example showcases the computation of the MMS metric over a synthetic dataset. The left side of the figure consists of the latent representation of both real samples (in blue) and generated samples (in green). The right side of the figure consists of the original series space of real and generated samples with the same associated colors. First, for each generated point in the latent space (left side), we find its nearest neighbor in the real set. Second, the Euclidean distance is calculated between each pair of generated point and its nearest neighbor. Third, the final MMS metric is the average of all computed distances. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)</figcaption></figure>

objective and subjective metrics in the GENEA Challenge, which provides further context for evaluating generative models, including the relationship between quantitative metrics and human perception of motion quality.

Villa-Monedero et al. (2023) used the value of the DTW metric in their evaluation of sign language datasets for automatic motion generation. While this demonstrates the utility of DTW-based metrics in motion generation tasks, in the following section, we argue that it is also important to consider the warping path information, not just the final DTW value. This provides deeper insights into the temporal distortions present in generated sequences, allowing for a more comprehensive evaluation of motion quality.

### 3. Proposed metric: Warping Path Diversity (WPD)

Every diversity metric detailed in Section 2.3 utilizes a pre-trained encoder $F$ to extract some latent features. This is necessary given most metrics assume an input latent space. However, for temporal data, particularly human motion sequences, this process may introduce temporal distortions that are significant in some contexts. Warping diversity can be caused by more than one factor such as temporal shifting, frequency change, etc. An example considering the left-hand $y$-coordinate of skeletons is illustrated in Fig. 9. It can be seen that the real samples (top of Fig. 9) from the HumanAct12 dataset start the action of drinking with left-hand at different frames. However, we can see in the generated space (bottom of Fig. 9) that the action of drinking with left-hand always starts at the same frame, highlighting a lack of warping diversity. As shown in the right side of Fig. 9 the pre-trained feature extractor $F$ produces very close latent representations for all these sequences, highlighting the fact that $F$ does not take into consideration the temporal distortion. The APD metric, for instance, calculates the average Euclidean distance in the latent space without taking into account the warping diversity.

For this reason, we proposed in this work a novel metric to quantify this type of diversity that relies on temporal distortions. We utilized the Dynamic Time Warping (DTW) (Müller, 2007) similarity measure to extract these temporal distortions between two sequences.

In what follows, we present a background on DTW before digging into how the metric is calculated and how it can evaluate diversity in warping.

**Algorithm 1: Dynamic Time Warping (DTW)**

**Data:** Sequence $x$, Sequence $y$

**Result:** DTW distance

1. Set $L_x = len(x)$; Set $L_y = len(y)$; Initialize a matrix $D$ of size $L_x + 1 \times L_y + 1$;

2. Set $D[\cdot, 0] = \infty$;

3. Set $D[0, \cdot] = \infty$;

4. Set $D[0, 0] = 0$;

5. **for** $i \leftarrow 1$ **to** $L_x$ **do**

6. **for** $j \leftarrow 1$ **to** $L_y$ **do**

7. Set $d = (x[i] - y[j])^2$;

8. Set $D[i][j] = d + \min(D[i-1, j], D[i, j-1], D[i-1, j-1])$;

9. **end**

10. **end**

11. **return** $D[L_x, L_y]$;

### 3.1. Background on Dynamic Time Warping (DTW)

Dynamic Time Warping (Müller, 2007) is a similarity measure between temporal sequences. It relies on finding the optimal temporal alignment between two sequences followed by a simple Euclidean distance on the aligned series. The computation of DTW is presented in Algorithm 1.

In simple words, DTW finds the optimal warping path that is admissible between two sequences. A path is a set of pairs of indices $\pi = \{(\pi_{1,1}, \pi_{1,2}), (\pi_{2,1}, \pi_{2,2}), \dots, (\pi_{L_x,1}, \pi_{L_x,2})\}$, where a pair of indices represent an alignment of these two points from $x$ and $y$. Given two sequences, $x$ and $y$ of lengths $L_x$ and $L_y$, respectively, a warping path $\pi$ between those two sequences is considered admissible if and only if:

* $\pi_0 = (0, 0)$

* $\pi_{L_x} = (L_x, L_y)$

* The sequence should be monotonically increasing in both $i$ and $j$, where all indices of both time series should appear at least once, so at element $l$ of $\pi$:

$$i_{l-1} \le i_l \le i_{l-1} + 1, \quad j_{l-1} \le j_l \le j_{l-1} + 1$$
<figure><img src="image_14.png" /><figcaption>Fig. 9. This figure illustrates the need for a temporal distortion diversity metric in certain applications. It showcases three real and three generated human motion sequences performing the “drink-with-left-hand” action. The top and bottom sections on the left display the y-axis projection of the subject’s left hand motion as univariate time series in blue, orange and green. The real samples demonstrate variability in the starting frame for lifting the bottle, whereas the generated samples show a consistent starting frame. On the right, the latent representation of both real (in blue) and generated (in red) samples is depicted using a pre-trained GRU classifier. This representation highlights that the pre-trained model does not account for temporal distortion diversity. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)</figcaption></figure>

Fig. 10 illustrates the DTW matrix calculated between two sequences and the optimal warping path is presented in gray. This optimal warping path is then used to align both series before applying the Euclidean distance between them.

## 3.2. Warping Path Diversity (WPD)

For simplicity, we consider in this section that both sequences x and y are of the same length $L_x = L_y = L$. For the two sequences in Fig. 10, three scenarios can occur:

1. Worst case scenario, with worst alignment and the length of the path equal to 2.L (through matrix’s edges);

2. Best case scenario, with best alignment and the length of the path equal to the length of the diagonal. Applying DTW is equivalent to directly use Euclidean distance on x and y;

3. Temporal distortions are present, the path is not on the diagonal but its length is smaller than 2.L.

In order to measure how much diverse both sequences are in terms of temporal distortions (warping), we propose to quantify how far the warping path is from the diagonal. For this reason, we need to calculate the sum of distances from each point in the warping path to the diagonal. A visual representation of this distance calculation is presented in Fig. 11. We consider each point on the path being in a integer coordinate space with both axis vary from 0 to L-1. Given the case of equal length sequences, the triangle presented in Fig. 11 on each point of the warping path is a right isosceles triangle. This information leads to the hypotenuse’s median being equal to half its length. The hypotenuse’s length can be simply calculated using the Pythagorean theorem. The distance of each point on the path π to the diagonal can be calculated as follows:

$$ distance(\pi_k, diagonal) = \frac{\sqrt{2}}{2} |\pi_{k,1} - \pi_{k,2}|. \qquad (16) $$

**Proof.** Using the annotations of a, b, c and d in Fig. 11 as the summit of points $\pi_k$ on the warping path $\pi$, we present the following formulation of distance($\pi_k$, diagonal):

$$ \begin{aligned} d &= \frac{1}{2} \sqrt{c^2} = \frac{1}{2} \sqrt{a^2 + b^2} = \frac{1}{2} \sqrt{2 * a^2} = \\ &= \frac{1}{2} \sqrt{2 * (\pi_{k,1} - \pi_{k,2})^2} = \frac{\sqrt{2}}{2} |\pi_{k,1} - \pi_{k,2}|. \end{aligned} $$

The WPD value between x and y is the average distance of all points on the warping path to the diagonal, as follows:

$$ WPD_d(x, y) = \frac{\sqrt{2}}{2 \cdot L_{\pi}} \sum_{k=1}^{L_{\pi}} |\pi_{k,1} - \pi_{k,2}|. \qquad (17) $$

Finally, the WPD metric of a generative model is calculated, like for the APD metric, between random subsets of samples from both real and generated samples:

$$ WPD_r = \frac{1}{S_{wpd}} \sum_{i=1}^{S_{wpd}} WPD_d(S_i, S'_i), \qquad (18) $$

where S and S' are two randomly selected subsets of $\hat{V} = F(\hat{X})$, i.e., S, S' ⊂ $\hat{V}$ and r ∈ {1, 2, ..., R} is the number of repetitions of this random experiment to avoid the bias of a random selection. The final WPD metric is calculated as:

$$ WPD = \frac{1}{R} \sum_{r=1}^{R} WPD_r, \qquad (19) $$

with WPD bounded between 0 and $\frac{\sqrt{2}}{4}(L+1)$, and L is the length of the sequences x and y.

The characteristics of our proposed WPD metric are summarized in Table 1 along with all the other metrics presented in this study.
<table><caption>**Table 1**
Summary of the Generative Models Metrics in this study.</caption><thead><tr><th>Metric</th><th>Category</th><th>Space</th><th>Hyper-Parameters</th><th>Bounds</th><th>Interpretation</th><th>Better Version</th><th>Used in Study</th></tr></thead><tbody><tr><td>FID</td><td>fidelity</td><td>latent</td><td>None</td><td>0 ≤ FID &lt; ∞</td><td>Higher but close to FID<sub>real</sub></td><td>None</td><td>yes</td></tr><tr><td>AOG</td><td>fidelity/accuracy</td><td>latent</td><td>None</td><td>0 ≤ AOG ≤ 1</td><td>Close to 1 (100% accuracy)</td><td>None</td><td>yes</td></tr><tr><td>density</td><td>fidelity</td><td>latent</td><td>• k: number of neighbors</td><td>0 ≤ density ≤ N/k<br>N being the number of real samples</td><td>closer to density<sub>real</sub> which is close to 1</td><td>None</td><td>yes</td></tr><tr><td>precision</td><td>fidelity</td><td>latent</td><td>• k: number of neighbors</td><td>0 ≤ precision ≤ 1</td><td>closer to 1</td><td>density</td><td>no</td></tr><tr><td>coverage</td><td>diversity</td><td>latent</td><td>• k: number of neighbors</td><td>0 ≤ coverage ≤ 1</td><td>closer to coverage<sub>real</sub> which is close to 1 − 1/2<sup>k</sup></td><td>None</td><td>yes</td></tr><tr><td>recall</td><td>diversity</td><td>latent</td><td>• k: number of neighbors</td><td>0 ≤ recall ≤ 1</td><td>closer to 1</td><td>coverage</td><td>no</td></tr><tr><td>APD</td><td>diversity</td><td>latent</td><td>• S<sub>apd</sub>: size of random subset<br>• R: number of random experiments</td><td>0 ≤ APD &lt; ∞</td><td>Lower but close to APD<sub>real</sub></td><td>None</td><td>yes</td></tr><tr><td>ACPD</td><td>diversity</td><td>latent</td><td>• S<sub>acpd</sub>: size of random subset<br>• R: number of random experiments</td><td>0 ≤ ACPD &lt; ∞</td><td>Lower but close to ACPD<sub>real</sub></td><td>None</td><td>yes</td></tr><tr><td>MMS</td><td>diversity/ novelty</td><td>latent</td><td>None</td><td>0 ≤ MMS &lt; ∞</td><td>Higher but close to MMS<sub>real</sub></td><td>None</td><td>yes</td></tr><tr><td>WPD (ours)</td><td>diversity/ warping</td><td>on raw data</td><td>• S<sub>wpd</sub>: size of random subset<br>• R: number of random experiments</td><td>0 ≤ WPD ≤ <sup>√2</sup>⁄<sub>4</sub> (L + 1)<br>L being the length of the sequence</td><td>Depends on the application. Higher could be better, or close to WPD<sub>real</sub></td><td>None</td><td>yes</td></tr></tbody></table>

<figure><img src="image_15.png" /><figcaption>**Fig. 10.** Distance matrix between two time series where each point in the heat map corresponds to the squared difference between the associated time stamps. The optimal DTW path (in gray) summarizes the temporal distortion between the two time series. The connections (in green) between the warping path and the diagonal (in black), represent how far these two series are from having no temporal distortion between them. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)</figcaption></figure>

<figure><img src="image_16.png" /><figcaption>**Fig. 11.** Basis of the mathematical formulation of the WPD metric. For each point on the warping path, the associated triangle is always a right isosceles triangle, in the case of equal length series.</figcaption></figure>

## 4. Experimental analysis

To quantitatively analyze each metric's behavior during evaluation, we present an experimental setup using Conditional Variational Auto-Encoders (CVAE) to generate human motion sequences. The conditioning aspect of the VAE helps to control the generation of new

examples by specifying the class of the action to generate. A visual representation of the training/generating phases of a CVAE on human motion sequences can be seen in Fig. 12.

### 4.1. Backbone architectures

The CVAE model consists on an encoder-decoder architecture with a symmetrical view. In this experiment, we choose three well known backbone neural network architectures used in the literature: (1) Convolutional Neural Networks (CNNs), (2) Recurrent Neural Networks (RNNs), and (3) Transformer Networks.

In the case of CNNs, the encoder is made of convolution blocks that contain a one-dimensional convolution layer, a batch normalization layer and an activation layer. The decoder of the CNN based CVAE contains de-convolution blocks consisting of one-dimensional convolution transpose layers, a batch normalization layer followed by an activation layer. The number of convolution and de-convolution blocks is the same for both encoder and decoder, with the same number of filters. The kernel sizes are chosen in a symmetrical view, i.e., if the CVAE encodes with $k_1, k_2, ..., k_n$ kernel sizes for each convolution block, it decodes
<figure><img src="image_17.png" /><figcaption>Fig. 12. Training and generation phases of the CVAE in our experiments. In the training phase, both the Encoder and Decoder are trained at the same time. The Encoder extracts the features from the input sequences, followed by a projection into a Gaussian latent representation, while being conditioned on the input action label. The Decoder takes as input a random sample from the learned Gaussian space and decodes the original input sequence, while being conditioned on the input action label. In the generation phase, a random sample is taken from a pure Normal distribution, fed to the decoder to generate a new sequence, while being conditioned on the action label we want the generated skeleton sequence to perform.</figcaption></figure>

with $k_n, \dots, k_2, k_1$ kernel sizes for each de-convolution block. In the rest of this work, we refer to the CNN based CVAE as CConvVAE.

In the case of RNNs, the encoder is made of Gated Recurrent Units (GRU) layers on top of each other, and outputs a vector representation. The decoder is made of the same number of GRU layers as the encoder, with the same parameters. However the input of the decoder is repeated as much as the number of frames in the sequence to be fed to the first GRU layer. We refer in the rest of this work to the RNN based CVAE as CGRUVAE.

In the case of Transformer Networks, the encoder and decoder both consist of self-attention mechanisms, with the same number of layers for both and the same parameters. We refer in the rest of this work to the Transformer Network based CVAE as CTransVAE.

## 4.2. Dataset

To train the generative models, we utilize a publicly available action recognition dataset, HumanAct12. This dataset, made available in Guo et al. (2020), contains 3D Kinect skeleton motion sequences of subjects performing 12 actions. These actions are: *warm up*, *walk*, *run*, *jump*, *drink*, *lift dumbbell*, *sit*, *eat*, *turn steering wheel*, *call with phone*, *boxing* and *throw*. Each of these actions contains a variety of ways of doing the action, e.g., the action *warm up* contains *warm up pectoral* and *warm up elbow-back*. The dataset contains 1191 examples in total, some of which are illustrated in Fig. 13. Each example in this dataset is a sequence of skeletons, where each skeleton is made of 24 body joints with their corresponding position in the $x-y-z$ 3D space. Moreover, each sequence can be seen as a 3D tensor of shape $(T, J, 3)$, where $T$ is the length of the sequence, and $J$ is the number of joints. These sequences can also be represented as a Multivariate Time Series of shape $(T, 3, J)$, which is then used as input shape for the CVAE generative models. Given that deep learning models require equal length sequences, we re-sample all of the samples of the HumanAct12 dataset to the average length (75)

using the scipy.signal.resample (Virtanen et al., 2020) functionality. Prior to training, we normalize all sequences in the dataset using a min-max scalar on each of the $x-y-z$ dimensions independently.

It is important to note that for all the metrics used in this work, no prior train-test splits are required, instead all the dataset can be used.

## 4.3. Implementation details

The three CVAE variants used in this work are implemented using `tensorflow` (Abadi et al., 2015) and trained for 2000 epochs with a batch size of 32. During training, we use a learning rate decay, the ReduceLROnPlateau approach.

For the CConvVAE, we use three convolution blocks and three de-convolution blocks for the encoder-decoder architecture with 128 filters for each layer and kernel sizes of 40,20 and 10 respectively (symmetrical for the decoder).

For the CGRUVAE, we use two GRU layers in the encoder. The input vector of the decoder is repeated multiple times to form an input sequence, followed by two GRU layers. The hidden GRU state size is set to 128 in all of the GRU layers in both encoder and decoder.

For the CTransVAE, we use a convolution embedding at first followed by two layers of Multi-Head Attention in the encoder. The decoder consists of two Multi-Head Attention layers followed by a de-convolution layer. The number of filters in the convolution and de-convolution embedding layers is set to 128 with a filter size of 40. The number of Attention heads is set to 4 with a head size of 32.

The latent space dimension of all of the three CVAE variants is set to the same value of 16. In this study, we selected a latent space dimension of 16 for all three CVAE variants, as the focus of the work is not on optimizing generative models but on providing a framework for their evaluation. Although larger latent dimensions may capture more complex motions, many papers in the field do not include ablation studies on this value, indicating that it likely does not have a major
<figure><img src="image_18.png" /><figcaption>Fig. 13. Visualization of some samples of the HumanAct12 dataset for four different actions: walk, lift dumbbell, warm up and throw. The temporal axis is the horizontal axis where we present different frames of the examples.</figcaption></figure>

impact on the generation process. Investigating the effect of different latent space sizes could be considered for future research.

We use the aeon-toolkit (Middlehurst et al., 2024), a Python package for time series machine learning in order to compute the DTW path.

## 4.4. Training on different loss parameters

As this experimental work aims at showcasing that a slight change in the model's parameters results in a change in the interpretation of the evaluation metrics, we choose to experiment on the model's loss parameters. A CVAE model is trained to optimize a weighted sum of two losses, the first being is the reconstruction loss and the second being the Kullback-Leibler divergence (KL) loss. The reconstruction loss aims to maximize the log likelihood by minimizing the mean squared error between the input and the reconstructed output. The KL loss aims to regularize the latent space and prevents the learned Gaussian distribution from over estimating the space. The total loss to minimize during training is as follows:

$$ \mathcal{L} = \alpha \mathcal{L}_{\text{rec}} + \beta \mathcal{L}_{\text{KL}} \qquad (20) $$

where $\alpha$ and $\beta$ are scalar weights between 0 and 1 for each of the reconstruction and KL loss respectively. In the ideal case, it is preferable to maintain the following constraint:

$$ \alpha + \beta = 1, \qquad (21) $$

in order to have a convex sum of both losses. From previous VAE based work in the literature, it has been seen that in most of the cases, the model converges to a more optimal state when the KL loss is assigned a smaller weight compared to the reconstruction loss.

In this work, we avoid choosing a specific couple of $(\alpha, \beta)$ and perform the experiment on different values while maintaining the constraint in Eq. (21). The set of values we choose for the couple $(\alpha, \beta)$ are: $(1E^{-1}, 9E^{-1})$, $(5E^{-1}, 5E^{-1})$, $(9E^{-1}, 1E^{-1})$, $(9.9E^{-1}, 1E^{-2})$, $(9.99E^{-1}, 1E^{-3})$, $(9.999E^{-1}, 1E^{-4})$.

## 4.5. Class imbalanced generation setup

In this section, we propose a generation setup that respects the class imbalanced problem of the training set. This setup is not necessarily

specific to our experiment with CVAE, but with any generative model in a supervised scheme, such as human action recognition datasets. Given that all metrics consist in comparing real and generated distribution, we believe it is important to make the evaluation as fair as possible. When generating new samples from a generative model, it is essential to preserve the same label distribution as the one of the training set of the real samples. The reason for this constraint is to avoid bias in the generative model for a specific sub-group of classes. For instance, suppose a class-imbalanced dataset used to train a generative model, there is a possibility that this model is only able to generate the most populated sub-class even when being told to generate the minority. In the case where higher number of generated samples is needed compared to the number of real samples, then we simply choose a factor to control the quantity, while preserving the label distribution.

## 4.6. Results and analysis

In this work, we present the results on the form of a radar chart, also known as a spider chart, given the nature of our metrics having different range of values. The usage of this kind of visualization is more feasible than simply mentioning that one model has a 0.003 value better on a given metric for example.

For each couple of $(\alpha, \beta)$, we present one radar chart on a polygon axes, containing each four polygons: one for each of three variants of CVAE and one for the metrics computed on the real samples. Prior to visualization, all metrics are first normalized between 0 and 1 independently from each other using a min max scalar. Then, all metrics are transformed with the following formula:

$$ \text{metric}_{\text{gen}} = \begin{cases} 1 - (\text{metric}_{\text{real}} - \text{metric}_{\text{gen}}) & \text{if } \text{metric}_{\text{gen}} > \text{metric}_{\text{real}} \\ 1 + (\text{metric}_{\text{gen}} - \text{metric}_{\text{real}}) & \text{if } \text{metric}_{\text{gen}} \le \text{metric}_{\text{real}} \end{cases} \qquad (22) $$

where *metric* is in the set of all metrics expected for *FID*. In the case of *FID*, the formula becomes:

$$ \text{metric}_{\text{gen}} = \begin{cases} 1 - (\text{metric}_{\text{gen}} - \text{metric}_{\text{real}}) & \text{if } \text{metric}_{\text{gen}} > \text{metric}_{\text{real}} \\ 1 + (\text{metric}_{\text{gen}} - \text{metric}_{\text{real}}) & \text{if } \text{metric}_{\text{gen}} \le \text{metric}_{\text{real}} \end{cases} \qquad (23) $$

This transformation facilitates the reading and understanding of the radar chart. It is based on the fact that all metrics calculated on the
<figure><img src="image_19.png" /><figcaption>**Fig. 14.** Radar charts comparing the performance of the three CVAE variants over the eight used metrics. For each sub-figure from a to d, we present the results of one parameter set for the three CVAE variants, by changing the values of $\alpha$ and $\beta$. Each radar chart contains 4 polygons, where three of these polygons are associated to each of the three CVAE variants, and the fourth polygon is for the metric values over the real set of data. For all metrics except for the *FID*, if the summit of polygon1 is higher than the summit of polygon2, than the metric value in reality of model1 is higher than model2. In the case of the *FID*, the opposite holds.</figcaption></figure>

generative set, should be close to the metric calculated on the real set, as explained before. The exception of the *FID* metric is because of its nature being the only metric we are interested in decreasing it instead of increasing it.

In Fig. 14, we present the radar charts for all pairs of $(\alpha, \beta)$ for the three variant CVAE and the real set. For each metric, a summit of a generative model's polygon being lower than the summit of the real polygon means that $metric_{gen} < metric_{real}$, except for the *FID* metric it signifies that $FID_{gen} > FID_{real}$. Unlike typical radar charts, where higher values indicate better performance, here the interpretation depends on the specific metric. Each metric reflects a different aspect of the quality of the generated motions, which may not uniformly define “better” across all metrics. To address this complexity, the radar charts are accompanied by detailed explanations in the text to guide the reader in understanding how each metric corresponds to the differences between the generated and real data.

In the case where an application is interested in being most optimal on all metrics, then the generative model's polygon should be most identical to the real polygon, except for the *MMS* metric summit. It is important to recall that for the *MMS* metric, *MMS<sub>gen</sub>* should be higher to a certain limit than *MMS<sub>real</sub>*, which is interpreted visually by the summit of the generative model's polygon being higher than the summit of the real polygon at the *MMS*.

Fig. 14 highlights that it may be difficult to find a generative model winning on all metrics simultaneously, as a change in the backbone architecture or in the loss parameters may result in a significant change in the value of some metrics. According to our experimental evaluation on the HumanAct12 dataset, the only case scenario where it may be possible to choose a best model, CConvVAE, is for a very specific set of values for the pair $(\alpha, \beta)$, corresponding to Fig. 14-d. However, this kind

of parameter search is not always feasible. This is why finding the best model on all possible metrics can be a difficult task. Instead, depending on the target application, we should expect to search for a best model on some specific set of, or even only one, metric.

In what follows, we present an interpretation of the results by comparing the three CVAE on each metric individually. For each metric, we detail the interpretation of a model being best on this metric.

### 4.6.1. FID

For some pairs of $(\alpha, \beta)$, it can be seen in Fig. 14 that CConvVAE has the smallest *FID* value that is still higher but close to the *FID<sub>real</sub>*. This means that the generated samples produced by CConvVAE are more reliable in terms of fidelity compared to CGRUVAE and CTransVAE. The CConvVAE model is able to learn a distribution $P_G$ that requires less energy to transform into $P_R$ compared to the distributions $P_G$ learned by CGRUVAE and CTransVAE. However in some cases, even being the variant with the smallest and closest value, the gap is large (Figs. 14-a-b-c).

### 4.6.2. Density

Similar to the interpretation of the *FID* metric, for some pairs of $(\alpha, \beta)$, the CConvVAE is the generative model with a *Density* value closer to the *Density<sub>real</sub>* compared to the two other CVAE variants. This implies that the likelihood of samples, generated by $P_G$, being generated by $P_R$ in the case of CConvVAE, is higher compared to that in the cases of CGRUVAE and CTransVAE. However, similar to the *FID* metric, even if in some cases the CConvVAE has the closest value to *Density<sub>real</sub>* a significant gap can be observed between *Density<sub>gen</sub>* and *Density<sub>real</sub>*.
<figure><img src="image_20.png" /><figcaption>Fig. 15. Generated sequences using the CConvVAE model, demonstrating successful adherence to the “walking” action constraint in all cases.</figcaption></figure>

### 4.6.3. AOG

The AOG metric is essential to assess if the generative model's conditional aspect is effective or not. This conditional mechanism may not be always efficient, while training generative models because of lack of data, underfitting or hyper-parameters settings. For instance, it can be seen from Fig. 14, that even though CConvVAE is the winning model on this metric for all parameters settings, it would not mean that the conditional mechanism is not failing. This can be seen in Fig. 14-f where the AOG value in the case of CConvVAE is far from the AOGreal, which means the conditional mechanism is failing in this hyper-parameter settings. In the case of CTransVAE and CGRUVAE the mechanism seems to fail for any hyper-parameter setting, which means the generative model is not able to learn how to control the sub-class of each sample being used to learn the model's hyper-parameters. We illustrate some generated exemplars using both CConvVAE and CGRUVAE in their best hyperparameter setups for the walking action in Figs. 15 and 16. In Fig. 15, the five randomly generated sequences show that the CConvVAE model consistently adheres to the walking action constraint, aligning with its high 91.14% score on the AOG metric. In contrast, Fig. 16 demonstrates that CGRUVAE struggles to maintain the constraint, as not all generated sequences depict the intended walking action, reflected in its lower 21.83% score on the AOG metric. This comparison illustrates the correlation between the metric values and the model's generative behavior. The metric allows us to quickly assess the model's performance across the dataset, without the need to visually inspect numerous random sequences to detect failures.

For all of the three above fidelity based metrics, the CConvVAE is considered most reliable compared to other CVAE variants.

### 4.6.4. APD

On the one hand, in terms of the APD diversity metric, the CConvVAE triumphs over this metric. On the one hand, the CConvVAE seems more diverse according to the APD metric by having the closest value to the APDreal on all hyper-parameters settings with an almost ideal APD value for Fig. 14-d. On the other hand, the CGRUVAE performs almost as good as the CConvVAE on the APD metric in some of the

parameters settings. This showcases that even though CConvVAE can be significantly better than CGRUVAE on one fidelity metric such as FID, it would not mean that CGRUVAE cannot perform well on other metrics.

### 4.6.5. ACPD

The ACPD metric evaluates the space diversity per sub-class of $P_R$ and $P_G$ on contrary to the APD metric, which evaluates space diversity in general over all classes. From Fig. 14 it can be seen that in almost half the cases of hyper-parameters settings, all three CVAE variants perform better on the ACPD metric compared to the real data (ACPDreal). This indicates that the generated space is more diverse per sub-class compared to the real space. This last interpretation is not entirely correct, as this phenomena where ACPDgen > ACPDreal is mostly a result of overfitting, instability and class imbalanced issue during training of the generative model. In addition, winning on one metric does not signifies winning on all metrics, for instance even though CGRUVAE outperforms CConvVAE in Fig. 14-a in terms of ACPD, it would not directly conclude that CGRUVAE is more diverse per sub-class than CConvVAE. This is due to the fact that on the same hyper-parameter settings, the FID value of CGRUVAE is much higher than FIDreal resulting in almost random generation, for which the ACPD values are not reliable. This highlights the importance of using different evaluation metrics.

### 4.6.6. Coverage

For most hyper-parameter settings, the CConvVAE outperforms other CVAE variants in terms of coverage. This signifies that in the case of CConvVAE more samples generated by $P_R$ can also be generated by the learned distribution $P_G$ compared to the case of CGRUVAE and CTransVAE. However, in the cases of Figs. 14-a and b, the CConvVAE does not outperform other variants on the coverage metric, instead CTransVAE wins on the coverage metric. This means, even though for the same hyper-parameters CConvVAE performs well in terms of APD diversity, it performs very badly in terms of coverage.

This raises the question: what is the difference between APD and coverage if they both quantify diversity? While both metrics rely on a
<figure><img src="image_21.png" /><figcaption>**Fig. 16.** Generated sequences using the CGRUVAE model, where the “walking” action constraint is not consistently respected, with some sequences failing to depict the intended action.</figcaption></figure>

latent space view with distance based measure (Euclidean distance), they differ in terms of quantification. On the one hand the APD metric relies on distance value between a random selection of pairs, quantifying the volume of space taken by $P_R$ and $P_G$. On the other hand, the coverage metric relies on the nearest neighbor portions between real and generated samples, quantifying how much volume do the generated samples take in $P_R$.

### 4.6.7. MMS

The MMS metric evaluates the diversity in terms of novelty of the generated samples, where the expected value of $MMS_{gen}$ should be higher but still close to $MMS_{real}$. This makes the interpretation of this metric a bit more difficult to the other metrics. In the case of CConvVAE, the MMS values seem to be the most stable between all three variants, as it is higher but still close to $MMS_{real}$, on contrary to CTransVAE, where MMS takes over the limit of the radar plot. This suggests that the CConvVAE model is able to generate more novel human motion sequences.

### 4.6.8. WPD

The WPD metric evaluates the diversity from a temporal perspective, as it evaluates how much warping and temporal distortions do exist between samples. For the interpretation of WPD, three cases do exist:

* Case 1: $|WPD_{gen} - WPD_{real}| < \epsilon$ for $\epsilon \ll 1$, which indicates a perfect re-creation of all temporal distortions in $P_G$ provided by $P_R$.

* Case 2: $WPD_{gen} \gg WPD_{real}$, which indicates the generative model found that some temporal distortions exist in $P_R$ and learned how to create similar but new ones.

* Case 3: $WPD_{gen} \ll WPD_{real}$, which is an indication of the model not being able to re-create temporal distortions and is rather always generating the same type of temporal distortions.

From Fig. 14, it can be seen that in all cases, except for CTransVAE in Fig. 14-d, all WPD values are smaller than $WPD_{real}$. This indicates

that in most cases the models can re-create some of the temporal distortions but not all of them. However, in one of the parameters settings, Fig. 14-d, the gap between $WPD_{real}$ and WPD for CConvVAE and CGRUVAE is very small, which indicates a perfect re-creation of the temporal distortions in $P_R$.

It is worth noting that in most cases presented in Fig. 14, when a model performs perfectly on the APD metric, the WPD metric does not exhibit the same behavior and instead shows an error margin. This discrepancy underscores the importance of using a warping based metric such as the proposed WPD, which operates directly on the raw data. In contrast, the APD metric is computed in a latent feature space that depends on the feature extractor, which may omit critical temporal information.

### 4.7. Choosing the right metric

In overall, our findings showcase that while the CConvVae model seems to be more appropriate when trained on the HumanAct12 dataset, some small changes in hyper-parameters can significantly affect its performance on most of the metrics. This suggests that it is not always easy to find the best model to win over all metrics, and we believe that it should always depend on the target application. For instance, if the workflow of generating new human motion sequences is dedicated for a gaming setup, fidelity metrics such as FID are useful but not as important as diversity metrics. In this scenario, generated samples are usually foreseen to have a very high number of possibilities of doing an action depending on the gaming environment of a user. Moreover, if the generation task is in the aspect of medical research such as generating rehabilitation movements for patients, than fidelity plays a huge role in this case as we want to show the patient an exact replica of how a human patient should do the exercise.

### 4.8. Sensitivity to feature extractor

In addition to the initial analysis using a supervised latent space for action classification, we extended our evaluation by incorporating
<figure><img src="image_22.png" /><figcaption>Fig. 17. Radar charts comparing the performance of the three CVAE variants over the eight used metrics using a self-supervised auto-encoding scheme instead of a supervised classification scheme. For each sub-figure from a to f, we present the results of one parameter set for the three CVAE variants, by changing the values of a and β.</figcaption></figure>

a self-supervised learning approach. Specifically, we developed a trans-
posed decoder and trained an auto-encoder on the same dataset using
the original encoding architecture (GRU based classifier used in Guo
et al., 2020). This allowed us to explore how the latent space derived
from unsupervised learning affects the performance of the generative
models. By recalculating the proposed metrics on the self-supervised
latent space, we observed that the results further emphasized the
strengths of the GRU-based CVAE. The new radar charts in Fig. 17 re-
flect a more pronounced “goodness” of the GRU-based CVAE compared
to the previous supervised version. This demonstrates the versatility
and general applicability of our evaluation framework across different
types of latent space models, whether supervised or self-supervised.

Unlike the previous results using a supervised feature extractor,
where the GRU-based CVAE did not consistently perform well across
all metrics compared to convolution-based models, the shift to a self-
supervised feature extraction method provided new insights. With
this change, the GRU-based CVAE now performs comparably to the
convolution-based models, as illustrated in Figs. 17 d and e. This
improvement underscores the flexibility of our evaluation framework,
which is independent of the feature extraction method used, making it
easy to adapt to different techniques. Furthermore, this analysis high-
lights the importance of not relying solely on one metric for evaluating
generative models, as many metrics appear to be sensitive to the feature
extraction method. This variability demonstrates that the evaluation of
generative models remains an open and challenging problem, with no
single solution fitting all cases. However, by providing a robust and
adaptable framework, we aim to contribute to tackling this complexity
in a meaningful way.

4.9. Application on human rehabilitation

The field of human rehabilitation stands to benefit significantly
from advancements in human motion generation. Generative models

that accurately simulate human movements can play a crucial role
in developing personalized rehabilitation programs, enhancing patient
outcomes, and improving the overall efficiency of therapeutic inter-
ventions. By generating realistic and diverse motion sequences, these
models provide detailed, actionable insights for physical therapists and
patients alike. For instance, they can create customized exercise rou-
tines tailored to an individual's specific needs and capabilities, ensuring
that rehabilitation exercises are both effective and safe.

For these reasons, we selected rehabilitation as a key application
to evaluate using our proposed framework. It is important to note
that, for this application, we usually aim to ensure that the generative
model does not exceed the real distribution's diversity. This is essential
both for feature space metrics and for our proposed warping based
metric. The constraint arises from the fact that most rehabilitation
exercises have a highly specific and constrained range of acceptable
motion patterns, resulting often in only one correct way to perform a
rehabilitation exercise effectively and safely.

In our experiments, we utilized the Kimore dataset (Capecci et al., 2019), which consists of five distinct rehabilitation exercises: (1) lifting of the arms, (2) lateral tilt of the trunk with the arms extended, (3) trunk rotation, (4) pelvis rotation on the transverse plane, and (5) squatting. Each sequence in the dataset contains three dimensional positions of 18 total body joints. To ensure consistency, all samples were resampled to have the same length, following the approach used in prior experiments on the HumanAct12 dataset. The exercises in the dataset were used as labels to control the generation of motion samples. The same labeling strategy was also employed to train the classifier feature extractor, which is necessary for metric calculation. Based on findings from previous experimentation and insights from the literature, it is clear that placing a higher weight on the reconstruction loss compared to the KL loss generally yields better results. Consequently, in this application setup, we conducted experiments with
<figure><img src="image_23.png" /><figcaption>Fig. 18. Radar charts comparing the performance of two CVAE variants over the eight used metrics on the Kimore dataset for the application of human rehabilitation. For each sub-figure from a to d, we present the results of one parameter set for the two CVAE variants by changing the values of α and β.</figcaption></figure>

four selected hyperparameter loss configurations for the CVAE variants. Additionally, given the challenges of data collection in rehabilitation contexts, which are constrained by both time and the need for expert supervision, we did not include results for the Transformer based CVAE variant. This decision was informed by the high data requirements of self-attention-based models, which are impractical for rehabilitation datasets of limited size. Instead, we focused on models that are more robust to smaller datasets, aligning with the realistic constraints of rehabilitation research and practice.

We present, similar to the previous experiments, the radar charts in Fig. 18 to compare both the convolution based and GRU based CVAE variants across eight metrics. From these figures, it can be observed that determining which model is “best” across all metrics remains challenging. For instance, even under the hyperparameter setup of Fig. 18-d, the GRU based model outperforms the convolution based model on the FID metric but falls short on the density metric, even though both metrics assess fidelity. Furthermore, neither model replicates the warping based diversity (WPD) observed in the real distribution. Instead, the convolution based model tends to generate less diversity and is prone to repeating the same temporal distortions across sequences, while the GRU based model tends to introduce new and varied distortions. This behavior can be attributed to the fact that the models were not trained with constraints to respect the boundaries of temporal distortions. In such cases, the convolution based model might be considered more favorable, as generating less diversity suggests that it adheres more closely to the original distribution, avoiding hallucinated changes to the exercise limits. For these reasons, we emphasize that the use of multiple metrics is essential and should be tailored to the specific application. The interpretation of these metrics will vary depending on the goals we aim to achieve.

It is crucial to note that the APD metric does not align with the WPD metric, underscoring a key difference in their behavior. As discussed in Section 3, the feature extractor used in the calculation of the APD metric removes temporal information, leading to a loss of critical details in the raw time series data. In contrast, the proposed WPD metric is specifically designed to identify temporal distortions in the raw data. This makes the WPD metric uniquely suited for capturing such distortions, which the APD metric cannot detect due to its reliance on features that omit temporal dynamics.

## 4.10. Reproducibility

In order to favor the reproducibility of our work, we publicly release our code, available at https://github.com/MSD-IRIMAS/Evaluating-HMG. We believe this repository is easily manageable and user-friendly. It allows a user to evaluate an existing or a new generative model on all the considered metrics using the same classifier pre-trained on the HumanAct12 dataset. The code provides different options from which a user can choose the metrics to compute as well choosing the set of parameters of each metric.

## 5. Conclusions

In this work, we presented a comprehensive review on evaluation metrics used to quantify reliability of generative models for the task of human motion generation. By leveraging the fact that human motion data are temporal data represented as multivariate time series, we also proposed a novel metric to assess diversity in terms of temporal distortion. Given the set of all the considered metrics, we proposed a unified evaluation setup that enables a fair comparison between different models. Through some experiments with three variants of generative models and a publicly available dataset, our experiments highlighted that it is not straightforward to find The One Metric To Rule Them All. Instead, our findings showcased that a set of different metrics is often more appropriate to conclude how reliable a model is compared to other models. Our work is supported by a user-friendly publicly available code that can be used to calculate all of the used metrics on any generative model with any possible parametrization. Although the current experiments focus on motions associated with discrete action labels, the framework can be extended to evaluate motions generated from other modalities, including those derived from natural language descriptions. While our framework is capable of assessing the quality of generated motion sequences, future work could explore methods to evaluate the alignment between generated motions and corresponding textual descriptions.

Finally, the proposed evaluation framework can be extended to test recent generative models such as Diffusion and auto-regressive transformers, as well as applied to additional datasets. This would provide further validation of the metrics and demonstrate their robustness across a wider range of models and data, enhancing the generalization and applicability of the framework.

We believe this work can be a good starting point for new comers into the field of human motion generation and can give a clear definition to a unified evaluation setup. However, we do not claim that the presented metrics in this paper are the only one available, as this research field is exponentially expanding. It is also important to note that most metrics suppose availability of labels in the real data, but this is not always the case and metrics in this case should be adapted.

### CRediT authorship contribution statement

**Ali Ismail-Fawaz:** Writing – review & editing, Writing – original draft, Visualization, Validation, Supervision, Software, Methodology, Investigation, Formal analysis, Conceptualization. **Maxime Devanne:** Writing – review & editing, Visualization, Validation, Supervision, Methodology, Investigation, Funding acquisition, Formal analysis, Conceptualization. **Stefano Berretti:** Writing – review & editing, Visualization, Validation, Supervision, Methodology, Investigation, Funding acquisition, Formal analysis, Conceptualization. **Jonathan Weber:** Writing – review & editing, Visualization, Validation, Supervision,
Methodology, Investigation, Funding acquisition, Formal analysis, Conceptualization. **Germain Forestier:** Writing – review & editing, Visualization, Validation, Supervision, Methodology, Investigation, Funding acquisition, Formal analysis, Conceptualization.

## Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

## Acknowledgments

This work was supported by the ANR DELEGATION project (grant ANR-21-CE23-0014) of the French Agence Nationale de la Recherche. The authors would like to acknowledge the High Performance Computing Center of the University of Strasbourg for supporting this work by providing scientific support and access to computing resources. Part of the computing resources were funded by the Equipex Equip@Meso project (Programme Investissements d'Avenir) and the CPER Alsacalcul/Big Data. The authors would also like to thank the creators and providers of the HumanAct12 and the Kimore datasets.

## Data availability

the data is publicly available and we cite the paper of the dataset in our manuscript for which they provide a link to download the dataset.

## References

Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G.S., Davis, A., Dean, J., Devin, M., Ghemawat, S., Goodfellow, I., Harp, A., Irving, G., Isard, M., Jia, Y., Jozefowicz, R., Kaiser, L., Kudlur, M., Levenberg, J., Mané, D., Monga, R., Moore, S., Murray, D., Olah, C., Schuster, M., Shlens, J., Steiner, B., Sutskever, I., Talwar, K., Tucker, P., Vanhoucke, V., Vasudevan, V., Viégas, F., Vinyals, O., Warden, P., Wattenberg, M., Wicke, M., Yu, Y., Zheng, X., 2015. TensorFlow: Large-scale machine learning on heterogeneous systems. URL: https://www.tensorflow.org/. Software available from tensorflow.org.

Aliakbarian, S., Saleh, F., Petersson, L., Gould, S., Salzmann, M., 2021. Contextually plausible and diverse 3d human motion prediction. In: IEEE/CVF Int. Conf. on Computer Vision. ICCV, pp. 11333–11342.

Asteriadis, S., Chatzitofis, A., Zarpalas, D., Alexiadis, D.S., Daras, P., 2013. Estimating human motion from multiple kinect sensors. In: Int. Conf. on Computer Vision/Computer Graphics Collaboration Techniques and Applications. pp. 1–6.

Bagnall, A., Dau, H.A., Lines, J., Flynn, M., Large, J., Bostrom, A., Southam, P., Keogh, E., 2018. The UEA multivariate time series classification archive. 2018. arXiv preprint arXiv:1811.00075.

Bansal, A., Borgnia, E., Chu, H.M., Li, J., Kazemi, H., Huang, F., Goldblum, M., Geiping, J., Goldstein, T., 2024. Cold diffusion: Inverting arbitrary image transforms without noise. Adv. Neural Inf. Process. Syst. 36.

Capecchi, M., Ceravolo, M.G., Ferracuti, F., Iarlori, S., Monteriu, A., Romeo, L., Verdini, F., 2019. The kimore dataset: Kinematic assessment of movement and clinical scores for remote monitoring of physical rehabilitation. IEEE Trans. Neural Syst. Rehabil. Eng. 27 (7), 1436–1448.

Cervantes, P., Sekikawa, Y., Sato, I., Shinoda, K., 2022. Implicit neural representations for variable length human motion generation. In: European Conference on Computer Vision. Springer, pp. 356–372.

Dau, H.A., Bagnall, A., Kamgar, K., Yeh, C.C.M., Zhu, Y., Gharghabi, S., Ratanamahatana, C.A., Keogh, E., 2019. The UCR time series archive. IEEE/ CAA J. Autom. Sin. 6 (6), 1293–1305.

Dowson, D., Landau, B., 1982. The Fréchet distance between multivariate normal distributions. J. Multivariate Anal. 12 (3), 450–455.

Fréchet, M., 1906. Sur quelques points du calcul fonctionnel.

Fréchet, M., 1957. Sur la distance de deux lois de probabilité. In: Annales de l'ISUP, vol. 6, (3), pp. 183–198.

Guo, C., Zuo, X., Wang, S., Zou, S., Sun, Q., Deng, A., Gong, M., Cheng, L., 2020. Action2motion: Conditioned generation of 3d human motions. In: Proceedings of the 28th ACM International Conference on Multimedia. pp. 2021–2029.

Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S., 2017. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Adv. Neural Inf. Process. Syst. 30.

Ho, J., Jain, A., Abbeel, P., 2020. Denoising diffusion probabilistic models. Adv. Neural Inf. Process. Syst. 33, 6840–6851.

Ismail-Fawaz, A., Devanne, M., Berretti, S., Weber, J., Forestier, G., 2023a. LITE: Light Inception with booSTing tTechniques for Time Series Classification. In: International Conference on Data Science and Advanced Analytics. DSAA, Thessaloniki, pp. 1–8. http://dx.doi.org/10.1109/DSAA60987.2023.10302569.

Ismail-Fawaz, A., Devanne, M., Berretti, S., Weber, J., Forestier, G., 2024a. Finding foundation models for time series classification with a pretext task. In: PAKDD International Workshop on Temporal Analytics. Taipei, pp. 1–8. http://dx.doi.org/10.1007/978-981-97-2650-9_10.

Ismail-Fawaz, A., Devanne, M., Berretti, S., Weber, J., Forestier, G., 2024b. A supervised variational auto-encoder for human motion generation using convolutional neural networks. In: 4th International Conference on Pattern Recognition and Artificial Intelligence. ICPRAI, Jeju Island, pp. 1–8. http://dx.doi.org/10.1007/978-981-97-8705-0_11.

Ismail-Fawaz, A., Devanne, M., Weber, J., Forestier, G., 2022. Deep learning for time series classification using new hand-crafted convolution filters. In: 2022 IEEE International Conference on Big Data. IEEE BigData 2022, Osaka, pp. 972–981. http://dx.doi.org/10.1109/BigData55660.2022.10020496.

Ismail-Fawaz, A., Devanne, M., Weber, J., Forestier, G., 2023b. Enhancing time series classification with self-supervised learning. In: 15th International Conference on Agents and Artificial Intelligence: ICAART 2023. Lisbon, pp. 1–8. http://dx.doi.org/10.5220/0011611300003393.

Ismail-Fawaz, A., Ismail Fawaz, H., Petitjean, F., Devanne, M., Weber, J., Berretti, S., Webb, G.I., Forestier, G., 2023c. ShapeDBA: Generating effective time series prototypes using ShapeDTW barycenter averaging. In: ECML/PKDD Workshop on Advanced Analytics and Learning on Temporal Data. Turin, pp. 1–8, URL: https://germain-forestier.info/publis/aaltd2023.pdf.

Karunratanakul, K., Preechakul, K., Suwajanakorn, S., Tang, S., 2023. Guided motion diffusion for controllable human motion synthesis. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 2151–2162.

Kingma, D.P., Welling, M., 2013. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114.

Kucherenko*, T., Wolfert*, P., Yoon*, Y., Viegas, C., Nikolov, T., Tsakov, M., Hen-ter, G.E., 2024. Evaluating gesture generation in a large-scale open challenge: The GENEA challenge 2022. ACM Trans. Graph. 43 (3), 1–28.

Kynkäänniemi, T., Karras, T., Laine, S., Lehtinen, J., Aila, T., 2019. Improved precision and recall metric for assessing generative models. Adv. Neural Inf. Process. Syst. 32.

Lee, D., Malacarne, S., Aune, E., 2023. Vector quantized time series generation with a bidirectional prior model. In: International Conference on Artificial Intelligence and Statistics. PMLR, pp. 7665–7693.

Lucas, T., Baradel, F., Weinzaepfel, P., Rogez, G., 2022. PoseGPT: quantization-based 3D human motion generation and forecasting. In: European Conf. on Computer Vision. ECCV, Springer, pp. 417–435.

Lun, R., Zhao, W., 2015. A survey of applications and human motion recognition with microsoft kinect. Int. J. Pattern Recognit. Artif. Intell. 29 (05), 1555008.

Mahmood, N., Ghorbani, N., Troje, N.F., Pons-Moll, G., Black, M.J., 2019. AMASS: Archive of motion capture as surface shapes. In: IEEE/CVF Int. Conf. on Computer Vision. ICCV, pp. 5442–5451.

Messina, N., Sedmidubsky, J., Falchi, F., Rebok, T., 2023. Text-to-motion retrieval: Towards joint understanding of human motion data and natural language. In: Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval. pp. 2420–2425.

Middlehurst, M., Ismail-Fawaz, A., Guillaume, A., Holder, C., Rubio, D.G., Bulatova, G., Tsaprounis, L., Mentel, L., Walter, M., Schäfer, P., et al., 2024. aeon: a Python toolkit for learning from time series. arXiv preprint arXiv:2406.14231.

Müller, K., 2007. Dynamic time warping. Inf. Retr. Music. Motion 69–84.

Naeem, M.F., Oh, S.J., Uh, Y., Choi, Y., Yoo, J., 2020. Reliable fidelity and diversity metrics for generative models. In: International Conference on Machine Learning. PMLR, pp. 7176–7185.

Petrovich, M., Black, M.J., Varol, G., 2021. Action-conditioned 3d human motion synthesis with transformervae. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 10985–10995.

Radford, A., Narasimhan, K., Salimans, T., Sutskever, I., et al., 2018. Improving language understanding by generative pre-training.

Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B., 2022. High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 10684–10695.

Sajjadi, M.S., Bachem, O., Lucic, M., Bousquet, O., Gelly, S., 2018. Assessing generative models via precision and recall. Adv. Neural Inf. Process. Syst. 31.

Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., Chen, X., 2016. Improved techniques for training gans. Adv. Neural Inf. Process. Syst. 29.

Sohn, K., Lee, H., Yan, X., 2015. Learning structured output representation using deep conditional generative models. Adv. Neural Inf. Process. Syst. 28.

Streijl, R.C., Winkler, S., Hands, D.S., 2016. Mean opinion score (MOS) revisited: methods and applications, limitations and alternatives. Multimedia Syst. 22 (2), 213–227.

Van Den Oord, A., Vinyals, O., et al., 2017. Neural discrete representation learning. In: Advances in Neural Information Processing Systems (Neurips), vol. 30.

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I., 2017. Attention is all you need. Adv. Neural Inf. Process. Syst. 30.
Villa-Monedero, M., Gil-Martín, M., Sáez-Trigueros, D., Pomirski, A., San-Segundo, R., 2023. Sign language dataset for automatic motion generation. J. Imaging 9 (12), 262.

Virtanen, P., Gommers, R., Oliphant, T.E., Haberland, M., Reddy, T., Cournapeau, D., Burovski, E., Peterson, P., Weckesser, W., Bright, J., van der Walt, S.J., Brett, M., Wilson, J., Millman, K.J., Mayorov, N., Nelson, A.R.J., Jones, E., Kern, R., Larson, E., Carey, C.J., Polat, İ., Feng, Y., Moore, E.W., VanderPlas, J., Laxalde, D., Perktold, J., Cimrman, R., Henriksen, I., Quintero, E.A., Harris, C.R., Archibald, A.M., Ribeiro, A.H., Pedregosa, F., van Mulbregt, P., SciPy 1.0 Contributors, 2020. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods 17, 261–272. http://dx.doi.org/10.1038/s41592-019-0686-2.

Xue, K., Seo, H., 2024. Shape conditioned human motion generation with diffusion model. arXiv preprint arXiv:2405.06778.

Yoon, Y., Cha B., Lee, J.H., Jang, M., Lee, J., Kim, J., Lee, G., 2020. Speech gesture generation from the trimodal context of text, audio, and speaker identity. ACM Trans. Graph. 39 (6), 1–16.

Yuan, Y., Song, J., Iqbal, U., Vahdat, A., Kautz, J., 2023. Physdiff: Physics-guided human motion diffusion model. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 16010–16021.

Zhang, M., Cai, Z., Pan, L., Hong, F., Guo, X., Yang, L., Liu, Z., 2024. Motiondiffuse: Text-driven human motion generation with diffusion model. IEEE Trans. Pattern Anal. Mach. Intell.

Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O., 2018. The unreasonable effectiveness of deep features as a perceptual metric. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 586–595.

Zhong, C., Hu, L., Zhang, Z., Xia, S., 2022. Learning uncoupled-modulation CVAE for 3D action-conditioned human motion synthesis. In: European Conference on Computer Vision. ECCV, Springer, pp. 716–732.