Beyond FVD: An Enhanced Evaluation Metrics for Video Generation Distribution Quality
Authors: Ge Ya Luo
Gian Mario Favero
ZhiHao Luo
Alexia Jolicoeur-Martineau
Christopher Pal
Venue: ICLR 2025 Poster (2025)
Keywords: Video quality metrics
Frechet Video Distance
Inflated 3D Convnet
VideoMAE
VJEPA
kernel metrics
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: The Fréchet Video Distance (FVD) is a widely adopted metric for evaluating video generation distribution quality. However, its effectiveness relies on critical assumptions. Our analysis reveals three significant limitations: (1) the non-Gaussianity of the Inflated 3D Convnet (I3D) feature space; (2) the insensitivity of I3D features to temporal distortions; (3) the impractical sample sizes required for reliable estimation. These findings undermine FVD's reliability and show that FVD falls short as a standalone metric for video generation evaluation. After extensive analysis of a wide range of metrics and backbone architectures, we propose JEDi, the JEPA Embedding Distance, 
based on features derived from a Joint Embedding Predictive Architecture, measured using Maximum Mean Discrepancy with polynomial kernel.  Our experiments on multiple open-source datasets show clear evidence that it is a superior alternative to the widely used FVD metric, requiring only 16% of the samples to reach its steady value, while increasing alignment with human evaluation by 34%, on average.

Project page: https://oooolga.github.io/JEDi.github.io/.

Published as a conference paper at ICLR 2025
BEYOND FVD: AN ENHANCED METRIC FOR EVALUAT- ING VIDEO GENERATION DISTRIBUTION QUALITY
Ge Ya Luo", Gian Mario Favero', Zhi Hao Luo
Alexia Jolicoeur-Martineuu Samsung SAIT Lab, Montreal
Mila Quebec Artificial Intelligence Institute
Christopher Pal
Mila Quebec Artificial Intelligence Institute Canada CIFAR AI Chair
ABSTRACT
The Fréchet Video Distance (FVD) is a widely adopted metric for evaluating video generation distribution quality. However, its effectiveness relies on critical assump- tions. Our analysis reveals three significant limitations: (1) the non-Gaussianity of the Inflated 3D ConvNet (I3D) feature space; (2) the insensitivity
I3D features temporal distortions; (3) the impractical sample sizes required for reliable esti- mation. These findings undermine FVD's s reliability and show that FVD falls short standalone metric for video generation evaluation. After extensive analysis of a wide range of metrics and backbone architectures, we propose JEDi, the JEPA Embedding Distance, based on features derived from Joint Embedding Predic- tive Architecture, measured using Maximum Mean Discrepancy with polynomial kernel. Our experiments on multiple open-source datasets show clear evidence that it is a superior alternative to the widely used FVD metric, requiring only 16% of the samples to reach steady value, while increasing alignment with human evaluation by 34%, on average.
Project page: https: petpss//ooe /tt:::///ooolll// gww.oouuuggggigubb.oo.... www.ub.t............ wt/.eei........... .gi trithub... 1o/
INTRODUCTION
Festure Space
V-JEMess
VideoMMEssa
Costre Siniturky (90) with human Evaluation
JEDi
Energy
FD
60
MMDrauy
Energy
FVD
Energy
KVD
2000
4000
5000
Samples Converge
Figure Comparing the number of samples that Fréchet Distance (FD), Energy, and MMDPOLY need to converge against its alignment with human evaluation on the UCF-101 dataset. JEDi, the feature space V-JEPA model (V-JEPAss-2) combination with a Maximum Mean Discrepancy (MMD) metric, is vastly more efficient framework for evaluating distributions of generated vidcos than conventional methods. The current standard, FVD (FD+13D), underperforms in terms both sample efficiency and alignment with human evaluation.

---

Published as a conference paper at ICLR 2025
Video generation research has experienced significant surge recently, vielding cutting- edge models that produce high-quality videos (Liu et al. 20224] Blattmann et al. 2023 Zeng et al. 2023 He ct al.
2023). However, evaluating their generation quality poses substantial challenge.
A good video generation model must produce high-quality frames, maintain temporal consistency, and generate varied content. For instance, at the object level, it is undesirable to have exclusively cars of specific brands or colors when generating automobile videos; at the motion level, it is undesirable to observe the same type of motion repeatedly when generating human action videos. Therefore, robust evaluation metric must address multiple facets to reliably assess generative video models.
Researchers have created a range of evaluation metrics and tools assess the quality of individual outputs from video generation models. Many of the older video generation metrics are derived from image quality assessments: LPIPS, MSE. SSIM, PSNR. These metries fail quantity the temporal consistency between frames (Zhang et al. 2018 Horé & Ziou 2010). Recent video distance metrics focus on assessing both temporal and spatial qualities, as well as diversity. by measuring distances in distribution with respect to the real videos. The most popular such metric for video is the Fréchet Video Distance (FVD) (Unterthiner et al., 2019| An analog exists for measuring image quality (but not temporal consistency): the Fréchet Inception Distance (FID) (Heusel et al. (017). These metrics have emerged as a leading tool for assessing the quality of video and image generation models.
FID, primarily image metric, is also used in video generation to compare feymffrmms. It computes the Fréchet distances of the frame features from Inception v3 (Szegedy et al.] 2014: Toffe & Szegedy 2015) trained on ImageNet. Building upon the principles of FID metrics, FVD evaluates the FD between the generated and data distributions in the Inflated 3D ConvNet (13D)'s 's feature space (Carreira & Zisserman| 2018), which is trained on the Kinetics dataset (Kay et al., 2017).
FVD's use of features extracted from 3D-ConvNet allows it to capture a more comprehensive range of visual and temporal information compared to FID.
Recent studies have highlighted limitations in the reliability of the FVD measure. Specifically, Brooks al. (2022) demonstrated that FVD is not effective in capturing long-term realism and more suitable for comparing generation model variants of the same architecture. Moreover, (Skorokhodoy al. (2021) showed that FVD overlooks motion collapse and is biased towards image quality, rather than video quality. Additionally, they pointed out that FVD is excessively sensitive to minor implementation details, such as the specific image storage formats used (e.g., JPEG compression levels or file encoding), which can lead to inconsistent and non-comparable results across different studies.
A comprehensive study on FVD was conducted by Gc et al. (2024), which compares Fréchet distances of features extracted by the 13D network (Carreira & Zisserman 2018) and VideoMAE network (Wang) et al. 2023a). The study shows that the FVD prioritizes per- pramf quality over temporal consistency when using 13D features, which they refer to as content-biat Further, they suggest that using features from self-supervised models trained
content-debiate data can effectively mitigate this bias in FVD. Our methodology draws inspiration from previous analysis that highlights shortcomings with
FID (Borji 2021 Kynkäänniemi et al. 2023 Soloveitchik et al. 2022; Sajjadi al. 2018).
A separate and distinct method of evaluating videos is on the sample level, rather than the distribu- tional level. For example, (Huang et al. (202)) recently developed VBench, comprehensive video benchmark that analyzes the evaluation of individual generated outputs on subject consistency, back- ground consistency, temporal flickering, motion smoothness, aesthetic quality, among others. VBench addresses the challenge of evaluating both temporal and spatial consistency in video generation, but naturally fails at efficiently evaluating the generational capabilities of a model on a distributional level, or the robustness of a model in generating videos outside its benchmark distribution.
In our proposed framework, JEDi, we address many of the problems affecting existing evaluation strategies:
1. JEDi employs a Maximum Mean Discrepancy (MMD) metric with a polynomial kernel, elimi- nating the need for parametric assumptions about the underlying video distribution, unlike FVD which relies on the Gaussianity assumption to make its metric feasible.
Unterthiner et ss work introduced Fréchet Video Distance (FVD) and Kernel Video Distance (KVD), both operating in the 13D feature space. FVD uses the Fréchet distance, while KVD employs polynomial kernel-based method, and Unterthiner et al. found that FVD aligns more elosely with human judgment.
2

---

Published as a conference paper at ICLR 2025
2. JEDi significantly reduces the number of samples needed make an accurate estimate by using an MMD metric in a V-JEPA feature space, enabling reliable use in smaller datasets that do not meet the requirement when using FVD.
3. JEDi leverages the robust representations of V-JEPA model, which are found to be more aligned with human evaluations compared to FVD.
2 BACKGROUND AND NOTATIONS
2.1 VIDEO FEATURE REPRESENTATION
Inflated 3D ConvNet: The Inflated 3D ConvNet (I3D) (Carreira & Zisserman 2018 a convolu- tional neural network model based on the pre-trained Inception-vl. It extends the 2D convolutional filters to 3D by replicating them along the temporal dimension. 13D, pre-trained on Kinetics, has demonstrated excellent classification performance on UCF-101 (Soomro et al. (2012), HMDB- 51 (Kuchne et al. 2011), and Kinetics datasets (Kay et al. 2017), proving to be a valuable network for video recognition tasks.
The original FVD work by Unterthiner et al. (2019) explores the use I3D features trained on the Kinetics datasets. They analyze the features from the logits layer, well as the features from the last pooling layer trained on the Kinetics-400 and Kinetics-600 datasets. Their experiments suggest that the features from the logits layer trained on the Kinetics-400 dataset are the most suitable for the FVD metric.
Video Masked Autoencoder: The Video Masked Autoencoder (VideoMAE-v2) (Wang al. 2023a is a self-supervised pre-training method that leverages a vision transformer (ViT) back- bone (Dosovitskiy et al. 2020) learn efficient video representations. According to Ge al. the jjant-VideoMAB-VV model, pretrained on a diverse set of unlabeled datasets and fine-tuned on Nomething-somethingg v2 (Goyal al. 2017) with a masked autoencoder objective, effectively captures both spatial and temporal distortions in its encoded feature space. We leverage two variants of the VideoMAE-v2 model in our study: (1) VideoMAEpr: the self-supervised pre-trained giant VideoMAE-v2 model and (2) VideoMAEss.2 the fine-tuned giant VideoMAE-v2 model.
Video Joint Embedding Predictive Architecture: Video Joint Embedding Predictive Architecture (V-JEPA) (Bardes et al.) 2024) is self- training paradigm that learns by predicting missing or masked parts of a video in an abstract representation space. V-JEPA excels "frozen evaluations", where its encoder and predictor are pre-trained through self-supervised learning and then left unchanged. For new tasks, only a small, lightweight layer or network is trained on top of the pre-trained components, enabling quick and efficient adaptation to new environments. In this study, we employ both (1) the pre-trained variant of the model trained with the self-supervised objective, V-JEPApr, well as (2) version that was fine-tuned et al. 2017) with an attentive classification probe such that its pre-logit features could be used for distributional analysis metrics, V-JEPAsSi2.
(Goyal)
2.2 FRÉCHET DISTANCE AND FRÉCHET VIDEO DISTANCE
Fréchet Distance (FD), also known as 2- 2asseersteinn distance (W₂), is a way of measuring how similar two distributions are (Frechet 1957 Dowson & Landau] 1982, Zilly et al. 2020). The Fréchet distance between two distributions P and Q is defined as the minimum distance between all pairs of random variables x and from the distributions. Assuming P and Q are multivariate Gaussian distributions, it can be expressed as:
Dirtcher (P,Q) = (pp- (o--kg) u2)² + Tr(Ep + -
where uP and μio are the means, while Ep and De the covariance matrices of the two Gaussian distributions. Without making this assumption, the Fréchet Distance is intractable and becomes much more arduous to obtain.
The Fréchet Inception Distance (FID) and Fréchet Video Distance (FVD) correspond to the above equations, but the distance is applied in the space of Inception V3 and I3D network features, respec- tively, instead of directly in raw image space in order to obtain more meanful distance that better align with human preferences (Szegedy et al. 2014
Unterthiner et al. 2019).
3

---

Published as a conference paper at ICLR 2025
2.3 OTHER DISTRIBUTION DISTANCE METRICS
This study also explores the application of alternative statistical methods compute probability dis- tribution distances in video feature spaces, including Mixture Wasserstein (MW2), Energy Statistics, and kernel-based methods such as Maximum Mean Discrepancy (MMD). The detailed backgrounds of these metrics are provided in Appendix A.4
3 EXAMINING FVD: FEATURE SPACES AND THE GAUSSIANITY ASSUMPTION
The Fréchet distance (FD) measures the difference between means and covariances. This can offer insights into the first two moments of the distributions, but fails do so with respect higher-order moments (e.g., skewness, kurtosis) that arise when either the real or generated data distribution is non-Gaussian, According to JayJsuuuaaaaa et al. (2024), the reliance on Gaussianity assumptions FID research can lead to substantial inaccuracies when the underlying image distribution does not come from such a distribution. This part of the study focuses on the video feature spaces, investigating the accuracy of Gaussian assumptions and considering the consequences of the Fréchet Video Distance (FVD) when those assumptions are not met.
Using each of the I3D, VideoMAE, and V-JEPA networks under comparison, we extract 48,501 features from 11 distinct video datasets, with each feature representing 32-frame clip. Specifically, we extract a maximum of 5000 features from the training set of each dataset, which include: Anime-
Run-v2 (Siyao et al. 2022). BAIR (Ebert et al.j (217)). BDD100K Yuu et al. 2020). DAVIS (Pont-
Tuset al. 2018), Fashion Modeling (Zablotskaia et al. 2019), HMDB-5T (Kuehne al., 201
How2Sign Duarte et al. (021), KITTI Geegeer et al. 2013), Something- Soannn---ommhhi-2222 (Goyal et al. 2017 Sky Scene (Xiong et al. 2018). and UCF-10T (oommr al. 2012).
The notion that I3D video features do not follow multivariate Gaussian distributions is investigated using the accepted Mardia's Skewness (Mardia, 1970). Mardia's Kurtosis (Mardia 1970). and Henze-Zirkler normality tests (Henze & Zirkler 1990), tollowing (Jayasumana et 20244) The null hypothesis that I3D features follow multivariate Gaussian distribution strongly rejected (p- = 0) across all datasets and normality tests.
We then normalize the aggregated training- set features and fit Principal Component Analysis (PCA) model and Linear Discriminant Analysis (LDA) model using dataset labels as classes. We apply the same pipelines to transform 5,256 I3D features (up to 500 samples from each of the eleven datasets' testing sets) into lower-dimensienn Appendix A.3] applying PCA or LDA transformations to Gaussian -annssiinndistiibueee data preserves their Gaussian properties. Figure 2 shows that the I3D features don't 't follow a single multivariate Gaussian distribution; rather, they cluster by dataset.
spaces for visualizations. As demonstrated in
I3D
VAdE
Video MA Egss2
V-JEPAPT
V-JEPAss22
LDA
PCA
#animerun
6 bair
bold
dovis
fashion
hmmbb
how2sign
kitti
sthv2
ucf
Figure 2: The dimensionally reduced video features of the datasets using LDA and PCA indicate that the video features are non-Gaussian in the combined dataset space. While individual dataset clusters may appear Gaussian in these plots, the low explained variance ratios (0. 134-0.2))) of the PCA-reduced spaces suggest that 2D projections in these plots may not capture the complexity of higher-dimeesional feature distributions within individual datasets. Figures 7 and g contain dataset- specific LDA and PCA plots, which reveal non-Gaussian characteristics within the datasets.
4

---

Published as a conference paper at ICLR 2025
Moreover, we conduct PCA on individual datasets and apply LDA to the HMDB-51 and UCF-101 datasets, incorporating classification labels and varying frame counts per clip. Our analysis shows that I3D clip features deviate from Gaussian distributions within each dataset. More notably, observe a positive correlation between clip duration and increase in FVD between the train and test sets from the same dataset, suggesting that higher-order moments may be essential for accurate characterization.
We replicate our experiments in the VideoMAE and V-JEPA feature spaces using the same datasets. Our results in these feature spaces mirror our findings in the I3D feature space. Additional details on the setup and results are provided in Appendix D
THE DUAL CHALLENGE OF CONVERGENCE: HIGH-DIMENSIONAL FEATURE SPACES AND LIMITED SAMPLES
The following section addresses two pivotal challenges in evaluating video distribution distances:
1. The Dimensionality Problem (Section 4.1 We examine the limitations of metrics relying on distribution assumptions (e.g., Fréchet distance, Mixture Wasserstein distance), highlighting the adverse impact of high dimensionality.
Sample Efficiency and Convergence (Section 4.2): We discuss the sample efficiency issue affecting all metrics and the necessary sample size for trustworthy measurements.
4.1 CHALLENGE #1: THE CURSE OF DIMENSIONALITY
IMPACT OF DATA DIMENSION ON FRÉCHET DISTANCE METRIC
In the previous section, we have shown that the Fréchet Distance (FD) can be used as a metric for comparing the discrepancy between the first two moments of two distributions. Consequently, the accuracy of the mean and covariance estimators is crucial for ensuring the validity of FD as a metric. In the following part, will explore the impact of data dimensionality and sample size the quality and precision of these estimators, and examine how this affects the reliability of distribution distance metrics.
The rank of the empirical covariance matrix (É) is tied to the number of samples (n) and the dimension (k). Given matrix X containing n observations, where each column vector represents k-dimensional multivariate sample, the empirical covariance matrix can serve as reliable estimator for the true covariance matrix. The empirical covariance matrix is calculated using the formula:
E = 1 -. (X, X) X) and it consistently converges to the true
rate of V
is crucial to recognize that when the number of samples less than the number of variables (n < k), the covariance matrix becomes singular. A good covariance estimator requires sample size that is sufficiently large, ideally at least several times greater than the data dimension. This is because estimating the covariance matrix involves estimating k-k + 1)/2 parameters, which requires sufficiently large number of samples to achieve accurate estimates (Bickel & Levina [2088] Marrenko Pastur 1967 Wang Jonsson 1982). Unfortunately, the nighedimensional nature of I3D (400),
VideoMAE (1408), and V-JEPA (1280) representation spaces exacerbates this issue.
Furthermore, optimal transport methods with complex distributional assumptions require more samples yet. The Mixture Wasserstein (MW2) experiment described in Appendix E.. highlights significant computational and practical limitations of optimal-transpor type metrics, making them impractical for this project. See Appendix E.I for further discussion.
DATA TRANSFORMATION: DIMENSIONALITY REDUCTION
To address the challenges posed by the curse of dimensionality, dimension reduction techniques such as PCA and autoencoders (Lecun] 1987) can be applied. Our preliminary investigation from Appendix the metric to converge with smaller number of samples using metrics like Fréchet Distance. We test that hypothesis by training autoencoders in various feature spaces to reduce dimensionality.
E.I suggests that decreasing the representation dimension could enable
5

---

Published as a conference paper at ICLR 2025
Our autoencoder architectures consist of simple multilayer perceptron networks, which compress feature dimensionality to either of original size for I3D features or of original size for Video- MAE and V-JEPA features. Additional information about the autoencoder training is available in Appendix E.2
Interestingly, dimension reduction significantly enhances the sample efficiency of the Fréchet Distance and energy statistic metrics in our experiments with Gaussian data (Figure T0). However, its benefits less pronounced for video features, resulting in only marginal improvements (Figures and Nonetheless, we retained autoencoder features in subsequent experiments to investigate other possible benefits.
12
4.2 CHALLENGE #2: SAMPLE EFFICIENCY AND DATA SCARCITY
we recognize the critical importance of sufficient sampling for accurate estimation of metrics. This section delves into the relationship between sample size and convergence rate for each metric, exploring its impact across various feature spaces. Here, we define convergence rate as the rate at which the distance between training and testing set feature stabilizes as the number of video sample increases. is a measure of sample efficiency.
Building the insights from Sections 4.1
Previous studies in the image and audio domains have shown that as the sample sizes N decrease, Fréchet Distances increase (Binkowski et al., [021]] Gui et al.. [024] Jayasumana et al.) 2024, Chong & Forsyth 2019). This sensitivity to sample size common phenomenon among distributional distance metries: As sample sizes increase, distance metrics become more reliable and accurate. However, while all metrics benefit from additional data, some converge the true underlying distance more quickly than others.
We investigate the sample size required to achieve convergence within 5% error margin average metric distance measured at 5,000 samples. Our comparative analysis in Figure sigure two diverse types of datasets: UCF-101, HMDB (human action recognition) and Something-Sommethngg Something-v2 (SSv2, hand gesture recognition). An extended analysis on more datasets, feature spaces, and metrics is found in Appendix
Our key findings include:
1. The sample sizes needed for convergence within the same feature space are similar across datasets. For instance, 4,350 samples on UCF-101 and 4,700 samples on SSv2 are required for FVD (I3D+FD) to converge, while 700 samples on UCF-101 and 800 samples on SSv2 are required for JEDi (MMDDoLy+Y.JJPASSE
converge.
2. MMDPOLY demonstrates the quickest convergence rate across non-13I) feature spaces. For ex- ample, requires only 1,500 and 700 samples reach steady metric value on UCF-101 in V-JEPAPT and V-JEPAssv2 feature spaces, respectively (Figure [4].
Fréchet Distance has the worst sample efficiency among metries, while I3D features exhibit the worst sample efficiency across feature spaces (Figure [4
16
15
FVD JECI
UCF
700
4350
SSv2
800
HMDB
600
4000
Figure 3: The number of samples required to achieve a 5% error margin in the distance measured from 5,000 samples using the training and testing sets of the 3 listed datasets. We assess the number of samples required for convergence at 100 sample intervals. Convergence at sample size N is achieved if: (1) the average metric value from 5 repeated samplings of features falls within a 5% error margin, and (2) all subsequent interval evaluations maintain average metric value within the 5% error margin. The results for other metrics, feature spaces, and different datasets are provided in the Appendix (Figure 15). We find that Fréchet Distance (FD) converges slowest, while MMDPOLY shows the highest sample efficiency.
As shown in Figure 3 it often takes thousands video clips for FVD features to converge; however, many datasets contain insufficient amount of unique videos reach this convergence, making

---

Published as a conference paper at ICLR 2025
it challenging to define robust distribution in such high-dimensional spaces (Pont- al. 2018; Geiger et al., 2013, Ebert et al., (017). This is often worked around by transforming videos into shorter, partly overlapping clips. This method is problematic and biases the metric due to the repetitiveness of the data. This fault has remained largely challenged. For instance, as shown in Figure T6. the BAIR dataset's size and sample efficiency issues are particularly noteworthy. Despite these limitations, BAIR dataset remains a widely-used benchmark for video generation, with numerous studies reporting FVD results on it YYu et al. 2023 Wu al., 2021 Voleti et al. (022).
We note three key hurdles stemming from this sample efficiency issue in video generation: (1) Data size: Limited samples compromise estimate reliability, undermining robust statistical analysis; (2) Computational resources: Generating samples is computationally expensive and time- -em-dddnn (3) Metric convergence speed: Slow convergence rates hinder accurate assessments. While dataset size and computational resources are largely beyond our control, we can address the third concern by selecting metrics with higher sample efficiency where convergence happens with less samples.
METRIC DISTANCE ANALYSIS: NOISE, GENERATIVE MODELS, AND HUMAN STUDY
This section explores the effects of videos distorted with noise and videos generated at varying model training checkpoints on metric reliability, assessing their impact (1) metric accuracy, (2) sample efficiency and (3) human metric alignment.
5.1 NOISE & GENERATION MODELS AND THEIR IMPACTS ON METRIC MEASUREMENT
Metric
No Noise
Blur (low)
Blur (medium)
Blur (high)
FVD
69.0 = 0.187
68.1 18.13 = 0.087
97.4 + 0.151
177.3 + 0.221
JEDi
0.017 0.000
0.038 + 0.000
0.256 + 0.000
0.571 0.000
Table I: The table shows average FVD and JEDi distances between training and testing set feature distributions under various blur distortions. The testing video dataset subjected to noise distortions, including low blur (a ~ [0.05, 0.75)). medium blur (a ~ [0.1, 1.5)), and high blur (a ~ [0.01, 3), where Ø represents the per-frame blur intensity, and a larger range indicates greater temporal inconsis- tency. The experiment is replicated 10 times account for variability. Our analysis reveals that FVD) fails to detect low blur noise and incorrectly suggests an improvement in video quality (highlighted in gray). Note: To improve readability, we standardize JEDi by applying a scaling factor of 100 to the V-JEPAss.++MM polynomial distance.
This study investigates metric reliability when presented with videos affected by three noise distortion types (salt and pepper noise, temporal blur and elastic distortion) and two image-to-video generation models (I2V-Stable Video Diffusion and Open-Sora).
Salt and pepper noise, a type of impulsive noise. spatially corrupts visual data by randomly altering pixel values to extreme intensities. Elastic noise distortion from (Ge et al. 2024) primarily introduces temporal distortions and occasionally deforms object shapes. In addition, we introduce temporal blur noise which involves applying Gaussian kernels of varying strengths blur frames, preserving appearance and shape integrity while focusing on temporal distortion. The two generative models we used were adopted from open-source repositories, utilizing the provided checkpoints (Zheng et al. 2024 von Platen et al. 2022). Detailed inference configurations for these models are in Appendix
We highlight some of our results in Table 1 and the remaining results are in Figures 17 and 18 The
key findings in these experiments include:
1. Metrics in the I3D feature space are impacted by salt and pepper noise (a spatial distortion) significantly more than by other types of distortions. This aligns with the findings of Ge et al. demonstrating that the I3D feature space is highly sensitive to spatial distortions but less responsive to temporal distortions. As shown in Scction 5.4 I3D does not align with human preferences with respect to distortions.
2. 13D and VideoMAErT are not ideal feature spaces for building video quality metrics, as they do not capture blur distortion well. Notably, they perceive testing distribution with slight artificial blur
7

---

Published as a conference paper at ICLR 2025
added to the frames as more similar to the training distribution than the original testing distribution. In fact, they estimate the low-blur distorted videos are 10%-20% closer to the ground-trun cotndttuthhhhh training distribution.
3. Our experimental evaluation reveals that the choice of feature space have greater impact on distance values than the choice of distribution distance metric. Moreover, the metrics exhibit consistent performance across diverse datasets, demonstrating robustness.
5.2 METRIC ROBUSTNESS ASSESSMENT WITH PROGRESSIVE DISTORTION LEVEL AND TRAINING DURATION
Distortion Level Figure we conduct study to evaluate the performance of metrics in assessing various blur noise distortions on the UCF-101 dataset. All metrics identified the decline in video quality as noise levels increased.
Hitt HUUT (
(a) FVD vs JEDi
(b) FVD vs VMAEss MMDD (c) FVD vs V.JEPAyZ+MMDpuL
Figure 4: How metric distance changes as temporal blur increases. Specifically, temporal blur distortion is controlled by varying the sigma range (a) using the distortion level (A), with [0.1 0.01A, 0.75 + 0.8A]. The study is carried out on the UCF-101 dataset.
Training Duration We further aim to evaluate metrics for generative models, focusing on their ability to track changes in video quality throughout training (Figure Due the computational expense of training vidco generation models from scratch, we fine-tune Stable Video Diffusion's s weights on the BDD dataset using Ctrl-V's code (Luo al. 2024). Ctrl-V uses a pre-trained SVD model but modifies the input padding strategy to enable mult li- frame conditioning. Initially, the visual quality of generated videos poor, but it improves over the training time. We utilize these fine-tuning steps to assess our metrics' robustness in evaluating fine-tuned model checkpoints. We expect good metric to decrease steadily over time and thus have a negative correlation close to in magnitude. We visualize several checkpoint generations in Figure 22
75 100)
73 F30
Menry with Setti PomO
MMO PAY recosnAA
-30
E10 trother
(a) FVD JEDi
(b) FVD vs
(c) FVD +JEPApT+MMDDrs
Figure 5: Ctrl-V is fine-tuned on BDD. Visual inspection show incremental improvements in gen- eration quality each training step. This is captured by, JEDi V-JEPAssz+MMDporr)) However, FVD (I3D+FD), VideoMAEss improvements. The Spearman coefficient correlation values for the X and Y axes are are- -1, -0.6, -0.9 and -0.8 for JEDi, FVD, only JEDi showing statistical significance.
and V.JEPAy+MMDDjo. fail to detect incremental
and V-JEPAyt+MMMPpo.. respectively, with
Results Only JEDi (VJEPAAs,+ successfully tracks incremental gains in all check- points, whereas FVD (I3D+FD), VideoMAFFss and +JEPApr+MMDDa do not.
8

---

Published as a conference paper at ICLR 2025
5.3 SAMPLE EFFICIENCY UNDER NOISE DISTORTION
investigate the sample efficiency of various metrics under noisy conditions. Specifically, we measure the number of samples it takes for the distance between the original training distribution and the noise-added testing distribution stabilize/converge. The noise-added testing set essentially simulates a set of generations. Our findings indicate that: JEDi remains much more sample efficient compared FVD in this condition. We present our experiment results in Appendix F.2
Alongside Section 5.1
5.4 HUMAN EVALUATION
To examine human perception of video quality degradation due noise distortions, we conduct small-scale survey using 24 randomly selected videos from the UCF-101 and Sky Scene test sets, originally 30 fps, subsampled to 3-second clips fps. We apply four noise distortions: two levels of blur, elastic distortion, and salt and pepper noise, with parameters detailed in Appendix F3 To reduce border effects from elastic distortion, all videos are center-cropped to 230 310 pixels.
Surveys for the UCF- 101 and Sky Scene datasets are conducted randomly, with participants evaluating anonymized video pairs that differ only in noise type. They rate each pair under four comparisons, choosing one video superior noting no difference. Each comparison is assessed by 20 indepen- dent raters from an academic community, who focus solely on visual quality without knowledge of the datasets distortions.
The Analytic Hierarchy Process (AHP) Saaty 1987) aggregates responses using pairwise com- parison matrix derive priority vector for noise distortion types. This vector is normalized and inverted to align with distribution distance metrics, where O represents ideal quality. Scores for each distance metrie-feature space combination normalized and compared human survey results using cosine similarity. Evaluation results are summarized in Figure and more details can be found in Appendix F3
ViceMMAS
VikonWAL
Vida
C0000 sduit with IeRlve
20
Exery
MM/Ow.
SwTr
(a) UCE-101
(b) Sky Scone
Figure 6: Alignment of human evaluation with distribution distance metrics. Metrics computed in V-JEPA feature spaces surpass that of both I3D and VideoMAE in terms of alignment with human evaluation.
Results While metrics within a feature space generally perform the same level, distances calculated in the feature space of V-JEPAss<2 or V-JEPApT model resoundingly outperform both 13D and VideoMAAEbassel metrics in terms of alignment with human evaluation. Among raters, there were agreements 83.70% and 53.54% on the UCF-101 and Sky Scene datasets, respectively. Perhaps intuitively, humans are much more confident assessing content with human activity than with more abstract visuals, where subjective interpretations can vary significantly.
5.5 STUDY ON ARTIFICIAL INTELLIGENCE GENERATED CONTENT
To further evaluate our model's performance on Artificial Intelligence Generated Content (AIGC), we leveraged the Text-to to- tee-tt--ieee Video Quality Assessment Database (T2 (12VQA-DB' (Kou et al. 2024) and
Text and Image Prompt Dalaset for Image-lo- Video Generation (TIP-12V) (Wang & Yang 2024

---

Published as a conference paper at ICLR 2025
Tune-A-Video
Show-1
Vie
MOS Rank
Worst
Median
Best
JEDi (#. conv samples)
4.427 (100)
2.164 (200) 0.722 (300)
FVD (#. conv samples)
884.1 (600)
445.6 (800) 279.9 (900)
Table 2: Quantitative comparison of JEDi and FVD Metries for T2VQA-DB models, stratified by MOS (Worst, Median, Best). Values represent measured distances; numbers in parentheses indicate sample sizes needed achieve a 5% error margin (estimated from 1,000 samples).
T2VQA-DB comprises 10,000 A Al-generated videos by 10 Text-to-Vide (T2V) Al models, each associated with a Mean Opinion Score (MOS) reflecting text-video alignment and fidelity (Section 3.2 (Kou et al. 2024). In particular, (I) MOS scores display substantial variance (u 50, 16.6), (2) model rankings are ambiguous due to score overlap (Figure from (Kou et al. 2024)). and (3) No "ground-truthe video set exists; we addressed this using WebVidIOM (Bain al. 202 videos with
matching prompts.
We assessed the outputs of three models from T2VQA-DB (Tune-A-Video (Wu et al. 2023) Show-
1 (Zhang et al. 2024). LaVie (Wang et al. 2023b)) with varying video quality, ranked 10 (worst), 5, and (best) based MOS scores. Both FVD and JEDi scores ranked these models consistently, with
Tune-A-Vider as the worst and Vie as the best. The quantitative findings are presented in Table 2 Notably, JEDi achieved convergence with significantly fewer samples on T2VQA-DB compared to FVD, requiring only 100, 200, and 300 samples to converge, while FVD required 600, 800, and 900 samples.
TIP-12V provides the videos generated by Pika Al 2024), SVD (Blattmann et al. 2023) and Open-SORA (Zheng et al. 2024), which we use for our subsequent analysis of the video feature spaces. Specifically, if video sets A and B are from the same distribution, their features extracted using any feature extractor should remain in a same distribution. However, if video sets A and B come from different distributions, robust feature extractor designed for evaluating distributional differences would highlight these distinctions. The results in Appendix demonstrate that V-JEPA extracted features reveal clear distributional distinctions between 3 AIGC video sets and 1 real video dataset, whereas I3D features do not exhibit such distinct clustering.
CONCLUSION
In this study, we carefully look at many aspects of video metrics and find JEDi to be the best choice. First, we show that the normality assumption made by the Fréchet Distance (FD) does not hold true in the video feature spaces. and this becomes more evident the duration of the videos increases. Second, we discuss two challenges with FD: 1) Estimating the covariance matrix for FD is difficult due high latent space dimensionality, and reducing this dimension with autoencoders showed no improvement: 2) The sample efficiency of FD is low; through extensive comparison, we find that Maximum Mean Discrepancy (MMD) with V-JEPA exhibits much higher sample efficiency across all datasets tested. Third, we investigate the impact of noise on feature spaces and found that 13D was more sensitive to image quality distortion than temporal distortion, and that I3D and VideoMAEsss2 does not capture blur distortion well. On the other hand, V-JEPAssvz stands out as the more robust feature space among them. Fourth, we observe the correlation between metric with distortion level and training duration. We show that while both FVD (FD+13D) and JEDi (MMDPOLY V-JFPAss+))))) are positively correlated with distortion level (higher distance with higher distortion), only JEDi is highly negatively correlated to training duration (lower distance with more training). Finally, our human study results indicate that our proposed metric aligns most closely with human preferences. Also, utilize the AIGC databases with human opinion scores to show that JEDi is a more sample-efficient metric with a superior feature space for analysis.
Based on comprehensive analysis, JEDi emerges the most effective and practical metric for guiding the current surge in video generation research. To facilitate the usage JEDi, we provide simple and easy-to-use code that, given its striking benefits, hope the community will embrace.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Max Bain, Arsha Nagrani, Gil Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. IEEE International Conference on Computer Vision, 2021.
Adrien Bardes, Quentin Garrido, Jean Ponce, Xinlei Chen, Michael Rabbat, Yann LeCun, Mahmoud Assran, and Nicolas Ballas. Revisiting feature prediction for learning visual representations from video. arXiv preprint arXiv:2404.000 08471, 2024.
L. Baringhaus and C. Franz. On a now multivariate two-sample test. Journal of Mul- fivariate Analysis, 88(1):190-206, 2004. ISSN 0047-259X. doi: https://doi.org/ 10.
101668004722500X0300007744 URL https: http://wwciieeeeeccmm rect com/ science/ att.icleep1.150044255500030
Peter Bickel and Hlizaveta Levina. Regularized estimation of large covariance matrices. The Annals
of Statistics, 36(1), February 2008. ISSN 0090-5364, 10.
10110111100000001000001111111110 00.2144009053600000000733 URL doi http:://kk.ool.ror....22222//000
Mikotaj Binkowski. Danica J. Sutherland, Michael Arbel, and Arthur Gretton. Demystifying mmd gans, 2021. URL https: /ttps::/azxiv.org// wtt.:///grabs//111... 01401
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Macicj Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, Varun Jampani, and Robin Rombach. Stable video diffusion: Scaling latent video diffusion models to large datasets, 2023.
Ali Borji. Pros and cons of GAN evaluation measures: New developments. CoRR, abs/2103.09396 2021.
Tim Brooks, Janne Hellsten, Mika Aittala, Ting-Chun Wang, Timo Aila, Jaakko Lehtinen, Ming-Yi Yu Liu, Alexci Efros, and Tero Karras. Generating long videos dynamic scenes, 2022.
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset, 2018.
Djalil Chafai. Wasserstein distance between two gaussians. https:://faalll.ccoomo chafai net/ o 04/00/0443//// www.eeeeeeiiiiindtteeeeeeee bww.een-two- www.ttwwwwgasnsss//// 2010. Accessed: 2024-06-26.
Min Jin Chong and David A. Forsyth. Effectively unbiased FID and inception score and where to
find them. CoRR, abs/1911.07023, 2019. htt/tttp://axxivcco wwwpaaawwwi..i..aaaaoooooooooora abs 19023
07023
Julie Delon and Agnes Desolneux. A wasserstein-type distance in the space of gaussian mixture models, 2020.
A. Dempster, M. Laird, and D. B. Rubin. Maximum Likelihood from Incomplete Data Via the EM Algorithm. Journal of the Royal Statistical Society: Series B (Methodological), 39(1): 1-22, 12 1977. ISSN 0035-9246. doi: 10.111112217--666..17777 org /
URL https
1977761111777.111111 www. 600.x
Yadolah Dodge. Mahalanobis Distance, pp. 325-326. Springer New York, New York, NY, 2008 ISBN 978-0-387-32833-1. doi: 10. 111110007-0-8877777 0-387-32883---- 1.240. URL httlhtttp//.....o org
240
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale.
CoRR, abs/2010.1 1929, 2020. BtRLbttps:// arwiv.xiivv///aa/ htttppps//aaxxx..oo//a////////// 11929
D.C Dowson and B.V Landau. The fréchet distance between multivariate normal distributions. Journal of Multivariate Analysis, 12(3):450-455 1982. ISSN 0047-259X. doi: https///oi.org/
22.006600047-22990x8888000077770 URL htt swwencedirect..........o. com/ htt.://att.coo/ciienne.e scinncee
atticcle/si110444223xX22200

---

Published as a conference paper at ICLR 2025
Amanda Duarte, Shruti Palaskar, Lucas Ventura, Deepti Ghadiyaram, Kenneth DeHaan, Florian Metze, Jordi Torres, and Xavier Giro i Nieto. How2sign: A large-scale multimodal dataset for continuous american sign language, 2021. /RLLhtttss//axxiv.ooor
08143
Frederik Ebert, Chelsea Finn, Alex X. Lee, and Sergey Levine. Self-supervised visual planning with temporal skip connections. CoRR, abs/1710.05268. 2017. URL httt::///axii....////// orgbas// /
1710 05268
Maurice Frechet Sur la distance de deux lois de probabilité. COMPTES RENDUS HEBDO- MADAIRES DES SEANCES DE ACADEMIE DES SCIENCES, 1957. ISSN 244(6):689-6))
Karl Pearson F.R.S. Liii. on lines and planes closest fit to systems of points in space. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, 2(11):559- 572, 1901. doi: 10.108011786440004477777
songwei Ge, Aniruddha Mahapatra, Gaurav Parmar, Jun- Yan Zhu, and Jia-Bin Huang. On the content bias in fréchet video distance. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets roboties: The kitti dataset. International Journal of Robotics Research (IJRR), 2013.
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuma Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, et al. The" something something" video database for learning and evaluating visual common sense. In Proceedings of the IEEE international conference on computer vision, pp. 5842-5850, 2017.
Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Scholkopf, and Alex Smola. A kernel two-sample test. J. Mach. Learn. Res., 13:723 773, 2012. URL https://apt
semanticscholie
Azalea Gui, Hannes Gamper, Sebastian Braun, and Dimitra Emmanouilidou. Adapting frechet audio distance for generative music evaluation, 2024. URL
wttp:/iiwrr//aaaoa abs/2311 /2111
01616
Yingqing He, Tianyu Yang. Yong Zhang, Ying Shan, and Qifeng Chen. Latent video diffusion models for high-fidelity long video generation, 2023.
N. Henze and B. Zirkler. A class of invariant consistent tests for multivariate normality. Com- munications in Statistics - Theory and Methods, 19(10):395--3117 1990. doi: 10.1080/ 036109999088834000.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, Günter Klambauer, and Sepp Hochreiter. Gans trained by two time-scale update rule converge nash equilibrium. CoRR, abs/1706.08500, 2017. httpppppp/aaxx...rg///////o
08500
Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pretraining for text-to-video generation via transformers, 2022.
Alain Horé and Djemel Ziou. Image quality metrics: Psnr vs. ssim. In 2010 20th International Conference Pattern Recognition, pp. 2366-2369, 2010. doi: 10.1 009/CCP.2000.577
Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, Yaohui Wang, Xinyuan Chen, Limin Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. Vbench: Comprchensive benchmark suite for vidco generative models, 2023.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by BtLthttpp//axi//
reducing internal covariate shift. CoRR, abs/1502.001667, abs/ fbss/1002.r 03167
Sadeep Jayasumana, Srikumar Ramalingam, Andreas Veit, Daniel Glasner, Ayan Chakrabarti, and Sanjiv Kumar. Rethinking fid: Towards better evaluation metric for image generation, 2024.
12

---

Published as a conference paper at ICLR 2025
Dag Jonsson. Some limit theorems for the cigenvalues of sample covariance matrix. Jour- nal of Multivariate Analysis, 12(1):1-38, 1982. ISSN 0047-259X. doi: https://doi.or////
1016/0047-299X00229000000 X. URL
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayaaaraimhann Fabio Viola, Tim Green, Trevor Back, Paul Natsev, Mustafa Suleyman, and Andrew Zisserman. The kinetics human action video dataset, 2017.
Tengchuan Kou, Xiaohong Liu, Zicheng Zhang, Chunyi Li, Haoning Wu, Xiongkuo Min, Guangtao Zhai, and Ning Liu. Subjective-aligned dataset and metric for text-to-video quality assessment, 2024. URL https: http://wwwww..aaaa////////////// 10031/1///2031156
H. Kuehne, H. Jhuang. E. Garrote, T. Poggio, and T. Serre. Hmdb: A large video database for human motion recognition. In 2011 International Conference on Computer Vision, pp. 2556-2563, 2011. doi: 20.1100//CCV.2211161666
Tuomas Kynkäänniemi, Tero Karras, Miika Aittala, Timo Aila, and Jaakko Lehtinen. The role of imagenet classes in frréchet inception distance, 2023.
Yann Lecun. PhD thesis: Modeles comeeionnistes de l'apprentissage (connectionist learning models). Universite P. et M. Curie (Paris 6), June 1987.
Yixin Liu, Kai Zhang, Yuan Li, Zhiling Yan, Chujie Gao, Ruoxi Chen, Zhengqing Yuan, Yue Huang, Hanchi Sun, Jianfeng Gao, Lifang He, and Lichao Sun. Sora: A review on background, technology, limitations, and opportunities of large vision models, 2024.
Ge Ya Luo, Zhi Hao Luo, Anthony Gosselin, Alexia Saiiceur-Martinese and Christopher Pal. Ctrl-v: Higher fidelity video generation with bounding-bow box controlled object motion, 2024. URL https://arxe org/ 2200..0663303
K. V. Mardia. Measures of multivariate skewness and kurtosis with applications. Biometrika, 57 (3)5119-530 530, 1970. ISSN 00063444, 14643510. URL http::/www httt:::/www..ssto..oooo jstor. www.ooo.rrgg/aaaall..ee.... 2334770
Aleix M. Martínez and Avinash C. Kak, Pea versus Ida. IEEE Trans. Pattern Anal. Mach. Intell., 23: 228-233, 2001. http::ttpp:/ppptt wttp://ssaaaattccccccocceeccco cotp:/aarr.ror/rroouusutuuuu : 267125089
Marčenko and Pastur. Distribution of eigenvalues for some sets of random matrices. Mathe- matics of the USSR-Sbornik, 1(4):457, apr 1967. doi: 10.
doi. URL |ttp:pppppppooooooooooooo//00010 SM1 010p:000000000000010000000000111
Anish Mittal, Rajiv Soundararajan, and Alan C. Bovik. Making a "completely blind" image quality analyzer. IEEE Signal Processing Letters, 20(3):209-21,, 2013. doi: 10.10091.SP2212.222772
G. Monge. Mémoire sur la théorie des déblais des remblais. Imprimerie royale, 1781. URL ptttps://boos..gooe www.:/ol.o...oooo..eeeooooc
Eduardo Fernandes Montesuma, Fred Ngolè Mboula, and Antoine Souloumiac. Recent advances in optimal transport for machine learning, 2023. URL sttps://arxiv..rgge wargg//b///////
16156
Pika-AI. Pika creative video editing platform, March 2024. URL httth::/ppiaaaartt
Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alex Sorkine-Hornung, and Luc Van Gool. The 2017 davis challenge on video object segmentation, 2018. URL https
7/arxiv. ww.. abs 700. 00675
Michele A. Saad and Alan C. Bovik. Blind quality assessment of videos using a model of natural scene statistics and motion coherency. In 2012 Conference Record of the Forty Sixth Asilomar Conference on Signals, Systems and Computers (ASILOMAR), pp. 332-336, 2012. doi: 10.1109/ ACSSC.2012.6489018
13

---

Published as a conference paper at ICLR 2025
Rozann W. Saaty. The analytic hierarchy process what it is and how it is used. Mathematical Modelling, 9:161-176, 1987. URL http::tp:::// www.stmaattti. sep.ss.maiiicccoo....oo wtt.ccchc.......ooooooroSs.S.... 73719597
Mehdi S. M. Sajjadi, Olivier Bachem, Mario Lucie, Olivier Bousquet, and Sylvain Gelly. Assessing generative models via precision and recall, 2018.
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang. Oron Ashual, Oran Gafni, Devi Parikh, Sonal Gupta, and Yaniv Taigman. Make-a-video: Text-to-video generation without text-video data, 2022.
Li Siyao, Yuhang Li, Bo Li, Chao Dong, Ziwei Liu, and Chen Change Loy, Animerun: 2d anima- tion visual correspondence from open source 3d movies. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
Ivan Skorokhodov, Sergey Tulyakov, and Mohamed Elhoseiny. Stylegan-v: continuous video generator with the price, image quality and perks stylegan2. CoRR, abs/2112. 14663,,,1.,,,, 2021. URL
Joram Soch and et al. Stapoocbookssttpooooooo..ggbbbbd The book of statistical proofs (version https::::://do http://ooooorgg//////// /
2023). Zenodo, 2024. doi: 10.5281/2ENN0000
ZENODO. 4.05944
Michael Soloveitchik, Tzvi Diskin, Efrat Morin, and Ami Wiesel. Conditional frechet inception distance, 2022.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. UCF101: A dataset of 101 human actions classes from videos in the wild. CoRR, abs/1212.0400 2012. URL http::/arxii brg/abs/ 0402
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions, 2014.
Gabor Szekely and Maria Rizzo. Testing for equal distributions in high dimension. InterStat, 5, 11 2004.
Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: A new metric & challenges, 2019.
Vikram Voleti, Alexia Jolicoer--Martineau, and Christopher Pal. Mevd: Masked conditional video diffusion for prediction, generation, and interpolation, 2022. URL
abs/2205.09853
Patrick von Platen, Suraj Patil, Anton Lozhkov. Pedro Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj. Dhruv Nair, Sayak Paul, William Berman, Yiyi Xu, Steven Liu, and Thomas Wolf. Diffusers: State-of-the-art diffusion models. https::/ http:///gitthbcccoo com//hggiig.g.ffccc.. di diffusers] 2022.
Jindong Wang al. Everything about transfer learning and domain adapation. sransferlearning,
http://
Lie Wang. Estimation of covariance matrix. htt.ps::/matt.mmmt.ccmm mit.
pdf
Limin Wang, Bingkun Huang, Zhiyu Zhao, Zhan Tong, Yinan Hc, Yi Wang. Yali Wang, and Yu Qiao. Videomae v2: Scaling video masked autoencoders with dual masking, 2023a.
Wenhao Wang and Yi Yang. Tip-i2v: million-scale real text and image prompt dataset for image-to-video generation, 2024. Itt.ttstt::///ara
org/ abs /2112 04709
Yaohui Wang, Xinyuan Chen, Xin Ma, Shangchen Zhou, Ziqi Huang, Yi Wang, Ceyuan Yang, Yinan He, Jiashuo Yu, Peiqing Yang, Yuwei Guo, Tianxing Wu, Chenyang Si, Yuming Jiang, Cunjian Chen. Chen Change Loy, Bo Dai, Dahua Lin, Yu Qiao, and Ziwei Liu. Lavie: generation with cascaded latent diffusion models, 2023b. URL https://///..........ttt..ss : aaxiv. hwgg/bb/
video
2309. 151003
14

---

Published as a conference paper at ICLR 2025
Chenfei Wu, Jian Liang, Lei Ji, Fan Yang, Yuejian Fang, Daxin Jiang, and Nan Duan. Nüwa: Visual synthesis pre-training for neural visual world creation. CoRR, abs/2111.12411, 2021. URL
https://arxiv: wwwp/rggg./.oooooo..o..o.t...ost /els//11ll.e. 12417
Jay Zhangjie Wu, Yixiao Ge, Xintao Wang. Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 7623-7633, October 2023.
Wei Xiong, Wenhan Luo, Lin Ma, Wei Liu, and Jiebo Luo. Learning to generate time-lapse videos using multi-stage dynamic generative adversarial networks, 2018. URL
abs 2oss/1709.0 07592
Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangehen Liu, Vashisht Madhavan, and Trevor Darrell. Bdd100k: A diverse driving dataset for heterogeneous multitask learning, 2020.
Lijun Yu, Yong Cheng, Kihyuk Sohn, José Lezama, Han Zhang, Huiwen Chang, Alexander G. Hauptmann, Ming- Haug- Yang. Yuan Hao, Irfan Essa, and Lu Jiang. Magvit: Masked generative video transformer, 2023, URL httt::ts:
wwwp:bsg//aaaooooooooo..oos..sas 2212.2222222 05199
Polina Zablotskaia, Aliaksandr Siarohin, Bo Zhao, and Leonid Sigal. Dwnet: Dense warp-based network for pose-guided human video generation, 2019. URL https://arxiiii wrg. abs/
19900.91133
Yan Zeng, Guoqiang Wei, Jiani Zheng, Jiaxin Zou, Yang Wei, Yuchen Zhang, and Hang Li. Make pixels dance: High-dynamic video generation, 2023.
David Junhao Zhang, Jay Zhangjie Wu, Jia-Wei Liu, Rui Zhao Lingmin Ran, Yuchao Gu, Difei Gao, and Mike Zheng Shou. Show-1: Marrying pixel and latent diffusion models for text-to-video generation. International Journal of Computer Vision, 2024. doi: 10. (2007//11126--022-022111 024 02271-2
URL https:://aii.org//0000 1007
Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness deep features as a perceptual metric, 2018.
Zangwei Zheng, Xiangyu Peng, Tianji Yang, Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. Open-sore Democratizing efficient video production for all, March 2024. URL PtR.sttts::// //g thub. com/hpca
Daquan Zhou, Weimin Wang, Hanshu Yan, Weiwei Lv, Yizhe Zhu, and Jiashi Feng. Magicvideo: Efficient video generation with latent diffusion models, 2023.
Julian Zilly, Hannes Zilly, Oliver Richter, Roger Wattenhofer, Andrea Censi, and Emilio Frazzoli.
The frechet distance of training and test distribution predicts the generalization gap, 2020. URL https:/ http:://ppeeeeeieee New net forum? (ttprummmddsJgsff1ffff
15

---

Published as a conference paper at ICLR 2025
A STATISTICAL DISTRIBUTIONS
A.1 MULTIVARIATE GAUSSIAN DISTRIBUTION
The probability density function of a k-dimensional multivariate Gaussian is given by:
1
1 (- 2/(xx (6--1) 11)
P(x) (2T) exp
(2)
where x Th is the k- d-dimensional random sample, μ R is the mean vector, R is the covariance matrix which is symmetric and positive definite, |Σ] denotes the determinant of the covariance matrix, and Σ denotes the inverse of the covariance matrix.
A.2 GAUSSIAN MIXTURE MODELS
Gaussian Mixture Models (GMM) with clusters is probabilistic model that assumes the data is generated from mixture of Gaussian distributions, each with its own mean and covariance. The probability density function weights π. sum up to 1. The parameters of GMMs often estimated using iterative algorithms, such as Expectation conertatiio-Maaiiizzzeee Maximization (EM) algorithm (Dempster et al., 1977). as there is no closed-form solution to maximize its likelihood function.
GMM is given by: P(x) = Σi= mxN(x/jt Σ.) where the cluster
A.3 PRELIMINARY: LINEAR TRANSFORMATION OF MULIIVARIATE GAUSSIAN DISTRIBUTION AND LINEAR DIMENSIONALITY REDUCTION METHOSS
A
Back to paper
Let (,, Σ) follow multivariate Gaussian distribution with mean vector Jr and covariance
matrix Σ. Let A be matrix and b be a vector. We are interested in the linear transformation
y Ax b. The following is the proof of y is also multivariate Gaussian distribution. Specifically, y ~ N ((Au
4 + b, AΣAT).
The moment generating function for
1 E M₂(1) = [[exp|["*x) x]) = exp (e" +
/ 2
(3)
and the moment generating function for y is given by
Mt( E exp 7t (Ax + b)])
explt b|E( explt" Ax])
explt =explt b)M,(A
exp [[T(A) b) 7 (AEAT)))
This indicates that the moment generating function of y aligns with the moment generating function of the multivariate Gaussian distribution. Therefore, is random variable that follows a multivariate Gaussian distribution (detailed proof can be found in [Soch & et al. (2024)).
Principal Component Analysis (PCA) and Linear Discriminant Analysis (LDA) are statistical meth- ods used reduce the number of variable dimensions in a dataset. PCA is process of linear transformation that involves mapping data from higher dimensional space to a lower dimensional space by identifying the directions which the data varies the most (FR.S.. 1901). LDA entails a linear transformation that maps data from a higher-dimensional space to lower-dimensional space in order to effectively separate multi-class objects (Martinez Kak 2001). Because PCA and LDA are linear transformations, and we' ve shown that data from multivariate Gaussian distribution remains Gaussian after linear transformations, we can conclude that applying PCA or LDA to data from multivariate Gaussian distribution preserves its Gaussian properties.
16

---

Published as a conference paper at ICLR 2025
A.4 DISTRIBUTION DISTANCE METRICS OVERVIEW
Back to paper
Optimal transport is a mathematical field that deals with finding transport plan that minimizes the total cost of moving the mass from the source distribution to the target distribution (Monge 1781; Montesuma et al. (023). Recently, Delon & Desolneux (2220) proposed Wasserstein-type distance within novel optimal transport framework for Gaussian Mixture Models (GMMs) with restricted couplings. By confining the set of possible coupling measures to GMMs, they derive simple, discrete formulation of the distance metric, making it computationally efficient for problems with high dimensions. The distance is called Mixture Wasserstein and is denoted 2W2. The MW2 distance is always upper bounded by Wasserstein distance (W2) plus the variances of the Gaussian components. In addition, its computational complexity is solely determined by the number of clusters.
Mixture Wasserstein (MW2) (Delon Desolneux 2020
Energy Statistic (Baringhaus & Franz| 2004 Szekely & Rizzo 2004): Energy statistic measures the difference between distributions based on pairwise distances between points. Given {xi,... Xm) are random samples generated from distribution P and f....... ... are random samples generated from distribution Q, the energy distance & ( () is given by:
m
m
E(P, Q)- mn
||
y,
|x
by:
- --yll-
(5)
m
π
1=1
1=1
1=
The energy statistic is appropriate for comparing complex distributions without making assumptions about a particular underlying distribution.
Maximum Mean Discrepancy (MMDs) (rreton el al. 2012): MMD is general class of kernel- based sample tests that maximize the mean difference between samples from two distributions by optimizing over all data transformations within a function space F. Some popular kernel functions used for MMD include: linear, polynomial, sigmoid, Laplace and RBF (Gaussian) kernels.
Given two sets of features, X = [xI,X2,.... )m and Y = Y2.. .2....... ......... .n sampled from P and Q, dh MD( (P, Q) with given kernel, k. is given by (Jayasumana al. 2024
2
L (XM) m(m
xx,,,,)
k
k( y2).
n(n
f=1
371
jf
(6)
Like the energy statistic, MMD is distribution-free requiring no assumptions about the underlying distributions of P or Q.
B COMPUTATION CONFIGURATIONS
B. FEATURE EXTRACTORS
Feature extraction is performed single NVIDIA RTX 4080 GPU with float32 precision. However, VideoMAE-v2 22 features require more specialized setup: single NVIDIA RTX A100 GPU with 80G memory. Notably, VideoMAE-v2 precision varies by clip length: float32 for clips under 64 frames and float 16 for longer clips. We use batch sizes of 10 (clips < 64 frames) and 2 (clips 2 64 frames) for feature extraction.
I3D Configuration et al. 2019: I3D logits features pre-trained on Kinetics-400.
We adopt the recommended feature extractor from the FVD paper (Unterthier
VideoMMAEp and VideoMAEsssa We compute VideoMAE features using the official PyTorch implementation (Wang al., 2023a), following the guidelines outlined in Ge et al.
Feature extractor for FVD
17

---

Published as a conference paper at ICLR 2025
V-JEPAPT Configuration We compute V-JEPA features using the official PyTorch implementa- tion [ardes et al., 2024). Specifically, our V-JEPA pretrained model, referred to as V-JEPApT. consists solely of the V-JPEA encoder. Consistent with previous research (Ge et al. 2024). we extract features from the encoder and average them across all feature tokens.
V-JEPAss<2 Configuration We compute V-JEPA features using the official PyTorch implementa- tion (Bardes et al. 2024). Specifically, our V-JEPA classifier model, referred to as V-JEPA SSv2 fine-tuned model or V-JEPAss consists of two components: (1) V-JEPA encoder, and (2) an adaptive probe with attentive pooler. Consistent with previous research (Ge et al. exploit the pre-logit features generated by the attentive pooler.
2024
B.2 DISTRIBUTION METRIC CONFIGURATIONS
Fréchet Distance We calculate the Fréchet Distance using torchaudio's functional API, passing mean and covariance statistics to Frechet.distance.
Energy statistic We utilize the http:s///gtthbbb..o/aappp/////op
repository to calculate the energy statistic.
Mixture Wasserstein We utilized the official implementation from Delon & Desolneux]s repositor to calculate the Mixture Wasserstein distance.
Mean Maximum Discrepancy We compute MMD distances using code from (Wang al.), with kernel-specific parameters:
RBF/Laplacian: 1/ndim
Polynomia degree 2, 1, coef = 0.
C FRÉCHET VIDEO DISTANCE: A SEMI-METRIC
distance metric on a set X is a function d X x - that satisfies the following prop- erties for all points y, z X: 1. Non-negativity: d(x,y) 0: 2. Identity of indiscernible: d(x,y) 0
x = 3. Symmetry: d(x.y) = d(y, x); and 4. Triangle inequality: d(x,z) S d(x.y) + dly z2)
Fréchet Distance (FD) satisfies all the properties of a metric, except for the triangle inequality, which establishes it as semi-metris The triangle inequality is crucial property that underpins the linearity of metric and offers valuable interpretability. We will explore this aspect in greater detail later. For now, we will demonstrate the proof for FD being a semi-metric.
The FD of multivariate Gaussian distributions, represented by Equation is composed of two terms: the squared-Euclidean matrices. In order to demonstrate that FD is semi-metric, we will examine these two components individually.
Euclidean distance between the mean vectors and term involving the covariance
The first-term, (uP (uP-~a)²²: By definition, the squared Euclidean distance between the mean vectors is non-negative and equals zero if and only if the mean vectors are identical, i.c., (μР mQ)² > 0 with equality iff p.- μQ. Also, the squared Euclidean distance symmetrie operation. Thus, it satisfies the first three properties. However, the triangle inequality is not satisfied. A counter- 11,11 HC [5, 5) and (H - 2c) 50 Z
example to prove this is: let μ^ = [0, T MB [1,
(MA =² + (uB 22)² 34.
The second-term, Tr [Ep + Σq ) According to Dowson & Landau 1982), the square root of the second term is considered a natural metric on the space of real covariance matrices of given order. This implies that the first three properties should hold. Below are the corresponding
FFettree extractor for JEDi
DDsttribution metric for FVD Distribution metric for JEDi
18

---

Published as a conference paper at ICLR 2025
proofs:
Epp + EQ
Tr
2 I1 (VE
Arithmetic mean is greater than or equal to geometric mean.
2
Ep
2iq + Ir
Fp Ex
=Ilt
(vEeZiP) -TT(Ep))
If the covariance matrices are the same, 2nd term becomes
Tr
Ep + EQ
I1 (vielio) + 2ip = Le Arithemetic mean equals geometric mean when covari- ance matrices are identical.
(7)
To illustrate the symmetric property of the second term, let's assume pr = Mq = 0 for simplicity. Let X represent column vectors sampled from normal distribution with mean jrr and covariance Ep, and Y represent column vectors sampled from a normal distribution with mean (4Q and covariance Σq-
There exists a linear transformation such that = -X. According to Chafai 2010 Direchel (P, Q) can be derived as:
Direcher (P, Tr(pp) + Tr(=z)) - E((X
(8)
To prove the symmetric property, we only need to demonstrate that E (X, I = B(((X)))) because the remaining terms are symmetric. Below is the corresponding proof:
(( rx) E() = E (XXTTT) EpTT, EppT E P = B (X, E((TXX)) Covariance matrices are symmetric: Ep pp= Σ3
B((TX, x)) E(TXXT) ПΣp
(9)
Lastly, the triangle inequality is not satisfied. A counter-example to prove this is: let ΣA (b
Σg = () Ec (8
Tr
- ",,=::/""-........-----..,,,,,,
-4
When combining the first and second terms with addition, their mathematical properties still hold because their input parameters are different and do not affect each other.
The triangle inequality property is crucial, as it ensures that the distance between points remains consistent and intuitive. In video generation, models aim to learn the underlying patterns of a real data distribution (R) by training on an empirical dataset (Rempirical.. This involves understanding a probability distribution over potential videos, allowing newly generated videos to closely resemble the structure and content of the observed data. Note that the empirical distribution is an approximation. potentially biased towards the specific sample. The discrepancy between the true distribution and empirical distribution can be quantified using metrics, such as:
Rempirical
N R = (ammpiiicatt R) N- O
(10)
where N is the number of samples. The equation demonstrates that the sample size approaches infinity, the discrepancy between the empirical and true distributions converges to zero. However, in practice, most datasets are finite, and training is typically done on limited number of samples. As result, the difference between the empirical and true distributions is bounded by specific value, rather than reaching zero. Triangle inequality provides insight into the opper-bound of the model generation quality based on the true distribution, expressed as D Rempirical, R) + D(G, Rempirical) Z D(G, Rt).
On the other hand, much of the video generation work involves using pre-trained models to perform zero-shot inference on different datasets in order to test the models' performance and domain adaptation abilities (Hong et al. 2022; Singer et al. 2022 Blattmann et al. 2023 Zhou et al., (023). However, measuring a model's generation quality another dataset requires setting up the model for the new dataset and generating numerous samples, which demands significant time, effort, and computational resources. If distribution distance metric follows the triangle inequality and one only wants to compute an upper bound to validate a model's ' generation quality on another dataset, is sufficient compute: D Remippaaa: G) + D Remipicaal Resuiplesca)) > D(G, Relate (emprical)
Finally, and most importantly, triangle inequality provides valuable insights into the magnitude of metric distances. Consider the example where D(GmodelA. Rempirica) = 2D Rempiricl). With the triangle inequality constraint, suboptimal as model B's. Without this constraint, we could only conclude that model A underperforms model B, but without precise measure of the extent of this underperformance.
deduce that model A's performance is at least twice as
19

---

Published as a conference paper at ICLR 2025
D NON-GAUSSIAN CHARACTERISTICS OF 13D FEATURES
Back to paper
D.1
VISUALIZATION IN LOWER DIMENSIONALITY
Figure 7 displays 2D PCA and LDA projections of I3D features, highlighting non-Gaussian charac- teristies within individual datasets. The numbers listed in the first column represent the quantity of frames per clip utilized for the experiments in the respective row. In each experiment, we randomly selected 5,000 video clips from each dataset and obtained their I3D features. We performed principal component analysis on each set of 5,000 samples, and the red point clouds in columns 2-5 represent the I3D features projected onto their first principal components.
The point clouds displayed in the last two columns depict the LDA dimensionally-reduee I3D features using the classification labels from the dataset. The markers' colors represent the class labels in the dataset. In each figure, we superimpose blue contour lines on the plot, which delineate the ellipsoidal contours of 2D Gaussian distribution. Specifically, these contours are computed directly from the mean and covariance matrices of the 2D point clouds, providing visual representation of multivariate Gaussian distribution suggested by the Fréchet Video Distance (FVD) metric. We conducted a similar analysis for V-JEPApT and V-JEPAssv2 features, as shown in Figure 8 and Figure
# F.
BDD
HMDB
Sky
UCF
HMDB
UCF
16
32
64
128
Figure 7: This figure provides empirical evidence of non-Gaussianity in I3D feature space across individual datasets.
Back to paper
#
BDD
HMDB
Sky
UCF
HMDB
UCF
16
32
Figure 8: This figure provides empirical evidence of non-Gaussianity in V-JEPApT feature space across individual datasets.
Back to paper
20

---

Published as a conference paper at ICLR 2025
#E
BDD
HMDB
Sky
UCF
HMDB
UCF
16
32
Figure 9: This figure provides empirical evidence of non-Gaussianity V-JEPAssv2 feature space across individual datasets.
D.2 MULTIVARIATE. NORMALITY TESTS
The notion that I3D video features do not follow multivariate Gaussian distributions was further investigated using Mardia's 's Skewness (Mardia 1970). Mardia's Kurtosis (Mardia 1970). and the Henze-Zirkler (Henze & Zirkler 1990) normality lests, following (Jayasumana al. (024). The anal- ysis was done on 16-, 32-, and 128-frame versions of the Anime-Run- Ruime-Run-v2, v2, BAIR, BDDI0OK, DAVIS, Fashion Modeling, HMDB-51, How2Sign, KITTI, Something-Something-
Sky Scene (Xiong
al. 2018), and 10F-101 datasets, as well on baseline dataset constructed by sampling from multivariate Gaussian distribution with 100 features.
The null hypothesis that the distributions of I3D features were drawn from multivariate Gaussian distribution was rejected for each of the datasets and normality tests. All three tests accepted the null hypothesis for the multivariate Gaussian baseline dataset.
E EXPERIMENTAL EVALUATION: CONVERGENCE RATES OF DISTRIBUTIONAL METRICS AND DIMENSIONALITY REDUCTION METHODS
FD
Energy
MW2
MMD RBF
MMD Polynomial MMD
Laplacian
MG
GMM
Figure 10: The figures illustrate the evolution of distance estimates between two identical distributions as function of sample size. The x-axes represent the number of samples drawn from each distribution, while the y-axes corresponds the distance measurements. The plots in the top-row experiments employ a 100-dimensional multivariate Gaussian distribution, denoted as MG. In this distribution, the first 50 dimensions follow standard normal distribution (μ4 0, I), while the remaining 50 dimensions are generated as the cumulative sum of the first 50 dimensions, resulting in a structured correlation pattern. In contrast, the bottom-row plots utilize 100-dimensional Gaussian mixture model comprising 5 clusters, mirroring the multivariate Gaussian setup, with the first 50 dimensions drawn from a GMM and the last 50 dimensions representing the cumulative sum of the first 50 dimensions. The blue lines represent the metrics calculated using samples directly drawn from the original distributions, while the orange lines represent the metrics computed using samples that have undergone PCA-based dimensionality reduction to 50 principal components. Since the compared distributions are identical, the ideal estimated distance between them should be zero.
Back to paper
21

---

Published as a conference paper at ICLR 2025
E.1 Toy EXPERIMENT: METRIC CONVERGENCE RATE
Back to paper
This section reports on simulated experiments analyzing two key questions: (1) How do different metrics converge? and (2) What role does dimensionality play in shaping convergence rates?
Figure 10 illustrates the findings of a series of simulated experiments examining the empirical distribution distances across various metrics, uncovering the intricate interplay between sample sizes, dimensionality-redue reducee features, and metric convergence rates. The experimental results shown in the figures demonstrate two key findings: (1) the accuracy of the metrics improves consistently as the empirical sample size increases, validating the proof of concept; and (2) dimension-reducek data can significantly enhance the convergence rate and accuracy of certain metrics (notably FD and energy), as evidenced by the faster convergence of the orange curves relative to the blue curves.
Our simulation experiments also yielded the notable observation that kernel-based distance metrics for distributions exhibit reduced sensitivity to data dimensionality, attributable to their inherent feature mapping, reliance on pairwise distance calculations, and regularization properties inherent in kernel selection. This diminished sensitivity confers significant advantage in mitigating the curse of dimensionality, thereby enhancing the robustness and reliability of distance metric estimates in higher dimensional space.
In contrast, GMMOT involves fitting separate Gaussian Mixture Models (GMMs) to the sample sets and then computing the optimal transport distance between the fitted GMMs. In the GMM fitting process, the mixtures. During the maximization step, cluster means and covariances are computed using the sample responsibilities calculated in the expectation step. Notably. the sum of the weighted responsibilities for each cluster less than the total number of samples, but the number parameters for each cluster is the same single multivariate Gaussian distribution. As result, because of the smaller effective sample size, it necessary to obtain more samples in order to accurately fit GMM. Therefore, Mixture Wasserstein (MW2) has the slowest rate of convergence compared to all other metrics, leading us to exclude from our subsequent video analysis.
(EM) algorithm updates the parameters of the Gaussian
E.2 AUTOENCODERS: MODELS AND TRAINING SPECIFICATION
Back to paper
In our study, we trained specialized autoencoders for each video representation space to account for variations in video length. Notably, we observed representation shift with differing video lengths: as shown in Figures 7 and [ features extracted from videos with different lengths have visible differences; features from 16f and 32f videos are more similar to each other while those from 64f and 1 128f videos are similar to each other. To address this, we divide the videos from 11 different datasets (UCF, HMDB, etc) into two groups: short clips (16-32 frames) and long clips (64-128 frames). For each group, we extract up to 10,000 features using five different feature extractors (e.g., I3D, V-JEPApr). These results in separate collections of features for short and long clips across all datasets. For instance, I3D yields up to 110,000 short-clip features and up 110,000 long-clip features from the 11 datasets. We then train two autoencoders for each feature extractor: one using the short-clip features and one using the long-elip features. This gives total of 10
22

---

Published as a conference paper at ICLR 2025
autoencoders.
The autoencoders' architectures specified in Algorithm and Algorithm 2
Algorithm 13D Autoencoder Config- uration
Algorithm 2: VideoMAE and V-JEPA
Autoencoder Configuration
input :in dim=400
input :in_dim=1408 if VideoMAE else 1280 encnder Sequential(
encoder Sequential(
Linear(in.ding in.dim//2).
Linear(in dim, dim//3),
ReLU0.
ReI.UO,
Lincar(in.dim//2 dimmi///))
Linear(in dim//3, in dim//4).
ReLUO,
ReLUO.
Lincar(in.dim/k) in di.ii///)),
Linear(in.dim//4, in.iim//8).
decoder = Sequential(
decoder Sequential(
Linear(in dim//6, in dim//4), ReLUO,
aancar(i_ddim//) in.dim//4) ReLUU.
Linear(in dim//4, dim//2),
Linear(in dim//44 in dim/mm//)),
ReLUO.
ReLUO,
Lincar(in_dim/1) in_dim),
Linear(in dim//3, in dim),
E.3
SAMPLE CONVERGENCE ANALYSIS
Back to paper
Figures included in this section are:
Figure The cvolution of UCF-101 train-test distances for all metrics in feature spaces.
Figure 12 A comparison of convergence rates of FVD and JEDi, comparing training and testing
sets on UCF101 and Srpethinn-Somehhingg".
Figure
13
Convergence rates UCF-101 train-test distances in VideoMAEpT and V-JEPApT feature spaces.
Figure 14
Visualization of the sample size required for VideoMAFPT and V-JEPAPT features to converge to a 5% error margin compared to the distance measured from 5,000 samples using the train and testing sets of UCF-101 and SSv2.
Figure
15
Visualization of the sample size required for V-JEPAPT and V-JEPApT features to verge to a 5% error margin compared to the distance measured using the train and testing sets of other datasets. The pink vertical lines in the plots denote the sample size used to compute the target metric distance (ideally 5,000 samples). Note in Figures and [4] that UCF-101 and SSv2 have sufficient samples (> 5, 000) in their training and testing sets. However, many datasets in this graph have fewer samples. Importantly, convergence estimates become less reliable as bars approach the pink line, due to fewer iterations meeting the second convergence criterion (referring to Figure 3 caption).
Figure 16
The BAIR dataset (Ebert et al.) 2017) demonstrates a perfect example of the convergence issue due insufficient samples. With 250 training videos, the estimated sample size for convergence (bars) nearly coincides with the target metric computation sample size (pink line). As convergence estimates degrade near this threshold due insufficient iterations meeting the second criterion, it is difficult to confirm whether metrics truly converge at the displayed sample sizes, especially for Fréchet Distance- based metrics.
23

---

Published as a conference paper at ICLR 2025
FD
Energy
MMDRBF
MMD Polynomial
MMD Taplacian
I3D
VMAESS22
V-JEPAPT
V-JEPAssv2
Figure 11: The figures depict the changes in distance estimates between the training and testing sets of the UCF-101 dataset. We extract features from video clips of 32-frame duration using the models indicated in the left-most column. The x-axes represent the number of samples drawn from each distribution, while the y-axes corresponds to the distance measurements. We repeat each experiment 10 times. The lighter shaded area on the plots indicate the variance across these 10 runs. The blue lines represent the metrics calculated using features directly extracted from the models, while the orange lines represent the metries computed using features that have been compressed by the autoencoder's s encoder. The autoencoder's structure is described in Appendix E.2
Back to paper
00
30
2000
4000
5000
1000
3000
(a) UCF-101
(b) SSv2
Figure 12: A comparison of convergence rates of FVD and JEDi, comparing training and testing sets on UCF101 and Bommthingg-Soeethnngee We evaluate convergence rate 100 intervals, from 50 to 5,000 samples, with 50-sample increments. The x-axes represent the number of samples drawn from the training and distributions, while the y-axes show the convergence rate, calculated as (5000) where e an arbitrarily small number and m is drawn from set of metrics M. Our (m00000== methodology involves sampling samples from the training and testing 10 times, computing the distance 10 times for each sampled sets, and calculating (mn) as the mean distance across the 10
Back to paper
24

---

Published as a conference paper at ICLR 2025
(a) 13D
(b) V-JEPAss-2
(d) VidcoMAEssv2
(c) VidcoMAEpr
(f) V-JEPArT
Figure 13: Convergence rates of distributional metrics on UCF-101, comparing training and testing sets in various feature spaces. The convergence rate computation in these figures follows the same configuration as Figure T2, with the x-axis representing the number of samples and the y-axis showing the convergence rate.
VideoMAEpT
VideoMAF
UCF
SSv2
pry_a*
pnly
Iplaciar
Ieplactin
energy
enrry
Figure 14: This figure shows the number of samples needed for VideoMAEpT and VideoMAEssv> to achieve 5% error margin of the distance measured from 5,000 samples using the training and testing sets. An "Lae" suffix indicates that the feature space has been compressed using an autoencoder. The convergence requirement is stated in Figure 3
A
Back to paper
25

---

Published as a conference paper at ICLR 2025
13D
V-JEPAPT
V-JEPASs22
UCF
SSv2
Anime
BDD
Davis
Fashion
HMDB
KITTI
Sky
1bT
poly 24
pory
applacinn
Ipraccar an
Figure 15: This figure shows the number of samples needed for V-JEPArT and V-JEPAss to achieve a 5% error margin of the distance measured from 5,000 samples using the training and testing sets on of the datasets presented this study. An " "ae suffix indicates that the feature space has been compressed using autoencoder. The convergence requirement is stated in Figure 3
Back to paper
26

---

Published as a conference paper at ICLR 2025
(a) 13D
(b) VidcoMAEpt
(c) VidcoMAEssy2
(d) V-JEPAPT
(e) V-JEPAss<2
poly
o0ly
Taplacian
leplacion
entrg
anergy
Figure 16:
The figures display convergence rates of distributional metrics BAIR, comparing training and testing sets in VideoMAEPT and V-JEPAssv2 feature spaces.
Back to paper
NOISE DISTORTION STUDIES
F.1 COMPLIMENTARY MATERIAL FOR THE NOISE AND GENERATIVE MODEL STUDY
Figure 18 Illustrates the impact of noise and generative models on the metrics in VideoMAEs and V-JEPAs spaces. The study demonstrates a distinction how different feature spaces rank these noise types. In Section 5.4] the findings from human survey to determine which model has the closest ranking with human perception are reported.
UCF-101
Sky Scene
leplecter ml
paly
erergy
segeer
(PayO
Figure 17: Comparing the FVD (I3D+FD) between training and test sets, using 5,000 samples from each, under various conditions, including noise and conditional generation. The train-test distance (gray) serves as a baseline for evaluating the reliability of the metric. For clarity, we normalize metric values using the train-test distance. The displayed bar values represent these scaled distances. Notably, the Sky Scene (Xiong et al.) (2088) experiment shows that low blur distortion brings the test distribution closer to the training distribution, highlighting a flaw in the FVD metric.
Back to paper
27

---

Published as a conference paper at ICLR 2025
UCF-101
Sky
pay
VMAEpT
belocin only
VMAE55v2
V-JEPAPT
V-JEPAssv2
094 10 PESLET
desh Shal0
dlakks Inno and
clocik dow)
Figure 18: Experiments were conducted on the feature spaces of VideoMAF and V-JEPA using con- figurations analogous those in T7] The train-test distance (gray) serves as baseline for eval- uating the reliability of the metric. For clarity, normalize metric values using the train-test distance. The displayed bar values represent these scaled distances. Both the V-JEPApT and VideoMAEpT im- plementations require averaging across patch embeddings. Intuitively, V-JEPApT metrics less affected by pixel-level salt and pepper noise as its training is done in an abstract representation space, in direct contrast to that of a VideoMAE.
Back to paper
28

---

Published as a conference paper at ICLR 2025
F.2
NOISE AND CONVERGENCE RATE
Back to paper
Our study investigates the impact of noise and generative model outputs on the convergence speed of FVD and proposed metric. Figure revellss two key findings:
1.
Our method exhibits significantly faster convergence than FVD even with added noise.
The presence of noise surprisingly accelerates convergence for both metrics.
No distortion
Blur (low)
Blur (med)
Blur (high)
UCF-101
303
SSv2
100
2002 380
Figure 19: The convergence speed of FVD and the proposed metric is evaluated with blur distortion noises on the UCF-101 and Something-Sometting-- follows the same configuration as Figure [2] with the x-axis represnting th enumber samples and the y-axis showing the convergence rate. The blue line in the figure represents the convergence rate of FVD, while the red line represents the convergence rate of the proposed metric.
datasets. The convergence rate computation
F.3 HUMAN EVALCATION
Back to paper
To investigate human alignment on the perception of video quality degradation under various noise distortions, we conduct small scale survey. The rater population consists 75% males, 25% females, with 55% under the age of 25, 35% between the ages of 25-35, and 10% above the age of 50. We randomly select 24 videos from each of the UCF-101 and Sky Scene test sets, originally captured at 30 frames per second (fps). and subsample them to 25 frames at fps to generate clips three seconds in length. Four types of noise distortions are systematically applied: high blur with a 7x7 kernel and Gaussian standard deviation ranging from 0.01 to medium blur with the same 7x7 kernel and a standard deviation ranging from 0.1 to 1.5, elastic distortion with a deformation strength of 30, and salt-and- pepper noise applied at a rate of 1%.
Following the Analytic Hierarchy Process (AHP) (Saaty 1987,, a pairwise comparison matrix is used aggregate the responses. It is important note that the sums of corresponding comparisons may not total 100% due to the inclusion of an option indicating no discernible difference in quality between videos (i.e.. all self-to-self comparisons yield value of 0%). To account for this, the columns are normalized before computing the priority vector, which captures human preference over the different types of noise included in our study. Raw metrie values can be found in Appendix F.
Noise Type
Blur (high)
Blur (medium)
Elastic (medium)
Salt and pepper
Blur (high)
0.00%
0.00%
12.50%
2.50%
Blur (med)
93.75%
0.00%
68.75%
16.25%
Elastic (med)
81.25%
16.25%
0.00%
3.75%
Salt and pepper
95.00%
69.68%
93.75%
0.00%
Table 3: Pairwise comparison matrix, noise distortion (UCF-101). Results are aggregated from 20 participants.
29

---

Published as a conference paper at ICLR 2025
Question 9/25
Left better
Right better
No difference
Figure 20: An example question in the human evaluation, noise distortion survey on the UCF-101 dataset.
Question 4:25
Left better
Right better
dffferencc qucullity
Figure 21: An example question in the human evaluation, noise distortion survey on the Sky Scene dataset.
Noise Type
Blur (high)
Blur (medium)
Elastic (medium)
Salt and pepper
Blur (high)
0.00%
7.63%
33.23%
37.50%
Blur (med)
68.23%
0.00%
41.25%
48.75 48.75%
Elastic (med)
56.78%
42.50%
0.00%
41.25%
Salt and pepper
53.75%
46.25%
51.25%
0.00%
Table Pairwise comparison matrix, noise distortion (Sky Scene). Results are aggregated from 20 participants.
F.4 PREVIOUS HUMAN STUDIES IMAGE AND VIDEO METRICS
Video Distributions FVD: To evaluate the alignment of FVD with human perception, Unterthiner et al. (2019) trained many variations four video prediction models- SV2P, SVP-FP, and SAVP on the BAIR robot pushing dataset. For evaluation, they selected 256 video sequences and
30

---

Published as a conference paper at ICLR 2025
Rank
Method
Feature Space
UCF-101 (%)
Sky Scene (%)
Energy
V-JEPAPT
83.26
84.06
MMDRUF
V-JEPAPT
83.02
84.07
3
MMDLAP
V-JEPArI+AE
83.02
85.60
4
FD
V-JEPApT
82.31
86.82
5
MMDPOLY
V-JEPAPT
81.82
81.07
Energy
V.JEPAPT+AE
80.88
83.17
FD
V.JEPAPT+AE
79.33
84.13
8
MMDPOLY
V-JEPAss4
78.96
78.39
9
MMD
V-JEPAPT
78.85
81.34
10
MMDFOIY
V.JEPArt+ÄE
78.78
82.33
MMDRDF
V-JEPAss<2
75,51
79.10
MMDLAP
V-JEPAssv2
75.49
79.98
13
Energy
V-JEPAss<2
75.42
79.85
14
MMDRAF
V-JEPArT+AE
75.32
94.67
15
FD
V-JEPAss22
74.63
86.32
MMDREF
13D+AE
70.32
75.43
17
MMD
V-JEPAss2+AE
63.78
84.03
18
FD
V.JEPAsSZtAE
61.15
84.12
19
Energy
V.JEPAss2+AE
59.12
79.09
MMDFOLY
V.JEPAsS2++E
58.47
78.60
21
MMDLAP
V.JEPAss-2+AF
57.93
80.34
22
FD
VideoMAEsso
55.72
79.08
23
Energy
VideoMA Fssa2
51.76
74.15
24
MMDRBF
13D
50.93
68.07
25
MMD
VideoMAEssv2
50.16
72.76
26
MMDrOLY
VideoMAEsss2+AAE
49.76
75,54
27
MMDPOLY
VidcoMAEss:
48.34
75.35
28
Energy
VideoMAEssserre
46.56
72.09
29
MMDREF
VideoMAEsso
46.36
73.47
MMDPOLY
VideoMAEPT
46.14
62.87
31
MMDREF
VideoMAEPT
45.98
62.29
32
MMDLAP
VideoMAEsssete
42.85
69.20
33
FD
VideoMAEssve
42.38
72.42
Energy
VideoMAERT
40.03
60.73
35
MMDREF
VideoMAEss.+AAE
37.90
73.47
36
Energy
AAdeoMABETTAF
37.64
58.12
37
MMDLAP
VideoMAEETTAF
37.47
57.99
38
MMDPOLY
VideoMAEET
36.88
58.06
FD
VideoMAEET
35.36
61.94
40
MMDRRF
VideoMAEETTAE
32.44
57.04
41
FD
VideoMAEET+AE
32.01
57.42
42
FD
13D+AE
31.66
57.54
43
FD
13D
31.32
58.07
MMDLAP
VideoMAErI
29,90
57.93
45
MMDPOLY
13D+AE
25.97
67.25
46
MMDLAP
I3D
25.55
54.57
47
Energy
13++AE
25.52
48.07
MMDLAP
13D+AE
25.17
57.30
49
Energy
B3D
23.74
47.25
50
MMDFOLY
I3D
22.38
67.17
Table 5: Human Evaluation:
Similarity Results (Ranked on UCF-101). VJEPAss:*MMDrous and FVD (I3D+FD) results are highlighted.
Cosine
computed SSIM, PSNR, FVD, and KVD scores for each model. Both FVD and KVD metrics were calculated in the 13D feature space to assess perceptual similarity and distributional alignment. Up to 3 humans evaluated videos from each model in one metric spread, metric equal comparisons, where both comparison types involved 10 models. They reported that FVD was most aligned with humans, while KVD was similar but slightly worse
31

---

Published as a conference paper at ICLR 2025
Image Distributions CMMD: In study the alignment of FID and CMMD (CLIP MMD) with human evaluation, Jayasumana et al. (2024) ran survey using two Muse models trained on the WebLI dataset, with one model (Model A) purposely trained for longer than the other (Model B). Side-by-side evaluations were performed where human raters were presented with two images, having select which looked more preferable. All image pairs rated by independent raters, with 1633 prompts used in total. They reported that human raters and CMMD preferred Model A, while FID preferred model B.
T2V Alignment Quality T2VQA: Kou et al. (024) constructed a 10,000 sample dataset of AI- generated videos from 10 Text-to- Video (T2V) Al models. To determine the Mean Opinion Score (MOS) for each model, 27 subjects rated its perceptual quality based on two main criteria: text-video alignment how well the video content matches the text description and video fidelity covering distortion, saturation, motion consistency, and content rationality. Subjects used slider from O to 100 to assign their final score for each video. To amalgemate the results, the authors we conducted normalization to avoid inter-subject scoring differences.
G GENERATIVE MODEL SPECIFICATIONS
Back to paper
G.1 OPEN-SOR:
For videos generated from the UCF-101 dataset, a single frame was provided as the image prompt with its corresponding video class label given as the text prompt. For videos generated from the Sky Scene dataset, a single frame was provided as the image prompt with "A sky timelapse" provided as the text prompt.
Videos were generated at output resolution of 240p with an aspect ratio of 3:4. The '4s" preset for the number of frames was used which translates to 102 frames 24 fps. These videos were converted to 16 frames at 7 fps for both the metric calculations. We used the open-source implementation loaded with the hpcai-tech/Opennoraa LaaeenSSrr---E-lll VAE-v1.2 checkpoint for inference.
G.2 CTRL-V
Frame 1
Frame 5
Frame 9
Frame 13
Frame 16
Iter O
Iter
Iter 1200
Figure 22: Training progression of Ctrl-v SVD generation at iteration 0, and 1200.
Back to paper
We use the original code and model configuration from Ctrl-V [uoo et al. 2024) in this study, and their Stable Video Diffusion's model backbone is HuggingFacce saabiiityaa/stableeviiieee
wewrussionimggvii-i-e model. Figure [2] contains demo of samples generated from the Ctrl-v's SVD checkpoints.
32

---

Published as a conference paper at ICLR 2025
H A STUDY OF THE FEATURE SPACE WITH AI-GENERATED CONTENT
Back to paper
In this section, we demonstrate compelling results showing that JEDi is superior metric compared to FVD when applied to AIGC.
To interpret these results, need agrec key principle:
If video sets A and originate from the same distribution, their features when extracted using any feature extractor-shoule remain in the same distribution. This is because consistent extractor should not introduce artificial distinctions between sets that inherently belong to the same distribution.
On the other hand, if video sets A and B come from different distributions, robust feature extractor designed for evaluating distributional differences would highlight these distinctions. A good extractor would encode the unique characteristics of each set, creating clearly separable distributions in the feature space. This separation critical for understanding the underlying differences and verifying that the extractor is effectively capturing meaningful distributional variations rather than noise or irrelevant features.
Validating feature extractor's ss ability consistently distinguish different distributions confirms its usefulness for evaluating distributional differences. Conversely, an extractor that fails to separate distinct distributions would be less effective in this context.
By this framework, we show that V-JEPA extracted features reveal clear distributional distinctions between AIGC video and real video dataset, whereas I3D features do exhibit such distinct clustering.
Specifically, we utilize the generated samples from the TIP-I2V (Wang Yang 2024) generated by the Pika (Pi 2024). SVD (Blattmann et al. 2023). and Open-SOR. (Zheng et al. 2024) models, well the real video samples from the VedvidIOM dataset (Bain al. (021)), and map them into 13D and V-JEPA feature spaces. Then, we employ LDA transformation project these features into 3D space, with the model label serving as conditioning factor. The slices the feature space provided in Figure 23 These illustrations show that JEPA-cxtrane features reveal clear distributional distinctions between AIGC video sets and real video dataset, whereas I3D features do not exhibit such distinct clustering.
IBD
VJEPA
Figure 23: The 3D slices illustrate different viewpoints of the feature spaces, where the blue points represent Pika samples, the teal points represent SVD samples, the green points indicate Open-SORA samples, and the orange points indicate WebVid10M samples.
33

---

Published as a conference paper at ICLR 2025
I ANALYSIS OF LONG-TERM VIDEOS
400
0.4
300
FVD
200
100
0.1
128
32
mid
high
128
16
mid
high
Clip length
Clip length
level
Figure 24: This figure illustrates the evaluated distances between the original training set and test sets with varying levels of blur distortion across different clip lengths. The bars are color-coded to represent different video clip lengths, while separate bars within each color group denote varying levels of distortion. The bar height indicates the measured distance. Notably, FVD exhibits monotonic increase in measured distance with clip length within each distortion group. In contrast, JEDi yields constant measured distances within each distortion group, irrespective of clip length.
We conducted study assess the FVD and JEDi metrics across various clip lengths under different levels of blur distortion. The blur distortion settings followed those depicted in Table [ with uniform application of distortion across each clip. The results in Figure show that for FVD, the evaluated distances increase clip length grows for the same blur level. In contrast, JEDi distances remain consistent regardless of clip length.
This demonstrates that the JEDi metric is sensitive solely to the amount of distortion applied to the clips. When the same level of distortion is applied, JEDi evaluates the distance consistently, regardless of the clip length. In contrast, FVD is influenced by clip length, with evaluated distributional distances increasing as the clip becomes longer.
J LIMITATIONS
J.1 JEDI: PERFORMANCE EFFICIENCY TRADEOFFS
While we performed extensive study of different feature space with different distance metrics on different datasets, there are more datasets be tested on and more types of noise distortions can be investigated. While we choose JEDi (MMDPoLY++ V-JEPAss)) other choices (see Table for the list) such as the Energy distance with V-JEPApT has slightly higher aligmment with human evaluation. It is decision made considering the large gain in sample efficiency from JEDi.
our main proposed method,
J.2 AVENUES FOR FUTURE RESEARCH
Similar to FVD, JEDi is a distribution based evaluation metric that measures the discrepancy between the generated and ground-truth distributions in feature space. While our study focuses on distribution evaluation, our findings have implications for assessing the quality of individual video samples.
In prior literature on single video quality blind assessment, the Natural Video Statistics (NVS) model, proposed by (Saad & Bovik employed multi-faceted approach, combining various models and techniques to extract features relevant to different aspects of video quality. This included hand- crafted temporal motion features, derived from differences between consecutive frames, and spatial characteristics captured using NIQE (Mittal et al. 2013) features. Our approach, on the other hand, integrates spatial and temporal aspects into single, unified framework: the V-JEPA feature space.
The V-JEPA feature space is particularly well-suited for evaluating video quality, as it is sensitive to both spatial and temporal distortions, enabling a comprehensive assessment. This characteristic
34

---

Published as a conference paper at ICLR 2025
presents a promising application: utilizing large-scale, diverse, and high-quality video dataset mapped to the V-JEPA feature space to establish a pre-defined guideline distribution.
The quality of individual videos can be quantitatively assessed by mapping them to the V-JEPA feature space and computing their distance or likelihood relative to the guideline distribution, which serves robust benchmark for high-quality videos. Possible evaluation methods include computing the Mahalanobis Distance (Dodge 2008) or the likelihood the sample with respect to the distribution.
35


Reviews:
[{"review_id": "VqklKYDJRh", "reviewer": "Reviewer_cX2Z", "summary": "This paper addresses the issues present in currently popular evaluation metrics in the field of video generation by proposing a new evaluation metric. It points out the assumptions underlying existing evaluation methods and the problems associated with these assumptions. The newly proposed metric shows better performance in terms of cosine similarity with human evaluations and demonstrates significant improvement in the number of samples needed to converge.", "questions": "Please refer to the Weakness section. \nAdditionally, the paper provides limited explanation of the proposed evaluation metric, JEDi. It might be beneficial to reduce the amount of reiteration about other evaluation metrics and instead increase the logical exposition of JEDi to provide a more comprehensive understanding of its strengths and limitations.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper comprehensively considers the strengths and limitations of existing metrics and systematically reviews the assumptions underlying these metrics, along with the problems associated with these assumptions. It delves into the accuracy of The Fréchet distance (FD) and its consequences when these assumptions do not hold. Building on this foundation, the paper addresses some of the issues present in current evaluation strategies. The new evaluation strategy proposed shows a significant improvement in consistency with Cosine Similarity with Human Evaluation.", "weaknesses": "The paper emphasizes that JEDi, as a video generation evaluation metric, has an advantage in the number of samples required for convergence, achieving faster convergence. However, the number of samples required for convergence should not be considered the primary concern of an evaluation strategy, as not all datasets face the issue of insufficient samples affecting precision; there are still many datasets that meet the necessary conditions. While JEDi performs well in consistency with human evaluations, further experimental validation is needed to assess its performance in other aspects as an evaluation metric.\nAlthough the author said in the experiment that the resolution can be arbitrary, since this work relies on the representation capability of SreamDiffusion[3] based on SD-Turbo or LCM, many \"high-resolution\" images are not included in the training set of SD-Turbo or LCM. Can Promptus effectively transmit videos with various resolutions and high resolutions that it has not seen during SD training?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cpho2KOaZa", "reviewer": "Reviewer_goxo", "summary": "This paper reveals that FVD falls short as a standalone metric for video generation evaluation, then analyzes a wide range of metrics and backbone architectures, and proposes JEDi, the JEPA Embedding Distance, based on features derived from a Joint Embedding Predictive Architecture, measured using Maximum Mean Discrepancy with polynomial kernel.", "questions": "Following the above weaknesses, some questions are suggested to be answered.\n1.\tAs we known, video generation quality covers many aspects, thus the authors are suggested to specify which aspect of quality this paper focuses on.\n2.\tThe possibility of applications on single video quality evaluation should be discussed.\n3.\tThe are many studies for single generated image or video quality assessment studies in the image/video quality assessment communities.\n4.\tIn some open AIGC video quality assessment databases, both generated videos and human labels are given, for example the Text-to-Video Quality Assessment DataBase (T2VQA-DB), Large-scale Generated Vdeo Quality assessment (LGVQ) dataset, Generic AI-generated Action (GAIA) dataset. The alignment with human ratings on these databases are suggested to be tested.\n5.\tFollowing the above comment, the performance comparisons with these AIGC video quality models are suggested to be given.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed JEDi eliminates the need for parametric assumptions about the underlying video distribution, unlike FVD which relies on the Gaussianity assumption to make its metric feasible.\nJEDi significantly reduces the number of samples needed to make an accurate estimate.\nJEDi has better alignment with human evaluations compared to FVD.", "weaknesses": "1.\tThe tile of the paper is relatively too broad, as video generation quality as well as its evaluation cover many aspects, however JEDi can only cover an aspect of quality.\n2.\tThe proposed JEDi is a distribution-based evaluation metric, which can not work for a single case, for example to evaluate the quality of a single generated video.\n3.\tThe single video AIGC video quality assessment should also be discussed and compared, including the state-of-the-arts and its differences and similarities with the distribution metrics.\n4.\tThe authors conduct a subjective evaluation which is a very small scale study. More evaluations on existing large-scale AIGC video quality databases should be conducted.\n5.\tMore comparisons with the single AIGC video quality assessment metrics are suggested to be included.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZKXQuUgnKO", "reviewer": "Reviewer_NmJ7", "summary": "This work analyzes the limitations of FVD and proposes the JEPA Embedding Distance based on features derived from a Joint Embedding Predictive Architecture. The authors commit to develop effective, robust metrics with small sampling size for guiding the current surge in video generation research. The experiments on open-source datasets show that compared to the widely used FVD metric, the proposed method requiring only 16% of the samples to reach its steady value, while increasing alignment with human evaluation by 34%, on average.", "questions": "see weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. The authors note three key hurdles in video generation: (1) Data size, (2) Computational resources, and (3) Metric convergence speed, and commence by developing metrics with higher sample efficiency with fast convergence.\n2. The authors investigate the correlation between metric with distortion level and training duration and gain valuable observations to further improve the FVD metric.", "weaknesses": "1. The content arrangement is not sufficiently appropriate. For example, Section 2.1 and 2.2 introduce a lot of consensus information about Inflated 3D ConvNet, video masked autoencoder, and the definition of Fr ́echet Distance, which should be moved to Appendix. In contrast, more related work about the evaluation metrics for video generation should be reviewed [1, 2]. \n2. The organization of Section 1 should be improved to enhance its readability. Line53: a video generation model …… produce ….. with diverse features? The two examples in Line53-57 are limited in scenarios. Line 59 - 60 only mentioned some full-reference video quality metrics,  however, lack of the introduction of no-reference video quality assessment (VQA) metrics ([1, 2]), which are more practical in real-world applications.\n3. Fig. 3 is difficult to interpret without zooming in. Improving the resolution, providing clearer annotations or changing the scale could make the results more convincing.\n4. What is the meaning of the items in red font in Table 1? Lack of explanation. And what is the meaning of blur (low, mid, high)? Are they mean the distortion Level? These should be noted in the Table caption.\n5. The experiment is not comprehensive enough. Only two I2V models and three noise distortion types are included. I recommend that the authors add more image-to-video models for evaluation, both proprietary and open source, to support your findings. Since there are low-level distortion (blur, noise, artifacts) and high-level distortion (semantic, composition) in generated videos, it is necessary to evaluate the proposed JEDi in these scenarios.\n6. The details of human evaluation in Section5.4 are vague. Since the normativity of the subjective study largely affect the reliability of these results. The content in App. E.3 is empty.\n7. Line378: I3D and VideoMAEPT are not ideal feature spaces for building video quality metrics, as they do not capture blur distortion well. How about other distortions?\n8. The equations in App. B should be better arranged to increase their readability.\n\n[1] Ghildyal, Abhijay, et al. \"Quality Prediction of AI Generated Images and Videos: Emerging Trends and Opportunities.\" *arXiv preprint arXiv:2410.08534* (2024).\n\n[2] Min, Xiongkuo, et al. \"Perceptual video quality assessment: A survey.\" *arXiv preprint arXiv:2402.03413* (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "w2hPKFWhQt", "reviewer": "Reviewer_gxbR", "summary": "This paper provides critical analysis about FVD, which is the commonly used quality evaluation metric for generated videos. Three major limitations of FVD are discussed: 1) over-reliance on Gaussianity assumptions, 2) high-dimensional feature spaces, and 3) low sample efficiency. Furthermore, this paper proposes a new metric named JEDi, which computes Maximum Mean Discrepancy (MMD) in a V-JEPA feature space. Extensive experiments are conducted and the results demonstrate its robustness and effectiveness in terms of distortion-sensitivity, high sample efficiency, and high correlation with human ratings.", "questions": "Can the monotonicity with training iterations in Fig. 5 imply the accuracy as well? Fig.5 only demonstrate the better monotonicity of JEDi during training over that of FVD and other metrics, yet it can not tell the superiority of quality evaluation accuracy of JEDi. Refer to the commonly used index, SROCC, PLCC, and RMSE, for measuring the performance of quality assessment metrics.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe limitations of FVD are systematically identified and analyzed, especially regarding the non-Gaussianity of the I3D feature space and the low sample efficiency. \n2.\tFindings and insights derived from comprehensive comparison experiments are interesting and convincing, such as the importance comparison of feature space and distance metrics, and the distortion-aware feature space selection.\n3.\tA well-justified metric name JEDi is proposed that addresses the main issues in FVD. The methodology is clearly explained, and the effectiveness of JEDi is well verified through extensive experiments.", "weaknesses": "1. Due to the limitations of the small-scale human evaluation, it would be better to evaluate the quality evaluation performance on large-scale video quality assessment datasets of generated content. In the human evaluation, only comparison ratings as the human preference are provided, lacking the overall quality ratings which can be obtained by single stimulus. Since JEDi is targeting quality evaluation for generated videos where the quality is practically measured without reference, i.e., no-reference quality assessment, it would be more comprehensive to evaluate JEDi on large-scale open-source video quality databases of AI-generated content, such as T2VQA-DB [1] and GAIA [2].  \n2. Lacking of reviewing and analyzing relevant literatures. Similar to FID ant its variants, NIQE [3] evaluates the quality by measuring distribution distance in the feature space. Targeting measure quality of non-generated images, NIQE extracts neurostatistical features, fit the features with Multivariate Gaussian Model, and compute distribution distance by Mahalanobis Distance. It would be very insightful to include the Mahalanobis Distance into the discussion of the effective distance metrics, and to analyze the difference of algorithm design among FID/FVD, NIQE, and JEDi. Quantitive experiments are expected to provide holistic analysis.\n\n[1] GAIA: Rethinking Action Quality Assessment for AI-Generated Videos, arXiv 2024\n[2] Subjective-Aligned Dateset and Metric for Text-to-Video Quality Assessment, arXiv 2024\n[3] Making a “completely blind” image quality analyzer[J]. IEEE Signal processing letters, 2012", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "npUy7FkIMp", "reviewer": "Reviewer_uZfY", "summary": "The paper makes a strong empirical case for the JEDi metric, which is based on features derived from a Joint Embedding Predictive Architecture, measured using Maximum Mean Discrepancy with polynomial kernel, as a superior alternative to FVD in evaluating video generation models.", "questions": "See the weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Interesting metric. The paper introduces the JEDi , a new metric for evaluating video generation quality, addressing several limitations of the existing FVD. JEDi is shown to improve alignment with human evaluations by 34%, which is a significant improvement.\n\n2. Clear Identification of FVD’s Limitations. The authors thoroughly analyze the shortcomings of FVD, including:\n    a.Non-Gaussianity in feature space.\n    b. Insensitivity to temporal distortions.\n    c.The impractically large sample sizes required for reliable metric estimation.\n\nThese are important challenges in the field of video generation, and addressing them demonstrates the paper's relevance.\n\n3. Good Efficiency. One of the key claims is that JEDi requires only 16% of the samples compared to FVD to reach its steady value, which could make it a more practical metric for real-world applications.", "weaknesses": "First I want to declare that I am not very familiar with this area and I am going to summarize some weaknesses that matter to me.\n1. Missing Analysis on Long-Term Temporal Distortions. Although the paper mentions the insensitivity of FVD to temporal distortions, the experiments primarily focus on shorter-term consistency and aesthetics. It would be beneficial to see deeper analysis on how JEDi handles long-term temporal coherence in video.\n\n2. Issues About Human Evaluation.  Are these human subjects trained and qualified to evaluate video quality? Do the randomly selected videos adequately represent the entire set, and is there sufficient diversity? Has there been consideration of increasing the number of evaluation videos and including more subjects to ensure the reliability of the conclusions?\n\n3.  In 5.1 section, how is the degree of blur determined, and are there any objective metrics for it? For the detection and assessment of noise and blur, many no-reference quality evaluation algorithms can perform this task. It is recommended to include some quality evaluation metrics for comparison, such as NIQE and Q-Align.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)