Beyond FVD: An Enhanced Evaluation Metrics for Video Generation Distribution Quality

Authors: Ge Ya Luo, Gian Mario Favero, ZhiHao Luo, Alexia Jolicoeur-Martineau, Christopher Pal
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: Video quality metrics, Frechet Video Distance, Inflated 3D Convnet, VideoMAE, VJEPA, kernel metrics

Abstract:
The Fréchet Video Distance (FVD) is a widely adopted metric for evaluating video generation distribution quality. However, its effectiveness relies on critical assumptions. Our analysis reveals three significant limitations: (1) the non-Gaussianity of the Inflated 3D Convnet (I3D) feature space; (2) the insensitivity of I3D features to temporal distortions; (3) the impractical sample sizes required for reliable estimation. These findings undermine FVD's reliability and show that FVD falls short as a standalone metric for video generation evaluation. After extensive analysis of a wide range of metrics and backbone architectures, we propose JEDi, the JEPA Embedding Distance, 
based on features derived from a Joint Embedding Predictive Architecture, measured using Maximum Mean Discrepancy with polynomial kernel.  Our experiments on multiple open-source datasets show clear evidence that it is a superior alternative to the widely used FVD metric, requiring only 16% of the samples to reach its steady value, while increasing alignment with human evaluation by 34%, on average.

Project page: https://oooolga.github.io/JEDi.github.io/.

Decision: accept (poster)
Reviewer scores: 3, 8, 6, 8, 6

Reviews and discussion:
<thread>
creation date:
1729062921238

thread_type:
official_review

<thread_object>
creator: reviewer_uzfy

<content>
summary:
The paper makes a strong empirical case for the JEDi metric, which is based on features derived from a Joint Embedding Predictive Architecture, measured using Maximum Mean Discrepancy with polynomial kernel, as a superior alternative to FVD in evaluating video generation models.

strengths:
1. Interesting metric. The paper introduces the JEDi , a new metric for evaluating video generation quality, addressing several limitations of the existing FVD. JEDi is shown to improve alignment with human evaluations by 34%, which is a significant improvement.

2. Clear Identification of FVD’s Limitations. The authors thoroughly analyze the shortcomings of FVD, including:
    a.Non-Gaussianity in feature space.
    b. Insensitivity to temporal distortions.
    c.The impractically large sample sizes required for reliable metric estimation.

These are important challenges in the field of video generation, and addressing them demonstrates the paper's relevance.

3. Good Efficiency. One of the key claims is that JEDi requires only 16% of the samples compared to FVD to reach its steady value, which could make it a more practical metric for real-world applications.

weaknesses:
First I want to declare that I am not very familiar with this area and I am going to summarize some weaknesses that matter to me.
1. Missing Analysis on Long-Term Temporal Distortions. Although the paper mentions the insensitivity of FVD to temporal distortions, the experiments primarily focus on shorter-term consistency and aesthetics. It would be beneficial to see deeper analysis on how JEDi handles long-term temporal coherence in video.

2. Issues About Human Evaluation.  Are these human subjects trained and qualified to evaluate video quality? Do the randomly selected videos adequately represent the entire set, and is there sufficient diversity? Has there been consideration of increasing the number of evaluation videos and including more subjects to ensure the reliability of the conclusions?

3.  In 5.1 section, how is the degree of blur determined, and are there any objective metrics for it? For the detection and assessment of noise and blur, many no-reference quality evaluation algorithms can perform this task. It is recommended to include some quality evaluation metrics for comparison, such as NIQE and Q-Align.

questions:
See the weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1729575230164

thread_type:
official_review

<thread_object>
creator: reviewer_gxbr

<content>
summary:
This paper provides critical analysis about FVD, which is the commonly used quality evaluation metric for generated videos. Three major limitations of FVD are discussed: 1) over-reliance on Gaussianity assumptions, 2) high-dimensional feature spaces, and 3) low sample efficiency. Furthermore, this paper proposes a new metric named JEDi, which computes Maximum Mean Discrepancy (MMD) in a V-JEPA feature space. Extensive experiments are conducted and the results demonstrate its robustness and effectiveness in terms of distortion-sensitivity, high sample efficiency, and high correlation with human ratings.

strengths:
1.	The limitations of FVD are systematically identified and analyzed, especially regarding the non-Gaussianity of the I3D feature space and the low sample efficiency. 
2.	Findings and insights derived from comprehensive comparison experiments are interesting and convincing, such as the importance comparison of feature space and distance metrics, and the distortion-aware feature space selection.
3.	A well-justified metric name JEDi is proposed that addresses the main issues in FVD. The methodology is clearly explained, and the effectiveness of JEDi is well verified through extensive experiments.

weaknesses:
1. Due to the limitations of the small-scale human evaluation, it would be better to evaluate the quality evaluation performance on large-scale video quality assessment datasets of generated content. In the human evaluation, only comparison ratings as the human preference are provided, lacking the overall quality ratings which can be obtained by single stimulus. Since JEDi is targeting quality evaluation for generated videos where the quality is practically measured without reference, i.e., no-reference quality assessment, it would be more comprehensive to evaluate JEDi on large-scale open-source video quality databases of AI-generated content, such as T2VQA-DB [1] and GAIA [2].  
2. Lacking of reviewing and analyzing relevant literatures. Similar to FID ant its variants, NIQE [3] evaluates the quality by measuring distribution distance in the feature space. Targeting measure quality of non-generated images, NIQE extracts neurostatistical features, fit the features with Multivariate Gaussian Model, and compute distribution distance by Mahalanobis Distance. It would be very insightful to include the Mahalanobis Distance into the discussion of the effective distance metrics, and to analyze the difference of algorithm design among FID/FVD, NIQE, and JEDi. Quantitive experiments are expected to provide holistic analysis.

[1] GAIA: Rethinking Action Quality Assessment for AI-Generated Videos, arXiv 2024
[2] Subjective-Aligned Dateset and Metric for Text-to-Video Quality Assessment, arXiv 2024
[3] Making a “completely blind” image quality analyzer[J]. IEEE Signal processing letters, 2012

questions:
Can the monotonicity with training iterations in Fig. 5 imply the accuracy as well? Fig.5 only demonstrate the better monotonicity of JEDi during training over that of FVD and other metrics, yet it can not tell the superiority of quality evaluation accuracy of JEDi. Refer to the commonly used index, SROCC, PLCC, and RMSE, for measuring the performance of quality assessment metrics.
</content>
</thread_object>

</thread>
<thread>
creation date:
1729680244186

thread_type:
official_review

<thread_object>
creator: reviewer_nmj7

<content>
summary:
This work analyzes the limitations of FVD and proposes the JEPA Embedding Distance based on features derived from a Joint Embedding Predictive Architecture. The authors commit to develop effective, robust metrics with small sampling size for guiding the current surge in video generation research. The experiments on open-source datasets show that compared to the widely used FVD metric, the proposed method requiring only 16% of the samples to reach its steady value, while increasing alignment with human evaluation by 34%, on average.

strengths:
1. The authors note three key hurdles in video generation: (1) Data size, (2) Computational resources, and (3) Metric convergence speed, and commence by developing metrics with higher sample efficiency with fast convergence.
2. The authors investigate the correlation between metric with distortion level and training duration and gain valuable observations to further improve the FVD metric.

weaknesses:
1. The content arrangement is not sufficiently appropriate. For example, Section 2.1 and 2.2 introduce a lot of consensus information about Inflated 3D ConvNet, video masked autoencoder, and the definition of Fr ́echet Distance, which should be moved to Appendix. In contrast, more related work about the evaluation metrics for video generation should be reviewed [1, 2]. 
2. The organization of Section 1 should be improved to enhance its readability. Line53: a video generation model …… produce ….. with diverse features? The two examples in Line53-57 are limited in scenarios. Line 59 - 60 only mentioned some full-reference video quality metrics,  however, lack of the introduction of no-reference video quality assessment (VQA) metrics ([1, 2]), which are more practical in real-world applications.
3. Fig. 3 is difficult to interpret without zooming in. Improving the resolution, providing clearer annotations or changing the scale could make the results more convincing.
4. What is the meaning of the items in red font in Table 1? Lack of explanation. And what is the meaning of blur (low, mid, high)? Are they mean the distortion Level? These should be noted in the Table caption.
5. The experiment is not comprehensive enough. Only two I2V models and three noise distortion types are included. I recommend that the authors add more image-to-video models for evaluation, both proprietary and open source, to support your findings. Since there are low-level distortion (blur, noise, artifacts) and high-level distortion (semantic, composition) in generated videos, it is necessary to evaluate the proposed JEDi in these scenarios.
6. The details of human evaluation in Section5.4 are vague. Since the normativity of the subjective study largely affect the reliability of these results. The content in App. E.3 is empty.
7. Line378: I3D and VideoMAEPT are not ideal feature spaces for building video quality metrics, as they do not capture blur distortion well. How about other distortions?
8. The equations in App. B should be better arranged to increase their readability.

[1] Ghildyal, Abhijay, et al. "Quality Prediction of AI Generated Images and Videos: Emerging Trends and Opportunities." *arXiv preprint arXiv:2410.08534* (2024).

[2] Min, Xiongkuo, et al. "Perceptual video quality assessment: A survey." *arXiv preprint arXiv:2402.03413* (2024).

questions:
see weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730623099022

thread_type:
official_review

<thread_object>
creator: reviewer_goxo

<content>
summary:
This paper reveals that FVD falls short as a standalone metric for video generation evaluation, then analyzes a wide range of metrics and backbone architectures, and proposes JEDi, the JEPA Embedding Distance, based on features derived from a Joint Embedding Predictive Architecture, measured using Maximum Mean Discrepancy with polynomial kernel.

strengths:
The proposed JEDi eliminates the need for parametric assumptions about the underlying video distribution, unlike FVD which relies on the Gaussianity assumption to make its metric feasible.
JEDi significantly reduces the number of samples needed to make an accurate estimate.
JEDi has better alignment with human evaluations compared to FVD.

weaknesses:
1.	The tile of the paper is relatively too broad, as video generation quality as well as its evaluation cover many aspects, however JEDi can only cover an aspect of quality.
2.	The proposed JEDi is a distribution-based evaluation metric, which can not work for a single case, for example to evaluate the quality of a single generated video.
3.	The single video AIGC video quality assessment should also be discussed and compared, including the state-of-the-arts and its differences and similarities with the distribution metrics.
4.	The authors conduct a subjective evaluation which is a very small scale study. More evaluations on existing large-scale AIGC video quality databases should be conducted.
5.	More comparisons with the single AIGC video quality assessment metrics are suggested to be included.

questions:
Following the above weaknesses, some questions are suggested to be answered.
1.	As we known, video generation quality covers many aspects, thus the authors are suggested to specify which aspect of quality this paper focuses on.
2.	The possibility of applications on single video quality evaluation should be discussed.
3.	The are many studies for single generated image or video quality assessment studies in the image/video quality assessment communities.
4.	In some open AIGC video quality assessment databases, both generated videos and human labels are given, for example the Text-to-Video Quality Assessment DataBase (T2VQA-DB), Large-scale Generated Vdeo Quality assessment (LGVQ) dataset, Generic AI-generated Action (GAIA) dataset. The alignment with human ratings on these databases are suggested to be tested.
5.	Following the above comment, the performance comparisons with these AIGC video quality models are suggested to be given.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730725183487

thread_type:
official_review

<thread_object>
creator: reviewer_cx2z

<content>
summary:
This paper addresses the issues present in currently popular evaluation metrics in the field of video generation by proposing a new evaluation metric. It points out the assumptions underlying existing evaluation methods and the problems associated with these assumptions. The newly proposed metric shows better performance in terms of cosine similarity with human evaluations and demonstrates significant improvement in the number of samples needed to converge.

strengths:
The paper comprehensively considers the strengths and limitations of existing metrics and systematically reviews the assumptions underlying these metrics, along with the problems associated with these assumptions. It delves into the accuracy of The Fréchet distance (FD) and its consequences when these assumptions do not hold. Building on this foundation, the paper addresses some of the issues present in current evaluation strategies. The new evaluation strategy proposed shows a significant improvement in consistency with Cosine Similarity with Human Evaluation.

weaknesses:
The paper emphasizes that JEDi, as a video generation evaluation metric, has an advantage in the number of samples required for convergence, achieving faster convergence. However, the number of samples required for convergence should not be considered the primary concern of an evaluation strategy, as not all datasets face the issue of insufficient samples affecting precision; there are still many datasets that meet the necessary conditions. While JEDi performs well in consistency with human evaluations, further experimental validation is needed to assess its performance in other aspects as an evaluation metric.
Although the author said in the experiment that the resolution can be arbitrary, since this work relies on the representation capability of SreamDiffusion[3] based on SD-Turbo or LCM, many "high-resolution" images are not included in the training set of SD-Turbo or LCM. Can Promptus effectively transmit videos with various resolutions and high resolutions that it has not seen during SD training?

questions:
Please refer to the Weakness section. 
Additionally, the paper provides limited explanation of the proposed evaluation metric, JEDi. It might be beneficial to reduce the amount of reiteration about other evaluation metrics and instead increase the logical exposition of JEDi to provide a more comprehensive understanding of its strengths and limitations.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732508609087

thread_type:
official_comment

<thread_object>
creator: area_chair_eeel

<content>
comment:
Hi Reviewers,

We are approaching the deadline for author-reviewer discussion phase. Authors has already provided their rebuttal. In case you haven't checked them, please look at them ASAP. Thanks a million for your help!
</content>
</thread_object>

</thread>
<thread>
creation date:
1732595249416

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Rebuttal Summary

comment:
Dear Reviewers,

We would like to extend our sincerest gratitude for your collaborative efforts in refining our work. Your constructive feedback has been invaluable, and we are confident that it has significantly enhanced the quality of our work.

**Below is a summary of the modifications made during the rebuttal period:**

### New experiments
1) STUDY ON ARTIFICIAL INTELLIGENCE GENERATED CONTENT: T2VQA-DB
    - This study utilizes the T2VQA database, comprising 10,000 AI-generated videos produced by 10 distinct Text-to-Video (T2V) AI models. Each video is associated with a human-rated Mean Opinion Score (MOS) that evaluates text-video alignment and fidelity.
We conducted an assessment of three T2V models from the T2VQA-DB dataset, representing a range of video qualities. Our analysis revealed that both FVD and JEDi scores consistently ranked these models in accordance with their MOS scores. Notably, JEDi demonstrated a significantly better sample efficiency rate.
    - *Section 5.5, Pg 9-10*
    - Suggested by Reviewer `goxo`, `NmJ7`, `gxbR`
2) STUDY ON ARTIFICIAL INTELLIGENCE GENERATED CONTENT: TIP-I2V
    - This study investigates the outputs of three state-of-the-art video generation models and a real video dataset’s samples within the FVD and JEDi embedding spaces. Our analysis reveals distinct distributional patterns in the features extracted by V-JEPA (used in JEDi), whereas features extracted by I3D (used in FVD) do not exhibit such clear distinctions. These findings suggest that V-JEPA's feature space is more suitable for video evaluation
    - *Appendix H, Pg 33*
    - Suggested by Reviewer `goxo`
3) LONG-TERM VIDEO STUDY
    - We investigate the impact of clip length (16, 32, 64, 128 frames) on metric performance. Our findings show that the JEDi metric is sensitive only to the level of distortion, providing consistent evaluations across different clip lengths under the same distortion. In contrast, FVD is influenced by clip length, with distributional distances increasing as the clip length grows, even when the distortion level remains unchanged.
    - *Appendix I, Pg 34*
    - Suggested by Reviewer `uZfY`

### New Discussions
1) Study on Artificial Intelligence Generated Content: *Section 5.5, Pg 9-10* [`goxo`, `NmJ7`, `gxbR`]
2) Computation Configuration: *Appendix B, Pg 17-18*
3) Human Study Demographics: *Appendix F.3, Pg 29* [`uZfY`, `NmJ7`, `goxo`]
4) Survey on Human Studies in Previous Literature: *Appendix F.4, Pg 30-32* [`NmJ7`, `goxo`]
5) Avenues for Future Researches (Implication for Single Video Assessment Metrics): *Appendix J.2, Pg 34-35* [`goxo`, `gxbR`]

### Revised Figures
We have fixed *Figure 2, 3, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19* suggested by Reviewer `NmJ7`

Thank you again for your insightful feedback and helpful suggestions. Your input has greatly improved our document, and we appreciate this opportunity to work with you on our paper's rebuttal.

Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper finds limitations of FVD for evaluating video generation distribution quality and proposes JEDi based on features derived from a Joint embedding predictive architecture, measured using maximum mean discrepancy with polynomial kernel. Experiments on several datasets show the proposed metrics is better than FVD.

This paper got 5 reviews with mixed scores, i.e., 2 accept, 2 marginally above the acceptance threshold and 1 reject. 

The strength of this paper are: 1) proposed JEDi eliminates the need for parametric assumptions about the underlying video distribution; 2) proposed methods significantly reduces the number of samples needed; 3) better aligned with human evaluations; 4) limitations of FVD are systematically identified; 5) experimental results are solid. 

Weaknesses are: 1) proposed JEDi is a distribution-based evaluation metric, can not work for evaluating a single generated video; 2) more evaluation on existing large-scale AIGC video quality datasets should be conducted; 3) more comparisons with the single AIGC video quality assessment metrics are suggested to be included; 4) paper structure is not good; 5) need more experimental results; 6) missing analysis on long-term temporal distortions; 

In the rebuttal, Reviewer goxo gave score 8 and didn't reply. Reviewer NmJ7 thought the authors addressed their concerns quite well and changed rating to 8. Reviewer gxbR also said their concerns are addressed and changed score to 8. Reviewer uZfY maintained positive rating 6. Reviewer cX2Z’s gave score 3 and didn't reply. AC checked reviewer cX2Z's comments and author's rebuttal and did find some parts are irrelevant. The rest are addressed by authors. Given these, AC decided to accept this paper.