Fréchet Wavelet Distance: A Domain-Agnostic Metric for Image Generation

Authors: Lokesh Veeramacheneni, Moritz Wolter, Hilde Kuehne, Juergen Gall
Primary area: generative models
Keywords: Frechet Distance, Wavelet Packet Transform, Frechet Inception Distance, Diffusion, GAN, ImageNet, Frechet Inception Distance, FD-DINOv2

Abstract:
Modern metrics for generative learning like Fréchet Inception Distance (FID) and DINOv2-Fréchet Distance (FD-DINOv2) demonstrate impressive performance. However, they suffer from various shortcomings, like a bias towards specific generators and datasets. To address this problem, we propose the Fréchet Wavelet Distance (FWD) as a domain-agnostic metric based on the Wavelet Packet Transform ($\mathcal{W}_p$). FWD provides a sight across a broad spectrum of frequencies in images with a high resolution, preserving both spatial and textural aspects. Specifically, we use $\mathcal{W}_p$ to project generated and real images to the packet coefficient space. We then compute the Fréchet distance with the resultant coefficients to evaluate the quality of a generator. This metric is general-purpose and dataset-domain agnostic, as it does not rely on any pre-trained network, while being more interpretable due to its ability to compute Fréchet distance per packet, enhancing transparency. We conclude with an extensive evaluation of a wide variety of generators across various datasets that the proposed FWD can generalize and improve robustness to domain shifts and various corruptions compared to other metrics.

Decision: accept (poster)
Reviewer scores: 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730275059271

thread_type:
official_review

<thread_object>
creator: reviewer_jm3n

<content>
summary:
This paper claims that FID/FD-DINOv2 suffer from the bias of generator and dataset, and proposes the FWD, a domain-agnostic metric based on the Wavelet Packet Transform. The FWD does not rely on any pre-trained network, and thus is general-purpose and dataset-domain agnostic. The author further claims this new metric is more interpretable due to its ability to compute Fréchet distance per packet.

strengths:
1. The proposed solution for enhancing FID is attractive, and the FWD is designed to be general-purpose and dataset-domain agnostic, with a solid foundation.
2. The experiments are well-considered.

weaknesses:
1. The paper appears to be missing comparisons with several competitors. While the FWD is compared with FID/FD-DINOv2, this seems insufficient. For instance, metrics such as IS (Inception Score), KID (Kernel Inception Distance), FID_\infty, IS_\infty, and Clean FID should also be considered. The authors should either include comparisons with these metrics or provide an explanation for their omission. Notably, FID_\infty and IS_\infty are designed to mitigate biases introduced by models, making their inclusion particularly relevant.

2. There is a lack of human evaluation. For a metric evaluating generative models, it would be beneficial to demonstrate alignment with human evaluation results.

3. The underlying mechanism is not entirely clear. Despite the authors’ claim that FWD offers improved transparency, it remains unclear why a hand-designed metric could yield better evaluation results than those based on deep learning techniques. Or you can also explain it.

4. The dataset selection is limited. It appears that all experiments are conducted on just four datasets (face, agricultural, and remote sensing datasets). Including a broader range of datasets, particularly natural images from sources like ImageNet, OpenImage, COCO, or Laion5B, would enhance the robustness of the findings. Or you can also explain it.

[1] Mikołaj Binkowski, Danica J Sutherland, Michael Arbel, and Arthur Gretton. Demystifying mmd gans. arXiv preprint arXiv:1801.01401, 2018.
[2] Min Jin Chong and David Forsyth. Effectively unbiased fid and inception score and where to find them. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6070–6079, 2020.
[3] Gaurav Parmar, Richard Zhang, and Jun-Yan Zhu. On aliased resizing and surprising subtleties in gan evaluation, 2022.

questions:
See weaknesses. As I am not an expert in this field, I would like to adjust my rating if all the reviewers’ concerns are addressed.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730547892514

thread_type:
official_review

<thread_object>
creator: reviewer_whkw

<content>
summary:
Modern generative models exhibit frequency biases, while commonly used metrics such as FID , KID and FD-DINOv2 are affected by domain bias.  
This paper proposes the Fréchet Wavelet Distance ( FWD ) as a dataset- and domain-agnostic metric for evaluation of generative approaches for image synthesis.    
It is  shown that the proposed method is robust to corruption, perturbation, and distractors.  
At the same time, its formulation is computationally efficient.

strengths:
The proposed metric Fréchet Wavelet Distance ( FWD ) is simple and has no trained parameters, thus it has high efficiency and is robust to domain, corruption, perturbation, and distractors.    

The calculation of FWD is much faster than FID and FD-DINOv2.  

On several dataset, FWD produces more reasonable results than FID and FD-DINOv2.  

Because of the packet and frequency design, FWD has some interpretability.

weaknesses:
To make the new FWD metric more convincing, a human subject experiment is necessay, as is conducted in the FD-DINOv2 paper. The human evaluation results in FD-DINOv2 paper might be a useful benchmark.

questions:
Many image quality metirc has been proposed for generative model evaluation, but to my knowledge, FID is still the mostly adopted one. I want to hear the authors opinion on this problem. Will evaluating on SOTA generative models like stable diffusion series help the advocation of a new image quality metric?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730612456583

thread_type:
official_review

<thread_object>
creator: reviewer_sdd9

<content>
summary:
The paper, "Fréchet Wavelet Distance: A Domain-Agnostic Metric for Image Generation", introduces the Fréchet Wavelet Distance (FWD) as a novel metric for evaluating generative models in a domain-agnostic way. FWD leverages the Wavelet Packet Transform (Wp) to project images into a frequency and spatially enriched domain, capturing both the frequency and spatial characteristics of images. This approach enables FWD to evaluate the similarity between generated and real images based on the Fréchet Distance (FD) of their wavelet packet coefficients, providing robustness against domain bias and dataset dependency issues seen in other metrics, such as FID and FD-DINOv2. The authors perform extensive evaluations across various datasets and demonstrate that FWD achieves consistent, domain-agnostic results and outperforms state-of-the-art metrics in terms of robustness and computational efficiency.

strengths:
1. Domain Independence and Robustness: FWD’s reliance on the wavelet transform offers a substantial improvement in domain-agnostic evaluation, avoiding the biases associated with pre-trained models like InceptionV3. This makes it a more universally applicable metric across datasets and model types.
2. Efficient and Scalable: FWD is computationally efficient, with much lower FLOPs compared to FD-DINOv2, making it suitable for large-scale evaluation. The method’s wavelet packet decomposition also enables interpretable results by isolating specific frequency bands for detailed analysis.

weaknesses:
1. Limited Comparison with Alternative Frequency-Based Metrics: Although the paper presents strong results, it lacks comparisons with alternative frequency-based metrics, such as those leveraging Fourier transforms or spectral analysis. This comparison could clarify the specific advantages of wavelet-based decomposition over other spectral methods in generative evaluation.
2. Potential Alignment with Human Evaluation: FID score has been long complaint against that it can not accurately evaluate the quality of generative models, specifically alignment with human preferences. This paper presents some improvement over FID scores on several GAN and Diffusion models, but a wider range of ablations and even human evaluations could better support the validity of this approach.

questions:
Please refer to the weakness part
</content>
</thread_object>

</thread>
<thread>
creation date:
1732019263518

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Overall rebuttal response

comment:
We appreciate the reviewers' constructive comments and valuable feedback. We are delighted to read that reviewers found our work novel (sdD9), efficient (sdD9, whKW), domain-agnostic (sdD9, Jm3n, whKw), robust (sdD9, whKW), interpretable (whKw, sdD9), with a solid foundation (Jm3n) and sound experimentation (Jm3n). Naturally, our reviewers also voice some concerns. We address the key points here.

First, we clarify the effect of pre-trained datasets (ImageNet) in the evaluation metrics of generative models. Generators utilizing ImageNet backbones often introduce unintended ImageNet-specific bias into the evaluation. Consistent with prior work [Kynkäänniemi et al., 2023], we confirm the domain bias in evaluating with FID and further show it also affects FD-DINOv2 (Table-1). To address this issue, we propose Frechet Wavelet Distance (FWD), a novel metric grounded by mathematical foundations of Wavelet Packet Transform (WPT). We demonstrate that FWD is free from domain bias problems.

To further substantiate our claims and also address the key concerns of reviewers, we conducted a user study involving around 50 participants and collected approximately 1000 responses. Following the user study methodology [George et al, 2024], participants were asked to select the more realistic image from a pair consisting of the generated image and the original dataset image. This study was conducted and deployed using Streamlit and Google Cloud Platform. We included the user responses and images in zip format in the paper's supplementary material as evidence.
The table below lists the results along with the Human Error Rates (HER) (higher is better) for each model and corresponding dataset.
||Generator|FID$\downarrow$|FD-DINOv2$\downarrow$|FWD(ours)$\downarrow$|HER$\uparrow$|
|-|-|-|-|-|-|
|CelebAHQ|Proj. FastGAN|**6.358**|685.889|1.388|20.0|
||DDGAN|7.641|**199.761**|**0.408**|**32.5**|
|DNDD|Proj. FastGAN |**4.675**|**171.625**|1.442|50.0|
||DDGAN|26.233|232.884|**1.357**|**57.0**|

The human evaluation asserts the effectiveness of FWD in detecting domain bias. Participants were able to identify unrealistic images generated by Proj. FastGAN at a higher rate than those from DDGAN, this is consistent with FWD's quantitative results across both CelebAHQ and DNDD datasets. In conclusion, our experiments suggest that FWD offers a computationally efficient, fast and reproducible metric and effectively detects domain bias, a prevalent issue in existing metrics. The combination of empirical validation and human evaluation results ensures that FWD addresses a critical gap in generative evaluation metrics.  We believe this work will serve a greater interest in the community, and we will incorporate this user study discussion into the paper at the end of the discussion period.

We hope the rebuttal has sufficiently addressed everyone's concerns. If that's indeed the case at the end of the discussion period, please consider raising your score.


*_References_*

[Kynkäänniemi et al, 2023] Tuomas Kynkäänniemi, Tero Karras, Miika Aittala, Timo Aila, and Jaakko Lehtinen. The role of imagenet classes in fréchet inception distance. _ICLR 2023_.

[George et al., 2024] Stein, George, et al. "Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models." _NeurIPS 2024_.
</content>
</thread_object>

</thread>


Metareview:
metareview:
Summary: The paper presents FWD as a novel, domain-agnostic metric for evaluating images in generative models. It uses the Wavelet Packet Transform and measures similarity between generated and real images based on the Frechet Distance of the wavelet coefficients, effectively addressing the domain bias problem. The proposed FWD metric is computationally efficient as shown by evaluations on various datasets.  

Strength: FWD is a simple, parameter-free metric, efficient to compute metric. It’s also shown to be robust to domain variations. The experimental results support the paper claims

Weakness: The scale of the experiments is rather smaller. Can this be directly used to re-evaluate text-to-image generation leaderboards? Human evaluation is provided only after requested by the reviewers. I encourage authors to include the human evaluation in the main body of the paper, rather than the supplementary. 

Acceptance Reason: It addresses an important problem in image quality evaluation in generative models. FID is studied to be susceptible to biases in the generator or the dataset. This work explores a domain-agnostic and efficient metric, which will benefit the community as an alternative metric, and inspire future research in studying related-metrics at a larger scale.