Fréchet Wavelet Distance: A Domain-Agnostic Metric for Image Generation
Authors: Lokesh Veeramacheneni
Moritz Wolter
Hilde Kuehne
Juergen Gall
Venue: ICLR 2025 Poster (2025)
Keywords: Frechet Distance
Wavelet Packet Transform
Frechet Inception Distance
Diffusion
GAN
ImageNet
Frechet Inception Distance
FD-DINOv2
Primary area: generative models
TL;DR: a novel domain agnostic metric to evaluate generated images.

Abstract: Modern metrics for generative learning like Fréchet Inception Distance (FID) and DINOv2-Fréchet Distance (FD-DINOv2) demonstrate impressive performance. However, they suffer from various shortcomings, like a bias towards specific generators and datasets. To address this problem, we propose the Fréchet Wavelet Distance (FWD) as a domain-agnostic metric based on the Wavelet Packet Transform ($\mathcal{W}_p$). FWD provides a sight across a broad spectrum of frequencies in images with a high resolution, preserving both spatial and textural aspects. Specifically, we use $\mathcal{W}_p$ to project generated and real images to the packet coefficient space. We then compute the Fréchet distance with the resultant coefficients to evaluate the quality of a generator. This metric is general-purpose and dataset-domain agnostic, as it does not rely on any pre-trained network, while being more interpretable due to its ability to compute Fréchet distance per packet, enhancing transparency. We conclude with an extensive evaluation of a wide variety of generators across various datasets that the proposed FWD can generalize and improve robustness to domain shifts and various corruptions compared to other metrics.

Published as a conference paper at ICLR 2025
FRÉCHET WAVELET DISTANCE: A DOMAIN-AGNOSTO METRIC FOR IMAGE GENERATION
Lokesh Veeramacheneni University of Bonn Iveerama@ uni-bonn.de
Moritz Wolter
Hildegard Kuehne
University of Bonn moritz. wolter @uni-bonn.de
University of Tuebingen,
MIT-IBM Watson Al Lab
h.kuehne @ uni-tuebingen.de
Juergen Gall
University of Bonn, Lamarr Institute for Machine Learning and Artificial Intelligence gall @ iai. uni-bonoonee
ABSTRACT
Modern metries for generative learning like Fréchet Inception Distance (FID) and DINOv2-Fréchet Distance (FD-DINOv2) demonstrate impressive performance. However, they suffer from various shortcomings. like bias towards specific gen- erators and datasets. To address this problem, we propose the Fréchet Wavelet Distance (FWD) as (oomain-agnostie metric based on the Wavelet Packet Trans- form (Wp)). FWD provides sight across broad spectrum of frequenniis in images with a high resolution, preserving both spatial and textural aspects. Specifically, we use Wp to project generated and real images to the packet coefficient space. We then compute the Fréchet distance with the resultant coefficients to evaluate the quality of generator. This metric is ceneral-purposss and dataset-doman agnostic, as does not rely on any pre-trained network, while being more interpretable due its ability compute Fréchet distance per packet, enhancing transparency. We conclude with an extensive evaluation of a wide variety of generators across various datasets that the proposed FWD can generalize and improve robustness to domain shifts and various corruptions compared to other metrics.
INTRODUCTION
With the surge of generative neural networks, especially the image domain, it becomes important to assess their performance in robust and reliable way (Heusel et al., 2017a; Binkowski et al., 2018; Salimans et al., 2016; Kynkäänniemi et al., 2019; Stein al., 2023). FID (Heusel al., 2017a) has emerged the de facto standard for comparing generative image synthesis approaches. However, it also shows various shortcomings, such as its reliance on a pre-trained classification backbone, i.c., Inception V3 trained on ImageNet. This, by design, introduces class dependency into FID leading to accidental distortions (Sauer et al., 2021). The FID scores improve if the evaluation set resembles ImageNet or the use of ImageNet pre-trained discriminator pushes the output distribution towards ImageNet, although the image quality remains the same in these cases (Kynkäänniemi et al., 2023).
To address the domain bias problem caused by the use of a pre-trained network, we propose an alternative metric based on the Wavelet Packet Transform (W2).. ). In contrast other pure frequency (Narwaria et al., 2012) or spatial (Wang et al., 2004; Horé & Ziou, 2010) metrics, wavelets have the advantage that they combine both frequency and spatial aspects in one metric. While frequency information is important (Durall et al., 2020; Dzanic et al., 2020; Rahaman et al., 2019; Schwarz et al., 2021; Wolter al., 2022), alone is insufficient to assess the quality of synthesized images without considering additional spatial information. Wavelets are thus ideal representation for metric comparing generative approaches for image synthesis. As FID, FWD utilizes the Fréchet distance of the real and generated set of images Inception V3 activation maps. Instead, it utilizes the wavelet- packet frequency band representations of Wr illustrated in Figures and 3. To this end, first use W, transform every image, where
distance measure, but it is not computed based on

---

Published as a conference paper at ICLR 2025
InceptionV3 Predictions
Packets (Ours)
MSE = 0.05
MSE 0.22
0.12
=
MSE =
MSE = 0.22
MSE = 0.06
Figure I: The first two images depict the same person, while the last image depicts a different person. Intuitively, the first images mean squared error between the images using the penultimate Inception activations or wavelet packets, we observe that the wavelet packets produce low distance for the first two images, as expected. Surprisingly, according to Inception V3, the last two images are similar since both images are classified as 'mi "microphone' whereas the first image as 'groom'. Images from Flickr.
similar than the other pairs of images. When computing the
we use the Haar wavelet transform fixed level. We then compute the Fréchet distance for each packet of the transform and average them over all packets. The proposed Fréchet Wavelet Distance (FWD) thus considers spatial information as well as all frequency bands.
To quantitatively assess those characteristics, we evaluate the proposed metric in terms of its domain bias and robustness. We further compare the proposed FWD existing state-of-the-ar metries like FID, Kernel Inception Distance (KID), and DINOv2-Fréchet Distance (FD-DINOv2) datasets. We show that FWD is more robust metric that does not suffer from the domain bias and can thus be applied to any dataset. Kynkäänniemi et al. (2023) experimented with optimizing FID by selecting a subset of images from 250k generated images, where the subset's InceptionV3 activations related ImageNet classes. Building this work, we observed significant improvement in FID by R 50%. when evaluated on this subset. FD-DINOv2 responded to ImageNet-feature optimization with improvement of 2% as well. This undesired improvement overlap between the ImageNet and the DINOv2 training set. In contrast, FWD remains the same despite the manipulation. We also show that some unexpected FID results be attributed the dataset bias. Furthermore, FWD is significantly faster to compute. The source code for computing FWD available at: https: / github........ com/ BonnBBnee/neeeeeeeee httpsyyttt///yooo--eteeeee
standard
likely be explained by the
In summary, this paper makes the following contributions:
We propose the Fréchet Wavelet Distance (FWD) as a dataset- and domain aagaosic metric for evaluation of generative approaches for image synthesis.
2. FWD is interpretable metric, the Wavelet Packet Transform (Wp) splits the frequency space into hierarchically organized, discrete subbands.
3. We show that the proposed method is computationally inexpensive and robust to corruption, perturbation, and distractors.
4. We show that FD-DINOv2 addresses the domain bias issue to extent but at very high computational cost. Furthermore, we provide evidence that still limited to its training data domain.
2

---

Published as a conference paper at ICLR 2025
liv
da
c
dv
da
Figure 2: Illustration of the Wavelet Packet Transform (Wp)). For visualization purposes, we depict
level-3 transform. All later experiments use a level-4 transform. Image from Jérémy Barande (2024).
2 RELATED WORK
2.1 METRICS FOR GENERATIVE LEARNING
generative model should generate novel image samples that mirror the training set sample distribu- tion, including data diversity. In vision context, Salimans al. (2016) proposed the Inception Score (IS) as measure of image quality, independent of the target dataset statistics. The IS is computed by measuring the entropy of the class probabilities of an InceptionV3. The score builds upon the assumption that a generative network that has converged to a meaningful solution will produce images that will allow Inception V3 make predictions with certainty. In other words, certain Inception V3 has a low prediction entropy. IS has been found to be sensitive to different ImageNet training runs (Barratt & Sharma, 2018). Furthermore, it does not use the statisties of the real data distribution Generative Adversarial neural Network (GAN) is trained to model (Heusel et al., 2017a). Heusel et al. (2017a) proposed FID in response. Instead of measuring the entropy at the final layer, FID is com- puted by evaluating the Fréchet distance (Dowson & Landau, 1982) between the penultimate network activations computed on both the true and synthetic images. Today, comparing high-level Inception features using an FID-score (Heusel et al., 2017a) enjoys widespread adoption and several variants exist. Kernel Inception Distance (KID) (Binkowski et al., 2018), for example, relaxes the multi- variate Gaussian assumption of FID and measures the polynomial kernel distance between Inception features of the generated and the training dataset. Binkowski et al. (2018) kept the Inception V3 backhone and replaced the Fréchet distance with kernel distance. While FID captures general trends well, the literature also discusses its backs. Kynkäänniemi et al. (2023) empirically studied the effect of ImageNet classes FID for non-ImageNet datasets by using GradCAM. Furthermore, Kynkäänniemi et al. (2023) examined ImageNet bias using Projected Fast GAN (Proj. FasIGAN) and StyleGAN2. Compared to StyleGAN2, Proj. FastGAN produces more accidental distortions like floating heads and artifacts (Sauer et al., 2021). Surprisingly. Proj. FastGAN's ' FID is compa- rable to StyleGAN2's in their experiment. Chong & Forsyth (2020) found a generator-dependent architecture bias, which limits the ability to compare samples for smaller datasets with 50K or fewer images. Additionally, Parmar et al. (2022) found that both FID and KID are highly sensitive to resizing and compression. Barratt & Sharma (2018) reported FID sensitivity with respect different Inception V3 weights. While comparing Tensorflow and PyTorch implementations, Parmar et al. (2022) measured inconsistent scores due to differing resizing implementations. Finally, FID scores are hard to reproduce unless all details regarding its computation are carefully disclosed (Hug. 2024). Stein et al. (2023) proposed an alternative over-reliance on Inception V3, by replacing it with the DINOv2-ViT-L/14 model (Oquab et al., 2024). This replacement partially addresses the domain bias problem but significantly increases computational cost. Unfortunately, DINOv2's training dataset is not publicly available. Furthermore. existing frequency-based metrics such as Sliced Wasserstein Distance (SWD) proposed in Karras al. (2018) involves multiple projections on a random basis. In spite of its ability to detect domain bias, it suffers from reproducibility issues (Nguyen et al., 2023) due to random projections. Consequently, gaps in the dataset remain hidden. This situation motivates the search for additional quality metrics. A detailed discussion of spectral methods and generative architectures is presented in supplementary Section B.
3 FRÉCHET WAVELET DISTANCE (FWD)
We want to tackle the problem of dataset-domain bias. To this end, we propose FWD, which in turn leverages the Wavelet Packet Transform (W2)). We require two-dimensional filters for image
3

---

Published as a conference paper at ICLR 2025
Haps Exp
LIDD ID, TDyll
M Yag
generased imagess
real amagges
Mog P2
Hog 200e
Mop in
FWD
Figure 3: Fréchet Wavelet Distance (FWD) computation flow-chart. Wp denotes the wavelet-packet transform. Not all packet coefficients are shown, dashed lines indicate omissions. We compute individual Fréchet Distances for each packet coefficient and finally average across all the coefficients.
processing. We start with single- dingel-dimeniional Haar wavelets. Next, we construct filter quadruples from the original single-dimensioaal filter pairs. The process uses outer products (Vyas et al., 2018):
ha heh h4 h h, h h2 h2gh
(1)
with a for the approximation filter, h for the horizontal filter, v for the vertical filter, and d for the diagonal filter (Lee et al., 2019). We construct W,-tree for images with these wo-dimensional filters, as illustrated in Fig. 2, using recursive convolution operations with the filter quadruples, i.e..
C₁ * h, = CF
(2)
at every recursion step where + denotes two-dimensional convolution with a stride of two. The filter codes Fi+1 are constructed by applying all j E [a, h, v, d] filters to the previous filter codes Fi. Initially, the set of inputs Fi will only contain the original image Cr, {X] shown in Fig. 2. At level one, we obtain the result of all four convolutions with the input image and have F1 = [a, h, v, d]. At level two, we repeat the process for all elements in Fi. F2 now contains two-character keys laa, ah, av, ad...., de, dd] as illustrated in Fig. 2. We typically continue this process until level 4 in this paper. We arrange the coefficients in CF, as tensors C₁ E R P.H,,.W W for the final layer. The total number of packages at every level is given by P = 4 Hp
and W,, = W. where we denote the image height and width as H and W. We provide more details on W, in the Supplementary.
Figure 3 illustrates how we compute the FWD. The process relies the wavelet packet transform, as previously discussed. We process N images with C' channels in parallel Wp L. E R XWW xc +
Ce RN XX where H and W denote image height and width as before. To facilitate the ensuing metric evaluation, we flatten the last axes into (Hp- W C). Before computing the packets, all pixels are divided by 255 to re-scale all values to [0.1]. The metric is computed in three steps. First, we compute the individual packet mean via
1 N
W(In)p
where I is the n th image in the dataset and p represent the corresponding packet from packets. Then we compute the covariance matrix as
N
[T(TN) N Σ ( -Mp(TN))
(4)
Here, / E H W, denotes the mean across the number of images, and Σ R''xCHH, W,, W,C,HH,,WW represents the covariance among all the coefficients. Now we are ready to compute the distances given the packet mean and covariance values.
(r. d(N(H,, Erg),.N(( Eg,))² |Pr, Mop 1l Tr(E,, + + -2 /V,,, E. (5)
with r and denoting the real and generated images and Tr denoting the trace operation. Utilising the above computed per-packer statisties for both real (H,, E,)) and generated samples (M,, Eg), measure the mean of Fréchet Distance (Equation 5) across all packets
P
FWD = P
Σ p=1
d(N(pr Er, Eg,
(6)
4

---

Published as a conference paper at ICLR 2025
Table Comparison of FID, FD-DINOv2 and FWD to depict domain bias. FID prefers Proj. FastGAN over Denoising Diffusion GAN (DDGAN) across all the datasets. Whereas FWD prefers Denoising Diffusion GAN (DDGAN). We find that FD-DINOv2 agrees with FWD across all datasets except Deep Nutrient Deficiency Dikopshof Dataset (DNDD-Dataset), This might be because agriculture data not part DINOv2's training set.
Dataset
Generator
FID
FD-DINOv2]
FWDJ (ours)
Proj. FastGAN
6.358
685.889
1.388
CelebA-HQ
DDGAN
7.641
199.761
0.408
Proj. FastGAN
4.106
593.124
0.651
FFHQ
StyleGAN2
4.282
420.273
0.312
Proj. FastGAN
4.675
171.625
1.442
DNDD-Dataset
DDGAN
26.233
232.884
1.357
Proj. FastGAN
8.96
424.898
0.755
Sentinel
DDGAN
23.615
404.700
0.115
(a) Proj. FastGAN on CelebA-HQ (FID: 6.358, FWD: 1,388)
(b) DDGAN on CelebA-HQ (FID: 7.641, FWD: 0,408)
Figure Samples from (a) Proj. FastGAN and (b) DDGAN on the Large-scale Celeb Faces Attributes High Quality (CelebA-HQ) dataset. The FID prefers Proj. FastGAN irrespective of visual artefacts and floating heads, whereas our metric (FWD) ranks DDGAN higher than Proj. FastGAN.
By averaging the distances of all frequency bands, the FWD captures frequency information across the spectrum.
EXPERIMENTS
Our first series of experiments demonstrates the effect of domain bias on learned metrics, demonstrat- ing the resilience of FWD to such bias. All experiments were implemented using the same code base. Implementation: We use PyTorch (Paszke et al., 2017) for neural network training and evaluation and compute FID using (Seitzer, 2020) as recommended by Heusel et al. (2017b). We work with the wavelet filter coefficients provided by Waveeeess (Lee et al., 2019). We chose the PyTorch- Wavelet-Toolbox (Wolter al., 2024) software package for GPU support. FD-DINOV2 and KID computed using the codebases from Stein et al. (2023) and Binkowski et al. 22018,, respectively.
4.1 EFFECT OF DOMAIN BLAS
Kynkäänniemi et al. (2023) observed that metrics based on ImageNet-traine trained network features emphasize ImageNet-related information. This behaviour is desired when we evaluate generators on ImageNet or similar datasets. When working with other datasets, this behaviour is misleading. Datasets: As datasets, we use Large-scale Celeb Faces Attributes High Quality (CelebA-HQ) (Karras et al., 2018), Flickr Faces High Quality (FFHQ), DNDD-Dataset (Yi et al., 2023), an agricultural dataset, and Sentinel (Schmitt et al., 2019), a remote sensing dataset. These datasets contain images that are very different from those in ImageNet. More information about the ONDD-Datased and the Sentinel dataset can be found in the supplementary material.
Generators: We study dataset domain bias effects using the Denoising Diffusion GAN (DDGAN),
5

---

Published as a conference paper at ICLR 2025
10
CelebAHQ DDGAN I Proj. FastGAN
8,000
DNDD
1
DDGAN
Fra
0.8
6,000
0.6
1,000
0.4
2,000
0.2
whiptail
meerlat
loals
Gila monster
consoome
dung beetle
sidewiudet
cliff
mampproon
got fddler crub
roncaal
water stabe- polerat
walf spider
w98 bikini
suil
lipstick
brassicre
seat belt
sunsereen
neck neck baice
lab cont Band Aid
hair spray stethoscope
sunglasses
bow
fur
(a) CelebA-HQ
(b) ONDD-Datasee
Figure 5: Distribution of ImageNet Top-1 classes, predicted by Inception for real images and images generated by DDGAN and Proj. FastGAN. (a) depicts the distribution for the CelebA-HQ dataset and (b) shows the distribution for DNDD-Dataset. Although irrelevant for visual quality, the class distribution of Proj. FastGAN aligns more closely with the real distribution than DDGAN for both the datasets, contributing to lower FID for Proj. FastGAN.
Proj. FastGAN and StyleGAN2 networks. Proj. FastGAN is particularly interesting since its discriminator relics on ImageNet weights to improve training convergence (Sauer et al., 2021). Prior work found this architecture to improve FID on image datasets far from ImageNet, without substantially improving image quality (Kynkäänniemi et al., 2023).
Hyperparameters: To examine the effect of dataset bias, require generators, which are tuned to produce output that resembles our datasets' distribution. Specifically, we trained Proj. FastGAN for 100 epochs on both the CelebA-HO dataset and DNDD-Dataset, respectively, using learning rate of le-4 and batch size of 64 with 8 A100 GPUs. For the Sentinel dataset, we trained Proj. FastGAN for 150 epochs, using the same hardware and hyperparameters. For FFHQ, pre-trained weights are available, as well as pre-trained weights for DDGAN on CelebA-HQ from Xiao et al. (2022). On DNDDD- we trained DDGAN for 150 epochs with a learning rate of le-4 and batch size of on the same hardware. We also trained DDGAN on the Sentinel dataset for 250 epochs, using learning rate of le-4 and batch size of 4 on A 100 GPUs. For StyleGAN2, we use the pretrained weights with the code from Karras et al. (2020).
Table presents the FID, KID, FD-DINOv2 and FWD values across all datasets for the afore- mentioned generators. Across all datasets, FID prefers images generated by Proj. FastGAN. When we compare images generated by Proj. FastGAN and DDGAN for CelebA-HQ, which are shown in Figures 4a and 4b, we observe that more deformations are visible in the images of Proj. FastGAN compared DDGAN images. DDGAN, in other words, produces more high-quality images. Sup- plementary Figures 16 and 17 illustrate this observation further. Consequently, it is surprising to see FID preferring Proj. FastGAN, as we would expect DDGAN come out on top. Following Kynkäänniemi et al. (2023), we compare the Inception V3 output label distribution of the original- CelebA-HQ images to their synthetic counterparts from DDGAN and Proj. FastGAN in Figure 5a. We observe that Inveption V3 produces a label distribution for Proj. FastGAN, which resembles the distribution from Inveption V3 for the original CelebA-HQ images. The label distribution for images from DDGAN differs significantly. This discrepancy, also reported by Kynkäänniemi et al. (2023), explains why FID produces a misleading verdict. FWD, in contrast, prefers DDGAN, as we would expect.
Results:
The same pattern repeats in the results for our FFHQ -xprrimmieem Generally, we see FID preferring Proj. FastGAN images, while FWD puts StyleGAN2 top. Our observations confirm the experiment in Kynkäänniemi et al. (2023). In next step, we study the effect of a larger network backbone for the neural Fréchet distance computations. Stein et al. (2023) proposed replace Inveption V3

---

Published as a conference paper at ICLR 2025
Table 2: Comparison of computational efficiency between FID, FD-DINOv2 and FWD. FWD exhibits the lowest FLOPs and highest throughput. FD-DINOv2 has the highest FLOPs and lowest throughput because of its large network, and FID is in between. FLOPs are calculated over individual feature extractors on single image, and throughput is measured over 50k images.
Metrie
GFLOPs|
Throughpur (imgs/sco)t
1.114
FD-DINOV2 FWD
15.566
53
0.006
1923
Table 3: Evaluation of FID (ImageNet), FID (CelebA) and FWD on the CelebA-HQ and FFHQ datasets. FID (ImageNet) prefers Proj. FastGAN in both datasets, whereas FID retrained on CelebA and FWD both prefer DDGAN in these datasets.
Dataset
Generator
FID (ImageNet)y
FID (CelebAy)
FWD1
Proj. FastGAN
6.358
5.602
1.388
CelebA-HQ
DDGAN
7.641
3.145
0.408
Proj. FastGAN
4.106
2.204
0.651
FFHQ
StyleGAN2
4.282
0.897
0.312
with the much larger pretrained DINOv2 network. Table 1 lists the resulting distance metrics. For CelebA-HQ and FFHQ, FD-DINOv2 prefers DDGAN and StyleGAN2 images respectively. Here, FD-DINOV2 and FWD agree.
To investigate further, consider the DNDDD-attast of agricultural images (Yi al., 2023) and the Sentinel (Schmitt et al., 2019) dataset. Samples from Proj. FastGAN for DNDI>-Dataset and Sentinel are provided in Figures 19 and 21, respectively. Correspondingly, Figures 18 and 20 represent samples from DDGAN for DNDD-Dataset and Sentinel, respectively. In both cases, FID consistently prefers Proj. FastGAN, which was also the case all prior experiments. Histograms of the Inception V3 label distribution are depicted in Figure 5b. The histograms indicate domain bias and resemble the observations reported above. On DNDD-Dataset and Sentinel, the verdicts of FD-DINOv2 and FWD particularly interesting. While both metrics correctly agree on the Sentinel dataset, only FWD correctly prefers DDGAN on the agricultural images.
We carefully chose the DNDD-Dataset, agriculture images are not commonly used, and the dataset does not resemble ImageNet. We speculate that the LVD-142M dataset may include satellite imagery, contributing to consistent ranking. Unfortunately, the closed source of the LVD-142M dataset used for training DINOv2 (Oquab et al., 2024) makes it difficult to investigate this domain bias more in detail. In this first set of experiments, to the domain bias problems, it still produces an inconsistent ordering for the DNDD-Dataset images. Furthermore, this partial remedy comes at tremendous computational cost. Table 2 shows that FWD is over 36 times faster to compute than FD-DINOv2.
observed that, while FD-DINOv2 provides partial remedy
In second series of experiments, we investigate the effect of retraining, another expensive solution to the domain bias problem. To this end, we train Inception on Large-scale Celeb Faces Attributes (CelebA). CelebA comes with 40 facial attributes, which we use train classifier. After conver- gence, we an exact match ratio of 90% and recalculate FID using this new backbone. The FID (CelebA) column of Table lists the corresponding scores, and FID (CelebA) and FWD provide the same order.
However, in the case the agricultural dataset, the retrained FID (DNDD) in supplementary Table 10 remains biased, while FWD produces meaningful domain agnostic results. DNDD-Dataset contains 3600 images with seven classes and the task requires detecting nutrient deficiency in winter wheat and winter rye, such nitrogen, phosphorous, and potassium deficiencies. Once more, we use retrained Inception V3 backbone for the FID computation. Compared to CelebA or ImageNet, this is small dataset and the retrained network does not provide meaningful features. This is interesting use case since it illustrates that FWD is not just free from data bias. It also provides meaningful feedback for low-resource tasks where retraining Inception V3 is not feasible.
7

---

Published as a conference paper at ICLR 2025
Boa
mah
sla
shis
han hab
hha hhd
10a
wav and ahv
phd
hav had hhs hhd
10
ava ava ada pdl
ava hda hdl
10
avy ava adv
sdd ndy hdd
10
10
vva den ddl
10-13
vvd ydv
ydd
dvdddy ddy
10-12
(a) Filter order
(b) StyleGAN2
(c) DDGAN
StyleGAN2
DDGAN
10g RDp
100
Packets
(d)
Figure 6: Interpretation of FWD. (a) represents the blueprint for level-3 Wp transformation. (b) and (c) depict the mean absolute packet difference between CelehA-HQ dataset and generated images by StyleGAN2 and DDGAN, respectively. (d) shows the per-packet Fréchet distances for StyleGAN2 in orange and DDGAN in blue.
In conclusion, experiments this section indicate that metrics like FID and FD-DINOv2, while helpful, prone domain bias when applied to datasets beyond the underlying training datasets. On the contrary, FWD offers a computationally efficient, consistent and domain-agnostic evaluation.
4.2 FWD INTERPRETABILITY
A generative metric is interpretable if and only if we can understand the underlying mechanics that produce the ranking. This section explains the decisions made by FWD one specific case where we focus samples from DDGAN and StyleGAN2 for CelebA-HQ.
Section formulates FWD as an average of per packet FWD scores. This design choice allows us to understand the overall FWD-score in terms of the individual packet coefficients for each frequency band. Figures 6b and 6c depict the mean absolute difference per packet between the original images of CelebA-HQ and generated samples from StyleGAN2 and DDGAN, respectively. Figure 6d presents both generators' per-pase per-packet FWD. Figure shows that DDGAN has a lower Fréchet distance for all packets and averaging the distances over all packages translates into meaningful metric.
4.3 EVALUATION ROBUSTNESS
The section follows up on prior work by Kynkäääniemi et (2023). The authors generate large set of samples and find specific combination of images with optimal FID. First, the weights of each image are optimized with FID as the objective function. Second, a subset of images is sampled based on the weights. We follow this process and sample 50k images from large set with optimized weights as probabilities. We employ generated images from StyleGAN2 and real-world images from the FFHQ dataset. Supplementary Table 6 lists the resulting FID, FD-DIN DO-DOI and FWD values. We observe that FWD is robust to FID optimization, whereas FD-DINOv2 showed a little reduction by optimizing FID.
8

---

Published as a conference paper at ICLR 2025
Table 4: Comparing various generative models using Fréchet Wavelet Distance (FWD), Fréchet Inception Distance (FID), DINOv2-Fréchet Distance (FD-DINOv2) and Kernel Inception Distance (KID) on the CelebA-HQ,
and ImageNet datasets.
Datusel Image Size
Method
FID.
KID,
FD-DINOv2L
FWD, (ours)
CelebAHQ
DDIM Song ct al. (2021)
32.333
0.0313
654,482
12.317
DDPM (Ho al., 2020)
19.101
0.0152
4,697
256
StyleSwin (Zhang et al., 2022)
23.257
0.0264
255,404
1.528
StyleGAN2 (Karras et al., 2020)
15.439
0.0155
593.344
0.476
DDGAN (Xiuo et al., 2022)
7.203
0.0034
199.761
0.408
Churches
DDIM (Song et al., 2021)
11.775
0.0043
538.400
4,919
256
DDPM (Ho al., 2020)
9.484
0.0036
3.546
StyleSwin (Zhang et al., 2022)
3.187
0,0005
435,967
2.835
StyleGAA2 (Karras et al., 2020)
4.309
0.0007
444.044
0.753
ImageNet Bedrooms
256
DDIM (Song et al., 2021)
25.857
0.0094
452.419
9.521
DDPM (Ho 2020)
16.251
0.0058
392.481
5.187
Imp. Diff. (VLB) (Nichol & Dhariwal, 2021)
33.522
0.0264
670.952
2.182
EDM (Karras et al., 2024)
12.295
0.0108
113.704
1.160
64
BigGAN (Brock et al., 2019)
5.128
0.0024
170.601
0,441
Imp. Diff. (Hybrid) (Nichol Dhariwal, 2021)
3.091
0.0006
96.208
0.392
In addition to FID optimization, we study the impact of image perturbation in supplementary Figure We find that FWD and FD-DINOv2 are closer to bijective mapping in the presence of perturbation than FID. This behaviour is desirable since would always expect larger distance if for example more noise is added. This is not always the case for FID. Consider for example the last quarter of the uniform noise intensity in (b), where FID falls even though more noise is added.
4.4 COMPARISON TO STATE OF THE ART
To understand the spectral qualities of existing generative methods for image synthesis, we evaluated various Diffusion and GAN models across wide range of benchmark datasets.
Datasets: We compare common metrics and our FWD on CelebA-HQ (Karras et al., 2018), the Church and Bedroom subsets of the Large-scale Scene UNderstanding (LSUN) dataset (Yu al., 2015), and finally ImageNet (Russakovsky et al., 2015). In order to retain consistent spatial and frequency characteristics across various image sizes, we use the level 4 packet transform for 256x256 images. For images that are smaller, we use fewer levels, i.c., 3 for 128x 128 and 2 for 64x64. Generators: For the evaluation, we use the diffusion approaches Denoising Diffusion Probabilistic Models (DDPM) (Ho et al., 2020). Denoising Diffusion Implicit Models (DDIM) (Song et al., 2021), Improved Diffusion (Nichol Dhariwal, 2021), DDGAN (Xiao et al., 2022), EDM (Karras et al., 2024), as well as the GAN approaches StyleGAN2 (Karras et al., 2019). StyleSwin (Zhang et al., 2022) and BIGGAN (Brock al., 2019).
Hyperparameters: All generators are evaluated with pretrained weights provided by the respective paper codebases.
Metrics: Table 4 reports the results for FID, KID, FD-DINOv2 and finally scores are obtained by the standard implementation by Seitzer (2020). The ImageNet numbers are computed with 50k images from the validation set. For CelebAHQ and LSUN, we work with 30k images.
FWD. FID-
Considering CelebA-HQ, FID, KID, FD-DINOV2 and FWD agree most of the time. Considering FID and FWD, only DDPM and StyleSwin are swapped. It is interesting to note that FWD ranks DDIM, DDPM. StyleSwin, and StyleGAN2 on CelebA-HO and LSUN churches the same way. whereas FID ranks StyleSwin differently on the two datasets. According to FID, StyleSwin performs worse than DDPM and StyleGAN2 on CelebA-HQ but better than these two approaches on LSUN churches. This is counterintuitive, but it can be explained by the domain bias of FID. The supplementary Figure la depicts the histograms of top-1 classes classified by Inception V3 on CelebA-HQ for DDPM and StyleSwin. We observe that DDPM matches the activation histograms of CelebA-HQ more accurately than the histograms of StyleSwin, whereas the histograms of both methods are very similar for LSUN-Church CelebA-HQ but better on ChU-Church. Our metric FWD is not biased by the class distribution and provides consistent metric for both datasets.
shown in Figure llb. As result, FID ranks StyleSwin worse

---

Published as a conference paper at ICLR 2025
Table 5: Comparison of existing metrics FID, FD-DINOv2 and FWD with Human Error Rate (HER). Higher HER means that participants find the generated images more realistic than the images of the original dataset. HER shows that DDGAN generates perceptually better images.
Dataset
Generator
FID
FD-DINÓv2]
FWD1
HER(%)
Proj. FasIGAN
6.358
685.889
1.388
20.0
CelcbA-HQ
DDGAN
7.641
199.761
0.408
32.5
Proj. FasIGAN DDGAN
4.675
171.625
1.442
50
DNDD-Datasct
26.233
232.884
1.357
57
We also consider the expect pristine performance for FID on ImageNet since this setting is perfectly in its data-domain. Yet, FD-DINOv2 places EDM (Karras et al., 2024) ahead of BigGAN, which is surprising since this does not match with the ranking from FID. FID and FWD agree and arrive at the same ranking.
and ImageNet 64 datasets, where FID and FWD agree. We
4.5
USER STUDY
To ensure that metric aligns with human perception, we conduct two types of user studies. The first study demonstrates that FWD does not suffer from domain bias. The second study supports FWD's alignment with human rankings on large-scale diffusion models.
Datasets and Generators: In case of the first user study, we use CelebA-HQ and DNDD-Datase to assess the pereeptual quality of images generated by Proj. FastGAN and DDGAN. For the second user study, we use Conceptual Captions (Sharma et al., 2018) as the evaluation dataset and work with pre-trained StableDifusion models, particularly versions 1.5, 2.1, 3.0 (Medium), and 3.5 (Large) from co/ st stabilityal
hugging face (https: :
Table presents the results of the first user study. higher Human Error Rate (HER) in the table implies that the participants find the generated images more realistic than the original images. The HER results show that the users identify DDGAN generated images more realistic than Proj. FasIGAN generated images. Predominantly, this table highlights FWD's s alignment with user preferences across both CelebA-HQ and DNDD-Datar in comparison to FID and FD-DINOv2. Supplementary Table exhibits the overall alignment of FWD with human perception large- scale diffusion models. We observe that FWD prefers the latest StableDiffusion-3.55 model over other models, same the users, whereas FID and FD-DINOv2 rank the StableDiiffusion115 model surprisingly better. Moreover, we observe that FWD and other metrics prefer StableDiffusion-1. images over 2.1 images. On careful observation of images from these models, we observe that the 2.1 model generates images with artifacts like deformed bodies, extra hands, improper artistic images (like paintings), and some images with white contrast more often than the 1.5 model. We provide the samples from all the StableDiffusion models in Supplementary Section C. Overall, the user study demonstrates FWD's alignment with human perception and that does not suffer from a domain bias.
Results:
5 CONCLUSION
Modern generative models exhibit frequency biases (Durall et al., 2020), while commonly used metrics such as FID, KID and FD-DINOv2 are affected by domain bias (Kynkäääniemi et al., 2023). To address these limitations, FWD accounts for frequency information without introducing domain- specific bias. Even though FD-DINOv2 offers a partial solution to this issue, it comes at a very high computational cost and has thus negative environmental impact. In response, this paper introduced FWD, a novel metric based on the wavelet packet transform. Our metric allows a consistent and domain-agnostic evaluation of generative models, and it computationally efficient. Our findings show that FWD is robust to input perturbations and interpretable through the analysis of individual frequency bands. FWD in conjunction with traditional metries ensures a comprehensive and accurate evaluation of generative models while also helping to mitigate domain bias.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This research was supported by the Federal Ministry of Education and Research (BMBF) under grant no. 011S22094A WEST-AI and 6DHBK 1022 InTr the Deutsche Forschungsgemeinschall (DFG, German Research Foundation) GA 1927/9-1 (KI-FOR 5351) and the ERC Consolidator Grant FORHUE (101044724). Prof. Kuehne is supported by BMBF project STCL 011S22067. The authors gratefully acknowledge the Gauss Centre for Supercomputing e. e.V. (www. wwww.gauss-cess www--guuusscctteeeee for funding this project by providing computing time through the John von Neumann Institute for Computing (NIC) on the GCS Supercomputer JUWELS Jüiiich Supercomputing Centre (JSC). The authors heartfully thank all the volunteers who participated in the user study. The sole responsibility for the content of this publication lies with the authors.
REFERENCES
Evaluating diffusion models.
http://cccoccc/cc//eeee////eee/e conceptual/evaluatt 2024. Accessed: 2024-04-11.
Shane T. Barratt and Rishi Sharma. A note on the inception score. ICML 2018 workshop on Theoretical Foundations and Applications of Deep Generative Models., 2018.
Mikolaj Binkowski, Danica J. Sutherland, Michael Arbel, and Arthur Gretton. Demystifying MMD gans. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings, 2018.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In International Conference on Learning Representations, 2019.
Min Jin Chong and David A. Forsyth. Effectively unbiased FID and inception score and where to find them. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pp. 6069-6078. Computer Vision Foundation / IEEE, 2020.
Ingrid Daubechies. Ten Lectures Wavelets. Society for Industrial and Applied Mathematics, 1992.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. Advances in neural information processing systems, 118888887944 2021.
DC Dowson and B BV666017 Landau. The fréchet distance between multivariate normal distributions. Journal of multivariate analysis, 12(3):450-455 1982.
Ricard Durall, Margret Keuper, and Janis Keuper. Watch your up-convolution: Can based genera- tive deep neural networks are failing to reproduce spectral distributions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 7890-7899, 2020.
Tarik Dzanic, Karan Shah, and Freddie Witherden. Fourier spectrum discrepancies in deep network generated images. Advances in neural information processing systems, 33:3022-3032, 2020.
Rinon Gal, Dana Cohen Hochberg, Amit Bermano, and Daniel Cohen-Or. Swagan: style-based wavelet-driven generative model. ACM Trans. Graph., 40(4), July 2021.
Florentin Guth, Simon Coste, Valentin De Bortoli, and Stephane Mallat. Wavelet score-based generative modeling. Advances in Neural Information Processing Systems, 35:478-491, 2022.
Alejandro Hernande7, Jurgen Gall, and Francesc Moreno-Noguer. Human motion prediction via spatio-temporal inpainting. In Proceedings of the Int#//CVFintemaaionnd Vision, pp. 7134-7143, 2019.
International Conference Computer
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update nule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017a
Martin Heusel, Thomas Unterthiener, Wendy Kan, Mark Hamilton, Zejian Li. Mare Uecker, Wang Penghui, and Partik Joshi. Two time-scale update rule for training gans. https: /ithtbb. com/bi oi in faaiinnjjjuuuuu 2017b.
11

---

Published as a conference paper at ICLR 2025
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
Alain Horé and Djemel Ziou. Image quality metrics: PSNR vs, SSIM. In 20th International Conference on Pattern Recognition, ICPR 2010, Istanbul, Turkey, 23-26 August 2010, pp. 2366- 2369. IEEE Computer Society, 2010.
Huaibo Huang, Ran He, Zhenan Sun, and Tieniu Tan. Wavelet-smet: A wavelet-based enn for multi- scale face super resolution. Proceedings of the IEEE international conference on computer vision, pp. (689-1697)) 2017.
Arne Jensen and Anders la Cour-Harbo. Ripples in mathematics: the discrete wavelet transform. Springer Science & Business Media, 2001.
via Wikimedia Commons Jérémy Barande.
Wikimedia commons: Yann lecun
2018 (cropped). https::/eet wikipedi www.///w///////.... Saaik//Yan__neuuiiia /media/File: Yann_LeCun__-00 - 8 (cropped) jpg. 2024. Accessed: 2024-05-22.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. In 6th International Conference on Learning Representa- tions, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings, 2018.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 4401-4410 2019.
Tero Karras, Samuli Laine, Miik Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 8110-8119, 2020.
Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-free generative adversarial networks. Advances in Neural Information Processing Systems, 34:852 863, 2021.
Tero Karras, Miika Aittala, Jaakko Lehtinen, Janne Hellsten, Timo Aila, and Samuli Laine. Analyzing and improving the training dynamics of diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 24174-24184 2024.
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In Yoshua Bengio and Yann LeCun (eds.), 2nd International Conference Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014.
Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc, Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 3929-3933 2019.
Tuomas Kynkäänniemi, Tero Karras, Miika Aittala, Timo Aila, and Jaakko Lehtinen. The role of imagenet classes in fréchet inception distance. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
Gregory Lee, Ralf Gommers, Filip Waselewski, Kai Wohlfahrt, and Aaron O'Leary. Pywavelets: A python package for wavelet analysis. Journal of Open Source Software, 4(36): 2019.
Jin Li, Wanyun Li. Zichen Xu, Yuhao Wang, and Qiegen Liu. Wavelet transform- adaptive generative modeling for colorization. IEFE Transactions Multimedia, 2022.
12

---

Published as a conference paper at ICLR 2025
Lin Liu, Jianzhuang Liu, Shanxin Yuan, Gregory Slabaugh, Aleš Leonardis, Wengang Zhou, and Qi Tian. Wavelet-based dual-branch network for image demoiréing. In Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK. August 23-28 2020, Proceedings, Part XIII 16, pp. 86-102. Springer, 2020.
Yunfan Liu, Qi Li, and Zhenan Sun. Attribute-awan face aging with wavelet-based generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 11877-11886, 2019.
Stéphane Mallat. A theory for multiresolution signal decomposition: The wavelet representation. IEEE Trans. Pattern Anal. Mach. Intell., 11(7):674-693, 1989.
Stéphane Mallat. A wavelet tour of signal processing. Elsevier, 1999.
Manish Narwaria, Weisi Lin, Ian Vince McLoughlin, Sabu Emmanuel, and Liang-Tiee Tien Chia. Fourier transform-base scalable image quality measure. IEEE Trans. Image Process., (18))3334-3377, 2012.
Khai Nguyen, Tongzheng Ren, and Nhat Ho. Markovian sliced wasserstein distances: Beyond independent projections. Advances in Neural Information Processing Systems, 36:39812-39841 2023.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic mod- els. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 8162-8171. PMLR, 18-24 Jul 2021.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby, Mido Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po- Yao Huang, Shang- Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Herve Jegou, Julien Mairal, Patrick Labatut, Ar- mand Joulin, and Piotr Bojanowski. DINOv2: Learning robust visual features without supervision. Transactions on Machine Learning Research, 2024. ISSN 2835-8856
Gaurav Parmar, Richard Zhang, and Jun-Yan Zhu. On aliased resizing and surprising subtleties in gan evaluation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14410-114200 2022.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary De Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In s Conference on Neural Information Processing Systems (NIPS 20/7), 2017.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the JEEF/CVF International Conference on Computer Vision, pp. 4195 2023.
Hao Phung, Quan Dao, and Anh Tran. Wavelet diffusion models are fast and scalable image generators. In IEEE/CVF Conference Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, pp. 10199-10208. IEEE, 2023.
Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In International Conference on Machine Learning, pp. 5301-5310. PMLR, 2019.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115 1):211-252. 2015.
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
Vishwanath Saragadam, Daniel LeJeune, Jasper Tan, Guha Balakrishnan, Ashok Veeraraghavan, and Richard G Baraniuk. Wire: Wavelet implicit neural representations. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pp. 18507-18516, 2023.
13

---

Published as a conference paper at ICLR 2025
Axel Sauer, Kashyap Chitta, Jens Müller, and Andreas Geiger. Projected gans converge faster. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
M. Schmitt, L. H. Hughes, C. Qiu, and X. X. Zhu. Senl2ms ndash; a curated dataset of georeferenced multi-spectral sentinel-1/2 imagery for deep learning and data fusion. ISPRS Annals of the Photogrammetry. Remote Sensing and Spatial Information Sciences, 4(2/W7):153- 16155366,,,,6- September 2019. ISSN 2194-9042.
Katja Schwarz, Yiyi Liao, and Andreas Geiger. On the frequency bias of generative models. Advances in Neural Information Processing Systems, 11111822-18866 2021.
Maximilian Seitzer. pytorch-fid: FID Score for PyTorch. https: / wttp:htwww..mmmmttteeeeeeeeeeeee pytorch- fid, August 2020. Version 0.3.0.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of ACI, 2018.
Jascha Sohl- Dickksiss,,,, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference machine learning, pp. 2256-2265. PMLR, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria. May 3-7, 2021, 2021.
George Stein, Jesse Cresswell, Rasa Hosseinzadeh, Yi Sui, Brendan Ross, Valentin Villecroze, Zhaoyan Liu, Authony L Caterini, Eric Taylor, and Gabriel Loaiza-Ganem. Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models. In Advances in Neural Information Processing Systems, volume 36, 2023.
Gilbert Strang and Truong Nguyen. Wavelets and filter banks. SIAM, 1996.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Aparna Vyas, Soohwan Yu, and Joonki Paik. Multiscale transforms with application to image processing. Springer, 2018
Jianyi Wang, Xin Deng, Mai Xu, Congyong Chen, and Yuhang Song. Multi-level wavelet-based generative adversarial network for perceptual quality enhancement of compressed video. In European Conference on Computer Vision, pp. 405-421. Springer, 2020.
Zhou Wang, Alan C. Bovik, Hamid R. Sheikh, and Eero P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process., 13(4):600-612 6120 2004.
Travis Williams and Robert Li. Wavelet pooling for convolutional neural networks. In International conference on learning representations, 2018.
Moritz Wolter, Felix Blanke, Raoul Heese, and Jochen Garcke. Wavelet-packed for deepfake image analysis and detection. Machine Learning, 111(11):4295 43277 2022.
Moritz Wolter, Felix Blanke, Jochen Garcke, and Charles Tapley Hoyt. ptwt the pytorch wavelet toolbox. Journal of Machine Learning Research, 25(80):1-7, 2024.
Zhisheng Xiao, Karsten Kreis, and Arash Vahdat. Tackling the generative learning trilemma with denoising diffusion gans. In International Conference on Learning Representations, 2022.
Jinhui Yi, Gina Lopez, Sofia Hadir, Jan Weyler, Lasse Klingbeil, Marion Deichmann, Juergen Gall, and Sabine Seidel. Non-invasive diagnosis of nutrient deficiencies in winter wheat and winter rye using uav-based rgb images. SSRN 4549653, 2023.
Jaejun Yoo, Youngjung Uh, Sanghyuk Chun, Byeongkyu Kang, and Jung- Woo Ha. Photorealistic style transfer via wavelet transforms. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 9036-9045, 2019.
14

---

Published as a conference paper at ICLR 2025
Fisher Yu, Yinda Zhang, Shuran Song, Ari Seff, and Jianxiong Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans in the loop. ArXiv, abs/ 15d./.366.33 2015.
Bowen Zhang, Shuyang Gu, Bo Zhang, Jianmin Bao, Dong Chen, Fang Wen, Yong Wang, and Baining Guo. Styleswin: Transformer- based gan for high-resolution image generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11304 11314, 2022.
15

---

Published as a conference paper at ICLR 2025
FID FWD FD-DINO22
FI-DINO0
iomalbed
0.0
0.4
FID
0.4
0.4
FWD
0.3
FD-DINOV2
0.2
0
Blur Intenety
Uniforno nooe intensity
PIL IPEG Quality
(b)
(c)
Figure 7: Figures depicting the effect of perturbations such (a) Gaussian blur, (b) uniform noise corruption and (c) JPEG compression on FID, FWD and FD-DINOv2.
A SUPPLEMENTARY
A.1 ACRONYMS
W Wavelet Packet Transform
CelebA Large-scale Celeb Faces Attributes
CelebA-HQ Large-scale Celeb Faces Attributes High Quality
CNN Convolutional Neural Network
DDGAN Denoising Diffusion GAN
DDIM Denoising Diffusion Implicit Models
DDPM Denoising Diffusion Probabilistic Models
DNDD-Dataset Deep Nutrient Deficiency Dikopshof Dataset
FD-DINOv2 DINOv2-Fréchet Distance
FFHQ Flickr Faces High Quality
FID Fréchet Inception Distance
FWD Fréchet Wavelet Distance
FWT Fast Wavelet Transform
GAN Generative Adversarial neural Network
HER Human Error Rate
IS Inception Score
KID Kernel Inception Distance
LSUN Large-scale Scene UNderstanding
MSE Mean Squared Error
Proj. FastGAN Projected Fast GAN
SWD Sliced Wasserstein Distance
VAE Variational AutoEncoder
A.2 FWD ROBUSTNESS
To supplement Section 4.3, we provide results for FWD's robustness towards various perturbations such Gaussian blur, uniform noise and JPEG compression in Figure 7. Furthermore, in Table 6, demonstrate that matching fringe features can be used to optimise FID and FD-DINOv2, whereas FWD does not improve.
16

---

Published as a conference paper at ICLR 2025
Table 6: Matching fringe features for 250k images generated using StyleGAN2 for the FFHQ dataset. By optimizing the sample weights for FID, FD-DINOv2 is also slightly improved. In contrast, FWD penalizes the manipulation of the sample distribution.
Metric FID
Random Images
FID-Optimized Images
Change
4.278 + 0.019
2.031 + 0.005
-52.53%
FD-DINOv2
420.223 0.563
414.048 = 0.905
-1.47%
FWD
0.338 = 0.017
0.398 + 0.009
+17.75%
A.3 EXTENDED USER STUDY
For the first user study, following Stein et al. (2023), we presented participants with pairs of real and generated images and asked them to select the realistic image. In this manner, collected over Ik responses from 50 volunteers. In the second user study, we generated images from the Conceptual Captions (Sharma al., 2018) validation set and compared our metric with user alignments taken from aittps://aree fi c cinaanalllllllaa ysis /ttetttttttttteee to- image/ arena ttab=Leade
Table 7: Comparison of metrics FID, FD-DINOv2 and FWD with HER. Higher HER represents a higher prompt alignment percentage according users. FWD aligns better with HER than FID and FD-DINOv2.
Generator
FID4
FD-DINOv21
FWD(
User Rating (
StableDiffussin-1.5
14.904
124.948
17,498
14
StableDiifusson-22.1
15.446
132.049
21.195
StableDifffssion-..0 (Medium)
18.709
158.572
6.645
45
StableDiffrusion-355 (Large)
17.907
157.798
4.979
A.4 THE FAST WAVELET AND WAVELET PACKET TRANSFORMS
This supplementary section summarizes key wavelet facts as convenience for the reader. See, for example, (Strang& & Nguyen, 1996; Mallat, 1999) or (Jensen & la Cour-Harbo, 2001) for excellent detailed introductions the topic.
The Fast Wavelet Transform (FWT) relies on convolution operations with filter pairs. Figure illustrates the process. The forward or analysis transform works with a low-pas, he and high-pass filter hy. The analysis transform repeatedly convolves with both filters
X, *1 h& = Ck,s+1
with #1 being the Id-convolution operation, & e [C, H] and e Ng, the set of natural numbers. While Xo is equal to the original input signal at higher scales, the FWT uses the low-pass filtered result as input, i.e., x, = cz.s if > O. The dashed arrow in Figure indicates that we could continue to expand the FWT tree here.
The Wavelet Packet Transform (Wp) additionally expands the high-frequency part of the tree. A comparison of Figures and illustrates this difference. Whole expansion way to construct a wavelet packet tree. See (Jensen & la Cour-Harbo, 2001) for a discussion of other options. both figures, capital letters denote convolution operators. These may be expressed as Toeplitz matrices (Strang & Nguyen, 1996). The matrix nature of these operators explains the capital boldface notation. Coefficient subscripts record the path that leads particular coefficient.
the only possible
We construct filter quadruples from the original filter pairs to process two-dimensional inputs. The process uses outer products (Vyas et al., 2018):
(8) With a for approximation. h for horizontal, v for vertical, and d for diagonal (Lee et al., 2019). We construct Wy-tree for images with these two-dimensional filters. Figure 10 illustrates the
h = hgh h, = heh h, = hygh h4 hąh
17

---

Published as a conference paper at ICLR 2025
Hc
42
C2
to Fc
H
4/
12 Fc
H
42
C1
1t F'
H Ht
42
Co
ta Fx
Figure 8: Overview of the Fast Wavelet Transform (FWT) computation. he denotes the analysis low-pass filter and by the analysis high pass filter. fe and fy the synthesis filter pair. 42 denotes downsampling with a factor of two, t2 means upsampling. The analysis transform relies on stride two convolutions. The synthesis inverse transform the right works with stride two transposed convolutions. Hg and F& with [c, H] denote the corresponding convolution operators.
Hc 22
cc.c
2 FE
He. $2
22 Fc.
H 42
C2 .....
1a Fx
Hc
Cxi.c ......
2 Fc
H
12
12 FR
H 2
1/ Fx
Figure 9: Schematic drawing of the full Wavelet Packet Transform (Wp) in single dimension. Compared to Figure 8, the high-pass filtered side the tree is expanded, too.
H J2
H 42
Hy 42
Ha 42
H J2
Hn J2
Hy J2
H4 4/
au
av
ad
12 Fa
12 Fn
12 Fv
1/ Fa
12 F.
12 Fh
12 Fv
12 Fa
Figure 10: Two dimensional Wavelet Packet Transform (W2) computation overview. X and X denote input image and reconstruction respectively. We compute the Fréchet Wavelet Distance (FWD) using the wavelet packet coefficients p. The transform is invertible, the distance computation is therefore based on a lossless representation.
18

---

Published as a conference paper at ICLR 2025
computation of full two-dimensional wavelet packet tree. More formally, the process initially evaluates
Xg * h, = Cj,1
with Xo equal to an input image X, E [a, h v, dl, and * being the two-dimensional convolution. At higher scales, all resulting coefficients from previous scales serve as inputs. The four filters are repeatedly convolved with all outputs to build the full tree. The inverse transforms work analogously. We refer to the standard literature (Jensen & la Cour-Harbo, 2001; Strang & Nguyen, 1996) for an extended discussion.
Compared to the FWT, the gegeffreyenccy half of the tree is subdivided into more bins, yielding fine greeggrrindd view of the entire spectrum. We always show analysis and synthesis transforms to stress that all wavelet transforms are lossless. Synthesis transforms reconstruct the original input based on the results from the analysis transform.
A.5 HISTOGRAM MATCHING INCEPTIONV3
To understand the results in Table 4 better, we present the histograms of Inception output labels for images in the datasets CelehA-HQ and LSUN-Church in Figures 11a and 1 1b,, respectively. In both figures, we compare the histograms of the generated images of DDPM and StyleSwin. While StyleSwin generates better images than DDPM, the class distribution of DDPM is closer to the real images compared to StyleSwin on CelebA-HQ. As result, FID is better for DDPM in Table 4. For LSUN- ChUUhhhu the distributions are more similar and FID correctly estimates that StyleSwin generates better images than DDPM. In contrast to FID, FWD is not fooled by the class distribution and provides consistent ranking for DDPM and StyleSwin on both datasets, as reported Table
-104
-10
CelebAHQ DDPM
LSUN-Church
I DDPM StyleSwin
0.8
StyleSwin
0.8
0.6
0.6
0.4
0.4
0.2
0.2
triumphal arch J
bencon +
wig
sbit
brosseeee bikini
neck bracce
tench
bow tie
soat belt aboe-
lipstick
ping- ball stethoscupe hair spray
ehureh
hell cale:
palace
moeque
custle
stupo
lakesside
dome
obelisk
vanls
bam
book jacket
lab
fur
(a) CelebAHQ
(b) ShhhChh
Figure 11: Histograms of predicted top-1 classes by the Inception V3 network.
A.6 COMPUTE DETAILS
While the proposed evaluation metric FWD is very efficient, some of the generative models expensive. We used 16 nodes with 4 Nvidia 1 GPUs to generate the samples in Table 4.
A.7 DNDD-DATASET
ONDD-Datasee contains 3600 images with 7 classes and the task requires detecting nutrient deficiency in winter wheat and winter rye, such nitrogen, phosphorous, and potassium deficiencies. The images were captured over the 2019 growth period at the long-term fertilizer experiment (LTFE) Dikopshof near Bonn and were annotated with seven types of fertilizer treatments. We preprocessed
19

---

Published as a conference paper at ICLR 2025
the dataset by splitting the 1000x1000 resolution image into 256x256 crops. This resulted in 57600 images overall. We trained Proj. FastGAN and DDGAN on this preprocessed dataset.
A.8 SENTINRI DATASET
The Sentinel dataset consists of 180,662 triplets of Synthetic Aperture Radar (SAR) image patches collected from Sentinel-1 and Sentinel-2 missions. From these, we only use the images from the ROIs_2011
subset, which contain 31,825 images. We train Proj. FastGAN and DDGAN on this subset. The original images are stored in the "tif" format and conversion to "jpg" is made using the official codebase provided by Schmitt et al. (2019).
A.9 ADDITIONAL METRICS
Here, present the comparison with additional metrics such FID (Chong & Forsyth, 2020), IS (Salimans et al., 2016), IS.0 (Chong & Forsyth, 2020), Clean-FID (Parmar et al., 2022), and KID (Binkowski et al., 2018). Table extends the results from Table The results show that all the stated metrics suffer from domain bias, as they share the same ImageNet pretrained Inception- V3 backbone.
Table 8: Extended comparison of metrics to detect domain bias. All the metrics which share the pretrained InceptionV3 backbone suffer from domain bias, whereas FWD is domain agnostic.
FID...
KIDI,
(lean-FIDD)
IS*
18
FID.
FD-DINOV22
FWD.
Pruj. FasiGAN
6.222
0.0020
6,729
2.925
2.545
6,358
685 685.889 889
1,388
CelcbA-HQ
DDGAN
6.961
0.0034
7.156
2.669
2.315
7.641
199.761
0,408
Proj. FastGAN
4.048
0.0006
4.206
5.358
3.732
4.106
593.124
0.651
FFHQ
StyleGAN2
4.782
0.0011
4.597
5.307
3,714
4.282
420.273
0.312
Proj. FastGAN
5.141
0.0032
5,597
2.461
2.142
4.675
171.655
1.442
DNDD-Dutaset
DDGAN
25.872
0.025
26.427
2.332
2.105
26.233
232.884
1,357
Proj. FasiGAN
5,216
0.0030
9.087
3.846
3.257
8.96
424.898
0.755
Sentinel
DDGAN
26.154
0.0248
23.358
3.647
3.329
23.615
404.700
In addition, Table presents the results of SWD (Karras et al., 2018) and FWD for generated CelebA-HQ images from Proj. FastGAN and DDGAN where we compute each metric five times independently. While SWD is robust to the domain bias, the randomized projections lead very high standard deviation, making this metric unreliable in practice. Our proposed metric FWD is deterministic and provides all runs the same result.
Table 9: Reproducibility of FWD and SWD. We report minimum and mean + standard deviation in brackets across independent runs.
Dataset
Generator
FWDJ
SWD1
Proj. FasIGAN
1.388 (1.388 + 0.00)
169.694 (175.292 3.54)
CelebA-HQ
DDGAN 0.408 (0.408 + 0.00)
99.198 (108.553166.8))
A.10 FID PRETRAINED WITH DNDD
As discussed in the results section, fine-tuning the Inception V3 backbone with not solve the domain bias problem. Since the dataset consists of only 3600 images, the Inception network fails to learn representative features to compute FID. Table 10 shows that fine-tuned FID still prefers Proj. FastGAN.
does
20

---

Published as a conference paper at ICLR 2025
Table 10: Comparison of FID (ImageNet), FID (DNDD) and FWD on DNDD-Dataset, After fine- tuning Inception V3 3 on DNDD-Dataset, FID (DNDD) still prefers Proj. FastGAN whereas FWD ranks DDGAN better.
Dataset
Generator
FID (ImageNet))
FID (DNDD)4
FWD1
Proj. FastGAN
4.675
20,937
1.442
DNDD-Datase
DDGAN
26.233
52.521
1.357
B EXTENDED DISCUSSION OF RELATED WORK
B.I SPECTRAL METHOOS
Prior work found neural networks are spectrally biased (Rahaman et al., 2019) and many architectures favor low- wwwfrrequnccy content (Durall et al., 2020; Gal al., 2021; Wolter et al., 2022; Zhang et al., 2022). Related articles rely on the Fourier or Wavelet transform to understand frequency bias. Wavelet transforms, as pioneered by Mallat (1989) and Daubechies (1992), have solid track record in signal processing. The Fast Wavelet Transform (FWT) and the closely related Wavelet Packet Transform (W2) are starting to appear more frequently in the deep learning literature. Applications include Convolutional Neural Network (CNN) augmentation (Williams & Li, 2018), style transfer (Yoo et al., 2019), image denoising (Liu al., 2020; Saragadam et al., 2023). image coloring (Li al., 2022), face aging (Liu et al., 2019), video enhancement (Wang et al., 2020). face super-resolution (Huang al., 2017), 7), and generative machine learning (Gal et al., 2021; Guth et al., 2022; Zhang et al., 2022: Phung et al., 2023). Hernandez. et al. (2019) use the Fourier transform to measure the quality of human motion forecasting. Zhang et al. (2022) use a FWT to remove artifacts from generated images. Phung et al. (2023) focuse on the FWT to increase the inference speed of diffusion models. This work proposes to use the Wavelet Packet Transform (Wp) as an interpretable metric for generators.
B.2 GENERATIVE ARCHITECTURES
Prior work mainly falls into the three GAN, Diffusion, and Variational AutoEncoder (VAE) archi- tecture groups. The StyleGAN architecture family (Karras al., 2019; 2020; 2021) among the pioneering architectures in generative vision. GANs allow rapid generation of high-quality images but suffer from training instability and poor mode coverage (Salimans et al., 2016). Sauer et al. (2021) proposed the Projected Fast GAN (Proj. FastGAN) architecture, which stabilizes and improves train- ing convergence by introducing ImageNet pre-trained weights into the discriminator. The upgraded discriminator pushes the output distribution towards ImageNet. VAE models (Kingma & Welling, 2014; Van Den Oord et al., 2017), on the other hand, enable the generation of diverse image sets, but are unable to produce high-quality images.
Diffusion models (Sohl-Dickstein et al., 2015; Ho et al., 2020; Peebles & Xie, 2023) have emerged very promising alternative and produce high-quality images (Ho et al., 2020: Dhariwal & Nichol, 2021) in an autoregressive style. DDPMs, for example, are Markovian processes that learn to gradually separate added noise from data during training. During inference. images are generated from Gaussian noise via reverse process that requires iterating through all steps to generate an image. Song et al. (2021) reduced the number of sampling steps by introducing DDIM, which relies on a deterministic non- non-Markovian sampling process. Furthermore, Nichol & Dhariwal (2021) proposed the use strided sampling, to reduce the sampling timesteps and also provide a performance improvement by using cosine instead of linear sampling. Moreover, Nichol & Dhariwal (2021) adopt a weighted variational lower bound to supplement the Mean Squared Error (MSE) loss. In an attempt to solve the generative learning trilemma (image quality, diversity and fast sampling), Xiao et al. (2022) proposed Denoising Diffusion GAN (DDGAN). The paper parameterizes a conditional GAN for the reverse diffusion process and demonstrates faster generation speed.
21

---

Published as a conference paper at ICLR 2025
chicksk
WOIDY WADEK
H
Joipp
Fin
poplers leate
busk de allaaioe
Figure 12: Samples from StableDiffussin v1.5 generated with prompts from the Conceptual Caption validation set.
C ADDITIONAL SAMPLES
C.1 STABLE DIFFUSION
In this section, we provide the generated samples from StableDiffusson models used for user study when evaluated on the Conceptual Captions dataset (Sharma et al., 2018). In particular, we use versions 1.5, 2.1, 3.0 (Medium), and 3.5 (Large), and Figures 12, 13, 14, and 15 represent the samples from these models respectively.
C.2 DDGAN AND PROI.FASTGAN
Here we present the additional samples generated from DDGAN and Proj. FastGAN trained on CelebA-HO, DNDD and Sentinel datasets individually. Figures 16, 17 represent CelebAHQ samples from DDGAN and Proj. FastGAN respectively. Similarly, Figures 18, 19 and Figures 20, 21 depict samples from DNDD and Sentinel datasets, respectively.
22

---

Published as a conference paper at ICLR 2025
180
S
Figure 13: Samples from StableDiffusion v generated with prompts from the Conceptual Caption validation set.
23

---

Published as a conference paper at ICLR 2025
OFD
AO
Figure 14: Samples from StableDiffusion v3.0 (Medium) generated with prompts from the Conceptual
Caption validation set.
24

---

Published as a conference paper at ICLR 2025
Toar dangerns
IN CBE
IN
PENNERE
Figure 15: Samples from StableDiffusion v3.5 (Large) generated with prompts from the Conceptual Caption validation set.
25

---

Published as a conference paper at ICLR 2025
U
20
Sa
Figure 16: Samples from DDGAN trained on the CelebA-HO HQ dataset.
26

---

Published as a conference paper at ICLR 2025
2
STC
Figure 17: Samples from Proj. FastGAN trained on the CelebA-HQ dataset.
27

---

Published as a conference paper at ICLR 2025
Figure 18: Samples from DDGAN trained on the DNDD-Dataset.
28

---

Published as a conference paper at ICLR 2025
Figure 19: Samples from Proj. FastGAN trained on the DNDD-Dataset.
29

---

Published as a conference paper at ICLR 2025
Figure 20: Samples from DDGAN trained on the Sentinel dataset.
30

---

Published as a conference paper at ICLR 2025
Figure 21: Samples from Proj. FastGAN trained on the Sentinel dataset.
31


Reviews:
[{"review_id": "n6jcQKtxIC", "reviewer": "Reviewer_sdD9", "summary": "The paper, \"Fréchet Wavelet Distance: A Domain-Agnostic Metric for Image Generation\", introduces the Fréchet Wavelet Distance (FWD) as a novel metric for evaluating generative models in a domain-agnostic way. FWD leverages the Wavelet Packet Transform (Wp) to project images into a frequency and spatially enriched domain, capturing both the frequency and spatial characteristics of images. This approach enables FWD to evaluate the similarity between generated and real images based on the Fréchet Distance (FD) of their wavelet packet coefficients, providing robustness against domain bias and dataset dependency issues seen in other metrics, such as FID and FD-DINOv2. The authors perform extensive evaluations across various datasets and demonstrate that FWD achieves consistent, domain-agnostic results and outperforms state-of-the-art metrics in terms of robustness and computational efficiency.", "questions": "Please refer to the weakness part", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Domain Independence and Robustness: FWD’s reliance on the wavelet transform offers a substantial improvement in domain-agnostic evaluation, avoiding the biases associated with pre-trained models like InceptionV3. This makes it a more universally applicable metric across datasets and model types.\n2. Efficient and Scalable: FWD is computationally efficient, with much lower FLOPs compared to FD-DINOv2, making it suitable for large-scale evaluation. The method’s wavelet packet decomposition also enables interpretable results by isolating specific frequency bands for detailed analysis.", "weaknesses": "1. Limited Comparison with Alternative Frequency-Based Metrics: Although the paper presents strong results, it lacks comparisons with alternative frequency-based metrics, such as those leveraging Fourier transforms or spectral analysis. This comparison could clarify the specific advantages of wavelet-based decomposition over other spectral methods in generative evaluation.\n2. Potential Alignment with Human Evaluation: FID score has been long complaint against that it can not accurately evaluate the quality of generative models, specifically alignment with human preferences. This paper presents some improvement over FID scores on several GAN and Diffusion models, but a wider range of ablations and even human evaluations could better support the validity of this approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "981xukR1l6", "reviewer": "Reviewer_whKw", "summary": "Modern generative models exhibit frequency biases, while commonly used metrics such as FID , KID and FD-DINOv2 are affected by domain bias.  \nThis paper proposes the Fréchet Wavelet Distance ( FWD ) as a dataset- and domain-agnostic metric for evaluation of generative approaches for image synthesis.    \nIt is  shown that the proposed method is robust to corruption, perturbation, and distractors.  \nAt the same time, its formulation is computationally efficient.", "questions": "Many image quality metirc has been proposed for generative model evaluation, but to my knowledge, FID is still the mostly adopted one. I want to hear the authors opinion on this problem. Will evaluating on SOTA generative models like stable diffusion series help the advocation of a new image quality metric?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed metric Fréchet Wavelet Distance ( FWD ) is simple and has no trained parameters, thus it has high efficiency and is robust to domain, corruption, perturbation, and distractors.    \n\nThe calculation of FWD is much faster than FID and FD-DINOv2.  \n\nOn several dataset, FWD produces more reasonable results than FID and FD-DINOv2.  \n\nBecause of the packet and frequency design, FWD has some interpretability.", "weaknesses": "To make the new FWD metric more convincing, a human subject experiment is necessay, as is conducted in the FD-DINOv2 paper. The human evaluation results in FD-DINOv2 paper might be a useful benchmark.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "B5lijR4Ey6", "reviewer": "Reviewer_Jm3n", "summary": "This paper claims that FID/FD-DINOv2 suffer from the bias of generator and dataset, and proposes the FWD, a domain-agnostic metric based on the Wavelet Packet Transform. The FWD does not rely on any pre-trained network, and thus is general-purpose and dataset-domain agnostic. The author further claims this new metric is more interpretable due to its ability to compute Fréchet distance per packet.", "questions": "See weaknesses. As I am not an expert in this field, I would like to adjust my rating if all the reviewers’ concerns are addressed.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. The proposed solution for enhancing FID is attractive, and the FWD is designed to be general-purpose and dataset-domain agnostic, with a solid foundation.\n2. The experiments are well-considered.", "weaknesses": "1. The paper appears to be missing comparisons with several competitors. While the FWD is compared with FID/FD-DINOv2, this seems insufficient. For instance, metrics such as IS (Inception Score), KID (Kernel Inception Distance), FID_\\infty, IS_\\infty, and Clean FID should also be considered. The authors should either include comparisons with these metrics or provide an explanation for their omission. Notably, FID_\\infty and IS_\\infty are designed to mitigate biases introduced by models, making their inclusion particularly relevant.\n\n2. There is a lack of human evaluation. For a metric evaluating generative models, it would be beneficial to demonstrate alignment with human evaluation results.\n\n3. The underlying mechanism is not entirely clear. Despite the authors’ claim that FWD offers improved transparency, it remains unclear why a hand-designed metric could yield better evaluation results than those based on deep learning techniques. Or you can also explain it.\n\n4. The dataset selection is limited. It appears that all experiments are conducted on just four datasets (face, agricultural, and remote sensing datasets). Including a broader range of datasets, particularly natural images from sources like ImageNet, OpenImage, COCO, or Laion5B, would enhance the robustness of the findings. Or you can also explain it.\n\n[1] Mikołaj Binkowski, Danica J Sutherland, Michael Arbel, and Arthur Gretton. Demystifying mmd gans. arXiv preprint arXiv:1801.01401, 2018.\n[2] Min Jin Chong and David Forsyth. Effectively unbiased fid and inception score and where to find them. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6070–6079, 2020.\n[3] Gaurav Parmar, Richard Zhang, and Jun-Yan Zhu. On aliased resizing and surprising subtleties in gan evaluation, 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)