Fréchet Wavelet Distance: A Domain-Agnostic Metric for Image Generation

## Abstract

Abstract Modern metrics for generative learning like \ac fid demonstrate impressive performance. However, they suffer from various shortcomings, like a bias towards specific generators and datasets.
To address this problem, we propose the \ac fwd as a domain-agnostic metric based on \acf wpt. \ac fwd provides a sight across a broad spectrum of frequencies in images with a high resolution, along with preserving both spatial and textural aspects.
Specifically, we use \ac wpt to project generated and dataset images to packet coefficient space.
Further, we compute Fréchet distance with the resultant coefficients to evaluate the quality of a generator.
This metric is general-purpose and dataset-domain agnostic, as it does not rely on any pre-trained network while being more interpretable because of frequency band transparency.
We conclude with an extensive evaluation of a wide variety of generators across various datasets that the proposed \ac fwd is able to generalize and improve robustness to domain shift and various corruptions compared to other metrics.

## 1 Introduction

With the surge of generative neural networks, especially in the image domain, it becomes important to assess their performance in a robust and reliable way [, , , ]. \acfid [] has emerged as the de facto standard for comparing generative image synthesis approaches, as \acfid scores align well with human perception [] when comparing different runs of the same architecture. However, it also shows various shortcomings, such as its reliance on a pre-trained classification backbone, i.e., InceptionV3 trained on ImageNet.
This, by design, introduces a class dependency into \acfid-scores leading to accidental distortions [] and improved scores if the evaluation set resembles ImageNet or if the use of ImageNet weights in the generator pushes the output distribution towards ImageNet [].

Figure: Figure 1: The first two images depict the same person, while the last image depicts a different person. Intuitively, the first two images are more similar than the other pairs of images. When computing the mean squared error between the images using the penultimate InceptionV3 activations or wavelet packets, we observe that the wavelet packets produce a low distance for the first two images, as expected. Surprisingly, according to InceptionV3, the last two images are similar since both images are classified as ‘microphone’ whereas the first image is classified as ‘groom’. All images are taken from [].
Refer to caption: x1.png

To address the domain bias problem caused by the use of a pre-trained network, we propose an alternative metric based on the \acfwpt. In contrast to other pure frequency [] or spatial [, ] metrics, wavelets have the advantage that they combine both frequency and spatial aspects in one metric.
While frequency information is important [, , , , ] it alone is insufficient to assess the quality of synthesized images without considering additional spatial information.
Wavelets are thus an ideal representation for a metric comparing generative approaches for image synthesis.
As \acfid, we utilize the Fréchet distance between the real and generated set of images as a distance measure, but it is not computed based on InceptionV3 activation maps. Instead, it utilizes the wavelet-packet frequency band representations of \acwpt as illustrated in Figs. 1 and 3.
To this end, we first use \acwpt to transform every image, where we use the wavelet transform at a fixed level. We then compute the Fréchet distance for each packet of the transform and average them over all packets. The proposed \acffwd thus considers the spatial information as well as all frequency bands.

To quantitatively assess those characteristics, we evaluate the proposed metric in terms of its domain bias and robustness. We further compare the proposed \acsfwd to domain-agnostic methods like the \acfssim and the \acpsnr as well as domain-dependent metrics like \acfid and \ackid on five standard datasets. We show that \acfwd is a more robust metric that does not suffer from the domain bias and can thus be applied to any dataset. We also show that some odd \acfid results can be attributed to the dataset bias.

In summary, this paper makes the following contributions:
(1) We propose the \acffwd as a dataset- and domain-agnostic metric for evaluating generative approaches for image synthesis.
(2) \acfwd is an interpretable metric, as the wavelet packet transform splits the frequency space into hierarchically organized, discrete subbands.
(3) We show in an extensive evaluation that the proposed method is robust to corruption, perturbation, and distractors.

## 2 Related Work

### 2.1 Metrics for generative learning

We consider domain-agnostic and domain-dependent metrics.
Traditional image distance metrics are a possible domain-agnostic option to evaluate the quality of generated images. The \acfssim, for example, is a longstanding pairwise image distance metric []. The score takes luminance, contrast, and structure into account.
Along with \acsssim, \acfpsnr is the most used metric for Image quality assessment. \acspsnr is computed as a logarithmic ratio between the maximum pixel value of the original image and the root mean square error between the original image and its noisy approximation [].
Previous work reports \acssim’s preference for blurry images [], which disagrees with human perception.

Having discussed domain-agnostic methods, we now turn to domain-dependent approaches. A generative model should generate novel image samples that mirror the training set sample distribution, including data diversity. In a vision context, [] proposed the \acfis as a measure of image quality, independent of the target dataset statistics. The \acis is computed by measuring the entropy of the class probabilities of an InceptionV3.
The score builds upon the assumption, that a generative network that has converged to a meaningful solution will produce images that will allow InceptionV3 to make predictions with certainty. A certain InceptionV3 has a low prediction entropy. \acis has been found to be sensitive to different Imagenet training runs []. Furthermore, it does not use the statistics of the real data distribution a \acgan is trained to model [].
In response [] proposes \acfid. Instead of measuring the entropy at the final layer \acfid is computed by evaluating the Fréchet distance [] the penultimate network activations computed on both the true and synthetic images. Today, comparing high-level inception net features using an \acfid-score [] enjoys widespread adoption.
Variants exist, \acfkid [], for example, relaxes the multi-variate Gaussian assumption of \acfid and measures the polynomial kernel distance between Inception features of generated and training dataset. [] kept the InceptionV3 backbone and replaced Frécet distance with kernel distance.
While \acfid captures general trends well, the literature also discusses its drawbacks.
[] empricaly studies the effect of ImageNet classes on \acfid for non-ImageNet datasets by using GradCAM. Furthermore [] examines ImageNet bias using \acprojgan and StyleGAN2. Compared to StyleGAN2, \acprojgan produces more accidental distortions. Surprisingly, \acprojgan’s \acfid is comparable to StyleGAN2’s in their experiment.
[] found a generator-dependent architecture bias, which limits our ability to compare samples for smaller datasets with 50K or fewer images.
Additionally, [] found that both \acfid and \ackid are highly sensitive to resizing and compression.
[] reported \acfid sensitivity with respect to different ImageNet training runs.
While comparing Tensorflow and Pytorch implementations, [] measured inconsistent scores due to differing resizing implementations.
Finally, \acfid scores are hard to reproduce unless all details regarding its computation are carefully disclosed [].
The motivates the search for additional quality metrics.

### 2.2 Spectral Methods

Prior work found neural networks are spectrally biased []. Many architectures favor low-frequency content [, , , ]. Related articles rely on the Fourier or Wavelet transform to understand frequency bias. Wavelet transforms as pioneered by [] and [] have a solid track record in signal processing. The \acffwt and the closely related \acfwpt, are starting to appear more frequently in the deep learning literature. Applications include \accnn augmentation [], style transfer [], image denoising [, ], image coloring [], face aging [], video enhancement [], face super-resolution [], and generative machine learning [, , , ]. [] uses the Fourier transform to measure the quality of human motion forecasting. [] uses a \acfwt to remove artifacts from generated images. [] focuses on the wavelet transform to increase the inference speed of diffusion models.
This work proposes to use the \acfwpt as an interpretable metric for image generators.

## 3 Fréchet Wavelet Distance (FWD)

We want to tackle the problem of dataset-domain bias. To this end, we propose \acfwd, which in turn leverages the \acfwpt.
We require two-dimensional filters for image processing where we use Haar wavelets. Consequently, we construct filter quadruples from the original single-dimensional filter pairs. The process uses outer products []:

$$ $\displaystyle\mathbf{h}_{a}=\mathbf{h}_{\mathcal{L}}\mathbf{h}_{\mathcal{L}}^{ T},\mathbf{h}_{h}=\mathbf{h}_{\mathcal{L}}\mathbf{h}_{\mathcal{H}}^{T},\mathbf {h}_{v}=\mathbf{h}_{\mathcal{H}}\mathbf{h}_{\mathcal{L}}^{T},\mathbf{h}_{d}= \mathbf{h}_{\mathcal{H}}\mathbf{h}_{\mathcal{H}}^{T}$ (1) $$

With $a$ for the approximation filter, $h$ for the horizontal filter, $v$ for the vertical filter, and $d$ for the diagonal filter [].
We construct a \acwpt-tree for images with these two-dimensional filters as illustrated
in Fig. 2.
Recursive convolution operations with the filter quadruples, i.e.,

$$ $\displaystyle\mathbf{C}_{\mathit{F}_{l}}*\mathbf{h}_{j}=\mathbf{C}_{\mathcal{F }_{l+1}}$ (2) $$

at every recursion step where $*$ denotes a two-dimensional convolution with a stride of two. The filter codes $\mathcal{F}_{l+1}$ are constructed by applying all $j\in[a,h,v,d]$ filters to the previous filter codes $\mathcal{F}_{l}$. Initially, the set of inputs $\mathit{F}_{l}$ will only contain the original image $\mathbf{C}_{\mathit{F}_{0}}=\{X\}$ as shown in Fig. 2.

Figure: Figure 2: Illustration of the \acfwpt. For visualization purposes, we depict level-3 transform. Even though all experiments later used level-4 transform. Image from [].
Refer to caption: x2.png

At level one, we obtain the result of all four convolutions with the input image and have $\mathit{F}_{1}=[a,h,v,d]$. At level two, we repeat the process for all elements in $\mathit{F}_{1}$. $\mathit{F}_{2}$ now contains two-character keys $[aa,ah,av,ad,\dots,dv,dd]$ as illustrated in Fig. 2. We typically continue this process until level 4 in this paper. We arrange the coefficients in $\mathbf{C}_{\mathcal{F}_{l}}$ as tensors $\mathbf{C}_{l}\in\mathbb{R}^{P,H_{p},W_{p}}$ for the final layer. The total number of packages at every level is given by $P=4^{l}$, and $H_{p}=\frac{H}{4^{l}}$ and $W_{p}=\frac{W}{4^{l}}$ where we denote the image height and width as $H$ and $W$.
We provide more details on \acwpt in the Supplementary.

Figure: Figure 3: \acffwd computation flow-chart. \acswpt denotes the wavelet-packet transform. Not all packet coefficients are shown, dashed lines indicate omissions. We compute individual Fréchet Distances for each frequency band.
Refer to caption: x3.png

Figure 3 illustrates how we compute the \acfwd. The process relies on the wavelet packet transform, as previously discussed. We process $N$ images with $C$ channels in parallel $\mathcal{W}_{p}:I_{s}\in\mathbb{R}^{N\times H\times W\times C}\rightarrow
\mathbf{C}\in\mathbb{R}^{N\times P\times H_{p}\cdot W_{p}\cdot C}$. $H,W$ denotes image height and width as before.
To facilitate the ensuing metric evaluation, we flatten the last axes into $(H_{p}\cdot W_{p}\cdot C)$. Before computing the packets, all pixels are divided by 255 to re-scale all values to [0,1].
The metric is computed in three steps. First, we compute the individual packet mean via

$$ $\displaystyle\mu_{p}(I_{N})=\frac{1}{N}\sum_{n=1}^{N}\mathcal{W}(I_{n})_{p},$ (3) $$

where $I_{n}$ is the $n^{th}$ image in the dataset and $p$ represent the corresponding packet form $P$ packets.
Then we compute the covariance matrix as

$$ $\displaystyle\Sigma_{p}(I_{N})=\frac{1}{N-1}\sum_{n=1}^{N}(\mathcal{W}(I_{n})_ {p}-\mu_{p}(I_{N}))(\mathcal{W}(I_{N})_{p}-\mu_{p}(I_{N}))^{T}$ (4) $$

Here $\mu\in\mathbb{R}^{P\times C\cdot H_{p}\cdot W_{p}}$ estimate the mean across the number of images, and $\Sigma\in\mathbb{R}^{P\times C\cdot H_{p}\cdot W_{p}\times C\cdot H_{p}\cdot W
_{p}}$ represents the covariance among all the coefficients.
Now we are ready to compute the distances given the packet mean and covariance values,

$$ $\text{FD}_{p}(r,g)=d(\mathcal{N}(\mu_{r_{p}},\Sigma_{r_{p}}),\mathcal{N}(\mu_{ g_{p}},\Sigma_{g_{p}}))^{2}=||\mu_{r_{p}}-\mu_{g_{p}}||_{2}^{2}+\text{tr}[ \Sigma_{r_{p}}+\Sigma_{g_{p}}-2\sqrt{\Sigma_{r_{p}}\Sigma_{g_{p}}}].$ (5) $$

With $r$ and $g$ denoting the real and generated images. tr denotes the trace operation. Utilising the above computed per-packet statistics for both real $(\mu_{r},\Sigma_{g})$ and generated samples $(\mu_{r},\Sigma_{g})$, we measure the mean of Fréchet Distance (Equation 5) across all packets

$$ $\displaystyle\text{FWD}=\frac{1}{P}\sum_{p=1}^{P}d(\mathcal{N}(\mu_{r_{p}}, \Sigma_{r_{p}}),\mathcal{N}(\mu_{g_{p}},\Sigma_{g_{p}}))^{2}.$ (6) $$

By averaging the distances of all frequency bands, the \acfwd captures frequency information across the spectrum.

## 4 Experiments

Our first series of experiments reevaluates \acfid, \ackid, PSNR, \acssim and \acfwd on six standard datasets. All our experiments rely on the same code base.
Implementation We use PyTorch [] for neural network training and evaluation and compute \acfid using [] as recommended by [].
We work with the wavelet filter coefficients provided by PyWavelets []. We chose the PyTorch-Wavelet-Toolbox [] software package for GPU support.
Furthermore, we employ scikit-image [] to compute \acssim and \acpsnr scores. \ackid is computed using the code from [].

### 4.1 Comparison to State-of-the-art

**Table 1: Comparing recent generative models using \acffwd,\acffid, \acfssim, \acfkid, and \acfpsnr.**
| Dataset | Image Size | Method | \acsfid$\downarrow$ | \ackid$\downarrow$ | PSNR$\uparrow$ | \acsssim$\uparrow$ | \acsfwd$\downarrow$ (ours) |
| --- | --- | --- | --- | --- | --- | --- | --- |
| CelebAHQ | 256 | \acsddim [] | 32.333 | 0.0313 | 9.467 | 0.255 | 12.317 |
| \acsddpm [] | 19.101 | 0.0152 | 9.538 | 0.277 | 4.697 |  |  |
| StyleSwin [] | 23.257 | 0.0264 | 9.105 | 0.275 | 1.528 |  |  |
| StyleGAN2 [] | 15.439 | 0.0155 | 8.906 | 0.253 | 0.476 |  |  |
| \acsddgan [] | 7.203 | 0.0034 | 8.976 | 0.258 | 0.408 |  |  |
| Churches | 256 | \acsddim [] | 11.775 | 0.0043 | 8.701 | 0.007 | 4.919 |
| \acsddpm [] | 9.484 | 0.0036 | 8.548 | 0.001 | 3.546 |  |  |
| StyleSwin [] | 3.187 | 0.0005 | 9.536 | 0.005 | 2.835 |  |  |
| StyleGAN2 [] | 4.309 | 0.0007 | 8.804 | 0.009 | 0.753 |  |  |
| Bedrooms | 256 |  |  |  |  |  |  |
| \acsddim [] | 25.857 | 0.0094 | 9.258 | 0.007 | 9.521 |  |  |
| \acsddpm [] | 16.251 | 0.0058 | 8.791 | 0.011 | 5.187 |  |  |
| CIFAR10 | 32 | \acsddim [] | 32.531 | 0.0239 | 8.461 | 0.011 | 14.803 |
| \acsddpm [] | 9.978 | 0.0075 | 8.975 | 0.013 | 2.506 |  |  |
| Imp. Diff. (VLB) [] | 15.067 | 0.3779 | 9.053 | 0.012 | 0.641 |  |  |
| StyleGAN2 [] | 17.313 | 0.0201 | 8.801 | 0.012 | 0.554 |  |  |
| Imp. Diff. (Hybrid) [] | 16.892 | 0.0395 | 8.873 | 0.016 | 0.303 |  |  |
| \acsddgan [] | 3.465 | 0.0008 | 8.811 | 0.012 | 0.211 |  |  |
| ImageNet | 64 |  |  |  |  |  |  |
| Imp. Diff. (VLB) [] | 42.257 | 0.1132 | 8.829 | 0.009 | 2.221 |  |  |
| BigGAN | 5.327 | 0.0731 | 8.529 | 0.054 | 0.441 |  |  |
| Imp. Diff. (Hybrid) [] | 3.947 | 0.0724 | 8.472 | 0.055 | 0.392 |  |  |

To understand the spectral qualities of existing generative methods for image synthesis, we evaluated various diffusion and \acgan models across a wide range of benchmarks.
Datasets We compare common metrics and our \acfwd on \acscifar10 [], \accelebahq [], the Church and Bedroom subsets of the \aclsun dataset [], and finally ImageNet []. In order to retain consistent spatial and frequency characteristics across various image sizes, we use level 4 packet transform for 256x256 images.
Furthermore, we reduce the transformation level by factor of 1 for reduction in image size by half.
Generators For the evaluation, we use the diffusion approaches \acddpm [], \acddim [], Improved Diffusion [], Diffusion Transformer [], as well as the GAN approaches DDGAN [], StyleGAN2 [], StyleSwin [].
We used the publicly available code for all approaches and compared the unconditionally generated images to the original dataset.
Hyperparameters No optimization is done in this section. All generators run with weights as provided by the respective paper authors.
Metrics We compute \acfid, \ackid, PSNR, \acssim [] and finally our own \acfwd. Table 1 lists all numbers. \acfid-scores are from the standard implementation by [].
For CIFAR10, we use 50K images to evaluate both metrics. The ImageNet numbers are computed with 50K images from the validation set. For CelebAHQ and LSUN we work with 30K images.

Considering \accelebahq, \acfid, \ackid, and \acfwd agree most of the time. Only, \acsddpm and StyleSwin are swapped. \acfwd therefore delivers a comparable quality signal in this case.
The ordering remains largely unchanged for LSUN churches. In terms of \acfwd, we observe a stable ranking across the two datasets. Except \acddpm and StyleSwin, which are swapped.
Given the different nature of both datasets, we argue small changes like these are reasonable. We observe larger changes in magnitude for \acfid-scores when making the switch from \accelebahq to
LSUN churches. The switch moves us towards ImageNet, since "church" is an ImageNet class, but "face" is not.
Figure 10(b) depicts the histograms of top-15 classes classified by InceptionV3 for LSUN churches, \acsddpm and StyleSwin.
We observe that StyleSwin matches the activation histograms of LSUN churches more accurately than the \acddpm. This observation is a manifestation of domain bias and explains the \acsfid inconsistency for both generators.

We also consider the LSUN-Bedrooms and ImageNet 64 data sets, where \acfid and \acfwd agree. We expect pristine performance for \acfid on ImageNet since this setting is perfectly in its data-domain.
Our \acfwd produces the same ranking on \acscifar10 and \acscelebahq, while \acfid is not consistent. We argue the inconsistency stems from a shift in the underlying data bias and does not truly reflect generator quality.

### 4.2 Effect of domain bias

We noticed in the previous section that metrics which rely on ImageNet features focus on ImageNet-related information. This behavior is desired when we score generators on ImageNet or similar datasets. When working with unrelated data sets, this behavior is misleading.
Data-Sets This section studies \accelebahq [], \acffhq and, \acsagriculture a dataset of Agricultural images []. These data sets contain images that are very different from those in the ImageNet-Set. More information about \acagriculture can be found in the supplementary material.
Generators We study data-set domain bias effects using the \acddgan, \acprojgan and StyleGAN2 networks. \acprojgan is particularly inteteresting. To improve training convergence its discriminator relies on ImageNet weights []. Prior work found this architecture to improve \acfid on image datasets far from ImageNet [].
Hyperparameters To examine the effect of data-set bias, we require generators, which are tuned to produce output that resembles our datasets’ distribution.
We train the \acprojgan for 100 epochs on \accelebahq and separately on the \acagriculture. In both cases with a learning rate of 2e-4 and a batch size of 64 on 8 A100 GPUs. For \acffhq, weights are available. Pretrained weights are also available for \acddgan on CelebAHQ []. On the \acagriculture, we train \acddgan for 150 epochs with a learning rate of 1e-4 and batch size of 8 on the same hardware.
For StyleGAN2 we the pretrained weights with the code from [].

Figure: (a) \acsprojgan on \acscelebahq \acsfid: 6.358 \acsfwd: 1.388
Refer to caption: x4.png

**Table 2: Comparison of biased and unbiased Frèchet distances. \acfid denotes the standard formulation using an InceptionV3 trained on ImageNet. To eliminate the imagenet bias we re-trained InceptionV3. The \acfid (CelebA) and \acfid (DNDD-Dataset) columns list the distances measured using the retrained networks as a backbone. We find that re-training is possible only for large datasets where sufficient data is available. \acfwd consistently provides a meaningful metric.**
| Dataset | Generator | \acsfid (ImageNet) $\downarrow$ | \acsfid (CelebA) $\downarrow$ | \acsfid (\acsagriculture)$\downarrow$ | \acsfwd(ours)$\downarrow$ |
| --- | --- | --- | --- | --- | --- |
| \acs<br>celebahq | \acsprojgan | 6.358 | 5.602 | 1.824 | 1.388 |
| \acsddgan | 7.641 | 3.145 | 0.558 | 0.408 |  |
| \acs<br>agriculture | \acsprojgan | 4.675 | 0.861 | 20.937 | 1.442 |
| \acsddgan | 26.233 | 2.325 | 52.521 | 1.357 |  |
| \acs<br>ffhq | \acsprojgan | 4.106 | 2.204 | 3.549 | 0.651 |
| StyleGAN2 | 4.282 | 0.897 | 0.393 | 0.312 |  |

Table 2 lists the \acfid, and \acfwd values for all datasets images from the aforementioned generators.
Across all datasets, ImageNet \acfid prefers \acprojgan images. Figure 4(a) and 4(b) depict images from the \accelebahq variants of \acprojgan and \acsddgan.
Deformations are visible in the images from \acprojgan on the left. Generally, we found more deformations in \acprojgan images compared to \acsddgan images. This confirms what we see in the random sample in Figures 4(a) and 4(b). Consequently, ImageNet \acfid’s preference is surprising, as we would expect DDGAN to come out on top. \acfwd correctly places DDGAN ahead of \acprojgan. Our results on \acffhq in the bottom row of Table 2 confirm this pattern.

Additional results on the \acagriculture further corroborate this argument. Once more \acfid prefers \acprojgan, this time with a significant margin. Figures 4(c) and 4(d) show both networks have converged and produce meaningful images.
To confirm the bias on \acagriculture, we perform histogram matching for the top-15 as recognized by InceptionV3 trained on ImageNet. The histograms appear in Figure 11.
We observe that \acprojgan pushes InceptionV3 to produce a more ImageNet-like activation distribution, which improves the score. However, in this case this distribution is irrelevant. Consequently, \acfid scores are unreliable.
\acfwd in turn, produces scores which place DDGAN slightly ahead of \acprojgan. A result consistent with earlier experiments.

Re-training InceptionV3 could be a solution. The network features will be relevant if we can find a classification problem close to the distribution of interest.
We train InceptionV3 on \acceleba. \acceleba comes with 40 facial attributes, which we use to train. Once converged we see an exact match ratio of 90%.
We proceed by computing \acfid with the retrained backbone. The \acfid (CelebA) column of Table 2, lists the corresponding scores. The order inverts. \acfid (CelebA) and \acfwd now agree.

However, in the case of the agricultural data set, the retrained \acfid (CelebA) remains biased, while \acfwd produces meaningful domain agnostic results. \acagriculture contains 3600 images with 7 classes. Networks are tasked with detecting nutrient deficiency in the soil, such as Sodium, Calcium, unfertilized, and 4 others. Once more, we use a re-trained InceptionV3 backbone for the \acfid computation. In comparison to \acceleba or ImageNet, this is a small data set. The small size explains the problems. We trained until convergence, but the data set is too small. The re-trained network does to provide meaningful features.
We believe this is a very interesting use case since it illustrates that the \acfwd is not just free from data bias, it also provides meaningful feedback for low resource tasks, where retraining InceptionV3 is not feasible.

### 4.3 \ac fwd interpretability

Figure: (a) Filter order
Refer to caption: x8.png

A generative metric is interpretable if and only if we can understand the underlying mechanics that produce the ranking. This section explains the decisions made by \acfwd in one specific case.
Dataset and Generators We use \accelebahq and we focus on samples from \acddgan and StyleGAN2.

Section 3 formulates \acfwd as an average of per packet \acfwd scores.
This design choice allows us to understand the overall \acfwd-score in terms of the individual packet coefficients for each frequency band.
Figure 5(b) and 5(c) depict the mean absolute difference per packet between the original \accelebahq and generated samples from StyleGAN2 and \acddgan respectively.
Figure 5(d) presents both generators’ per-packet \acfwd.
Figure 5(d) shows that \acddgan has a lower Fréchet distance per packet than StyleGAN2 across the entire spectrum. (b) and (c) confirm \acfwd because the mean packet difference is larger for StyleGAN2 (b) than \acddgan (c).
Overall, we observe that the mean absolute packets translate into per packet Frécet distance-scores, which validates the \acfwd overall.

### 4.4 Evaluation of Robustness

The section follows up on prior work by Kynkäänniemi et al. [].
The authors generate a large set of samples and find a specific combination of images with an optimal \ac*fid. Weighted sampling produces possible combinations. The process chooses images according to a corresponding weight. The weights or drawing probabilities are optimized with \acfid as the objective function. We follow this process and sample 50K images from a large set with optimized weights as probabilities. We employ generated images from StyleGAN2 and real-world images from the \acffhq dataset.
Table 3 lists the resulting \acfid and \acfwd scores. We observe that \acfwd is robust to \acfid optimization. We speculate this may be the result of its independent vision space.
It is of course possible to optimize our metric this way, but it will be tougher to optimize \acfid and \acfwd jointly as their feature representation space is different.

**Table 3: StyleGAN2-FFHQ matching fringe features with 250K images with mean and std for 5 runs.**
| Metric | Random Images | FID-Optimized Images | Change |
| --- | --- | --- | --- |
| FID | 4.278 $\pm$ 0.019 | 2.031 $\pm$ 0.005 | -47% |
| KID | 0.00118 $\pm$ 0.00019 | 0.0003 $\pm$ 0.00019 | -25% |
| FWD | 0.338 $\pm$ 0.017 | 0.398 $\pm$ 0.009 | +2.5% |

Figure: (a)

In addition to \acfid optimization, we study the impact of image perturbation in Figure 6. We find that \acfwd is closer to a bijective mapping in the presence of perturbation than \acfid. This behavior is desirable since we would always expect a larger distance if for example more noise is added. This is not always the case for \acfid. Consider for example the last quarter of the uniform noise intensity in (b), where \acfid falls even though more noise is added. \acssim responds in a very clean way to image perturbations in this case. But the metric is pairwise in nature, which limits its appeal in unconditional settings such as the one we are considering here.

## 5 Limitations and Impact

The \acfwd is a useful addition for tasks far from ImageNet or when a frequency-based metric is desired. For generative tasks on ImageNet, its utility will be limited since we expect \acfid or \ackid to perform well in this case. Furthermore, for datasets close to ImageNet \acfid captures semantic meaning, where \acfwd will instead rely more on frequency and spatial information.
While \acfwd is robust to \acfid, optimization is presented by []. It is certainly also possible to optimize \acfid and \acfwt jointly. We expect this task to be harder since \acfwd must also appear in the cost function.
This paper re-evaluates generative learning methods. We do not expect our evaluation to negatively impact society.

## 6 Conclusion

Modern generative models are biased in frequency[], while modern metrics like \acfid and \ackid come with a domain bias [].
Addressing both problems requires metrics that consider frequency information without introducing a domain bias. Existing domain-agnostic metrics like \acssim
and \acpsnr are pairwise metrics and, therefore, unsuitable in unconditional generation settings.
Consequently, this paper proposed a robust, interpretable, and domain-agnostic metric built on the wavelet packet transform for comparing image synthesis approaches.

## 7 Acknowledgements

This research was funded by Bundesministerium für Bildung und Forschung (BMBF) via WestAI under grant no. “01IS22094E WEST-AI” and BNTrAInee project under “16DHBK1022”.
We thank Gauss Centre for Supercomputing e.V. (www.gauss-centre.eu) for supporting this project by providing computing time through the John von Neumann Institute for Computing (NIC) on the GCS Supercomputer JUWELS at Jülich Supercomputing Centre (JSC).
Finally, we would like to thank Dr. Bartosz Kostrzewa for helping with the experiments on JSC and Michael Kuckertz from the HPC team for providing last-minute resources.

## Appendix A Supplementary material

### A.1 Acronyms

[heading=None]

### A.2 The fast wavelet and wavelet packet transforms

This supplementary section summarizes key wavelet facts as a convenience for the reader.
See, for example, [, ] or [] for excellent detailed introductions to the topic.

The \acffwt relies on convolution operations with filter pairs.

Figure: Figure 7: Overview of the \acffwt computation. $\mathbf{h}_{\mathcal{L}}$ denotes the analysis low-pass filter and $\mathbf{h}_{\mathcal{H}}$ the analysis high pass filter. $\mathbf{f}_{\mathcal{L}}$ and $\mathbf{f}_{\mathcal{H}}$ the synthesis filter pair. $\downarrow_{2}$ denotes downsampling with a factor of two, $\uparrow_{2}$ means upsampling. The analysis transform relies on stride two convolutions. The synthesis or inverse transform on the right works with stride two transposed convolutions. $\mathbf{H}_{k}$ and $\mathbf{F}_{k}$ with $k\in[\mathcal{L},\mathcal{H}]$ denote the corresponding convolution operators.

Figure 7 illustrates the process. The forward or analysis transform
works with a low-pass $\mathbf{h}_{\mathcal{L}}$ and a high-pass filter $\mathbf{h}_{\mathcal{H}}$.
The analysis transform repeatedly convolves with both filters,

$$ $\displaystyle\mathbf{x}_{s}*_{1}\mathbf{h}_{k}=\mathbf{c}_{k,s+1}$ (7) $$

with $k\in[\mathcal{L},\mathcal{H}]$ and $s\in\mathbb{N}_{0}$ the set of natural numbers, where $\mathbf{x}_{0}$ is equal to
the original input signal $\mathbf{x}$. At higher scales, the \acfwt uses the low-pass filtered result as input,
$\mathbf{x}_{s}=\mathbf{c}_{\mathcal{L},s}$ if $s>0$. And $*_{1}$ as the 1d-convolution operation.
The dashed arrow in Figure 7 indicates that we could continue to expand the \acfwt tree here.

The \acfwpt additionally expands the high-frequency part of the tree.

Figure: Figure 8: Scematic drawing of the full \acfwpt in a single dimension. Compared to Figure 7, the high-pass filtered side of the tree is expanded, too.

A comparison of Figures 7 and 8 illustrates this difference.
Whole expansion is not the only possible way to construct a wavelet packet tree. See [] for a discussion of other options.
In both figures, capital letters denote convolution operators. These may be expressed as Toeplitz matrices [].
The matrix nature of these operators explains the capital boldface notation.
Coefficient subscripts record the path that leads to a particular coefficient.

We construct filter quadruples from the original filter pairs to process two-dimensional inputs. The process uses outer products []:

$$ $\displaystyle\mathbf{h}_{a}=\mathbf{h}_{\mathcal{L}}\mathbf{h}_{\mathcal{L}}^{ T},\mathbf{h}_{h}=\mathbf{h}_{\mathcal{L}}\mathbf{h}_{\mathcal{H}}^{T},\mathbf {h}_{v}=\mathbf{h}_{\mathcal{H}}\mathbf{h}_{\mathcal{L}}^{T},\mathbf{h}_{d}= \mathbf{h}_{\mathcal{H}}\mathbf{h}_{\mathcal{H}}^{T}$ (8) $$

With $a$ for approximation, $h$ for horizontal, $v$ for vertical, and $d$ for diagonal [].
We can construct a \acwpt-tree for images with these two-dimensional filters.

Figure: Figure 9: Two dimensional \acfwpt computation overview. $\mathbf{X}$ and $\hat{\mathbf{X}}$ denote input image and reconstruction respectively. We compute the \acffwd using the wavelet packet coefficients $\mathbf{p}$. The transform is invertible, the distance computation is therefore based on a lossless representation.

Figure 9 illustrates the computation of a full two-dimensional wavelet packet tree.
More formally, the process initially evaluates

$$ $\displaystyle\mathbf{x}_{0}*\mathbf{h}_{j}=\mathbf{c}_{j,1}$ (9) $$

with $\mathbf{x}_{0}$ equal to an input image $\mathbf{X}$, $j\in[a,h,v,d]$, and $*$ for two-dimensional convolution. At higher scales, all resulting coefficients from previous scales serve as inputs. The four filters are repeatedly convolved with all outputs to build the full tree. The inverse transforms work analogously. We refer to the standard literature [, ] for an extended discussion.

Compared to the \acfwt, the high-frequency half of the tree is subdivided into more bins, yielding a fine-grained view of the entire spectrum.
We always show analysis and synthesis transforms to stress that all wavelet transforms are lossless. Synthesis transforms reconstruct the original input based on the results from the analysis transform.

### A.3 Histogram matching - InceptionV3

Figure: (a) CelebAHQ

Figure: Figure 11: Top-15 classes from InceptionV3 network for \acagriculture dataset, and images generated using Projected FastGAN and DDGAN. The distribution of top-15 classes for \acprojgan matches with the dataset better than the \acddgan.

### A.4 Generative Architectures

Prior work mainly falls into the three \acgan, Diffusion, and \acvae architecture groups. The StyleGAN architecture family [, , ] is among the pioneering architectures in generative vision. \acgan’s allow rapid generation of high-quality images but suffer from training instability and poor mode coverage []. [] proposes the \acfprojgan-architecture, which stabilizes and improves training convergence by introducing ImageNet pre-trained weights into the discriminator. The upgraded discriminator pushes the output distribution towards ImageNet.
\acvae models, on the other hand, [, ] enable the generation of diverse image sets, but are unable to produce high-quality images.

Diffusion models [, , ] have emerged as a very promising alternative and produce high-quality images [, ] in an autoregressive style. \acpddpm, for example, are Markovian processes that learn to gradually separate added noise from data during training. During infernce images are generated from Gaussian noise via a reverse process that requires iterating through all steps to generate an image. [] reduced the number of sampling steps by introducing \acddim, which rely on a deteministic non-Markovian sampling process.
Furthermore, [] proposed the use of strided sampling, to reduce the sampling timesteps and also provide a performance improvement by using cosine- instead of linear sampling. Moreover, [] adopt a weighted variational lower bound to supplement the \acmse loss.
In an attempt to solve the generative learning trilemma (image quality, diversity and fast sampling), [] proposed \acfddgan. The paper parameterizes a conditional GAN for the reverse diffusion process and demonstrates faster generation speed.

### A.5 Compute details

In principle, our experiments run on single GPUs. For some experiments, we used up to 16 nodes with 4 Nvidia A40 GPUs each at a large scientific computing centre.

### A.6 \ac agriculture

agriculture contains 3600 images with 7 classes. Networks are tasked with detecting nutrient deficiency in the soil, such as Sodium, Calcium, unfertilized, and 4 others.
The images of sugar beet are captured over the 2019 growth period. Capturing occurred at the long-term fertilizer experiment (LTFE) Dikopshof near Bonn. The images were annotated with seven types of fertilizer treatments. The dataset is used for image classification and domain adaptation.

We train the \acprojgan and \acddgan models on this dataset.
We further preprocessed the dataset by splitting the 1000x1000 resolution image to 256x256 resolutions. This resulted in 57600 images overall.
The training details are further provided in the main paper.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: Please refer to the weakness part
soundness: 3
strengths: 1. Domain Independence and Robustness: FWD’s reliance on the wavelet transform offers a substantial improvement in domain-agnostic evaluation, avoiding the biases associated with pre-trained models like InceptionV3. This makes it a more universally applicable metric across datasets and model types.
2. Efficient and Scalable: FWD is computationally efficient, with much lower FLOPs compared to FD-DINOv2, making it suitable for large-scale evaluation. The method’s wavelet packet decomposition also enables interpretable results by isolating specific frequency bands for detailed analysis.
summary: The paper, "Fréchet Wavelet Distance: A Domain-Agnostic Metric for Image Generation", introduces the Fréchet Wavelet Distance (FWD) as a novel metric for evaluating generative models in a domain-agnostic way. FWD leverages the Wavelet Packet Transform (Wp) to project images into a frequency and spatially enriched domain, capturing both the frequency and spatial characteristics of images. This approach enables FWD to evaluate the similarity between generated and real images based on the Fréchet Distance (FD) of their wavelet packet coefficients, providing robustness against domain bias and dataset dependency issues seen in other metrics, such as FID and FD-DINOv2. The authors perform extensive evaluations across various datasets and demonstrate that FWD achieves consistent, domain-agnostic results and outperforms state-of-the-art metrics in terms of robustness and computational efficiency.
weaknesses: 1. Limited Comparison with Alternative Frequency-Based Metrics: Although the paper presents strong results, it lacks comparisons with alternative frequency-based metrics, such as those leveraging Fourier transforms or spectral analysis. This comparison could clarify the specific advantages of wavelet-based decomposition over other spectral methods in generative evaluation.
2. Potential Alignment with Human Evaluation: FID score has been long complaint against that it can not accurately evaluate the quality of generative models, specifically alignment with human preferences. This paper presents some improvement over FID scores on several GAN and Diffusion models, but a wider range of ablations and even human evaluations could better support the validity of this approach.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: Many image quality metirc has been proposed for generative model evaluation, but to my knowledge, FID is still the mostly adopted one. I want to hear the authors opinion on this problem. Will evaluating on SOTA generative models like stable diffusion series help the advocation of a new image quality metric?
soundness: 3
strengths: The proposed metric Fréchet Wavelet Distance ( FWD ) is simple and has no trained parameters, thus it has high efficiency and is robust to domain, corruption, perturbation, and distractors.    

The calculation of FWD is much faster than FID and FD-DINOv2.  

On several dataset, FWD produces more reasonable results than FID and FD-DINOv2.  

Because of the packet and frequency design, FWD has some interpretability.
summary: Modern generative models exhibit frequency biases, while commonly used metrics such as FID , KID and FD-DINOv2 are affected by domain bias.  
This paper proposes the Fréchet Wavelet Distance ( FWD ) as a dataset- and domain-agnostic metric for evaluation of generative approaches for image synthesis.    
It is  shown that the proposed method is robust to corruption, perturbation, and distractors.  
At the same time, its formulation is computationally efficient.
weaknesses: To make the new FWD metric more convincing, a human subject experiment is necessay, as is conducted in the FD-DINOv2 paper. The human evaluation results in FD-DINOv2 paper might be a useful benchmark.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 2
questions: See weaknesses. As I am not an expert in this field, I would like to adjust my rating if all the reviewers’ concerns are addressed.
soundness: 2
strengths: 1. The proposed solution for enhancing FID is attractive, and the FWD is designed to be general-purpose and dataset-domain agnostic, with a solid foundation.
2. The experiments are well-considered.
summary: This paper claims that FID/FD-DINOv2 suffer from the bias of generator and dataset, and proposes the FWD, a domain-agnostic metric based on the Wavelet Packet Transform. The FWD does not rely on any pre-trained network, and thus is general-purpose and dataset-domain agnostic. The author further claims this new metric is more interpretable due to its ability to compute Fréchet distance per packet.
weaknesses: 1. The paper appears to be missing comparisons with several competitors. While the FWD is compared with FID/FD-DINOv2, this seems insufficient. For instance, metrics such as IS (Inception Score), KID (Kernel Inception Distance), FID_\infty, IS_\infty, and Clean FID should also be considered. The authors should either include comparisons with these metrics or provide an explanation for their omission. Notably, FID_\infty and IS_\infty are designed to mitigate biases introduced by models, making their inclusion particularly relevant.

2. There is a lack of human evaluation. For a metric evaluating generative models, it would be beneficial to demonstrate alignment with human evaluation results.

3. The underlying mechanism is not entirely clear. Despite the authors’ claim that FWD offers improved transparency, it remains unclear why a hand-designed metric could yield better evaluation results than those based on deep learning techniques. Or you can also explain it.

4. The dataset selection is limited. It appears that all experiments are conducted on just four datasets (face, agricultural, and remote sensing datasets). Including a broader range of datasets, particularly natural images from sources like ImageNet, OpenImage, COCO, or Laion5B, would enhance the robustness of the findings. Or you can also explain it.

[1] Mikołaj Binkowski, Danica J Sutherland, Michael Arbel, and Arthur Gretton. Demystifying mmd gans. arXiv preprint arXiv:1801.01401, 2018.
[2] Min Jin Chong and David Forsyth. Effectively unbiased fid and inception score and where to find them. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6070–6079, 2020.
[3] Gaurav Parmar, Richard Zhang, and Jun-Yan Zhu. On aliased resizing and surprising subtleties in gan evaluation, 2022.

Meta-review: Summary: The paper presents FWD as a novel, domain-agnostic metric for evaluating images in generative models. It uses the Wavelet Packet Transform and measures similarity between generated and real images based on the Frechet Distance of the wavelet coefficients, effectively addressing the domain bias problem. The proposed FWD metric is computationally efficient as shown by evaluations on various datasets.  

Strength: FWD is a simple, parameter-free metric, efficient to compute metric. It’s also shown to be robust to domain variations. The experimental results support the paper claims

Weakness: The scale of the experiments is rather smaller. Can this be directly used to re-evaluate text-to-image generation leaderboards? Human evaluation is provided only after requested by the reviewers. I encourage authors to include the human evaluation in the main body of the paper, rather than the supplementary. 

Acceptance Reason: It addresses an important problem in image quality evaluation in generative models. FID is studied to be susceptible to biases in the generator or the dataset. This work explores a domain-agnostic and efficient metric, which will benefit the community as an alternative metric, and inspire future research in studying related-metrics at a larger scale.

Decision: Accept (Poster)