Controlling the Fidelity and Diversity of Deep Generative Models via Pseudo Density
Authors: Shuangqi Li
Chen Liu
Tong Zhang
Hieu Le
Sabine Susstrunk
Mathieu Salzmann
Venue: Accepted by TMLR ()

Abstract: We introduce an approach to bias deep generative models, such as GANs and diffusion models, towards generating data with either enhanced fidelity or increased diversity. Our approach involves manipulating the distribution of training and generated data through a novel metric for individual samples, named pseudo density, which is based on the nearest-neighbor information from real samples. Our approach offers three distinct techniques to adjust the fidelity and diversity of deep generative models: 1) Per-sample perturbation, enabling precise adjustments for individual samples towards either more common or more unique characteristics; 2) Importance sampling during model inference to enhance either fidelity or diversity in the generated data; 3) Fine-tuning with importance sampling, which guides the generative model to learn an adjusted distribution, thus controlling fidelity and diversity. Furthermore, our fine-tuning method demonstrates the ability to improve the Frechet Inception Distance (FID) for pre-trained generative models with minimal iterations.

Published in Transactions on Machine Learning Research (10/2024)
Controlling the Fidelity and Diversity of Deep Generative Models via Pseudo Density
Shuangqi Li
diuonggi.l@@epf...
School of Computer and Communication Sciences, EPFL
Chen Liu
chen. Mien.iu@esttyy..ee edu.
Department of Computer Science, City University of Hong Kong
Tong Zhang
Bingzhhang@eff.
School of Computer and Communication Sciences, EPFL
Hieu Le
minh.l@@efft.c
School of Computer and Communiication Sciences, EPFL
Sabine Süisstrunk
sabine.sustruwkkeepp.ddd
School of Computer and Communication Sciences, EPFL
Mathieu Salzmann
muthice...llmannneepp...
School of Computer and Communication Sciences, EPFL
Reviewed on OpenReview: https: // openreviewwee.e. net/ forum? 8VkIBmg3s
Abstract
We introduce an approach to bias deep generative models, such as GANs and diffusion models, towards generating data with either enhanced fidelity or increased diversity. Our approach involves manipulating the distribution of training and generated data through novel metric for individual samples, named pseudo density, which is based on the nearest- neighbor information from real samples. Our approach offers three distinct techniques to adjust the fidelity and diversity of deep generative models: 1) Per-sample perturbation, enabling precise adjustments for individual samples towards either more common or more unique characteristics; 2) Importance sampling during model inference to enhance either fidelity or diversity the generated data; 3) Fine-tuning with importance sampling, which guides the generative model learn an adjusted distribution, thus controlling fidelity and diversity. Furthermore, our fine-tuning method demonstrates the ability improve the Frechet Inception Distance (FID) for pre-trained generative models with minimal iterations.
1 Introduction
The advent deep generative models has revolutionized the field of image generation. Key developments marked by Variational Autoencoders (Kingma Welling 2013), Generative Adversarial Networks (Goodfel-
low et al. 2014h, and the more emergent diffusion models (Ho et al. 2020 Sohl-Dickstein
al. 2015), have
demonstrated umprecedented capability in producing high-quality images. They have been followed by
remarkable results in applications such classification (Xu et al. 2021 Xu & Le 2022 Xu et al.
2023a
super-resolution [Ledig et al. 2017, Li et al., 2022). face generation Karras et al. 2019
2020
2021 and
lexl-to-image generation (Ramesh el al. 2022 Saharia et al. 2022 Rombach et al. 2022) However, despite
these advancements, the critical challenge of balancing fidelity (the realism of the generated images (Xul ct al. 2024)) and diversity (the variety in the generated images) remains unaddressed, despite this ability being essential for the practical applicability of these models. Our work focuses on tackling this crucial need for improved control mechanisms in generative models.

---

Published in Transactions on Machine Learning Research (10/2024)
To this end, we propose to enhance control over generated images through the probability density of image data. Facing the challenge of directly estimating the density on the complex manifold of image data, we intro- duce a novel surrogate metric called pseudo density, which leverages the nearest-neighbo information of real samples in the feature space, utilizing an image feature extractor such as a Vision Transformer (Dosovitskiy et al. 2020p. By calculating the pseudo density of the generated images, we can estimate their likelihood of appearing in the real distribution. Notably, we observe a correlation between pscudo density and the realism as well as the uniqueness of generated images. Based on the pseudo density of samples, we can modulate the probability of specific samples in both the real data and generated data. Specifically, we introduce three distinct methods to enhance the fidelity and diversity of deep generative models: 1) Per-sample perturbation, allowing precise manipulation of realism and uniqueness in any individual generated image via adversarial perturbation with pseudo density as the objective; 2) Importance sampling during model inference, enabling to enhance the fidelity or diversity in the generated data by accepting with higher or lower probability the gencrated samples that have higher pscudo density; 3) Fine-tuning with importance sampling, guiding the generalive model to learn an adjusted distribution skewed towards data with higher or lower pseudo density, thereby controlling the fidelity and diversity.
Our extensive experiments across diverse datascts and various generative models, including different GANs and diffusion models, demonstrate the effectiveness and generality of our proposed approach in controlling the trade-off between fidelity and diversity. Additionally, our fine-tuning method can be employed to improve the Fréchet Inception Distance (FID) (Heusel et al. 2017) scores by fine-tuning pre-trained models and adjusting their fidelityddiveriity trade-off. Aside from the Inception Score (IS) (Salimans al.; 2016)) and the Fréchet Inception Distance (FTD); as step towards more comprehensive evaluation, employ precision and recall (Kynkäänniemi al. 2019) as discntangled metrics to separately assess the fidclity and diversity of generalive models.
The main contributions of this work can be summarized as follows:
1. We introduce novel metric, termed pseudo density, for estimating the density of image data. It correlates effectively with the realism and uniqueness of individual images.
2. We propose a per-sample perturbation strategy based on pseudo density that enables adjustment of realism and uniqueness for individual samples.
3. We propose importance sampling based on pscudo density during inference, which controls the fidelity and diversity of generative models at inference time, without re- re-training or fine-tuning the model.
4. We propose fine-tuning with importance sampling based on pseudo density, which balances the trade-off between the fidelity and diversity of generative models without additional computational overhead during inference.
We demonstrate that our fine- fineituning method able to improve the Frochet Inception Distance (FID) of deep generative models by adjusting their precision (fidelity) and recall (diversity) trade-off.
2 Background and Related Work
Generative adversarial networks. GANs (Goodfellow et al., 2014) are popular generative models to learn the underlying distribution of the given training data and generate new samples. They typically consist of generator G(-) parameterized by 6g that transforms random vector, called the latent code 2, into data sample, and discriminator D(-) parameterized by 02 that aims to distinguish the real training data from the generated samples. Both the generator and discriminator are deep neural networks that can be trained in an end-to-end manner. Therefore, training GANss involves solving a min-max opl imiiatiion problem by alternatively updating the generator and the discriminator. While the original GANs may suffer from training instability, mode collapse, and poor scalability to high-rcsolution images, many variants (Arjovsky et al. 2017 Gulrajani et al., 2017]: Miyato et al.] 2018 Brock et al.) 2018 Karras et al., [0088 2019 have been developed to address these problems and enable impressive high-resolution image generation.
2

---

Published in Transactions on Machine Learning Research (10/2024)
Diffusion models. Diffusion models have emerged as a prominent class of generative models, renowned for their ability to create highly detailed and varied images, especially in text-to-image fashion (Ramesh et al.
2022 Saharia et al. 2022 Rombach et al. 2022). These models, introduced in (Sohl-Dickstein et al. (2015). simulate a physical diffusion process in thermodynamics: Starting with an image, noise is incrementally added over several steps until the image translormed into pure noise. The model then learns to reverse this process, effectively reconstructing the original image from its noised state. Subsequently, the Denoising Diffusion Probabilistic Model (DDPM) (Ho et al. 2020) introduced the e parameteriiation and adopted new architecture based on U-Net (Ronneberger et al., 2015), obtaining significantly improved sample quality. Improved Denoising Diffusion Probabilistic Models (IDDPM) (Nichol & Dhariwal 2021 refined the
noise addition process and optimized the model's architecture to improve both the sampling efficiency and
the quality of the generaled images. Ablated Diffusion Models (ADM) (Dhariwal & Nichol 2021) further
improved the architecture over IDDPM and demonstrated that diffusion models can achieve image sample quality superior to that of GANs.
Fidelity and diversity of generative models. Quantitatively evaluating the performance of generative models is still open and challenging question. Popular metrics such as the Inception Score (IS) (Salimans ct al. 2016) and the Fréchet Inception Distance (FID) (Heusel ct al.
2017 are based on the overall quality
(fidelity and diversity) of the generated samples. Kynkäānniemi et al. 2019 proposed the improved precision and recall metries for generative models to separately evaluate the fidelity and the diversity. The precision is the proportion of generated samples that fall within the manifold estimated by the K nearest neighbors (KNN) of the real data, whereas the recall measures the proportion of real samples that are within the generator-induced manifold. A low precision indicates that many of the generated images do not resemble the training data, suggesting poor fidelity. By contrast, low recall indicates lack of diversity in the generated images. An extreme case of low recall is mode collopse, i.e., the failure to generate some particular categories of the multi-modal data. Several works have proposed to control the precision (fidelity) and recall (diversity) of generative models, including by truncating the latent distribution (Brock et al. 2018 Karras et al. 2019) and developing latent space samplers (Humayun et al.) 2022a by. Furthermore, Discriminator Rejection CrmplinngAzaadl et. al., 2018)) and Discriminator Optimal Transport (Tanaka 2019) utilize the information retained in the GAN discriminator to improve the fidelity of the generated data during inference. However, these approaches either come with high computational cost during the inference phase or are limited to specific GAN archilectures.
Realism score and rarity score. Both the realism score (Kynkäänneemi et al. 2019) and the rarity score (Han et al. 2022), along with our new pscudo density metric, are based on the features extracted from image data using an image feature extractor. To calculate the realism score for an input image, one finds the real image sample that has the highest ratio of its k-NN radius to its distance to the input sample. The rarity score is essentially the smallest possible k-NN radius of real sample whose k-NN hypersphere contains the input sample. The proposed pseudo density is calculated by simple neural network that directly fits the estimated densities of real samples in the feature space. Unlike the other two metries, our metric is specifically tailored for our control approach that manipulates data probability density. Morcover, the realism score is ineffective for samples that are rare in the real distribution, and the rarity score is not computable for outliers that are too far away from other real samples.
3 Pseudo Density
Our control approach aims to explicitly manipulate the occurring likelihood of data samples, ideally based on their probability density in the real data distribution. However, estimating the density of high-dimensional data such images is not straightforward, primarily due to the complexities introduced by the high dimen- sionality and the fact that image data exist in low-dimensional manifold. In this section, introduce an effective metric, termed pscudo density, for estimating the density of image data. Specifically, we first em- ploy a pre-trained image feature extractor to lower the dimensionality of the real image data while capturing the semantics. Then, for each real sample, the average distance d to its nearest neighbors is calculated. Based on the average neighbor distances, we can estimate the volume "occupied" by each sample, and the density is inversely proportional to the volume. Finally, we train a weiht folly-connected network to fit. the estimated density given the extracted feature as input. To compute the pseudo density of any
3

---

Published in Transactions on Machine Learning Research (10/2024)
(8.1) Estimate the real distribution Extrectfestures real eamples
Estimate the real distribution Compute density in the feature space (k-3)
Feature Extractor
Train a density predictor with extracted features and computed densities
Density Predictor
MSE
density
Loss
Gradion
(c) Use the trained predictor to compute density for generated samples
Feature
Dansity
0.61
d (a))== (d, d2 dz)
Extractor
Predictor
density( (#
Figure 1: Overview of the proposed pscudo density metric. (a) The pscudo-density of cach sample is inversely proportional to its average distance to its K-Nearest Neighbors in feature space. (b) We use these pseudo density to train a density prediction network, which then can be used on any generated samples (c).
5OOD
Lower Density
Higher Density
Figure 2: Images generated by StyleGAN2 (Top), ProjectedGAN (Middle), and IDDPM (Bot- tom). The training datasets are FFHQ, LSUN-Chured and LSUN-Bedroom. respectively. For cach row, the left four images obtained the lowest pseudo density out of 1000 generations, and the right four obtained the highest.
generated sample, feed it into the cascade of the feature extractor and the trained network. Note that this pseudo density metric differs from the Fréchet Inception Distance (FID) (Heusel et al. 2017): Pseudo density evaluates individual samples while the FID evaluates the overall performance of the generative model by calculating the distributional distance between the real and generated images. We illustrate the steps of computing pseudo density in Figure I and provide more details in the following.
Estimating the real data distribution. We estimate the real data distribution in feature space, using the extracted features of samples from the dataset. First, the features of all real samples in the dataset.

---

Published in Transactions on Machine Learning Research (10/2024)
be extracted by a pre-trained feature extractor, such as a Vision Transformer (ViT) Dosovviskki et al. 2020 trained on ImageNet (Deng et. al.) 2009). Then, for each data sample I;, we find its top-k nearest neighbors based on the Euclidean distance in the feature space and compute its average distance to these k samples dā With the assumption that the sample and its neighbors are locally uniformly distributed within a sphere of radius d, we can compute the volume sample "occupies" as V F(Y
gn/² + -c- (df)", where is the
dimensionality of the feature space, I is Euler's gamma function, and C is a constant only conditioned on n. The density at the data point 24 is hence inversely proportional to the volume, i.e, Pf x YE.
V.. Finally, for where Σ. P
numerical stability, we compute the normalized density across all data points as P ===0 N. N. P
N is the lotal number of samples. Note that although image data (as well their extracted features) are in high-dimensional space, they reside in manifold of much lower dimensionality. Hence, in practice, we set to be a small number (e.g., 1 or 2), preventing the volume V from the curse of dimensionality.
Learning a model that predicts the density. To efficiently compute the density for any data point x and enable gradient hack-propagatton, we train a simple regression network p(-) to fit the density Po,,1,.... PN- with the extracted features of real samples as imput F(xo), F(r)),.., F(æN-1). We observed that a basic fully-connected network consisting of only three hidden layers with fewer than one million parameters is capable of fitting well to the real data while generalizing very effectively. We term the output of the network as pscudo density. Compared to directly computing the density of a generated sample by injecting it into all real samples, this approach has lower computational overhead and ensures that the gradient can be back-propagated from the metric to features.
Compute pscudo density for generated samples. To calculate the pscudo density of a generated sample H; we sequentially feed it to the image feature extractor and the learned regression model p p(F())). Intuitively, an image with a high density typically exhibits more common characteristics, whereas an image with a low density is likely to feature more unique attributes. Moreover, in the context of generated images, the lack of training on low-density data often results in reduced realism for low-density images. In Figure we illustrate the effectiveness of pseudo density by showcasing generated images with either high or low pseudo density. For each model under consideration, we generate 1, 000 images, sort them according to their pscudo density, and select the ones with the highest and lowest extremes. The results evidence that the proposed metric aligns well with human perception in terms of realism and uniqueness.
4 Censity-Based Perturbation
While adversarial perturbation was originally used to attack an input image to fool a classification model, their formulation applies to any other type of input to a neural network as long as an objective function is defined. In this work, we utilize pseudo density as the objective function to adversarially perturb GAN's latent code or the input noise vector of diffusion model, both of which we refer to as latent vector z below.
In the context of image synthesis, such a perturbation optimizes the latent vector within its neighbor- hood such that it leads to a higher-density or lower-density generated image, hence increasing the realism or uniqueness of the generated image without drastically changing the image content. Let us consider a randomly-sampled latent vector z. We explore the neighborhood of z using a perturbation 8 based on the pseudo density. To achieve this, we employ the PGD attack (Madry et al.) 20188) strategy to minimize the pseudo density of the features of G(z +)) under the constraint | I c, where e is the predefined adversar- ial budget. The pseudo-code of our per-sample perturbation method is provided in Algorithm [ In Madry ct al. (2018), the perturbation is applied directly to the image and the magnitude of e should be kept small to ensure that the changes are imperceptible in the image; in our context, however, the perturbation is applied to the latent code and the resulted changes in the image are intended to be noticeable. Additionally, e determines the magnitude of the pscudo density shift and the extent of the content changes in the generated image. A larger e yields a large change in pseudo density and also more significant changes in the content and realism/ uniqueness of the generated image.
5

---

Published in Transactions on Machine Learning Research (10/2024)
Algorithm 1 Per-Sample Perturbation Based on Pseudo Density
Require:
The image generator G(-), the image feature extractor F(.), the pseudo density function p(-): the number of PGD iterations K, the step size a, the adversarial budget e, and the initial latent vector z.
1: Set an initial perturbation = 0
2: for t= 1, K do
3: Calculate the pseudo density of the perturbed sample G( + 0):
f(2,0, 6) p(F(G(z 8)))
Update ò:
t 8 VVf 6) if the goal is increase realism 8 8 aVsf 8) if the goal is to increase uniqueness 8 4 -c,to, 6: end for
5:
Ensure: The perturbed latent vector # + 6.
5 Density-Based Importance Sampling
With the help of pseudo density, not only distributions, assigning adjusted probability to samples with different densities. To achieve this, we proceed as follows. For the given dataset, we estimate its data distribution and compute the pscudo density for all samples, as described in Section 3 and assign two different importance weights for the samples whose densities are above and below a pre-defined density threshold, respectively. Intuitively, by assigning higher weights to above-threshold or below-threshold samples for importance sampling, we can either increase or decrease the proportion of high-density images in the manipulated data distribution.
perturb individual samples, but also manipulate data
To edit the generated data distribution in streaming fashion, ct al. 2018). Let the original generator's 's output distribution given the random latent vector be p(x), and the desired distribution be ((x). Furthermore, let M be the upper bound of the ratio 2(x)
employ rejection sampling (Azadi
p(a)" i.e. Vz, q(x) Mp(x).
Rejection sampling proceeds by drawing a sample x" from p(z) and only accepting xt with probability g(z")
Myla") In practice, we formulate the desired distribution q(x) by assigning an importance weight to the samples that have pseudo density higher than a pre-defined threshold and 1 to the other generated samples. In the case where w > 1,, which means the high-density samples are highlighted, ((x) can be obtained through iterative execution of two steps: 1) Generate a sample from the generator; 2) accept this sample if its pseudo density is higher than the threshold t, otherwise, reject it with a probability of w The case where w S 1 can be handled similarly by rejection sampling.
5.1 Inference with Importance Sampling
A simple yet effective method for controlling the fidelity and diversity of generative models is to employ the above importance sampling strategy generated images during inference, directly manipulating the generated data distribution. To perform inference in streaming manner, we employ rejection sampling. Different combinations of density threshold and importance weight result in different effects the data distribution, increasing either high-realism samples or high-uniqueness ones. In practice we set the threshold to be the {20, 50, 80} percentile of the pseudo density values of all real samples, and the weight w ranges from 0.01 to 100. Intuitively, adopting an importance weight w > 1 leads to more samples with a pseudo density higher than the threshold т, hence more samples of better realism, while an importance weight w < results in more samples with pseudo density lower than т, hence more variety of samples and better overall diversity. The pseudo-code of this method is provided in Algorithm We demonstrate our experimental results in Scction 0
5.2 Fine-tuning with Importance Sampling
In contrast our previously introduced method, which manipulates the generated data distribution inference time, the current method employs importance sampling during the fine-tuning process to adjust

---

Published in Transactions on Machine Learning Research (10/2024)
Algorithm 2 Importance Sampling during Inference
Require:
The image generator G(:), the image feature extractor F(.), the pseudo density function p()). the number of images to generate K, the density threshold the importance weight w for high-density samples.
1: Sct the number generated images k +- 0. 2: Initialize an empty list to store generated images I +-
while & < K do
Randomly sample latent vector from the prior distribution. Calculate the psendo density of the generated sample G(z):
f(z) p(F(((()))
6: Generate a random number 10)
7:
if w > then
8: if f(=) > T or u then
Accept the generated sample G(z) and push the list I.
10: k k+1
11: end if
12: else
13: if f(z) < or u < w then
14:
Accept the generated sample G(z) and push it to the list T.
15: k t ++
16: end if
17: end if
18:
end while
Ensure: K generated images.
the data distribution at training time. As result, fine-tuning does not introduce any computational overhead during inference. Furthermore, it enables the realism paalimm/uiqquen /uniqueness adjustment of any generated images, given their latent vectors. Similarly to the inference-time method, fine-tuning with importance sampling requires the same hyperparameters, the density threshold and the importance weight w, to yield different controlling effects. Before fine-tuning, we compute the pseudo density for every sample in the dataset, and then determine the density threshold т based on the computed densities of all samples. During fine-tuning, training examples sampled from the dataset using the importance sampling strategy described before based the pre-computed pseudo density.
Algorithm 3 Fine-tuning GANs with Importance Sampling
Require: The batch size B, the generator G(-10,), the discriminator D(-10a), the number of iterations T., the density threshold the importance weight w.
1: for t 1,.., do
2: Perform importance sampling with т, w to obtain B real samples (x)
(BB)).
3: Perform importance sampling with r, w until generating B accepted samples
4: Update by minimizing the loss:
B I (D( ()G()
5: Perform importance sampling with r, w until generating B accepted samples G(2(B)))
6: Update G by minimizing the loss:
D - -D(((())
end for Ensure: The finc-tuned generator G and discriminator D
In general, our fine-tuning approach does not modify the training algorithm except for how images are sampled from the data distribution in each iteration. Therefore, compatible with any generative models including GANs and diffusion models. Moreover, can be further enhanced for certain generative models, particularly GANs, for which importance sampling be applied not only to the real data distribution

---

Published in Transactions on Machine Learning Research (10/2024)
Higher
Density
Original
Lower
Density
Higher
Density
Original
Lower
Density
Figure 3: Per-sample perturbation, applied to pre-trained models. Given random latent vectors 2 and their generated images (Middle in each panel), we perturb = to achieve higher pseudo density (Top each panel) and lower pseudo density (Bottom in each panel). Groups from left to right, and from top to bottom: StyleGAN2 on FFHQ, ProjectedGAN on LSUN-Bedromm; StyleGAN3-T FFHQ, IDDPM LSUN-Bedroom:
ProjectedGAN on LSUN-Church, ADM on LSUN-Bedroom.
pr(z), but also to the generated data distribution Py(x), which serves as input to the discriminator during training. We provide the pscudo-code for the method for GANs in Algorithm 3 Specifically, when the generator and the discriminator reach their equilibrium, we have pr(+))p.(=)
P.(x) = [Goodfellow et al. 2014).
and thus the learned generated data distribution Pz(x) = (x)). In the case where we perform importance sampling to the real data distribution pr(r), we have Py (x) = (pr)) upon convergence, where we use Pert((-) (-) to represent the perturbation function that maps the original distribution to the sampled one. If we also apply importance sampling to the generated data distribution during training, the equilibrium becomes (z) = Pert, (x) This allows a stronger control over the learned generated data distribution Pa(x) after training. A practical choice to perform importance sampling on Pz(x) and p)(x)) with reciprocal importance weights and 1, as in Algorithm
Note that the proposed fine-tuning strategy also be applied to training from scratch. However, we observed that applying fine-tuning yields comparable effectiveness but with significantly less computational overhead compared to training a model from scratch.
Additionally, our fine-tuning strategy is compatible with inference- time techniques, including truncation and the proposed importance sampling stratogy detailed in Scction When inferenco-time methods are applied in conjunction with our fine-tuning strategy, stronger enhancement over fidelity or diversity can be achieved. Empirical evidence supporting this combined effect is presented in Appendix A.4
8

---

Published in Transactions on Machine Learning Research (10/2024)
0.85
StlecAA3, mhi 302541026
0.75
DojectedGAA, LSUN Bodroom 230K256
LSUN- 256x256
9.775
0.80
0.79
0.75
Precttion
0.70
0.65
Precision
0.725
0.63
Pro traned
Ours, 1-5
0.60
Pre traitedd
0.700
Ours, 20%
Ours r- 199
0.00
Ours:
0.55
T- 50%
Ous. T 20%
Predre
0.55
Data,
Ours Frontier
Ours, T- 90%
Durs. 80%
9.650
Oun. T 80%
Ous. T- 50%
Trunc elkin
6.50
Durs fronlier
Gura Fronterr Truncation
0.50
Polanity Sampling Frenie
Iruncaioon
0.55
Recall
0.80
StyloGaN3, FHH 102401044
0.70
ProkectodGeN LSun-Charch 256:256
LSUli-Bedsoom 258
0.74
0.75
0.05
0.72
0.70
0.60
Predtiion
0.65
Predsion
Predsion
0.68
0.60
Ours, r=
Ours:
0.50
Ours, = 10%
he-ttaited
0.68
0.55
Durs, 1 50%
Duns r 20%
Pre-traied
Ours. 8080
Outs T- 50%
Ours, E- 20%
0.50
9-90
0.40
80%
0.82
Truncation
Ours Rontor
0.45
Polanity Sampling Fromlee
Trancation
Irancatior
0.30
6.35
0.40
0,45
0.50
0.55
0.80
0.65
0.25
0.30
0:35
0.40
0.45 Recall
0.50
0.55
0.60
0.22
0.24
0,26
0.88
0.30
0.32
0.34
Recall
Recall
Figure 4: Precision-recall trade-off. For all methods, the precision and recall metrics compete with each other. For polarity sampling, we used the values reported in Hummyun et al. (2022a) due to replication difficulty. For our method, we adopted different density thresholds т for cach dashed line with varying importance weights ranging from 0.01 to 100. For better visualization, we only report part of the results for polarity sampling and truncation. We visualize the Pareto frontier for polarity sampling and ours, illustrating the optimal trade-offs achieved across various hyperparameter configurations.
6 Experiments
In this section, we present the results of our methods on various generative models and datasots. We se- lect. Improved Denoising Diffusion Probabilistic Model (IDDPM) (Nichol & Dhariwal 2021 addition Ablated Diffusion Model (ADM) (Dhariwal & Nichol 2021). StyleGAN2 (Karras et al.) 2020), Style- GAN3 (Karras et al. 2021), and ProjectedGAN {Sauer et al. 2021 to demonstrate the generality of our methods different types of generative models. The datasets consist of LSUN-Bedroon. (Yu et al. 2015), LSUN-Church Yuu et al. 2015). and FFHQ (Karras et al., 2019) The images from LSUN-Beeroom and LSUN-Church are resized to 256 x 256, while the FFHQ images are of resolution 1024 x 1024. For all exper- iments, we use the pre-trained checkpoints provided by the authors and use the same syper-parametens as in the original papers for fine-tuning. More details regarding the datasets are deferred to the supplementary matcrial, as well as more examples of generatod images.
In Figure g we showcase examples of images generated by the pre-trained models to illustrate the effects of our proposed per-sample perturbation strategy, which achieves large density change while preserving content of the images. By applying perturbation that increases density, the generated images exhibit greater realism, characterized by simplified backgrounds, fewer objects, elc. By contrast, perturbation to lower density leads to images with higher uniqueness, featuring more complex backgrounds, the presence of rare features the datasots, etc. Appendix A.2 we consider perturbation that incrcases the output of a GAN's discriminator and demonstrate its inferior performance compared to pseudo density. Additionally, we compare approach with Discriminator Rejection Sampling (Azadi et al. 2018) and Discriminator
9

---

Published in Transactions on Machine Learning Research (10/2024)
Table 1: Improved precision/reeall/PDD obtained by fine-tuning with importance sampling, with different importance weights. stands for precision and R stands for recall. Bold denotes the best value in each column.
Model
Pt
R
FID4
Model
Pf
R1
FID4
256 x 256
FFHQ
1024 X 1024
Improved DDPM
0.727
0.172
10.13
StylcGAN2
0.685
0.493
2.79
fine-lune (w 33.0)
0.749
0.168
9.54
fine-tune (w 33.0)
0.811
0.387
36.24
fine-tune (w = 0.03)
0.648
0.231
13.48
+ fine-tune (w 0.03)
0.558
0.564
10.58
fine-tune (w 2.0)
0.737
0.175
9.53
+ fine-tune (w 0.5)
0.675
0.500
2.60
ADM
0.715
0.290
6.35
StyleGAN3-T
0.658
0.538
2.81
fine-tune (w- 33.0)
0.729
0.284
7.70
+ fine-tune (w 33.0)
0.824
0.382
60.89
fine-tune (w = 0.03)
0.627
0.343
8.15
+ fine-tune (w 0.03)
0.482
0.624
22.34
fine-tune (w 2.0)
0.697
0.302
5.83
+ fine-tune (w 1.5)
0.657
0.535
2.73
ProjectedGAN
0.608
0.366
2.31
LSUN-Church 256
256
fine-tune (w = 33.0)
0.694
0.290
3.00
ProjectedGAN
0.577
0.512
1.62
fine-tune (w = 0.03)
0.524
0.419
7.65
+ fine-tune (w = =33.0)
0.658
0.418
3.95
fine-tune (w = 1.05)
0.617
0.359
2.22
+ fine-tune (w = =0.03)
0.494
0.540
7.47
+ fine-tune (w =2.0)
0.586
0.507
1.58
Optimal Transport (Tanaka 2019) both of which rely on the trained discriminator's output in GANs as an indicator for image realism.
In Figure we demonstrate the trade-off between precision (fidelity) and recall (diversity) achieved by density-basee importance sampling during inference. Our approach achieves a better overall precision- recall trade-off than the other two inference-time methods, polarity sampling (Humayun al. 2022al and truncation (Karras et al. 2019). especially in the pmprovedd-reecision area. It's important to note that polarity sampling may face compatibility issues with diffusion models due to the computational complexity of the denoising process. The truncation method is adaptable to the intermediate W+ space for StyleGAN (Karras ctt al. 2019 2020 2021) models, but is generally limited to the input 2 space for others, which could result in suboptimal performance. For instance, for Improved DDPM, the truncation method struggles around the pre-trained trade-off point. Unlike these methods, neither the computational complexity nor particular model architectures restricts the applicability of method since approach directly manipulates the output data.
In Table [ we demonstrate that our fine-tuning approach can improve the precision, recall, or FID across various datasets and generative models of different types, depending on the choices of hyperparameters, particularly the importance weight w. With larger w > 1, the fine-tuning procedure encourages the models to generate higher-density data, and vice versa with w < By fine-tuning with w close to 1, the model can end up slightly altered trade-off point that leads improved FID scores. Note that the density threshold that we use may vary for the same model. We refer the reader to the appendix for more details about the byperparameters and fine-tuning configurations. In Figure 5 further present examples of images generated by the pre-trained models compared with images produced by their finc-tuned versions to visually demonstrate the effects of our fine-tuning method.
7 Conclusion
In this study, we have roduced a simple and effective approach for estimating the densily of image data, enabling us to devise inference-time methods and a fine-tuning strategy for biasing deep generative models into outputting data with either higher fidelity or higher diversity. We have shown that our method can greatly improve the fidelity or diversity in the generated data without significantly altering their primary subject and structure, making it of great interest to applications such image editing. We have also
10

---

Published in Transactions on Machine Learning Research (10/2024)
Fine-tuned > 1
Pre-trained
Fine-tuned
< 1
Fine-tuned > 1
Pre-trained
Fine-tuned
< 1
Figure Images generated by pre-trained and fine-tuned models. For each column in a panel, the same latent vector is used by the pre-trained model and the fine-tuned versions for image generation. Fine-tuning with w biases the generative model output more high-density data. Groups from left right, and from top to bottom: StyleGAN2 FFHQ, ProjectedGAN on LSUN-Bedroom; StyleGAN3-T FFHQ, IDDPM on LSUN-Bedroom; ProjectedGAN LSUN-Church, ADM on LSUN-Bedroom.
demonstrated that adjusting the balance between precision (fidelity) and recall (diversity) through finc- tuming can improve the Frechet Inception Distance (FID) for models pre-trained in a standard manner. This underscores the importance of considering both fidelity and diversity in the evaluation of generative models instead of relying solely FID performance metric. Future research may aim to evaluate and improve various density-based sampling strategies for optimized their efficacy. Another potential research interest lies in adapting the proposed coltrol approach to conditional generation tasks such as text-to-image synthesis. hesis.
References
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International Conference on Machine Learning, pp. PMLR, 2017.
Samaneh Azadi, Catherine Olsson, Trevor Darrell, Ian Goodfellow, and Augustus Odena. Discriminator rejection sampling. In International Conference on Leorning Representations, 2018.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. In International Conference on Learning Representations, 2018.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A largo-scale hicrarchical image database. In 2009 IEEE conference on computer vision and pottern recognition, pp. 248-256 Iece, 2009.
11

---

Published in Transactions on Machine Learning Research (10/2024)
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural 2021.
information processing systems,
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprinl Xrx::20010. 11929, 2020.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12873-12883, 2021.
Tan Goodfellww, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in Neural Information Processing Systems, 27, 2014.
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Improved training of wasserstein gans. Advances in Neural Information Processing Systems, 30, 2017.
Jiyeon Han, Hwanil Choi, Yunjey Choi, Junho Kim, Jung- Woo Ha, and Jaesik Choi. Rarity score: A new metric to evaluate the uncommonness of synthesized images. arXiv preprint arXiw:2206. 08549, 2022.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to local nash equilibrium. Advonces in Neural Infor- mation Processing Systems, 30, 2017.
Jonathan Ho, Ajay Jain, and Picter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840- 13-8800-6851, 2020.
Ahmed Imtiaz Humayun, Randall Balestriero, and Richard Baraniuk. Polarity sampling: Quality and diversity control of pre-trained generative networks via singular values. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognition, pp. 10641 10650, June 2022a.
Ahmed Imtiaz Humayun, Randall Balestriero, and Richard Baraniuk. Magnet: Uniform sampling from deep
generative network manifolds without retraining. In International Conference on Learning Representations, 2022b. URL
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. In International Conference Learning Representations, 2018.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVE Conference Computer Vision and Pattern Recognition, pp. 4401-4410 2019.
Tero Karras, Samuli Laine, Miia Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of StyleGAN. In Proceedings of the IEEE/CVF Conference on Computer Vision and Paltern Recognition, 2020.
Tero Karras, Mika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-free generative adversarial networks. Advances in Neural Information Processing Systems, 34: 852 863, 2021.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes arXiv preprint arXiv:1312.611. 2013.
Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. Advances in Neural Information Processing Systems, 32, 2019.
12

---

Published in Transactions on Machine Learning Research (10/2024)
Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, An- drew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al. Photo-realistic single image super- resolution using a generative adversarial network. In Proceedings of the IEEE/CVE Conference on Com- puter Vision and Pattern Recognition, pp. 4681-4690, 2017.
Haoying Li, Yifan Yang, Meng Chang, Shiqi Chen, Huajun Feng, Zhihai Xu, Li, and Yueting Chen. Srdiff: Single image super-resolution with diffusion probabilistic models. Neurocompating, 479:47-59, 2022.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International conference on learning representations, 2018.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for gener- ative adversarial networks. In International Conference on Learning Representations, 2018.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. International Conference on Machine Learning, pp.
PMLR, 2021.
Adilya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint 2rX1v:2204.061125, 1(2):3, 2022.
Robin Rombach, Andreas Blattmann, Dominík Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pallern recogniilion, pp. 10684- 10695, 2022.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assie Assisted Intervention- 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pp. 234 241. Springer, 2015.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text- to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 11364799366494 2022.
Tim Salimans, Ian Goodfellow, Wojeicch Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in Neural Information Processing Systems, 29, 2016.
Axel Sauer, Kashyap Chitta, Jens Müller, and Andreas Geiger. Projected gans converge faster. Advances in Newral Information Processing Systems, 34:17480 17492, 2021.
Jascha Sohl-Dickstedd Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequiliriiun thermodynamics. In International conference on machine learning, pp. 2256 2265. PMLR, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXie preprinl arXiv:2010.0220 02502, 2020.
Akinori Tanaka. Discriminator optimal transport. Advances in Nearal Information Processing Systems, 32, 2019.
Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al. Conditional image generation with pixelcnn decoders. Advances in newral information processing systems, 29, 2016.
Yan Wu, Jeff Donahue, David Balduzzi, Karen Simonyan, and Timothy Lillicrap. Logan: Latent optimisation for generative adversarial networks. arXie preprinl arXim:1912.0000 2019.
Jingyi Xu and Hieu Le. Generating representative samples for few-shot classification. In Proceedings of the IRFF/CVF conference on compuler vision and pattern recognition, 2022.
13

---

Published in Transactions on Machine Learning Research (10/2024)
Jingyi Xu, Hieu Le, Mingzhen Huang, ShahRukh Athar, and Dimitris Samaras. Variational feature disen- tangling for fine-grained few-shot classification. In International Conference on Compuler Vision, 2021.
Jingyi Xu, Hieu Le, Vu Nguyen, Viresh Ranjan, and Dimitris Samaras. Zero-shot object counting, pp. 155577 2023a.
Jingyi Xu, Hieu Le, and Dimitris Samaras. Generating features with increased crop-related diversity for few-shot object detection. In Proceedings of the LEEE/CVF conference on computer vision and pattern recognition, 2023b.
Jingyi Xu, Hieu Le, and Dimitris Samaras. Assessing sample quality via the latent space of generative models. In European Conference on Compuler Vision, 2024.
Fisher Yu, Ari Scff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Con- struction of a large-scale image dataset using deep learning with humans in the loop. or preprinl arXiv:1506. 033555556600355, 2015.
Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversarial networks. In International conference machine learning, pp. 7354-7363. PMLR, 2019.
14

---

Published in Transactions on Machine Learning Research (10/2024)
A Appendix
A.1 Training Details, Evaluation Details, and Hyper-parameters
Datasets The FFHQ (aces-)Fa dataset (Karras et al. 2019) is high-quality collection of human face images consisting of 70k images with the resolution of 1024 x 1024. The LSUN-Church and LSUN- Bedroom are subsets the LSUN (Large-scale Scene Understanding) dataset. (Yu et al. 2015). For LSUN- Church, we use all 126k images in the dataset and resize them to 256 256 resolution. For LSUN-Bedroom, we sample the first 200k images from the dataset and apply center crop, resizing them to 256 x 256 resolution well. We followed the same pre-procesiing procedures as in previous works.
Training details We follow the same training hyper-parameters published in the original papers, except the number of GPUs, while keeping the same batch size per GPU. For GANs models in our experiments, we used 2 GPUs, and a single GPU for diffusion models. For StyleGAN2/3 models, we fine-tuned for, measured in thousands of real images fed to the discriminator, 80 thousand images. For Projected GAN, we fine-tuned for 40 thousand images. To improve FIDs, we only fine-tuned all GANs models for 12 thousand images. As for diffusion models in our experiments, we fine-tuned them for 128 20000 2560k images.
Regarding the computation of pseudo density, use 1 for all datasets and K 10 for all except LSUN-Bedroom where we set K 20.
Evaluation details To evaluate the FID, precision, and recall for a generative model, we use the model to generate 50, 000 images every time and then calculate the metrics against the training dataset. For diffusion models, however, we only generate 10, 0000 images every time for computational efficiency. In addition, we use the uniform stride from DDIM (Song et al. 2020 and each sampling process takes 50 steps.
Per-sample perturbation hyper-parameters We employed PGD attack (Madry et al. 2018) whose eyper-parameters involve the number of steps K, the step size and the adversarial budget e. For all GANs models in our experiments, we adopted K = 10, = 0.025, and = 0.1. For diffusion models, we adopted K 5, = 0.0025, and c 0.0125.
Density-based importance sampling hyper-parameters The two relevant cyper-parametten are the density threshold and the importance weight w of above-threshold samples. Their optimal values may vary across different datasets and models. We conducted sweep for with values from {20, (20,50,8) 80) percentiles of the estimated densities of real images. We also performed sweeps for w with values from {0.01, 0.03, 0.1, 10, 33.0, 1003. We show the optimal values found in Table 2 When aiming to improve FIDs, we kept density threshold of 50 percentile and performed importance sampling only on the real data. We conducted a sweep for w from {0.5, 0.6, 0.7, 0.8, 0.9, 0.95, 1.05, 1.1, 1.3, 1.5, 1.7, 2.0) for GANs models and only [0.5, 2.0} for diffusion models.
Table 2: Optimal hyper-parameters for importance sampling. Optimal Syper-parameters vary with different goals, which are improving precision (Precision/>) /)), improving recall (Rocall and improving FID (FID)). In the first. two scenarios, the selection was not based on maximizing the precision/ /recall gain but the (subjective) optimal trade-off.
Precision x
Recall
FID
Model, Dataset
w
T
w
T
W
StyleGAN2, FFHQ StyleGAN3-TT T, FFHQ
80%
33.0
20%
0.03
50%
0.5
80%
33.0
20%
0.03
50%
1.5
ProjectedGAN, LSUNNChurch
50%
33.0
50%
0.03
50%
2.0
ProjectedGAN- LSUN-Bedroom
50%
33.0
50%
0.03
50%
1.05
IDDPM, LSUN-Bedroom
80%
33.0
20%
0.03
50%
2.0
ADM, -eeeooo
80%
33.0
20%
0.03
50%
2.0
15

---

Published in Transactions on Machine Learning Research (10/2024)
StyleGAN2, 1024
VOGAN, FFHQ 1024x1024
0.80
Precision
0.70
Precision
0.65
Pre-trained
0.60
Pre-trained DRS DOT
Density, == 20%
Density. -- 50%
Density, += 80%
Frantier
0.60
Likelhood. 20%
Truncation
Likelhhodd 50%
0.50
Polarity Sampling Frontier
Likelihood, r- 80%
0.35
0.40
0.45
0.50
0.55
0.60
0.275 0.300 0.325 0.350 0.375 0.405 0.450 0.475 Recall
Recall
Figure 6: Precision-recall trade-off for StyleGAN2 on FFHQ.
Figure 7: Precision-recall trade-off for VQGAN on FFHQ.
A.2 Discriminator Output versus Pseudo Density
Many works, including LOGAN (Wu et al. 2019 Discriminator Rejection Sampling (Azadi et al., 2018). and Discriminator Optimal Transport (Tanaka, 2019), have explored the output of trained discriminator as metric for sample realism. Ideally, discriminator output may indicate sample realism in the context of Generative Adversarial Networks since the discriminator learns to tell if sample is real or generated. DRS (Azadi et al. (2188) and DOT (Tanaka) 2019) demonstrated that discriminator output aligns well with realism in low-dimensional datasots and their methods improve FID for SNGAN (Miyato et al. 2018 and SAGAN (Zhang et al. 2019). However, our experiments suggested that works badly for higher- utoo image data and larger modern GAN architectures such as StyleGAN (Karras et al. 2019 2020). In Figure S we showcased the results of per-sample perturbation based on pseudo density and discriminator output using StyleGAN2 on FFHQ dataset. Additionally, compared to random perturbation, which replaces the gradient-descent direction with a random direction while keeping the same magnitude in each step of the PGD attack.
We implemented DRS and DOT for StyleGAN2 and followed the original hyperparameters reported in the papers. As shown in Figure G where we plot the precision and recall for DRS and DOT alongside our approach and others, both DRS and DOT achieved performances similar to using the pre-trained model naively. Additionally, DRS and DOT obtained FID scores of 2.95 and 3.44, respectively, which are worse than that of the pretrained. These results indicate that discriminator-basse methods like DRS and DOT do not achieve ideal precision-recall trade-off for high-resolution image data and larger, modern GAN models.
A.3 Extra Computational Overhead
The proposed rejection sampling strategies, as described in Section Algorithm 2 and Algorithm 3 result in extra computational cost since only proportion of the generated samples are accepted. Given the importance weight < 1 and let the density threshold be p% of estimated densities of the data, assuming the generated data perfectly fit the real data, a generated sample is accepted with probability 0.01p + (1 0.01p) x w on average, hence the compulational cost is approximately 6.01p+(1-0.01pxx) times as before. The case where w can be handled similarly.
In our experiments, generating samples using StyleGAN2 with importance sampling (p = 50, w = 0.01)
slowed down from the original 13 sec/kings to 26 sec/kings while generating with = 20, w = 0.01 slowed down to 60 sec/kimgs. In contrast, generating samples with DRS (Azadi et al.] (2088) and DOT TTnaka 2019) slowed down to 23 see/kings and 290 boe/kimgs respectively. We were unable obtain statistics for polarity sampling (Humayun et al. 2022a) due difficulties in replicating the method, but the authors
16

---

Published in Transactions on Machine Learning Research (10/2024)
reported that preprocessing for StyleGAN2 took daunting 14 days. In comparison, method and DOT took several minutes, while DRS took less than an hour.
On the other hand, our fine-tuning method maintains the original inference efficiency but compromises training efficiency. Fine-tuning StyleGAN2 using importance sampling with p 80, w = 33.0 lowered the sec/kings. However, fine-tuning diffusion models did not exhibit a significant
speed from 56 sec/kings slowdown in training speed since involves importance sampling only on the real data, avoiding the time- consuming rejection sampling. Notably, fine-tuning process typically requires less than one hour to complete due to the small number of iterations needed.
A.4 Combination of Fine-tuning and Inference-Tinee Methods
Not only the pre-trained models but also the models fine-tuned by our methods can be readily combined with any dfference-time methods such as our proposed importance sampling strategy and truncation (Karras et al. 2019) to further boost the precision/recal improvement. In Table 3 we demonstrate that applying them to fine-tuned models can achieve even higher precision/recall than either fine-tuning or solely applying inference-time methods.
Table 3: Applying truncation importance sampling strategy during inference further improves the precision/recat P stands for precision and R stands for recall. Bold denotes the best value in each column.
Model
P1
R
StyleGAN2
0.685
0.493
+ Fine-tune (r 80%, w 33.0)
0.811
0.387
+ Truncation (œ 0.7)
0.854
0.233
+ Sampling (r 90%, 100)
0.825
0.397
+ Fine-tune (T 80%, 33.0) + Truncation (Φ 0.7)
0.938
0.163
+ Fine-tune (r 80%, w = 33.0) + Sampling (T 90%, w 100)
0.872
0.198
+ Fine-tune 20%, w 0.03)
0.558
0.564
+ Truncation ( 1.3)
0.491
0.571
+ Sampling (r 20%, 0.01)
0.548
0.574
+ Fine-tune (T = 20%, w 0.03)
+ Truncation (Φ =133)
0.386
0.617
+ Fine-tune (r = 20%, w = 0.03)
+ Sampling (r = -20%, w = 0.01)
0.473
0.597
A.5 Autoregressive Generative Models
Autoregressive generative models, such as PixelCNN (Van den Oord et al. 2016) and VQGAN [sser et al. 2021), have the unique capability of generating images while providing their exact likelihood in the learned distribution at the same time. This suggests the potential use of sample likelihood as an indicator of image realism, potentially preventing the need for pseudo density computation in the proposed importance sampling strategies. To investigate this possibility, based importance sampling using the autoregressive VQGAN (Esser et al.) 2021 As shown in Figure 7 log-likelihood-bane based sampling yiclds an inferior precision-reeall tradcoff compared to pseudo-density-base sampling and fails to improve the recall metric even when precision drops below that of the pretrained model. These results indicate that while images with higher likelihood tend to exhibit better realism, those with lower likelihood do not nccessarily demonstrate greater uniqueness. We conclude that the likelihood metric provided by autoregressive models less effective than pseudo density for controlling fidelity and diversity in generated samples.
compared pseudo-density based and log- log-likelhhood-
17

---

Published in Transactions on Machine Learning Research (10/2024)
A.6
Additional Qualitative Result
In Figures 10
2
13
we present images generated by per-sample perturbation, compared to the
original generated images. In Figures
15
16
18
19
20
we present images generated by pre-trained
models and finc-tuned ones.
18

---

Published in Transactions on Machine Learning Research (10/2024)
Figure 8: Images generated by StyleGAN2 pre-trained on FFHQ with per-sample perturbation. (Top pancl): Pscudo Density the objective for PGD. (Middle pancl): Discriminator output as the objective for PGD. (Bottom panel): Random perturbation. In each panel, the middle row displays images generated by the pre-trained model, and the top corresponds to images perturbed for better realism while the images in the bottom
perturbed for better diversity.
19

---

Published in Transactions on Machine Learning Research (10/2024)
Figure 9: Images generated by StyleGAN2 pre-trained on FFHQ with per-sample perturbation. (Middle): Images generated by the pre-trained model. (Top): Generated by the same model with perturbed latent vectors for better realism. (Bottom): Generated by the same model with perturbed latent vectors for better diversity.
Figure 10: Images generated by StyleGAN3-T pre-trained on FFHQ with per-sample pertur- bation. (Middle): Images generated by the pre-trained model. (Top): Generated by the same model with perturbed latent vectors for better realism. (Bottom): Generated by the same model with perturbed latent vectors for better diversity.
20

---

Published in Transactions on Machine Learning Research (10/2024)
Figure 11: Images generated by ProjectedGAN pre-trained on -SUN-Bedroom with per-sample perturbation. (Middle): Images generated by the pre-trained model. (Top): Generated by the same model with perturbed latent vectors for better realism. (Bottom): Generated by the same model with perturbed latent vectors for better diversity.
Figure 12: Images generated by ProjectedGAN pre-trained on LSUN-Church with per-sample perturbation. (Middle): Images generated by the pre-trained model. (Top): Generated by the same model with perturbed latent vectors for better realism. (Bottom): Generated by the same model with perturbed latent vectors for better diversity.
21

---

Published in Transactions on Machine Learning Research (10/2024)
Figure 13: Images generated by IDDPM pre-trained on LSUN-Bedroome Bedroom with per-sample per- turbation. (Middle): Images generated by the pre-trained model. (Top): Generated by the same model with perturbed latent vectors for better realism. (Bottom): Generated by the same model with perturbed latent vectors for better diversity.
Figure 14: Images generated by ADM pre-trained on bation. (Middle): Images generated by the pre-trained model. (Top): Generated by the same model with perturbed latent vectors for better realism. (Bottom): Generated by the same model with perturbed latent vectors for better diversity.
with per-sample pertur-
22

---

Published in Transactions on Machine Learning Research (10/2024)
Figure 15: Images generated by StyleGAN2 trained on FFHQ after fine-tuning with importance sampling. Within each column, images are generated with the same latent code. (Middle): Images generated by the pre-trained model. (Top): Generated by the model fine-tuned for better fidelity. (Bottom): Generated by the model fine-tuned for better diversity.
Figure Images generated by StyleGAN3-T trained on FFHQ after fine-tuning with im- portance sampling. Within each column, images are generated with the same latent code. (Middle): Images generated by the pro-trained model. (Top): Generated by the model fine-tuned for better fidelity. (Bottom): Generated by the model fine-tuned for better diversity.
23

---

Published in Transactions on Machine Learning Research (10/2024)
Figure 17: Images generated by ProjectedGAN trained on LSUN-Bedroom after fine-tuning with importance sampling. Within each column, images are generated with the same latent code. (Middle): Images generated by the pre-trained model. (Top): Generated by the model fine-tuned for better fidelity. (Bottom): Generated by the model fine-tuned for better diversity.
Figure 18: Images generated by ProjectedGAN trained on LSUN-Church after fine-tuning with importance sampling. Within each column, images are generated with the same latent code. (Middle): Images generated by the pre-trained model. (Top): Generated by the model fine-tuned for better fidelity. (Bottom): Generated by the model fine-tuned for better diversity.
24

---

Published in Transactions on Machine Learning Research (10/2024)
Figure 19: Images generated by IDDPM trained on LSUN-Bedroom after fine-tuning with importance sampling. Within each column, images are generated with the same latent code. (Middle): Images generated by the pre-trained model. (Top): Generated by the model fine-tuned for better fidelity. (Bottom): Generated by the model fine-tuned for better diversity.
9l0. R
Figure 20: Images generated by ADM trained on LSUN-Bedroom after fine-tuning with im- portance sampling. Within each column, images are generated with the same latent code. (Middle): Images generated by the pre-trained model. (Top): Generated by the model fine-tuned for better fidelity. (Bottom): Generated by the model fine-tuned for better diversity.
25


Reviews:
[{"review_id": "yzkwYF1vKB", "reviewer": "Reviewer_gu77", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n-\tThe proposed framework is a simple and intuitive way to bias generative models towards higher fidelity or diversity. \n-\tThe experimental results are largely positive and, show interpretability in the controls provided by pseudo density.\n-\tAs someone who is not very familiar with this area, I enjoyed reading this paper. It is generally well written and easy to follow. \n\n**Weaknesses**\n\nI do not have any major weakness to point out but would like to clarify something about the result in Figure 4.\n\nUnlike in other cases, some of the face images appear to provide less realistic outputs on fine tuning with a higher w, when one would expect otherwise. Did the authors explore why that is in their experiments?\n\nI wonder if the nearest neighbors of these images in the training dataset (after feature embedding) are more further apart than usual, thereby resulting in poor approximation of pseudo density at these test images.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a method of modifying the distribution of samples produced by a generative model with the goal of varying the realism of generated images or the overall diversity in the overall distribution. This is achieved by proposing a metric called pseudo density that aims to approximate the underlying probability distribution of the dataset samples. \n\nTo do so, probability densities are first approximated at individual images of the training dataset using a k-nearest neighbor based scheme. Then, a fully connected network is fitted over them to infer the densities at arbitrary images. This approximation obtained using the fully connected network at any data point $x$ is termed pseudo density.\n\nThis proposed metric can be used to modify the individual generated samples and make them more realistic by locally perturbing them with the goal of increasing their pseudo density. Additionally, the generated distribution can be altered during inference by assigning different weights to images depending on their pseudo density followed by using an importance sampling based strategy. A similar approach is proposed to fine-tune a trained model to produce samples with greater realism vs greater diversity.\n\nThe efficacy of the proposed approach is validated experimentally using different generative models.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see the section above", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Bzj29AqXfy", "reviewer": "Reviewer_fPN1", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n- The paper is well-written and provides a clear description of the proposed methods and obtained results.\n- The proposed methods are simple and can be directly applied to a learned generative model from a wide class of models.\n- The authors provide examples of their approach applied to different generative models, including several GANs and one diffusion model. \n- The authors demonstrate that their approach is, in some cases, superior to other baseline methods in terms of the fidelity-diversity Pareto frontier.\n\n**Weaknesses:**\n- The comparison is limited. The authors compare their methods only with the truncation method and polar sampling. Furthermore, polar sampling is considered only with a fraction of the models used for comparison.\n- Some of the competitive methods [1,2] are not considered. The authors state that such methods have relatively higher computational costs than their method. However, they do not support this claim by studying computational costs. \n\n[1] Tanaka A. Discriminator optimal transport //Advances in Neural Information Processing Systems. – 2019. – Т. 32.\n\n[2] Azadi S. et al. Discriminator rejection sampling //arXiv preprint arXiv:1810.06758. – 2018.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose several methods to control the fidelity-diversity tradeoff of a wide class of unconditional generative models. The core of all the proposed methods is transforming real data to the feature space, empirically estimating density in this feature space based on the nearest neighbors approach, and then approximating this density by a neural network. Thus, the authors propose to learn a model that assigns a so-called pseudo-density for any given data point. Then the authors propose 3 specific methods to control fidelity-diversity tradeoffs based on this pseudo-density model. The first one is applicable to the learned model and implies optimizing latent code z to move it to the regions of high pseudo-density to improve fidelity or to the regions of low pseudo-density to improve diversity. The second and third methods combine rejection and importance sampling using pseudo-density. As well as in the previous case, using objects from the region of high density leads to better fidelity, while using objects from the region of lower density leads to better diversity. The second and third methods are similar, but the second method is used at the inference stage with the learned model, while the third method is used for fine-tuning.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I am not deeply aware of related works in the subfield of controlling the fidelity-diversity trade-off in generative models so I can not truly judge how comprehensive the provided comparison is. Anyway, considering the other mentioned methods [1, 2] and studying the computational cost is highly appreciable since the proposed approach is mostly empirical. (This would strengthen the work in my view).\n\n[1] Tanaka A. Discriminator optimal transport //Advances in Neural Information Processing Systems. – 2019. – Т. 32.\n\n[2] Azadi S. et al. Discriminator rejection sampling //arXiv preprint arXiv:1810.06758. – 2018.", "broader_impact_concerns": "No", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "xiNV5gc3zV", "reviewer": "Reviewer_vAPv", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strenghts**\n\n- The paper reads well and is scientifically sound. \n- The results presented in this work are interesting, and the approach of using importance sampling seems quite effective in the analysis performed in the experiment section. \n\n**Weaknesses**\n- I think the paper requires an intuitive visualization of the pseudo-density to enhance clarity and make it easier for the general reader to grasp the basic idea immediately. \nI don't have further weaknesses to highlight. The paper appears to be solid work. \n- However, I do have changes to request, as well as some more conceptual questions to address the author. Please see the section below **Requested Changes**.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a technique to leverage importance sampling to increase fidelity or diversity in newly generated samples from generative models. Crucially, the authors introduce the concept of pseudo-density, which they present as a way to attain a probability density for the target distribution. With this tool at hand, importance sampling, e.g., reweighting/rejection sampling, can be applied both at inference and training (fine-tuning) time to manipulate the model's output to achieve a more diverse or realistic sampling of new images. \n\nWhile I refrain from judging the novel of this work as my knowledge of related literature/work in the field of computer vision is somehow limited, I do find this work to be, in general, very interesting and its contribution to be of valuable interest to the machine learning community.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- It would be important to have some cartoons that sketch the basic idea behind the pseudo density and how it is computed. This would help the reader to have an intuitive understanding of this new metric. In this regard, I find the first paragraph of section 3 a bit hard to parse. The clarity of this paragraph can be easily enhanced by elaborating a bit more on the details and by providing a visual example. \n\n- At the bottom of page 3, the authors introduce the volume a sample occupies $V_i^k$, which is proportional to a constant $C$. This constant is said to depend on the dimensionality of the feature space $n$, although this dependence is not made explicit in the formulae. This might be a little oversight unless I misunderstood something. It'd be great if the author could edit the formulae making the dependence of $C$ on $n$ clear. On the other hand, I find this a bit counterintuitive because having a large feature space should increase the capability of the pseudo-density network to be more expressive. Or is my intuition off here?\n\n- In Fig. 1, the caption indicates that the four left-most images have the highest pseudodensity, while the four right-most images have the lowest. I suggest making this distinction clearer in the figure itself. As a recommendation, the authors could separate the two blocks of 3x4 images, inserting dots in the middle to indicate that there are images lying in between. Additionally, they could add an \"x-axis\" at the bottom displaying pseudodensity as an increasing quantity from right to left (or rearrange the blocks and depict the arrow moving from left to right for increasing pseudodensity, which I find more natural). The authors can refer to Fig. 2 as an example of what I mean.\n\n-  Small typo: $F(x_1 \\to F(x_1)$ (page 4)\n- Small typo: where $\\epsilon$ is the predefined the adversarial budget $\\to$ remove **the** (page 5)\n\n- The first sentence of section 5.2 is not clear to me. It'd be great if the authors could rephrase it. \n\n- In the last sentence on page 6, the authors state that GANs not only have access to the real data distribution but also to the **generated data** distribution. What do the authors mean here? Neither GANs nor diffusion models (in general) give you access to the probability density (or likelihood) for the generative distribution. In this respect, I don't see any difference between those two approaches. Am I misunderstanding something? \n\n\n- The top paragraph on page 7 is not very clear to me. First, I am confused by $Pert_g(\\cdot)$ in the third line. Should not this be $Pert_r(\\cdot)$? Second, I do not fully understand why we need to perform importance sampling of the real data distribution. Importance sampling is used to  *reweight* a probability density so that it matches closely to a target density. In that sense, I do expect to use IS only to *reweight* generated samples based on the pseudodensity as introduced in the paper. I would appreciate the authors elaborating on this and clarifying this paragraph. Again, some more detailed formulae and/or a cartoon would help clarify this crucial part of the paper as well. \n\n- The last paragraph of section 5 seems to suggest that high fidelity and high diversity can be achieved when the importance sampling strategy and other fine-tuning strategies, such as truncation, are applied together. However, not many details are provided in the main manuscript. The authors defer to Appendix A.4, but I did not find much information there. It'd be great if the authors could expand on this, elaborating on the limitations of this method. If there are no limitations in principle I wonder why the authors have not dedicated a greater focus on this combined approach in their experiments. \n\n- small typo: manipulate $\\to$ manipulates (page 8)\n\n- In Figure 3, what do the authors mean with *Ours Frontier* in the plot legend?\n\n## More conceptual questions\n\n- Does the network computing the pseudo-density from the extracted features need to be trained independently for each dataset considered, or is it rather a one-fits-all kind of network that can be trained once (e.g., on Imagenet) and applied to different datasets?\n\n- In page 4, below the figure, the author say that $n$ is set small, e.g., 1,2. Is there any practical heuristics to choose the dimensionality of the feature space? What's the effect of setting $n$ to larger values? Does the quality deteriorate substantially? I presume the authors want to keep the Volume of the ipersphere $V^k_i$ to be small as this is inversely proportional to the pseudodensity. Is my intuition correct? \n\n- As a naive question to the authors: indeed, manipulating the density to produce more diverse samples is likely increasing the number of defects and weird artifacts in the image. Does the author have in mind any technique that could help achieve a high diversity while limiting this artifact, which deteriorates the quality of the generated sample?\n\n- Throughout the paper, the authors discuss different types of generative models such as GANs and DDPMs. However, they never mention autoregressive models, such as PixelCNN. While those have their limitations and may not perform as well as SOTA GANs and DDPMs, they give access to the exact likelihood of the generative distribution. Therefore, combined with the pseudodensity applied to the real data, one can efficiently compute the importance weights and, therefore, perform the same type of rejection sampling and/or fine-tuning, even in a more natural way (since the generative likelihood is given). Is there any specific reason why the authors have not considered such kind of generative models?", "broader_impact_concerns": "I couldn't identify any specific ethical concerns related to the submitted work that need explicit mention. However, all general ethical implications associated with training and utilizing a generative model for image generation, such as potential bias embedded in the model, copyright issues, and privacy concerns within the training data, still hold true.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper provides a sound and intuitive approach to controlling fidelity and diversity in generative models. The introduction of pseudo density as a control mechanism is novel in its application and shows clear empirical benefits across a range of models. In particular, the results are well-supported by extensive experiments across multiple generative models, including GANs and diffusion models. The approach demonstrates flexibility and control, and the empirical evidence for the claims is largely convincing. While the paper is more empirical than theoretical, the experiments validate the effectiveness of the method.

The authors have effectively addressed most of the reviewers' concerns in the rebuttal. The authors are encouraged to incorporate all the requested changes from reviewers as well as their clarifications in the response. For example, further clarity in explanation of certain sections, especially around the pseudo-density concept and the impact of feature space dimensionality would be great. Some visual improvements could be made to make the figures more intuitive. Overall, the paper makes a valuable contribution with well-supported claims. Experimental validation is strong, and most of the reviewers' concerns have been addressed. A final round of polish could enhance the clarity and presentation of the paper.
Decision: Accept (poster)