Don't Play Favorites: Minority Guidance for Diffusion Models
Authors: Soobin Um
Suhyeon Lee
Jong Chul Ye
Venue: ICLR 2024 poster (2024)
Keywords: diffusion models
fairness
generative models
minority generation
Primary area: generative models
TL;DR: We develop a new framework that can make the generation process of diffusion models focus on low-likelihood minority samples.

Abstract: We explore the problem of generating minority samples using diffusion models. The minority samples are instances that lie on low-density regions of a data manifold. Generating a sufficient number of such minority instances is important, since they often contain some unique attributes of the data. However, the conventional generation process of the diffusion models mostly yields majority samples (that lie on high-density regions of the manifold) due to their high likelihoods, making themselves ineffective and time-consuming for the minority generating task. In this work, we present a novel framework that can make the generation process of the diffusion models focus on the minority samples. We first highlight that Tweedie's denoising formula yields favorable results for majority samples. The observation motivates us to introduce a metric that describes the uniqueness of a given sample. To address the inherent preference of the diffusion models w.r.t. the majority samples, we further develop *minority guidance*, a sampling technique that can guide the generation process toward regions with desired likelihood levels. Experiments on benchmark real datasets demonstrate that our minority guidance can greatly improve the capability of generating high-quality minority samples over existing generative samplers. We showcase that the performance benefit of our framework persists even in demanding real-world scenarios such as medical imaging, further underscoring the practical significance of our work. Code is available at https://github.com/soobin-um/minority-guidance.

Published as a conference paper at ICLR 2024
DON'T PLAY FAVORITES: MINORITY GUIDANCE FOR DIFFUSION MODELS
Soobin Um, Subyeon Lee & Jong Chul Ye KAIST, Daejeon, Republic of Korea {sum, suhyeon. lee, jong. ye) yew.keistt.eeeee
ABSTRACT
We explore the problem of generating minority samples using diffusion mod- els. The minority samples are instances that lie on low-density regions of a data manifold. Generating sufficient number of such minority instances is important, since they often contain some unique attributes of the data. How- ever, the conventional generation process of the diffusion models mostly yields majority samples (that lie on high-density regions of the manifold) due to their high likelihoods, making themselves ineffective and time-consuming for the mi- nority generating task. In this work, we present a novel framework that can make the generation process of the diffusion models focus on the minority sam- ples. We first highlight that Tweedie's denoising formula yields favorable re- sults for majority samples. The observation motivates us to introduce a met- ric that describes the uniqueness of a given sample. To address the inherent preference of the diffusion models w.r.t. the majority samples, we further de- velop minority guidance, a sampling technique that can guide the generation process toward regions with desired likelihood levels. Experiments on bench- mark real datasets demonstrate that our minority guidance can greatly improve the capability of generating high-quality minority samples over existing gener- ative samplers. We showcase that the performance benefit of our framework persists even in demanding real-world scenarios such as medical imaging, fur- ther underscoring the practical significance of our work. Code is available at https::/gitte com soobin wtwp:///ooooooumm/iimoaaaaaaaaea
guidance
INTRODUCTION
Conventional large-scale datasets typically exhibit long-tailed distributions, where the majority of samples are concentrated in high-probability regions of a data manifold (Ryu et al. 2017 al.) 2019]. While in low-likelihood regions, there are instances called minority samples, which con- tain unique attributes rarely observed in the majority of the data. The ability to generate minority data holds substantial relevance in critical real-world applications, particularly in enhancing the predictive capabilities of medical diagnosis models by introducing additional instances of rare ditions. This augmentation is also significant in promoting fairness, aligning with social vulnera- bilities often associated with minority instances. Furthermore, the unique features contained within low-likelihood instances are pivotal in applications like creative AI (Han et al. 2022)) where the generation of samples with exceptional creativity an essential component of the task.
One challenge is that generation focused on such minority samples is actually difficult to per- form (Hendrycks et al.] 2021). This holds even for diffusion-based generative models (Sohl- Dickstein et al. (2015] Ho et al. 2020) that provide strong coverage for a given data distribution (Se- hwag et al. (022). The generation process of the diffusion models can be understood as simulating the reverse of a diffusion process that defines a set of noise-perturbed data distributions (Song & Ermon 2019). The principle guarantees their generated samples to respect the (clean) data distribu- tion, which naturally makes the sampler become majority-oriented i.e., producing higher likelihood samples more frequently than lower likelihood ones (Sehwag et al. 2022), when deployed on the long-tail talg--aal-distibutee data. Consequently, collecting even handful number of minority samples ing diffusion models often requires meticulous curation and substantial investments in terms of time and computational resources (Sehwag al. 2022

---

Published as a conference paper at ICLR 2024
Original
Perturbed
Reconstructed
Majority
Vx, the (
Va, log So, (2t, zi)
Minority
40 (. t)
20
E|
Minority
Majority
(a) Biased denoising on CelebA (Luu al. 2015)
(b) Geometric intuition
Figure 1: Inherent bias of prreedie-based denoiser toward majority features. (a) (Left column) Clean images Zy from CelebA; (Middle column) Noised samples Iy made by the DDPM per- turbation (i.e., Eq. (T) with 1. 0.97 on the clean versions in the left column; (Right col- umn) Denoised samples Bgg using Tweedie's formula (i.e., Eq. (6) on the perturbed ones. The top (bottom) represents the perturbation-demoiiinn sequence performed on majority (minor- ity) featured samples. (b) Geometric interpretation of the bias based on optimal score expression sg (zz, t) Eg,, (mo)
vw log qa, (2: [za]].
Contribution. In this work, we propose a novel framework to counteract the majority-focused nature of diffusion models. Our approach begins with the development of new metric describe the uniqueness of features in a given sample. Our metric, which we call minority score, is based on Tweedie's ' formula (Robbins 1992) that yields the posterior mean of clean sample given noise-coruuped one (Kim & Ye 2021 Chung et al. 2022a). We emphasize that given a pretrained diffusion model trained on long-tailed data, Tweedie's 's formula leads to biased denoising against minority samples. Specifically for strong noise-perturbation,, it often incurs significant information define loss for low-likelihood minority samples (see Figure Ta for instance). This motivates our metric as a discrepancy between clean samples and its reconstructed versions via Tweedic's formula. We highlight that the proposed metric is efficient to compute, requiring fewer network evaluations than other diffusion-basse outlier metrics (Wolleb et al.) 2022) Wyatt et al. 2022 Teng al. 2022). We also establish connection to the noise-matching error of diffusion models, offering an additional insight on the effectiveness of our metric for detecting low-density instances.
Given the proposed metric at hand, we develop a sampling technique that can used for encour- aging diffusion models to produce minority-featured featured samples. Our sampler, which we call minority guidance, conditions the sampling process on a desired level of minority score using the classifier guidance technique (Dhariwal & Nichol 2021). One key advantage of minority guidance lies in its label-agnostic nature, which is contrary to existing minority samplers that rely upon the use of labeled data YYu et al. 2020 Lin al. 2022 Sehwag al., (022).
To validate the efficacy of our proposals, we conduct comprehensive experiments across a diverse range of real-world benchmarks, which encompasses both unconditional and conditional datasets. Specifically to highlight the practical importance of our work, we also explore the domain of medical imaging where generating minority instances is a demanding issue in the area (e.g., synthesis of highly-distinctive lesion images). To that end, we first demonstrate that minority score is effective for identifying low-density instances, sharing similar trends with existing criteria like Average k- Nearest Neighbor (AvgkNN) and Local Outlier Factor (LOF) (Breunig et al. 2000). We also exhibit that our minority guidance can serve as an effective knob for controlling the uniqueness of features for generated samples. We further demonstrate that our sampler greatly improves the ability to generate high-quality minority samples on the considered real benchmarks including the medical imaging dataset, as demonstrated by high values of outlier measures (such AvgkNN and LOF) and better quality metrics like Fréchet Inception Distance (Heusel et al. 2017).
Related work. Generating minority samples that contain novel features has been explored under
a number of different scenarios (Sehwag et al. 2022 Lee et al. 2022 Yu et al. 2020: Lin et al,
2022 Qin et al. 2023). The closest instance to work is Sehwag et al. (2022) wherein the authors propose a sampling technique for diffusion models, which can encourage the generation process to
move toward low-density regions w.r.t. specific class using class-predictor and a conditional
2

---

Published as a conference paper at ICLR 2024
model. The key distinction w.r.t ours is that their method is limited to class- conditionl settings and therefore does not work with unlabeled data. Another notable work that bears an intimate connection to ours is Lee et al. (2022). As in Schwagg et al. (2022), they also leverage diffusion models yet for a different modality, graph data (e.g., lying in chemical space). The key idea therein is to produce out-of-distribution (OOD) samples via customized generation process designed for maintaining some desirable properties w.r.t. the focused data space (e.g., plausible chemical structures). Since is tailored for a particular modality which is inherently distinct from our focus (e.g., image data), hence not directly comparable to our approach. The authors in (Yu et al. 2020 Lin et al. 2022] Qin et al., 2023; Huang& & Jafari (023) focus on improving data coverage for low-density instances via the use of labels that indicate minority instances. A distinction w.r.t. ours is that their methods
require the knowledge on the minority labels. Saauel et al. (2023) develops sampler for text-to- image (T2I) diffusion models (Rombach et al. 2022) to specifically enhance the quality of generated samples prompted with unique concepts (c.g., shaking hands). However, their method is confined to text-paired datasets and not applicable to more general benchmarks like unconditional data.
Leveraging diffusion models for identifying uncommon instances (e.g., OOD samples) has recently been proposed, especially in the context of medical imaging (Wolleb et al. 2022. Wyatt al.. 2022 Teng et al.) (022). Their methods share a similar spirit as ours: measuring discrepancies between the original and reconstructed images. However, their focus is for the anomaly detection for given data by generating the majority (normal) samples, which is different from our focus.
BACKGROUND
Before delving into details of our work, we briefly review several key concepts related to diffusion- based generative models. We provide overview of diffusion models with a particular focus on Denoising Diffusion Probabilistic Models (DDPM) (Ho et al. 2020) that our framework is based upon. Also, cover Tweedie's 's denoising formula (Robbins 1992) that plays important role in developing our metric. We review the classifier guidance (Dhariwal & Nichol (021) which provides basic principle for our sampling technique.
2.1 DIFFUSION-BASED GENERATIVE MODELS
Diffusion- based generative models -icks-Dicsttein et al., 2015 Song & Ermon, 2019) are latent variable models defined by forward diffusion process and the associated reverse process. The forward process is a Markov chain with a Gaussian transition where data is gradually corrupted by Gaussian noise in accordance with (positive) variance schedule {3.)2-1 g(æ; xx-1) = (a /1 - B, 10 2, I) where {#:) are latent variables having the same dimensionality as data 26 (xx). One notable property is that the forward process enables one-shot sampling of x, at any desired timestep t:
qa (z) xa) = Naxx Varzo, (1)
where 04 == (1'' B.). The variance schedule is designed to respect oT 0 so that ar be- comes approximately distributed as NO. 1). The reverse process is another Markov Chain that is parameterized by a learnable Gaussian transition: pa xt) := N( po t), B,I). One way to express mo( is to employ noise-conditioned score network se s0(x;; 6) = Vzz log Po (x:) that approximates the score function V.. log 9.., (x.). Then, we have μe (zz. t) -T (z, + S, (2.,t)) (Song & Ermon, 2019 Song et al. 2020b The score network is trained
with weighted sum of denoising score matching (DSM) (Vincent 2011 objectives:
T
min
wi E Be(aa)gn (wx (2w) so (2 1) Vn, m, log Lo, (æx 20)
(2)
where We := 1 ar. Notably, this procedure is equivalent to building noise-prediction network
c,(2), t) that regresses noise added on clean data 2n through the forward process in Eq. (1) (Vin- cent 2011 Song et al. 2020b). This establishes an intimate connection between the two networks: So( (2, t) Eg (2, t) VI ag implying that the score model is closely related to a denoiser. Once obtaining the optimal model via the DSM training, data generation can be done by starting
3

---

Published as a conference paper at ICLR 2024
from 2T NO I) and following the reverse Markov Chain down Zy:
Lt-1 1
-I- (e, + B1 (2r, ()) + BEz,
2~ ~~,
(3)
which is often called ancestral sampling (Song et al. 2020b). This process corresponds to discretized simulation of stochastic differential equation that defines (po(2c:))_o (Song et al. 2020b), which guarantees to sample from (ato)) Rs g(2ro).
2.2 TWEEDIE'S S FORMULA FOR DENOISING
Given perturbed sample with exponential family noise, classic result of Tweedie's formula gives the Bayes optimal solution for denoising (i.e., the posterior mean) using the score function (Robbins) Kim & Ye 2021). Under Gaussian perturbation qo (æ) x)) [N æ o² the formula reads:
1992
E T æ + o' V log (Z) where 4a (Ø) = 9. ( (() dæ. Tweedic's 's formula can be
extended to our focused DDPM perturbation (i.e., Eq. T (Chung et al. 2022a (b:
E E[
(x: + (1 at) Va, log 1og4a,(4 (x:)),
(4)
v(z
Notice that once having the optimal score model si (3r, Væ, log qa, (2t), can implement the formula and get the posterior mean by using the optimal model in place of the score function Eq.
2.3 CLASSIFIER GUIDANCE FOR DIFFUSION MODELS
Suppose we have access to auxiliary classifier p (v(=z) that predicts class y given perturbed input æe. The main idea of the classifier guidance is to construct the score function of conditional density w.r.t. by mixing the score model s0 1) and the log-gradient of the auxiliary classifier:
V ns log pa (æt y) V.. (log (ax:) + (o)
so (2 8) + w V., log (y))) =: S0 f, y),
(5)
where w is a hyperparameter that controls the strength of the classifier guidance. Employing the mixed score $0 f, y) in place of sg t) in the generation process (e.g.. in Eq. (3) enables the conditional sampling w.r.t. Po (x:yly) ε po (o())))pp (v) Increasing the scaling factor affects the curvature Pa (3)(2x))) around given y to be more sharp. i.c., gives more strong focus on some noticeable features w.r.t y, which often yields improvement of fidelity w.r.t. the corresponding class at the expense of diversity (Dhariwal & Nichol 202T).
METHOD
We present our framework herein that specifically focuses generating minority samples lying on low-density regions of a data manifold. To this end, we start by pinpointing the inherent bias of Tweedie's denoising formula against low-density samples. In light of this, we come up with measure for describing the uniqueness of features. We subsequently develop sampler that can guide the generation process of diffusion models toward minority regions. Throughout the section, we follow the setup and notations presented in Section 2
3.1 INHERENT BIAS OF TWEEDIE-BASER DENOISER
Consider data distribution y (xx)) where unique features data are contained in low-density regions Suppose we have access to diffusion model 8 (m,, i) trained on dataset 2o y(mo) via DSM (in Eq. (2). Then for Gaussian Gaussian-perturbel perturbed sample æ; ~ 9a, (xe, Φo), one can obtain the posterior mean
to by implementing Tweedie's 's formula in Eq. (4)
to == E CCo
(xe + (1 an (2),
WWe note that such long-tailed property is common in conventional large-scale henchmarks (Ryu et al.
2017
Liu al. 2019 Schwag et al. 2022)
4

---

Published as a conference paper at ICLR 2024
Smallest minority score
Moderate minority score
Highest minority score
CelebA
CIPAR-10
LSUN
Figure 2: Real samples with the smallest (left column), moderate (middle column), and the highest (right column) minority score, our proposed metric for identifying minorities.
While it is the best effort in a statistical sense, this denoising has critical downside in terms of fair- ness (Choi et al. [2200] Xiao et al. (02I). The issue is that the expectation induced by the formula makes the denoising process strongly affected by majority samples (that mostly contribute to the average), thereby yielding biased results against low-likelihood minority instances. The discrimina- tion often severe under strong perturbation (e.g.. Rs T) where denoised outputs become close to dataset mean (Karras al. 2022).
Figure Ta illlsttaates an example of this observation for groups of majority and minority samples in CelebA (Liu et al.) 2015). As we can see, the denoiser based on Tweedie's formula successfully restores some key structural properties (e.g., pose and gender) when perturbation is made upon ma- jority samples. However, it fails to do so for minority- denoiser embeds some common features in place of the unique attributes that are originally con- tained in the clean minority samples, exhibiting its inherent bias toward majority features. Figure Tb gives geometric illustration of the phenomenon. In the figure, we employ another expression for the optimal score function provided in the following proposition:
samples. Instead, the Tweedie-based
Proposition Consider the DSM optimization in Eq. (2) Assume that a given noise- coise-conditioned score network so(x,,? t) have enough capacity. Then for each timestep t, the optimality of the score network is achievable at:
(3 1) Egg, (2eg (x,) [ mc log qo, (wx [o))]
See Section A. for the proof. Observe in Figure Ib that the optimal score function gives the aver- aged conditional score over the data distribution (conditioned noised input æ;2), thereby producing directions being inclined to the manifold w.r.t. the majority samples.
3.2 MINORITY SCORE: MEASURING THE UNIQUENESS
Inspired by the discriminative treatment against minority features, we develop metric that quan- tifies the uniqueness of features of a given data instance. Remember that Tweedie's 's formula often incurs significant loss of information for minority samples (as seen Figure la). Hence, we employ a distance between original clean sample 2o and its denoised version an::
I(x0; x6) := Eg.,, (2, u.)) [d((ar, @o)];
where d(..) denotes a discrepancy measure (e.g.. Learned Perceptual Image Patch Similarity (Zhang et al.] 2018), squared-eree loss), and t is timestep used for perturbation; see Sections C and D.I for details on our choices of d( d(...) and f.. The expectation is introduced for randomness that comes from the DDPM perturbation? We call this metric minority score, since it would yield high values for minorities that contain novel attributes.
Figure 2 visualizes the effectiveness of minority score on several benchmarks that have diverging characteristics. Notice that the samples in the left column exhibit features that are commonly ob- served in the corresponding datasets (e.g., frontal-view faces in CelebA (Liu et al. 2013)). On the
We empirically found that the metric performs well even when the expectation is computed with a single sample from Jau (z:(mo) See Figure for instance, where the computations based such single-sampling.
5

---

Published as a conference paper at ICLR 2024
other hand, some unique features in the right column, such "Eyeglasses" and "Wear- ing. that are famously known as minority attributes of CelebA (Amini et al. 2019 Yu et al. 2020). We found that minority score shares similar trends with existing outlier metrics, exhibiting its effectiveness for identifying minorities; see Section D.2 for empirical evidence. Thanks to the one-shot reconstruction offered by Tweedie's formula, minority score is more efficient to compute than previous methods that rely upon iterative forward and reverse diffusion processes so requiring a number of evaluations of models (Wolleb al. 2022 Wyatt et al. 2022 Teng et al.) 2022). con- trast, our metric requires only a few number of function evaluations. Given a conditional model and
labeled data, minority score can be extended to class-conditional settings for identifying low-density
samples w.r.t. specific class; see Section B for details.
Connection to noise-prediction error. We found that minority score bears an intimate connection to the noise-estimation error of diffusion models. Specifically with the squared-error distance loss, minority score is equivalent (up to scaling) to the noise-matching error for timestep This offers additional insights on why low-density instances could yield high minority scores, e.g., since they would experience significant error noise-estimation (compared to the majorities) due to their scarcity in data. Below we provide formal statement of the claim. See Section A.2 for the proof.
Corollary 1. Minority score, when defined with the squared-ers error distance loss || 11. is equivalent the noise prediction error w.E.t. timestep t up scaling:
[aoi sT) = Epeep()) [1|5 -E
where W. = (1 (1-aa)/a an)/au-
3.3 MINORITY GUIDANCE: TACKLING THE PREFERENCE
Here a natural question arises. Given minority score at hand, what can we do for tackling the majority-focused generation of diffusion models so that they become more likely to generate the novel-featured samples? To address this question, we take a conditional generation approact[]whre where we incorporate minority score as a conditioning variable into the generation process, which can then serve produce the miiqu--featrre condition the given framework with minimal effort, we employ the classifier guidance (Dhariwal & Nichol 2021) that does not require re-building of class-conditional models. Below we describe in detail how we develop the conditional generation framework for minority score based on the classifier guidance technique.
samples by conditioning w.r.t. high minority score values. To
(N) Lid.
Let us consider dataset To ((xo)) and pretrained diffusion model sy (x:, t).
~
Impact minority class
Impact of classifier scale
Real
Real
For each sample, we compute minority score via Eq. 7 and obtain (()) ..... ((N) where (() = l(a)" (0) s E (I....,Ny N.. We pro- cess the (positive-valued) minority scores as or- dinal data with L categories by thresholding them with L 1 levels of minority score. This yields the ordinary categorized minority scores j()) J(N) E (O..... L. 1} where ī and i = L indicate the classes w.r.t. the most common and the rarest features, respec- tively (see Section details on the categorization). The ordinal minority scores are then coupled with the associated data samples to yield paired dataset (= (,(())),...............)) (x) (((,)) which is subsequently used for training (noise-condittoned) classifier Pep (Tla.)) that predicts for input æ
Ltide=0
w=100
"D....,((N),
Density
Ltide-50
Density
w 0
1idc-99
2
100
1.1
12 LOF
1.3
1.4
1.0
3.1
1.2 LOF
1.3
14
Figure 3: Impacts of minority class ī (left) and classifier scale w (right) on the density of Local Outlier Factor (LOF). The other parameters fixed: 2.0 (left) and = 0 (right).
(perturbed via Eq. (T). After training, we blend the given score model with the log- geedgradient of the classifier as in Eq. (5) yield modified score:
so t,), = (z,,, t) + V., log log (i xx.),
where zb indicates parameterization for the classifier, and w is a scaling factor for the guidance; Figure 3 for details on its impact. Incorporating this mixed score into the sampling process
We also explored another natural strategy that concerns sampling from ((zo) x (<zo))((oo; so). See Section D.5 for details.

---

Published as a conference paper at ICLR 2024
Generated samples by StyleGAN (Karras al. 2019) (left), ADM (Dhariwal& & Nichol 202T) with ancestral sampling (middle), and minority guid- ance (right) are exhibited. We use the same random seed for the diffusion-bai based samplers.
Figure 4: Sample comparison on
Figure 5: Sample comparison on the brain MRI data. Generated samples by StyleGAN2- (arr- ras al. 2020) (left), ADM (Dhariwal & Nichol 202T with ancestral sampling (middle), and minority guidance (right) are exhibited. We use the same seed for the diffusion- based samplers.
(in Eq. 3 then enables conditional generation w.r.t. (xx xx po po(ar))p (1 (]ax))". We call our technique minority guidance, it gives guidance w.r.t. minority score in the generation process.
Generating unique-featured samples is now immediate with minority guidance, e.g. by condition- ing an arbitrarily high I with a properly chosen w to incorporate the strength of the associated class I. Even more, our sampler enables free control of the uniqueness of features for generated sam- ples, which has never been offered in literature so far our best knowledge. This uniqueness- controllability could be particularly instrumental in many practical scenarios including medical imaging; see Figure 14 in Section D.7 for one such instance. We found that minority guidance is not confined to the unconditional setting we explored thus far, and can be extended to the class- conditional context for producing low-density samples w.r.t. a specific class. See Section for de- tails on the extension. This broad applicability that encompasses both unconditional and conditional settings is a key advantage of our approach compared to existing methods for minority generation, which are limited to supervised (i.e., -avilablon-availabll)) settings only (Yu et al. 2020 Lin et al. 2022 Sehwag et al. 2022).
4 EXPERIMENTS
In this section, we provide empirical demonstrations that validate our proposals and arguments through the image generation task, considering both unconditional and class-conditional scenarios. To this end, we first clarify the setup used in our demonstrations (see Section for more details) and then provide results for our approach comparison with existing frameworks. A comprehensive collection of our experimental results can be found in Section
4.1 SETUP
Datasets. Our experiments are conducted on six real benchmarks: four unconditional and two class-conditional datasets. For the unconditional settings, we employ CelebA 642 (Liu et al. 2015).
7

---

Published as a conference paper at ICLR 2024
Real Styleorw
CDPM (eandard) DDM louns)
coNN Iitandand) coursi
COrM Istendard) tours)
sty
StylcbMM
0.20
0.020
0.15
0.015
Density
Density
0.30
0.010
0.005
0.00
0.000
120
AvgkNN
Rarity Score
Figure 6: Comparison of neighborhood density on of LUNN-Beedoooms. "DDPM (standard)" indicates ADM (Dhariwal & Nichol (02T) with ancestral sampling, while "DDPM (ours)" is ours. The higher values, the less likely samples for all measures.
"Real" refers to the training set
CIFAR-10 Krizhevsky et al., 2009), and L.SUNBeedrooms 256 Yu et al., 2015). In addition to the three unconditional natural image benchmarks, employ medical dataset to demonstrate the practical importance of our work. Specifically, we consider an in- ho-house (IRB-approved) brain MRI dataset containing 13, 5,640 axial slice images, where minorities are instances that exhibit degenerative disease like cerebral atrophy. The MRI images are standard 3T T1-weighted with 256² resolution. We use ImageNet 64 and 256" (Deng al. 2009) for the class-conditional generation. To further investigate the performance on long-tailed data, we additionally incorporate a famous benchmark specifically manipulated to exhibit long-tailed tailed characteristics: CIFAR- 10-LT
(Cao et al.) 2019).
Baselines. We consider three prominent GAN-based frameworks for the unconditional cases: (i) BigGAN (Brock et al.] 2019); (ii) StyleGAN (Karras et al. 2019); (iii) StyleGAN2-ADA (Karras)
et al. 2020). In addition to the GAN baselines, we employ a standard sampler for diffusion models, ancestral sampling (a. the DDPM sampling) (Ho et al. 2020), which we implement with the
pretrained diffusion models used in each benchmarks (e.g., ADM (Dhariwal & Nichol (021) on CelebA). The BigGAN models are employed for CelebA and CIFAR-IO, and we use StyleGAN for L.SUN-Bedrooms, while StyleGAN2-ADA is compared in our brain MRI experiments. The DDPM sampling is employed in all three unconditional datasets. To explore potential benefits that may come from the use of minority labels, we incorporate additional baseline on the CelebA dataset, which conditionally generates minority instances using the classifier guidance and minority annota- tions available in the data (e.g.. "Eyeglasses") Yu et al., (0020). We also incorporate unconditional version of [Sewaa et al. (002)) on CelebA, which we specifically tailor the original method (devel- oped for class-conditional datasets) for the use in unconditional data. For LSUN-Bedrooms, we additionally consider LDM (Rombach et al., (022), a famous latent-based framework extensively used in various applications. For the class class-conditional ImageNet. we consider two baselines. the standard DDPM sampler and (Sehwag et al. (2022). On CIFAR- 10-LT, our approach is compared with Qin et al. (023) and ancestral sampling.
Evaluation metrics. For the purpose of evaluating the generation capability of low-density sam- ples, we incorporate three distinct measures for describing the density of neighborhoods: (i) Average
k-Nearest Neighbor (AvgkNN); (ii) Local Outlier Factor (LOF) (Breunig et al.) (000); (iii) Rarity Score (Han et al. 2022). For all three measures, a higher value indicates that a given sample less likely than its neig neighboring samples (Sehwag et al. 2022 Han et al.) 2022). Additionally to investi- gate fidelity and diversity of generated samples, we employ a set of quantitative metrics: (i) Clean Fréchet Inception Distance (cFID) (Parmar et al. 2022); (ii) Spatial FID (sFID) (Nash et al. 2021);
(iii) Improved Precision & Recall (Kynkäänniemi et al. 2019). Since we are interested in producing high-quality diverse samples close real low-density data, yielding the highest AvgkNNs) as baseline real data for computing these quality metrics.
the most unique samples (e.g.,
4.2 RESULTS
Validation of the roles of i and w. Figure 3 illustrates the impacts of the control parameters of minority guidance on likelihood distributions of generated samples. Observe in the left plot that increasing the minority class i yields the LOF density shifting toward high-LOF (i.e., likelio)))))od) regions, demonstrating the capability of our sampler to encourage generations of minorities. On the other hand, the right plot of Figure exhibits that an increase of the classifier scale w makes the density squeezing toward high likelihood regions. This showcases the controllability of minority
8

---

Published as a conference paper at ICLR 2024
Method
cFID
sFID
Prec
Rec
Method
cFID
sFID
Prec
Rec
CelebA 64 x
LSUN Bedrooms 256 x 256
ADM
75.05
16.73
0.97
0.23
ADM
62.83
7.49
0.89
0.16
ADM-ML
51.66
13.04
0.94
0.31
LDM
63.61
7.19
0.90
0.13
BigGAN
80.41
16.51
0.97
0.19
StyleGAN
56.45
5.96
0.87
0.13
Sehwag et al.
27.97
10.17
0.82
0.42
Ours
41.52
6.67
0.87
0.10
Ours
26.98
8.23
0.89
0.34
ImageNet 256x2550
Brain MRI 256x
ADM
17.41
12.85
0.87
0.35
ADM
21.71
14.30
0.82
0.37
Sehwag et al.
13.43
9.88
0.85
0.42
StyleGAN2a
22.69
23.07
0.53
0.28
Ours
13.83
7.82
0.85
0.45
Ours
19.73
13.36
0.84
0.47
Table 1: Comparison of sample quality and diversity for generating minority samples.
"ADM-
ML" refers to classifier-guided sampler implemented on ADM (Dhariwal & Nichol 2021 which conditions on Minority Labels. The best results marked in bold.
guidance on the strength of features associated with the selected minority class (e.g.. i = 0 in Figure 3), which also aligns with the well-known role of w emphasized in Dhariwal & Nichol 20211 See Figures [5) and 16 (in Section E for generated samples that visualize these impacts.
Comparison with the baselines. Figure 4 compares generated samples on LSUN-Bedrooms. OB- serve that our sampler produces more exquisite and unique images than the two baseline methods. Notably, it often encourages some monotonous-lookin features generated by the standard DDPM sampler to be more novel, c.g., with inclusions of additional objects. We found that such sophisti- cated attributes are in fact novel features that yield high minority score values LSUN-Bedrooms (see Figure @ for instance). Figure exhibbis generated samples on the brain MRI benchmark. We see that our sampler are more likely to generate minority instances of the dataset (e.g.. containing severe brain atrophy) compared to the baseline methods (see Figure 13 for visualizations of the minorities). We highlight that the minority- -ennrrty-geeeratingg capability of our sampler persists even on this particular type of data containing distinctive visual aspects, further demonstrating the practical significance of our method. We leave the generated samples on the other datasets in Section
Note that minority guidance outperforms the baselines in generating unique samples for all three measures, which corroborates the visual inspections made Figure We leave the density results the other datasets in Sec- tion [E] Table exhibits quality and diversity evaluations on four challenging benchmarks (see Ta- ble for the results the CIFAR-10 10 settings and ImageNet-64).. For baseline real data, we employ the most unique samples that yield the highest AvgkNN values. Additional evaluations using other criteria for collecting baseline data can found in Section E see Table g therein. Observe that our sampler yields better (or comparable) results than the basclines in all metrics, demonstrating its ability to produce realistic and diverse samples close to the real minorities. For a more robust evaluation on the MRI data containing distinctive features (hence may not be interpretable in the ImageNet- basee feature space), we provide evaluations based on domain-specific feature space that could admit our MRI data; see Section for details.
Figure 6 compares our focused density measures
4
5 CONCLUSION AND LIMITATIONS
We emphasized that Tweedie- based denoiser is inherently biased to produce high-likelihood major- ity samples. In light of this, we introduced novel metric to evaluate the uniqueness of features and developed sampling technique that can serve to generate novel-featured minority samples. We demonstrated that the proposed sampler greatly improves the capability of producing high-quality minority samples even on high-stake applications like medical imaging.
One disadvantage is that our approach requires the construction of a minority classifier, which is often computationally expensive when the size of given dataset is huge. Hence, believe one promising future direction to push the boundary towards the challenging scenarios that are data- restricted while addressing the computation issue.

---

Published as a conference paper at ICLR 2024
ETHICS STATEMENT
The brain MRI images employed in experiments were obtained exclusively for in-house pur- poses and are not publicly accessible. We emphasize that our MRI dataset has been rigorously reviewed and approved by the Institutional Review Board (IRB), ensuring strict compliance with ethical standards. This includes meticulous attention to informed consent procedures and robust data anonymization protocols.
A potential risk due to our sampler is that it could be used maliciously to suppress the generation of minority-featured samples. This could be achieved, for instance, by using low values of 7 in Eq. (8). ie.. focusing on producing instances containing majority features. important be aware of this risk and to ensure that our proposed framework is used responsibly to promote fairness and inclusivity in generative modeling.
REPRODUCIBILITY
To ensure the reproducibility of our experiments, we provide thorough description regarding our four publicly available benchmark real datasets: CelebA (Liu et al. 2015), CIFAR-10 (Krizhevsky itt al. 2009). LSUN-Bedrooms (Yu et al. 2015). and ImageNet (Deng et al. 2009). For our in- house brain MRI dataset which is unable to open to the public, we describe in detail on its char- acteristics and the way of gathering. All these settings including the choices of hyperparameters are presented in Section C We also provide the average running time of our algorithm together with specific computer configuration details in the same section. Lastly, to facilitate replication, we make our code available in public repository: https:// com/ soobin iinoinin- um/ minority guiiiiiannn dance
ACKNOWLEDGMENTS
This work supported by National Research foundation of Korea (NRF) (No. RS-2023- 00262527), by Field-oriented Techrieeeeeeeeholoooooooo Technology Development Project for Customs Administration through National Research Foundation of Korea (NRF) funded by the Ministry of Science & ICT and Korea Customs Service (No.
1097933), by the Korca Medical Device Development Fund grant funded by the Korea government (the Ministry of Science and ICT, the Ministry Trade, In- dustry and Energy. the Ministry of Health & Welfare, the Ministry of Food and Drug Safety) (Project Number: 171 771133889, KMDF PR 20200990 1001)), by Culture, Sports and Tourism R&D Program through the Korea Creative Content Agency grant funded by the Ministry of Culture, Sports and Tourism in 2023, and by Institute for Information & communications Technology Promotion (ITTP) grant funded by the Korea government (MSIP) (No. 2019-0-00075 Artificial Intelligence Graduate School Program (KAIST)).
REFERENCES
Alexander Amini, Ava P Soleimany, Wilko Schwarting, Sangeeta N Bhatia, and Daniela Rus. Un- covering and mitigating algorithmic bias through learned latent structure. In Proceedings of the 2019 AAAI/ACM Conference AI, Ethics, and Society, pp. 289-299 2019.
Markus M Breunig, Hans- Kriegel, Raymond T Ng, and Jörg Sander. Lof: identifying density- based local outliers. In Proceedings of the 2000 ACM SIGMOD international conference Management of data, pp. 93-104, 2000.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In International Conference on Learning Representations, 2019.
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with label-distribution aware margin loss. Advances neural information processing systems, 32, 2019.
10

---

Published as a conference paper at ICLR 2024
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties self-supervised vision transformers. Proceedings of the International Conference Computer Vision (ICCV), 2021.
Kristy Choi, Aditya Grover, Trisha Singh, Rui Shu, and Stefano Ermon. Fair generative modeling via weak supervision. In Proceedings of the 37th International Conference on Machine Learning (ICML). PMLR, 2020.
Hyungjin Chung, Jeongsol Kim, Michael T Mccann, Marc I. Klasky, and Jong Chul Ye. Diffusion posterior sampling for general noisy inverse problems. arXiv preprint arXiv:2209. 14687, 2022a.
Hyungjin Chung, Bycongsu Sim, Dohoon Ryu, and Jong Chul Ye. Improving diffusion models for inverse problems using manifold constraints. arXiv preprint arXiv:2206.0094 2022b.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. leee, 2009.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. Advances in Neural Information Processing Systems, 34:8780-8793 8794, 2021.
Izrail Moiseevitch Gelfand, Richard A Silverman, et al. Calculus of variations. Courier Corporation, 2000.
Jiyeon Han, Hwanil Choi, Yunjey Choi, Junho Kim, Jung- Woo Ha, and Jaesik Choi. Rarity score: A new metric to evaluate the uncommonness synthesized images. arXiv preprint arXiv:2206.08544 2022.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adver- sarial examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15262-15271, 2021.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
Gaofeng Huang and Amir Hossein Jafari. Enhanced balancing gan: Minority- image genera- tion. Neural computing and applications, 35(7):5145-5155) 2023.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative the JEEE/CVF conference on computer vision and pattern
adversarial networks. In Proceedings the recognition, pp. 4401-4410, 2019.
Tero Karras, Miika Aittlla, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. In Proc. NewrIPS, 2020.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. arXiv preprint 22X.:2206.003666 2022.
Dongjun Kim, Yeongmin Kim, Wanmo Kang, and Il-Chul Moon. Refining generative process with discriminator guidance in score-based diffusion models. arXiv preprint arXiv:2211. 2022.
Kwanyoung Kim and Jong Chul Ye. Noise2score: tweedic's approach to self-supervieed image denoising without clean images. Advances in Neural Information Processing Systems, 34:864- 874, 2021.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. Advances Neural Information Processing Systems, 32, 2019.
11

---

Published as a conference paper at ICLR 2024
Scul Lee. Jaehyeong Jo, and Sung Ju Hwang. Exploring chemical space with score-based out-of- distribution generation. arXiv preprint orXiv:2206.0763,, 2022.
Zinan Lin, Hao Liang, Giulia Fanti, Vyas Sekar, Rahul Anand Sharma, Elahe Soltanaghaci, Anthony Rowe, Hun Namkung, Zaoxing Liu, Dachyeok Kim, al. Raregan: Generating samples for rare classes. arXiv preprint ar Xxxv::223.106111 2022.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X Yu. Large- scale long-tailed recognition in an open world. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 2537-2546, 2019.
Muhammad Ferjad Naeem, Seong Joon Oh, Youngjung Uh, Yunjey Choi, and Jaejun Yoo. Reliable fidelity and diversity metrics for generative models. In International Conference on Machine Learning, pp. 7176-7185. PMLR, 2020.
Charlie Nash, Jacob Menick, Sander Dieleman, and Peter W Battaglia. Generating images with sparse representations. arXiv preprint arXiv:2103.03844 2021.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffiusion probabilistic models. In International Conference on Machine Learning, pp. 8162-8171. PMLR, 2021.
Gaurav Parmar, Richard Zhang, and Yan Zhu. On aliased resizing and surprising subtleties in gan evaluation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11410-11 11420, 2022.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances in neural information processing systems, 32, 2019.
Yiming Qin, Huangjie Zheng, Jiangehao Yao, Mingyuan Zhou, and Ya Zhang. Class-balancing diffusion models. arXiv preprint arXiv:2305.00562, 2023.
Herbert E Robbins. An empirical bayes approach to statistics. Breakthroughs in statistics, pp- 388-394 Springer, 1992.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image syuthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Hee Jung Ryu, Hartwig Adam, and Margaret Mitchell. Inclusivefacenet: Improving face attribute detection with race and gender diversity. arXiv preprint arXiv: 2017.
Dvir Samuel, Rami Ben-Ari, Simon Raviv, Nir Darshan, and Gal Chechik. is all about where you start: Text-to-image generation with seed selection. arXiv preprint arXiv:2304.1455 2023.
Vikash Sehwag. Caner Hazirbas, Albert Gordo, Firat Ozgenel, and Cristian Canton. Generating high fidelity data from low-density regions using diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11492-1155 15011 2022.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learn ing, pp. 2256-2265. PMLR, 2015.
Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 2020a.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems, 32, 2019.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:201 134466, 2020b.
12

---

Published as a conference paper at ICLR 2024
Yapeng Teng, Haoyang Li, Fuzhen Cai, Ming Shao, and Siyu Xia. Unsupervised visual defect detection with score-based generative model. arXiv preprint arXiv:2211.16092, 2022.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Deep image prior. In Proceedings of the
IEEE conference on computer vision and pattern recognition, pp. 9446-9454, 2018.
Pascal Vincent. connection between score matching and denoising autoencoders. Neural compu- tation, 23(7):1611-177, 2011.
Julia Wolleb, Florentin Bieder, Robin Sandkühler, and Philippe C Cattin. Diffusion models for medical anomaly detection. arXiv preprint arXiv:2203.04366 2022.
Julian Wyatt, Adam Leach, Sebastian M Schmon, and Chris G Willcocks. Anoddpm: Anomaly detection with denoising diffusion probabilistic models using simplex noise. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 650-656, 2022.
Zhisheng Xiao, Karsten Kreis, and Arash Vahdat. Tackling the generative learning trilemma with denoising diffusion gans, arXiv preprint urXiv:2 2021.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv 2015.
preprint 1rXiv:5006.000
Ning Yu, Ke Li, Peng Zhou, Jitendra Malik, Larry Davis, and Mario Fritz. Inclusive gan: Improving data and minority coverage in generative models. In European Conference on Computer Vision, pp. 377-393. Springer, 2020.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness deep features as perceptual metric. In CVPR, 2018.
Yue Zhao, Zain Nasrullah, and Zheng Li. Pyod: A python toolbox for scalable outlier detec- tion. Journal of Machine Learning Research, 20(96):1-7, 2019. URL http: httta::///mlr.ooo/ papers/v20 10000/119900000 011.htmml
13

---

Published as a conference paper at ICLR 2024
A PROOFS
A.1 PROOF OF PROPOSITION 1
Proposition [ Consider the DSM optimization in Eq. 22) Assume that given coise-conditioned score network 80 a2, have enough capacity. Then for each timestep i, the optimality of the score network is achievable at:
(a t) = Bag (
Vx. xe log qa, (xt æo)].
Proof. Since the score network sø (z t) is assumed to be large enough, can achieve the global optimum that minimizes the DSM objectives for all timesteps regardless of their weights in Eq. (2) (Song et al. 2020a). Hence, we can safely ignore the weighted sum and focus on the DSM objective for a specific timestep to derive (z t):
s ( t) arg min En(aoo) (m) se ( or, V log qa. (xx xo)
#gES
where we consider the optimization over function space 80 E S instead of the parameter space ④ for the of analysis. The objective function in Eq. (9) can be rewritten as:
E g ) (me mo) (x t) Væ, log 9a, (xx xo)]
/ / q( ( ((x0))a. (xt æo)) se (t, Va, log qa, (xt; xo)) l| dæ, dæy
/ la (as) [x)) so t) v. log qa, (x:. 2o)ll day dæy.
((
Since the objective function Eq. (9) a functional and convex w.r.t. se, we can apply the Euler-
Lagrange equation (Gelfand al. 2000) to come up with a condition for the optimality:
ac
f
0
+
go. (2x, (z. (wo) xx.) 2 (30 t) - Vas, log do. (æ. xoo) dæc = 0.
Ase
Rearranging the RHS of the arrow gives:
/ go. (xx)) (wo xx)) 2 (2 t)
Va, log 9o, (xe xo0) dæo = 0
S (xxt,)) t) / qae (xo xx) dæy
ga. (2o xx)) Ve, log Ja, (a: To) dæy
(a,, t) Erg, (molare) Van, log aco (xx, xo)]]
This completes the proof.
A.2 PROOF OF COROLLARY
Corollary 1. Minority score, when defined with the squared-error distance loss || 112. is equivalent the noise prediction error w.r.t. timestep up to scaling:
l(xo: s/) (.rEp(e) (xl, =112]
where W; : -1- (1-aa)) a1-a))/an.
Proof. We start from the definition of minority score in Eq. (7):
l(xn 8%) : Ey., (m/ mm) [d([to 2n)].
14

---

Published as a conference paper at ICLR 2024
Plugging the squared-enve error loss and further manipulations then yield:
1( (ao; 88))
Bua, (oea (02) [d(
.bo]]
Bg (#. (mm) ( 20
17
Bya (ctelozu)
-0000 val
f vI - -a [3,
M- az (E - -}
[I
Bp(e))
Vat
ae
B((s)) [lle t) -c1
at
= p(e) [1 1) =112]
(10)
where tü (1 aa.. The second equality is due the Tweedie's formula (i.e., Eq. 4
gether with the noise-predicting expression s6 (zr,, = ((vi.)/vVI-- II- Ozz, and the third equality follows from the DDPM forward process in Eq. (D). This completes the proof.
B EXTENSION TO CLASS-CONDITIONAL SETTINGS
We continue from Section 3 of the main paper and explore how to extend our proposed framework toward class-conditional settings. To this end, we first provide brief background on conditional diffusion models that will be used throughout this section. After that, we elaborate extensions of our metric and guided sampler to our interested dasss-conditonall context.
B. BACKGROUND ON CONDITIONAL DIFFUSION MODELS
Consider joint density g( c) = g(c)g(xolc) where c is a random variable (or vector) that in- dicates specific condition The reverse process to model conditional density g(æole) can be defined as pa( Ce, C)
data (e.g., "Water tower" in ImageNet (Deng et al. 2009
( (# 4, c), B) where (3:) is the same noise schedule in Section 2 Similar to the unconditional setting, we can express the mean of the reverse conditional density by employ- ing score network: po( c) vI-3, (x, + 8 c)). The score network is designed to approximate class-conditional score function Væ, log 9a, (x//c) and therefore parameterizable as se to,,,,,, c) Va, log.oo (x The conditional score network can be trained with a variant of denoising score matching similar to the unconditional case:
T
min
w, (o (m. (l (z Va. log qa, (xx, [o)
(11)
where Wi := 1 ag. Given the optimal score network s (xx, f, c) obtained via Eq. ancestral sampling can be used for producing samples from po (xulec x c))ulc)):
11 the same
Pt-1
(xt + (, t, + BIz, 10,10.
(12)
V1
Br
B.2 EXTENSION OF MINORITY SCORE
Our first step extending Tweedie's formula to the class-conditional setting that we described in Section B. Let us consider labeled sample (æo, c) q(æ0, c) and its noised version at ~ lee (i, (o)) For denoising, one can naturally think of using a variant of Tweedie's formula by replacing s (æx, t) in Eq. (6) with sz t. e):
1
(o(c) = Va
(x. + (z)
(13)
In fact, this natural extension yields the posterior mean the clean sample E[w, x,, c): see the following proposition for formal description and the proof.
15

---

Published as a conference paper at ICLR 2024
Smallest minority score
Moderate minority score
Highest minority score
Water tower
Bald cagle
Joystick
ESP maker
Figure 7: Real samples with the smallest (left column), moderate (middle column), and the highest (right column) minority score. An extended version minority score (i.e., Eq. (16) is used on the class-conditional ImageNet 64 x 64. We employ Learned Perceptual Image Patch Similarity (Zhang et al. 2018) for d(-,-) in Eq. (16). The considered classes are as follows: water tower (top row), bald eagle (top-middle jow),joyticc (middle-bottom row), and espresso maker (bottom row).
t, c) trained via the DSM optimiza- Proposition 2. Suppose that the optimal score network (z,,,, tion in Eq. (77) is available. Then for a given labeled sample (Zn. c) ~ g(Zo, c) and its noised version Dt qa, (x, (2)), the posterior mean E[za| [z,, c) is achievable at:
co(C) = E[xoola c)
(a+ + (2), c))
Va
Proof. As a first step, we obtain an analytic form of the optimal score network sg L, e) via the following lemma which is in fact extension of Proposition into the class-conditional context:
Lemma Consider the DSM optimization in Eq. (7)) Assume that a given noise-conditioned score network sø (x,,,,c) c) have enough capacity. Then for each timestep and given condition c, the optimality of the score network is achievable at:
sg (xt, L c) Byg, (w))
Vm, log qo, (xr mo)]-
(14)
Proof. The proof techniques are essentially the same as the ones used Proposition Since we assume the score network so s0,,,,,, c) to have enough capacity, the global optimal point for all timesteps and classes is achievable regardless of (w: LT Eq. 11). Therefore, we can focus on the DSM optimization w.r.t. specific timestep and a particular class obtain s5( (Fx c):
(at, c) arg min Eygsolleea,, (m, (ea) [|] (0 f, c) Va, log qa, (æ+ x=))
(15)
ASES
Note that we consider the optimization over function space se E S rather than the parameter space 0 è for simplicity. Rewriting the objective function in Eq. (15) gives:
Eg (c) (m, [m) se (zr, c) Væ. log ac. (x) xo)]
/ q(<0 c)q.. (xx (0))||s(( t, - Va. log 9a, (x; 1 dx; dæ
= -/ To. (x: c)go. (20
Tt. c) se 1r,, t, c) Va, log (o, (x. 2o) dæy dæy
((00.00)
16

---

Published as a conference paper at ICLR 2024
As in Proposition I employ the Euler- Euler-Lagrange equation for L(se, xx) to up with dition for the optimality:
Oc
f qa. c)qa, (20o æt. c) (s) tt), - Va, log
=0 +
lag ga, (x) xo)) dæg =
Ose
The RHS of the arrow can rewritten as:
/ qai (xt c) æı, c) (2, Va, log qex (x) Φoo} dæy = 0
t, c) /
qai gailzu (2o Iz. c) dxy- = / gai (xv Iee V log qa, (x, xp) dæy
(z t, c) = Beg (malm
a log Ja, (xt æo)..
This completes the proof of Lemma]
The remaining part is straightffrward. Plugging the optimal score formula (i.e., Eq. RHS of Eq. (13) yields:
(4) into the
(x: + 04 ())8)
Var
{et + (1 (xr) c) [v, log Qox (xte wo)]}
var
00
+ (1 - (ceal ())
v Va
at
En (mal [an]]
[
3t; cl
This completes the proof of Proposition 2 Note that when considering the case where e applies to all data, this also serves formal proof of Eq. (4). confirming that Tweedie's formula yields
posterior mean (Chung et al. 2022a).
Then by employing the same intuition in the unconditional setting, we develop a variant of mi- nority score tailored for the class conditional context:
((vo, C; 86) = Eq,,, (m.1o)))(() ((())
(16)
where sg now indicates the class-conditioned score model (t,,,,t, Notice that when geared with this conditional version, one can figure out whether given sample wo minority regarding specific class e, thus being capable to serve as an outlier metric in the class-conditional context. See Figure for visualization of the effectiveness of the extended minority score.
B.3 EXTENSION OF MINORITY GUIDANCE
As the unconditional case, we employ the classifier guidance (Dhariwal Nichol, 2021) for imposing the condition the uniqueness. A distinction here is that now have handle two conditioning variables, i.c., e and which involves careful consideration and coordination to en- sure effective guidance and control over the sampling process. Below describe in detail approach to attain this.
) lid. g( e) and conditional diffusion model s (a t, c) pretrained on that dataset. Similar to the unconditional case, one can compute minority score for each paired sample via Eq. (16) and get (e1,.... ...., [(N) where ( = 1(=x))) (s) s (1,..., N).
Consider labeled dataset (x
For categorizing minority score values, we take an intra-class approach to better capture the unique- ness of features associated with each individual class-condition. Specifically for each class C (available in the dataset), first aggregate the associated minority score values {1()) where
Se (i c(f) c). We then categorize them by applying T. distinct threshold levels where
17

---

Published as a conference paper at ICLR 2024
Low minority class
Middle minority class
High minority class
Water lower
Bald eagle
Joystick
BSP maker
Figure 8: Generated samples by the extended minority guidance over various minority classes. Three different minority classes are considered: = O (left column), i 12 (middle column), and ī = 24 (right column). The same ImageNet classes in Figure ? are considered for generation: water tower (top row), bald eagle (top-middle row). joystick (middle-bottom row), and espresso maker (bottom row). We share the same random seed for each row.
L is shared across all classes E d This yields the categorized minority scores w.r.t. condi- tion e: (((,)((())))) E {o..... I. 1} where i(c) = and ?(e) = indicate the minority classes w.r.t the most common and the rarest features regarding condition c, respec- tively. We then couple the ordinal minority scores with the associated paired samples to construct ( (()),,((()))). We employ this dataset for training a (class-conditioned) minority predictor Pe [), e) that classifies for perturbed input æ; given class-condition c. After training, we mix the given score model 8y (æ:, L, c) with the log-gradient of the classifier as in Eq. (5) to yield a modified score:
tripled dataset (ar)) (1) c(1) _ea)), ((),,((())),,
$o fx c, (z V... log De, F),
(17)
Using this blended score as drop-in replacement of (a f, c) Eq. 12) then implements con- ditional generation w.r.t. both e and i, e.g., according to pa (x x Po (o(2,|c)p 21, c)a For hyperparameters and in Eq. (17).
provide visualizations their respective influences in Figures [ and [7] see details therein. We empirically found that our extended minority guidance offers better (or comparable) performance existing class-conditions low-density samplers such as Sehwag et al. (0022). See Sections 4.2 and for explicit details.
C ADDITIONAL DETAILS ON EXPERIMENTAL SETUP
Categorization strategy. For the threshold levels to compart the raw minority scores (i.e., ones that are used to turn 166) into ((0) in Section 3.3 we observed that naive equally-spaced thresh- olds yield significant imbalance in the size of classes (e.g., extremely small numbers samples in highly unique classes), which would then yield negative impacts in the performance of the classifier especially against the small-sized classes. Hence, we resort to splitting the minority scores based on their quantiles. When L = 10 for instance, we categorize the minority scores such that / = 0)
'We may assign different numbers of thresholds for distinct classes, say T(c),, especially when given dataset contains highly imbalanced distrihutions of classes.
18

---

Published as a conference paper at ICLR 2024
becomes the samples with the top (bottom) 10% of the mniquely-featured samples. For the number of classes L, using ones that yield benign numbers of samples per each class (e.g., over 50 samples) chosen based on the size of given dataset usually offers good performances. See Section D.3 for demonstration. We also found that I. can serve as a control knob for balancing the faithfulness of the guidance with the controllability over the uniqueness of features; see Section D.3 for details.
Medical imaging dataset. The brain MRI benchmark used in our experiments is a subset of an IRB- approved volumetric dataset that consists of 236,288 2D images with 923 volumes (i.e., images from 923 distinct subjects). All volumes are in the shape of 256 x 256 x 256 cube and standard 3T T1- weighted images with Imm thickness. The raw (volumetric) dataset has two different data sources, where 482 (out of 923) volumes are from Alzheimer's Disease Neuroimaging Initiative (ADNI) dataset, comprising 271 subjects with probable dementia and 211 subjects with normal cognition. The remaining 441 volumes were collected from university hospital's voluntary health screening program, with a group of participants that have normal cognition. For the use of this volumetric data in our experiments, we chose 20 specific slice positions (per each volume) that prominently exhibit abnormal features in the dataset (e.g., brain atrophy). We extracted these selected 20 slices from all 652 volumes with normal features, while for the 271 abnormal volumes, we deliberately restricted our selection to only 30 volumes to ensure a low likelihood of encountering the abnormal features. The selected 13,640 slice images are normalized to fall within the range of (0, (per slice).
Pretrained models. The backbone diffusion model for the CelebA results is constructed using the architecture and the training setting used for ImageNet 64 x64 in Dhariwal & Nichol (2021 For the unconditional CIFAR-I0 10 model, we employ the checkpoint provided in Nichol & Dhariwal 2021 without fine-tuning or modification. The pretrained models for LSUN-Bedrooms and ImageNet were taken from Dhariwal & Nichol (202T) and used without modifications. For the brain MRI results, we train the backbone model by employing the same architecture and the setting used for LSUN- BUUN-Bedrooms in Dhariwal & Nichol (2021).
Classifiers for minority guidance. The classifiers used for minority guidance are based on the encoder architecture of U-Net used in Dhariwal & Nichol (2021). Except for LSUN-Bedroome Bedrooms, we employ all training samples for constructing the minority classifiers (e.g., N = 50000 for CIFAR- 10). On the other hand, only 10% of the training samples are used for LSUN-Bedroomt number of minority classes L, we take L = 100 for the three unconditional natural image datasets and L 25 for ImageNet and CIFAR-10-LL
For the
We use L = 50 for the brain MRI experiments.
We construct the minority classifier used on LSUN--edroomee Bedrooms by employing a 10% of the training samples provided in publicly available Kaggle repository [ Specifically for the LSUUN-edroome and ImageNet classifiers, we employ pretrained feature extractors to encourage efficient training as in (Kim et al. 2022). More precisely, we put (I2,, t) to a feature extractor, and extract the latent 2; of 2, from the Tast pooling layer of the extractor network. We then feed (z,, t) to a minority classifier and predict its minority class For the feature extractors, we employ the pretrained ImageNet classifiers provided in DDhrrwaal Nichol (202T) and freeze them during training (as in (Kim al.) 2022). The classifier architecture for L.SUN-eddromss is the same as the shallow U-Net used for ImageNet 256 256 in (Kim et al.] 2022). For ImageNet, we employ the shallow U-Net architecture used for CelebA in (Kim al., (022) with class-conditioning (to incorporate in Eq. (17h). The classifier architecture for the brain MRI experiments is the same as the one used in Wolleb et al. 2022). See Table for explicit details.
Baselines. The BigGAN model for the CIFAR- 10 experiments are trained with the setting provided in the author's official project page" For the CelebA model, we respect the architecture used in Choi ct al. [2020] [0] and follow the same training setup as the CIFAR-1O model. We employ StyleGAN
for LeUN-oedsooms using the checkpoint provided in Karras et al. 2019 To obtain the baseline
hhttps://githut http://itthhb..coom/peeaaea wtt.:////ppppaammmmpeeeededddddd ffusio
Tound that the performance of minority guidance is indeed affected by the number of samples employed
for constructing the minority classifier. See Section D.4 for details.
bedroom
https://gittbbb.cc/////////////
19

---

Published as a conference paper at ICLR 2024
CelebA
CIFAR-10
LSUUN
ImageNet
Brain MRI
Feature extractor
Model
Unused
Unused X
ADM (8,256,256.3)
ADM
Unused
Input shape
(B,64,64,3)
Output shape
X
(B,8,8,512)
(B,8,85512)
U-Net encoder architecture
Input shape
(B.64.64.3)
(B,32,32,3)
(B,8,85112)
(B,8,85512)
(B,256,256,)
Output shape
(B,100)
(B,100)
(B,100)
(B,25)
(B,50)
Diffusion steps
1000
4000
1000
1000
1000
Noise schedule
cosine
cosine
linear
cosine
linear
Channels
64
32
128
128
32
Depth
2
2
4
Channels multiple
1,2,3,4
1,2,2,4
1,1,2,2,4,4
Heads channels
64
64
64
64
64
Attention resolution
32,16,8
16,8
8
32,16,8
BigGAN up/down
X
Attention pooling Class-conditional
/
V
V
X
Training setup
Weight decay
0.05
0.05
0.05
0.2
0.05
Batch size Iterations
256
128
256
1024
256
20K
12K
60K
60K
5K
Learning rate
3e-4
3e-4
3e-4
6e-4
3e-4
Sampling
Minority class (i) Classifier scale (w)
(990, 99)
((85,99)
99
24
U(45,
4.0
8.0
3.5
2.5
1.5
NFE
250
250
250
250
250
Table Training and sampling configurations for our proposed sampler, minority guidance. "ADM" is the diffusion model framework proposed in Dhariwal & Nichol (2021). "cosine" refers to noise schedule proposed in Nichol & Dhariwal (2021) and "linear" indicates linearly-spaced noise sched- ule used in the original DDPM (Ho et al. 2020).
StyleGAN2-ADD model for our MRI experiments, we respect the settings provided in the official codebase and train the model by ourselves. The DDPM baseline and Sehwag et al. (2022) share the
same pretrained diffusion models as ours. Since there is no open codebase for Sehwag et al. (0022), implement their method by ourselves by following the descriptions provided in the original pa- per (Sehwag et al. (002).. More specifically. we employ the same pretrained diffusion model as ours, i... the ImageNet checkpoint provided in (Dhariwal & Nichol 202T). The classifier used in their sampler is imported from Dhariwal & Nichol (2021). and the real-fake discriminator is constructed via the same architecture and the training setting used [Shwwag et al. (2022). For the additional CelebA baseline that uses the classifier guidance targeting minority annotations, we construct pre- dictor that classifies four minority attributes: (i) "Bald"; ; (ii) "Eyeglasses"; (iii) "Mustache": (iv) "Wearing Hat". During inference time, produced samples with random combinations of the four attributes (e.g.. bald hair yet not wearing glasses) using classifier guidance. We use the same pretrained model DDPM and ours for the additional CelebA baseline.
Evaluation metrics. We employ implementation from PyOD Zhao et al. 2019 to compute Av- erage k-Nearest Neighbor (AvgkNN) and Local Outlier Factor (LOF) (Breunig al., 2000). We respect the conventional choices for the numbers of nearest neighbors for computing the measures:
1n "ttptarss/tthaaocooo/////aaaaaaa pytorch
20

---

Published as a conference paper at ICLR 2024
5 (AvgkNN) and 20 (LOF). As in (Sehwag et al. 2022). , the two metries are computed in the feature space of Re.Net-50. We compute Rarity Score (Han et al. 2022) with k = 5 using implementation provided in the official project page 14 To evaluate the three neighborhood measures for the brain MRI dataset whose modality is distinctive (from that of natural images), we employ a random em- bedding space which has been demonstrated data type different from that of natural images (Ulyanov et al. 2018) Nacem et al. 2020). More specifically, we employ randomly-initialize
being particularly instrumental when interpreting
feature space of ResNet-55. Additionally to the ran- dom embedding space, we further investigate the neighborhood measures on an adapted feature space which is constructed by fine-tuning pretrained DINO (Caron et al. (002T) on our brain MRI dataset. The fine-tuning is performed on ViT-B/16 architecture respecting the same training setting provided in the official codebase
1
yet with decreased learning rate by tenth.
The computations of clean Fréchet Inception Distance (cFID) (Parmar et al. 2022) are due to the official implementation 16 For computing spatial FID (Nash et al.) 2021), we modify the official pytorch FID (Heusel et al.
to use spatial features (i.e., the first 7 channels from the inter- mediate mi xed_6/ conv feature maps), rather than the standard pool_3 inception features. The results of Improved Precision & Recall (Kynkäänniemi et al. 2019) are obtained with it 5 based on the official codebase of (Han et al. 2022). For the brain MRI data, we additionally report the quality metrics evaluated in the DINO fine-tuned feature space (see the above paragraph for de- tails on the fine-tuning). In an effort to evaluate the closeness to our focused low-density data, we employ the most unique samples as baseline real data for computing the quality metries. More pre- cisely, we employ the 10K and 5K real samples yielding the highest AvgkNN values for CelebA and CIFAR-10, respectively. For the highest AvgkNNs are employed for baseline real data. We employ the most unique IK sam- ples for constructing the MRI baseline data. All quality metrics are computed with 50K generated samples.
2017 17
and ImageNet, the most unique 50K samples with
Computational complexity. The inference time of our minority guidance is line with that of classifier guidance by Dhariwal & Nichol (202T). which takes approximately 0.15 seconds per image with 250 NFE in our CIFAR-I0 experiments. On the other hand, the baseline ancestral sampling (with the same 250 NFE) spends 0.12 seconds per image on the same dataset, which is relatively cheap due to the absence of the classifier guidance in the sampling process. In addition to the inference time, our sampler requires some time for the construction of minority classifier, which totals around 23 minutes for the CIFAR-10 dataset. Specifically, 3 minutes are dedicated to the minority-class labeling process (which is described in Section 3.3) while 20 minutes are spent on the classifier training. All these results were obtained using a single A100 GPU.
Other details. For the distance measure in minority score (i.e., d(. .) in Eq. (7). we employ Learned Perceptual Image Patch Similarity (LPIPS) (Zhang et al.] 2018) for all of our experiments; see Section D.I for empirical evidence that supports this choice. For the perturbation timestep used for computing minority score (i.e.. t in Eq. (7), we employ t = 0.9T for the models pretrained with the cosine schedule (Nichol & Dhariwal, (202T) and t 0.6T for the ones built with the linear schedule (Ho et al.] 2020). where 7 indicates the total number of steps that a given backbone model trained with. For instance, we use = 900 on CelebA where the pretrained model is configured with T = 1000. We leave in Section D.6 for details on ablating 1.
For each dataset, the range of minority class for generation (i.e., /n Eq. 1 is chosen the one that achieves the best balance among fidelity, diversity, and low-densitiness of generated samples. We swept over {1.0, 1.5, 2.0, 2.5, 3.0, 3.5, 4.0, 6.0. 10.0) for the classifier scale w. We employ 250 timesteps to sample from the baseline DDPM and [Shwagg et al. (2022) for all the results. For minority guidance, the same 250 timesteps timesteps for speeding-up sampling (e.g.. when conducting ablation studies), since we found that it yields little degradation as reported in Dhariwal & Nichol (202T). See Table 2 for details on the sampling parameters.
mostly employed, yet occasionally use 100
21

---

Published as a conference paper at ICLR 2024
(a) L1
(b)
(c) LPIPS
Figure 9: Performance comparison of minority score with various distance metries. The most novel (top row) and the most common (bottom row) CelebA real training samples determined by minority scores based on (a) L1; (b) L2; and (c) LPIPS are exhibited.
Our implementation is based on PyTorch (Paszke et al., 2019). and all experiments are per- formed on twin A100 GPUs. Code is available at https: // /g/gtttttt github. com/ soobin-un um/ minority-guidance
D ABLATION STUDIES, ANALYSES, AND DISCUSSIONS
We continue from Sections 3 and 4 of the main paper and provide ablation studies and discussions on the proposed approach. We first make an investigation for choosing proper distance metric that can yield well- pell-performing minority score. We subsequently present additional evidence to reinforce the effectiveness of the minority score as a detector of minority instances. After that, we ablate the number of classes L and investigate its impact on the performance of minority guidance. We also explore the sensitiveness of our algorithm w.r.t. the number of available samples N. We then explore generation, which is naturally given by our minority score.
another strategy for the minority-focuss We further perform an ablation study on the perturbation timestep for minority score. Lastly, we conduct additional investigations on our in-house brain MRI dataset to highlight the practical significance of our work.
D.1 THE DISTANCE METRIC FOR MINORITY SCORE
Figure compares the performances of several ver- sions of minority score employing distinct distance metrics. We see that samples containing bright and highly-saturated visual aspects obtain high values from the LI and L2 versions. This implies that the minor- ity scores with such distances sensitive to less- semantic information like brightness and saturation. On the other hand, observe that the minority score
Unique- IK
Common- IK
L1
121.33
69.24
L.2
119.21
68.80
LPIPS
102.66
130.57
Table 3: Sensitiveness to brightness of the three versions of minority score.
22

---

Published as a conference paper at ICLR 2024
vs AvgkNN
vs Rarity Score
0.3
Density
Density
Density
0.04
0.2
0.03
0.1
2.5
5.0
7.5
10.0 12.5 15.0 17.5
20.0
1.0
1.2
1.4
1.6
1.5
20
40
60
60
100
120
AugkNN
LOF
Rarity Score
Figure 10: Correlations of minority score with existing outlier metrics. For the CelebA test samples, we compute minority score and the three outlier metrics, and display the correlations between mi- nority score and each outlier metric. A higher value of indicates a density of more unique samples in terms of minority score. For instance, = 9 denotes a density of instances that yield the top 10% values of minority score. "MS" denotes minority score, our proposed metric. "AvgkNN" refers to Average k-Nearest Neighbor, and "LOFF is Local Outlier Factor (Breunig et al. 2000).
based LPIPS does not exhibit such sensitiveness, yielding results that are disentangled with brightness and saturation of images. Moreover, it suc- ceeds in identifying some weird-looking abnormal samples latent in the dataset (see the samples in the first row of the right column) while the other versions fail to do so due to their distraction, e.g., to bright and highly-saturated instances. compares the brightness measured on images that are collected using the three versions of minority scores with distinct metrics: I.I, L,2, and LPIPS. The average brightness values of the most and least unique IK samples determined via each of the three versions are exhibited. Notice that the minority scores with LI and L2 distances sensitive to changes in such less semantic information compared to the one based on LPIPS. We therefore choose LPIPS the distance metric in computing minority score and use it for all of our experimental results.
D.2 EFFECTIVENESS OF MINORITY SCORE AS AN OUTLIER METRIC
We argued in the manuscript that minority score is effective for mining minority samples (by giv- ing them high values), and provided empirical evidence for the claim by visualizing samples with various levels of minority score (e.g.. Figure 2 The argument was further strengthened by our experiments that exhibit improved capability of our sampler (that actually focuses on high- minority-scores instances) for producing low-density samples (see Figure 6 for explicit details). To further validate the success of our metric, additionally provide empirical evidence of the effee- tiveness of our metric by conducting quantitative evaluations comparing minority score with other well-established outlier measures, such Average k-Nearest Neighbor (AvgkNN) and Local Outlier Factor (LOF).
Figure 10 illustrates correlations of minority score with existing well-known outlier metrics. Notice that minority score has positive correlations with other metrics, further demonstrating its ability to reliably capture the uniqueness and minority characteristics of samples.
D.3 THE NUMBER OF CLASSES L FOR MINORITY GUIDANCE
We first discuss how the number of classes L can be used for balancing between the faithfulness of the guidance and the controllability over the uniqueness of features. We then provide an empirical study that ablates the number of classes I, and explores its impact on the performance of minority guidance.
Consider a case where we use small L for constructing ordinal minority scores. This leads to large numbers of samples per class and therefore helps to build a faithful classifier that well captures class-wise representations. However, if we use too small L, then it induces samples containing wide variety of features being flocked into small categorical ranges, thus making difficult to produce them separately (i.e., losing the controllability). One extreme case that helps understand this is when = 1, where the sampler then becomes just the standard one (that uses Eq. (3)) where we don't t have any control over the uniqueness of features. On the other hand, employing large T. would
23

---

Figure la compares results obtained with a variety of minority classes. Observe that for all con- sidered L, our approach improves generative capability of the minority samples over the baseline DDPM sampler, which demonstrates its robustness w.r.f. the number of minority classes L. How- ever, we note that 100 offers better guidance for the minority features than 1000, yielding Published as a conference paper at ICLR 2024
DDPR (stancerdi
Curs (L-130)
w-b (DOPH slandard)
Clrru (L-10)
Curs (L-1200)
DCPP
Oure taill
Ours (30N)
Density
Density
Density
1. LOF
1.2
LOF
LOF
(a) Impact of L
(b) Impact of
(c) Impact of w in Eq.
18
Figure 11: Impact of design choices the density of Local Outlier Factor (LOF) (Breunig al. 2000). The influences of three parameters are exhibited: (a) the number of minority classes L; (b) the number of available samples N; (c) the mixing intensity w in the mixture score approach (i.c.,
Eq. 18 The results are obtained on CIFAR-10. The higher LOF, the less likely samples.
make the generation highly controllable the uniqueness of features. However, large results in vast numbers of small-sized classes, which makes the classifier hard to capture class- wise representations properly, thereby resulting in crude predictor that rarely gives meaningful guidance. Considering the other extreme case L N where the number of classes is the same as the number of samples, the classifier therein would never be able learn representations over classes well enough for making meaningful guidance for the uniqueness of features. Therefore, can say that important to set a proper L that well balances between the controllability and the faithfulness. Nonetheless, we empirically observed that our minority guidance is robust to the choice of L, and a properly selected (yet not that difficult to come up with) L yielding benign numbers of samples for each classes (e.g., containing over 50 samples per class) performs well regardless of L, i.e., effectively producing samples with desired features while maintaining realistic sample quality. Below provide demonstrations that validate the above descriptions.
more tilted LOF density toward minority regions (i.c., high LOF values). This validates the role of L as balancing knob for the faithfulness of the guidance and the controllability over the uniqueness.
D.4 THE NUMBER OF AVAILABLE SAMPLES N
Figure 11b and Table 4 provide empirical results that ablate the number of available samples N specifically on CIFAR-10. Ob- serve that even with limited numbers of samples that much smaller than the full training set, minority guidance still offers significant improvement in the minority-focused generation over the baseline DDPM sampler. In fact. this benefit is already demonstrated via our experiments on LSUN-Bedrooms presented in Section 4.2 where we only take a 10% of the total training set in obtaining great improvement; see Figures 27 and 20 therein. However, see degraded diversity compared to the full dataset case, which is reflected in a lower recall (Kynkäänniemi et al. value; see Table for explicit details.
Method
Recall
Ours (all)
0.6254
Ours (20%)
0.5646
Impact of N on sam- ple diversity. The results are obtained on CIFAR-10 using 10K generated samples for each method.
Table 4:
2019
D.5 GENERATION FROM ((xn) x g(20)
(N) iid. g(xo), a pretrained diffusion model s ( t), and minority score values associated with the dataset (1),...,
Recall our setting considered in Section 3.3 where have dataset 2 (1)
....., ((N) where (60 = 1(xb6) 88), i (1, (I,..., N). Since interest is make samples with
24

---

Published as a conference paper at ICLR 2024
MS
t=0.1T
MS
0.4
0.4
0.3
0.3
0.3
Density
Density
Density
0.2
0.2
0.1
0.1
0.0
255
5.0
7.5
10.0 12.5 15.0 AugkNN
17.5
20.0
0.0
2.5
5.0
7.5
10.0
15.0 AugkNN
12.5
17.5
20.0
0.0
2.5
5.0
7.5
10.0
12.5 15.0
17.5 20.0
AvgkNN
Figure 12: Comparison of minority scores with different choices of the perturbation timestep l. For the CelebA test samples, we compute Average k-Nearest Neighbor (AvgkNN) and minority scores with three different timesteps: and exhibit the correlations between AvgkNN and each version of minority score. T indicates the total number of steps that given backbone model is trained with (e-g..
0.17 (left): (ii 0.97 (middle); (ii) 0.997 (right),
1000 for CelebA). The perturbations are based on the cosine noise schedule (Nichol & Dhariwal 202T). A higher value of I indicates density of more unique samples based on the respective minority score. For instance, I denotes density of instances that yield the highest 10% values of minority score. "MS" refers to minority score, our proposed metric. "AvgkNN" refers to Average k-Nearest Neighbor.
high (66) (i.e., minority instances) more likely to be produced, one naturally think of sampling from a blended distribution ((xo) x (<x)))(Zz; sğ). However, simulating the generation process w.r.t. this mixed distribution is not that straightforward, since do not have access to the score functions its perturbed versions (Va, log fa, (==))== where 9a, (zx) == J ((2r))a. (zx (2g dzz-
To circumvent the issue, we take approximate approach for obtaining the perturbed mixtures. Initially, we take an approximation ÿa, (xx) ) of the perturbed distribution áa, (xx) as aa, (2x) x qo, (æ.) (xx,) where ((x.) denotes minority score w.r.t. noised instance æe, (describing the unique- ness of xt). This gives Va, log Lo, (ate) = (a l) + V., log (xxx). Since ((x.)) is intractable, we resort to considering approximated version ((x:) by computing empirical average via the use of the minority classifier pw (I(x)))
((x))= Σ To pppp = xx),
!- where {T)) denote the threshold levels used for categorizing minority score (see Section 3.3 for details). The score function of the perturbed mixture at timestep then reads:
V u. log 9a, (æx) ( t) + wV w. log i(x:))
(18)
where w is a parameter introduced for controlling the intensity of the mixing; see Figure TIc for demonstrations of its impact. We found that using the mixed score as drop in replacement of sg (Z., t) in the standard sampler (i.e., Eq. (3b) improves the capability of diffusion models to gen- crate low-density minority samples.
Figure ITe exhibits the ability of the mixed score approach in Eq. (18) to generate minority sam- ples concerning a variety of w. Notice that increasing w yields the LOF density shifting toward high-valued regions (i.e., low-density regions), implying that w plays similar role as i in minority guidance and therefore can be used for improving the generation capability for minority features. While this alternative sampler based on the mixed distribution ((x)) x ((zr)) sā) yields sig- nificant improvement over the standard DDPM sampler (i.e., = 0.0 case), minority guidance still offers better controllability over the uniqueness of features thanks to the collaborative twin knobs i and w. Hence, we adopt it as our main gear for tackling the preference of the diffusion models w.r.t. majority features.
D.6 THE PERTURBATION TIMESTEP FOR MINORITY SCORE
Figure exhibits performances of minority score with distinct choices of the perturbation timestep I. Notice that when using small (or too high) values of f, the corresponding minority score does
25

---

Published as a conference paper at ICLR 2024
Figure 13: Real samples in our focused brain MRI dataset across minority score values. Data instances that yield the smallest (left), moderate (middle), and the highest (right) minority scores exhibited herein.
Figure 14: Generated samples by minority guidance on the brain MRI dataset over various minority
classes. Three different classes are considered herein: (right). The classifier scale is fixed to w 2.5 for all three settings. We share the same random seed for all generations.
0 (left), = 35 (middle), and = 49
not well correlate with Average k-Nearest Neighbor (AvgkNN), meaning its lack of effectiveness in terms of outlier detection; see the next paragraph for theoretical intuition on this phenomenon. Conversely, a suitable choice of f (e.g.. t = 0.97' in the figure) yields a well-performing metric that shares a clear trend aligned with AvgkNN, We found that adopting similar perturbation strength in the linear schedule scenarios (e.g., t = 0.67) yields good performances as well. Below we provide an intuition why a properly-
t is important for the performance of minority score.
Rationale on the failures of small and too large t's. Employing weak perturbation (like t = 0.17) leaves little ambiguity in predicting clean samples based on perturbed ones, which enables Tweedie's denoising formula (i.e., Eq. (4)) to offer high-fieelity reconstructions both for majority and minority instances, thereby yielding marginal differences between the their reconstruction losses. On the other hand, when the perturbation is too strong (e.g., t = 7), it would destroy all information in given clean samples. This makes Tweedie's formula unable to reconstruct information even for majority samples, thus inducing little gap between reconstruction losses of majority and minority samples.
D.7 ADDITIONAL INVESTIGATIONS ON THE BRAIN MRI DATASET
Figure B visualizes real brain MRI images yielding various levels of minority score. Observe the trend that as minority score increases, instances more likely to contain features associated with lesions of the dataset such as cerebral atrophy, demonstrating the effectiveness of our metric even on this distinctive data modality. Moreover, we that such brain impairments often more severe
26

---

Published as a conference paper at ICLR 2024
Low minority class
Middle minority class
High minority class
ColebA
L.SUN CIFAR-10
Figure 15: Generated samples from minority guidance over various minority classes. Three different classes considered: i= 0 (left column), i = 50 (middle column), and i = 99 (right column). We consider three benchmarks: (i) CelebA (top row); (ii) CIFAR-10 (middle row); (ii) LSUN- Bedrooms (bottom row). We share the same random seed for each row.
Small classifier scale
Moderate classifier scale
High classifier scale
L.SUN CI CelebA
Figure 16: Generated samples by minority guidance over various classifier scales. Three different scales are considered: w 0.0 (left column), w 1.0 (middle column), and we = 5.0 (right column). The minority class is fixed to the most common one (i.e., i = 0). As in Figure 15) we consider three benchmarks: (i) CelebA (top row); (ii) CIFAR-10 (middle row); (iii) LSUN- Bedrooms (bottom row). We share the same random seed for each row.
in samples with high minority scores, which aligns well with the dataset constitution that rarely contains instances with such significant brain atrophy.
Figure 14 exhibits generated samples by minority guidance considering a range of minority class I. Observe that as increases, the samples tend to have impaired features that correspond to low- density attributes of the data. In the generated samples, we observe the similar correlation as we saw in Figure [3 between the severity of the lesion and minority score, i.e., producing severer impairments with higher minority classes.
We argue that this capability of controlling uniqueness of features particularly instrumental in medical imaging. In fact, there are numerous cases that demand medical images exhibiting highly unique diseases, such as significant brain atrophy (e.g., the right-side images in Figure [4] which is rarely seen in normal populations. On the other hand, there are instances where we need to generate images with lesions that are not exceedingly rare, like mild cerebral atrophy (c.g., the middle-side images in Figure 14). or even normal images (e.g., the left-side images in Figure [4] without any health issues. Our sampler's 's ability to control the level of uniqueness may empower us to handle such diverse requirements in medical image generation.
27

---

Published as a conference paper at ICLR 2024
D.8 CAPABILITY TO LEARN THE GROUND- TRUTH DATA DISTRIBUTION
Since our approach focuses on producing minority instances ly- ing in the tail of data distribution, one may concern that it could introduce the performance loss against instances within high-density regions. We address this concern herein by inves- tigating the capability our approach to reproduce all regions of the data manifold not limited to low-density regions. Specifically on CelebA, we evaluate the FID metrics of generated samples by our approach and compare the values with the performances due to the standard sampler (i.e., ancestral sampling). The base- line real data used for the evaluation herein is the entire CelebA dataset (not minority instances used in Table [ To encourage generation that covers wide range of data manifold,
Method
cFID
sFID
ADM
2.93
2.88
Ours
2.91
2.86
Table 5: The capability of producing the ground-truth data distribution.
condition our sampler on all ranges of minority classes uniformly at random with fixed scale w = 1.0. See Table 5 for explicit details. Notice the table that our sampler yields better performance than the baseline DDPM sampler, demonstrating its robustness against the potential performance loss in learning non-m con-minority repre- sentations.
D.9 INTERACTION WITH OTHER GUIDANCE
One significant benefit diffusion models lies in the controllable nature of their sampling process, which enables arbitrary condi- tioning unseen at the training time. Hence, it is important for guidance technique to preserve such controllability and to harmo- nize well with other forms of guidance signal. To further evaluate the practical significance of our proposed sampler in this context, we investigate the interactive aspect of our approach with other incorporate addi-
Scale (w)
Accuracy (%)
0.0
46.86
1.0
87.44
2.0
96.26
4.0
99.16
forms of guidance herein. Specifically, tional (class-conditional) classifier guidance on ImageNet-64 and explore the impact of the supplementary guidance. Table ( ex- hibits the accuracy outcomes of an ImageNet classifier applied generated samples across spectrum of additional classifier strengths. Notice that as the classifier scale increases, the generated samples increasingly mirror their intended dlass-oonitions, demonstrating the adaptable nature of our approach in accommodat- ing supplementary guidance.
Table 6: The impact of addi- tional classifier guidance.
E ADDITIONAL EXPERIMENTAL RESULTS
We continue from Section 4.2 of the main paper and provide some additional results. We first exhibit generated samples that visualize the impacts of I and w. We then provide the complete results on all five benchmarks where we compare ours with the baselines in thorough manner.
E. VISUALIZATION OF THE IMPACTS OF i AND w
Figure [5ehhibiss generated samples by minority guidance considering a variety of the (categorized) minority class Ī. Observe that increases, the samples tend to have more rare features that appear similar to the ones observable in the minority samples (e.g., in Figure 2 See the left plot in Figure for a quantitative analysis.
Figure 16 showcases generated samples illustrating the impact of the classifier scale (see the right plot in Figure| 3 for a quantitative analysis). We see that as w increases, the generated sam- ples are more likely to exhibit representative features of the associated minority class (i.e., ī 7= 0), such frontal-view attribute in CelebA. This observation aligns well with the quantitative result shown in Figure and with the findings presented in Dhariwal & Nichol (202T), which extensively investigates the impact of
28

---

Published as a conference paper at ICLR 2024
Small classifier scale
Moderate classifier scale
High classifier scale
Water tower
Bald eagle
Joystick
ESP maker
Figure 17: Generated samples by the class-condittonal minority guidance over various classifier scales. Three different scales are considered herein: w = 0.0 (left column), w = 1.0 (middle column), and w = 2.5 (right column). The minority class is fixed to the most unique one (i.e., = 24). The same ImageNet classes as in Figure 8 are considered for generation: water tower (top row), bald eagle (top-middl row), ), joystick We share the same random seed for each row.
row), and espresso maker (bottom row).
Figure 17 further demonstrates the influence of the classifier scale w in the extended minority guid- ance (i.c., Eq. (17). is evident that increasing leads to a more focused generation of unique features associated with the targeted class I 24, exhibiting a similar trend in the unconditional setting.
E.2 COMPARISON WITH THE BASELINES: COMPLETE RESULTS AND ADDITIONAL SAMPLES
Figure 24 exhibits generated samples by approach and their associated nearest neighboring real samples on CelebA. Notice in the figure that generated features due to our method exhibit a notable level of novelty distinct from those observed in real samples.
Figure 25 visualizes generated samples on CelebA, Observe that the proposed sampler is more likely generate novel-featured samples that exhibited in real minority data; Figures 2 and 9 for instances of the minority features of the dataset. Figure 18] gives quantitative validation to this observation. We see that minority guidance outperforms the other baselines in terms of generating low-density samples, as demonstrated by yielding more data instances with high values of outlier metries. The same trend observed in our CIFAR-10 10 results; see Figures 26 and 19 for details.
Figure 21 provides comparison of the performances of neighborhood density measures on the cons-conditional ImageNet 64 64. It is noteworthy that our extended minority guidance (to the class-conditional setting) outperforms the baselines in generating low-density unique samples for all three measures. Notably, our sampler demonstrates superior performance compared the tailored algorithm by (Schwag al. (2022), which is specifically designed for the class-condition and not applicable in other contexts like unlabeled-data settings. For completeness, we exhibit the neighborhood density results of -ed--oedo which are already reported in the manuscript; Figure 20 for details.
setting
Figures and 23 exhibit the density measures on the brain MRI dataset. Inspired by Ulyanov al. (2018) and Nacem et al. (2020) that showcase the power of random embedding spaces for admitting distinctive data modality, employ andomly-initialzzen neural networks (e.g., ResNet-50, VGG-
29

---

Published as a conference paper at ICLR 2024
16) for computing density measures for the results in Figure 22 On the other hand, the performance values Figure are are computed in feature space of DINO (Caron et al. [02220221 which fine-tuned on our brain MRI data (see Section C for details on the fine-tuning setting). Notice that minority guidance outperforms the baselines in generating unique samples for all cases, which corroborates observations made Figure (in Section (4.2).
Table 7 provides quality and diversity evaluations on all our considered benchmarks including CIFAR-10. We additionally provide the results on the brain MRI data evaluated based on fine- tuned feature space of DINO (Caron et al. 202T on our available brain MRI images. Observe that the same performance benefit of our sampler also applies to the scenarios that are not covered in Table 1 which further strengthen the robustness of our sampler. We found that the performance values StyleGAN2-AD4 (Karras et al. 2020) sometimes diverge on our brain MRI dataset, yield- ing poor performance values in some metrics (e.g.. Precision, cFID on the fine-tuned feature space). This diverging phenomenon is also observed in density distributions of the method, which are widely spreaded over the axes of neighborhood metric values (c.g., as seen in the middle plot in Figures 23 We conjecture that this comes from the inconsistent modalities between natural and medical images, which may hinder proper interpretations of generated MRI features in the ImageNet- -eeetrriid fea- ture space (e.g.. hence yielding low precision). Another possible reason is that the fine-tuned feature space (wherein the high-cFIID value is measured) may still suffer from the lack of knowledge on the brain MRI modality. While we employed all the available samples (i.e., 13,640 slices) for fine- tuning, may not be sufficient construct a well-structured feature space that can admit diverse aspects of the target data, which often requires huge amount of data to achieve.
Table exhibits the evaluation results employing variety of density metrics for collecting base- line real data. Observe that our sampler yields better (or comparable) performance values under various criteria limited AvgkNN, demonstrating the effectiveness our method in producing instances that are generally unique.
To facilitate more comprehensive qualitative comparison among the samplers, we provide an ex- tensive showcase of generated samples for all the considered datasets. See Figures 25 for details.
30

---

Published as a conference paper at ICLR 2024
Heal
DCPM (ctindant)
DOPM (ML)
Kcal BigUAN
DOPM (ntnnannt DOPM touns)
DOPM (ML)
DDPM (standand)
COPN (R)
DGMM (ours)
DoPM lours)
0.30
0.01
0.25
0.03
Density
0.20
Density
Density
0.15
0.02
0.01
0.05
0.00
6
12
14
1.0
1.1
1.2
1.3
1.5
0.00
20
40
50
60
70
80
AvgkNN
LDF
Rarity Score
Figure 18: Comparison of neighborhood density on CelebA. "Real" refers to our real-data baseline, the test samples of CelebA. "BigGAN" is our GAN-based generative baseline, BigGAN (Brock et al. 2019). "DDPM (standard)" indicates the diffusion- based generative baseline: ADM (Dhariwal & Nichool with the standard sampler (i.e., Eq. (3)) (Ho et al. 2020). "DDPM (ML)" refers to a classiier-ggiier DDPM sampler conditioning on Minority Labels. "DDPM (ours)" denotes DDPM with the proposed sampler, minority guidance. "AvgkNN" refers to Average k-Nearest Neighbor, and "LOF" is Local Outlier Factor (Breunig et al. 2000).
Real DigCAN
DDPM (standad)) DDPM (ours)
D00 Real
DOPM (standard)
200 Real
DOPK (standard) DDPM (ours)
DiGGAN
DOPM tours)
LOT BIpGAN
0.20
0.03
Density
0.15
Density
Density
0.02
0.10
0.02
0.05
0.00
10
12
14
16
10
20
1.00
1.65
1.10
1,15
1.20
1.25
0.00
20
30
40
50
60
70
80
AvgkNN KNN
LOF
Rarity Score
Figure 19: Comparison of neighborhood density on CIFAR-10. "Real" refers to our real-data base- line, the test samples of CIFAR-10. All the remaining settings are the same as those in Figure 18
StyleG4N
DDPM (ours)
StyleGAN
DOPM tours)
DDPM (ours)
0.20
0.020
0.15
0.015
Density
Density
Density
0.10
0.010
0.05
0.005
0.00
0.000
6
10
12
14
16
18
20
1.0
111
12 LOF
1.3
144
40
60
80
100
120
AvgkNN
Rarity Score
Figure 20: Comparison of neighborhood density on those in Figure 6
The results the same as
Sehwag et DDPM (ours)
Schwag et DORM (ours)
Real
Sehwog et al. DOPM touns)
DDPM (standard)
DDPM (standard)
DDPM (standard)
0.020
Density
0.15
Density
Density
0.035
0.010
0.05
0.005
10
12
14
18
20
22
8.05
1.00
1.05
1.10
1.15
1.20
1.25
0.000
30
45
60
75
90
105
120
AugkNNN
LOF
Rarity Score
Figure 21: Comparison of neighborhood density on the class-conditional ImageNet. "Real" refers to the validation samples of ImageNet. "Sehwag et al." refers to the low-density sampler proposed in Sehwag et al. (002). All the remaining settings are the same as those in Figure 18
31

---

Published as a conference paper at ICLR 2024
DDPM (standard)
Real
DDPM Istandard)
Real SQYIeGAN
(standord)
0.5
15
Density
Density
Density
10
0.2
0.2
0.0
6
8
8,95
1.08
1.12
0.16 814
0.10
0.20 0.22 0.24 0.26 Rarity Score
AugkNN
LOF
Figure 22: Comparison of neighborhood density the brain MRI dataset. The density measures
computed in randomly-initialized feature spaces (Ulyanov et al. 2018 Naeem et al., 2020). "Real" refers to the training dataset (i.e., all 13,640 slices). "DDPM (standard)" indicates ADM (Dhariwal & Nichool 202I) with ancestral sampling. "DDPM (ours)" denotes minority guidance. The higher values, the less likely samples for all three measures.
Real StyleGAR2-AB ADA
DDPN Intendent)
102 Real StyleGAN2 ADA
IDPM Intendird) CCPM lours)
200 Real
DDPN Istanderd)
DCPN jours)
1073
StyleGNG ADA
CDPN (ours]
0.30
12-
0.4
0.25
Density
0.20
Density
0.15
0.2
0.10
0.1
0.5
12
0.00
1.15
AugkNN
Ranity Score
Figure 23: Comparison of neighhorhood density on the brain MRT dataset. The density measures exhibited herein are evaluated in a fine-tuned DINO (Caron et al. 202T) feature space on our brain MRI dataset. All the remaining settings are the same as those in Figure 22
Figure 24: The novelty of generated samples by minority guidance. Generated samples by our approach (left) and their corresponding nearest neighbors in the training dataset of CelebA (right) are visualized.
32

---

Published as a conference paper at ICLR 2024
Method
cFID
sFID
Prec
Rec
Method
cFID
sFID
Prec
Rec
CelebA 64x64
ImageNet 64 64
ADM
75.05
16.73
0.97
0.23
ADM
17.85
4.90
0.79
0.52
ADM-ML
51.66
13.04
0.94
0.31
Schwag et al.
10.76
4.12
0.80
0.52
BigGAN
80.41
16.51
0.97
0.19
Ours
11.95
2.62
0.76
0.55
Sehwag et al.
27.97
10.17
0.82
0.42
Ours
26.98
8.23
0.89
0.34
CIFAR-10
ImageNet 256x256
IDDPM
31.58
10.61
0.93
0.35
ADM
17.41
12.85
0.87
0.35
BigGAN
34.08
14
0.94
0.19
Sehwag et al.
13.43
9.88
0.85
0.42
Ours
29.95
9.63
0.92
0.41
Ours
13.83
7.82
0.85
0.45
CIFAR-10-LT
Brain MRI 256
DDPM
75.71
44.26
0.95
0.23
ADM
21.71
14.30
0.82
0.37
Qin et al.
64.93
41.41
0.92
0.36
StyleGAN2a
22.69
23.07
0.53
0.28
Ours
63.34
42.53
0.95
0.24
Ours
19.73
13.36
0.84
0.47
LSUN Bedrooms 256 256
Brain MRI
256x256 (FT f-space)
ADM
62.83
7.49
0.89
0.16
ADM
20.64
0.87
0.77
LDM
63.61
7.19
0.90
0.13
StyleGAN2a Ours
315.19
0.69
0.61
StyleGAN
56.45
5.96
0.87
0.13
11.83
0.86
0.79
Ours
41.52
6.67
0.87
0.10
Table 7: Comparison of sample quality and diversity for generating minority samples. The results on all considered benchmarks are exhibited herein. "ADM-ML." refers to classifier-guided ances- tral sampling conditioning on Minority Labels. "StyleGAN2a" indicates StyleGAN--ADA (Karras) et al., 2020). "FT f-space" refers to the evaluations made an MRI-adapted feature space con- structed by fine-tuning the pretrained DINO (Caron al. 202 on our brain MRI benchmark. The best results are marked in bold.
33

---

Published as a conference paper at ICLR 2024
Method
cFID
sFID
Prec
Rec
Method
cFID
sFID
Prec
Rec
CelebA 64x64 (LOF)
LSUN Bedrooms 256 256 (MS)
ADM
32.50
10.21
0.94
0.30
ADM
28,20
11.74
0.54
0.33
ADM-ML
30.37
9.76
0.88
0.39
StyleGAN
24.19
8.43
0.57
0.33
BigGAN
35.05
10.16
0.95
0.29
Ours
6.92
5.53
0.70
0.36
Ours
25.41
8.90
0.80
0.43
CelebA 64 (RS)
ImageNet 64x (LOF)
ADM
60.68
12.75
0.98
0.22
ADM
11.55
4.05
0.83
0.52
ADM-ML
32,45
8.16
0.97
0.31
Sehwag et al.
8.28
3.45
0.83
0.52
BigGAN
67.07
13.12
0.98
0.18
Ours
8.96
2.42
0.80
0.55
Ours
20.68
7.71
0.93
0.32
CelebA 64x64 (MS)
ImageNet 64x (MS)
ADM
58.15
16.37
0.91
0.34
ADM
14.67
6.09
0.70
0.62
ADM-ML
30.32
10.99
0.87
0.43
Sehwag et al.
12.57
5.30
0.70
0.62
BigGAN
63.16
16.15
0.91
0.29
Ours
5.88
2.34
0.71
0.69
Ours
8.88
5.37
0.85
0.49
CIFAR-10 (LOF)
Brain MRI 256 (MS)
IDDPM
17.31
8.56
0.93
0.37
ADM
24.02
21.10
0.78
0.35
BigGAN
31.05
10.74
0.93
0.22
StyleGAN2a
26.36
19.82
0.47
0.28
Ours
17.05
8.43
0.91
0.42
Ours
14.50
16.99
0.84
0.40
CIFAR-10 (MS)
Brain MRI 256x256 (MS; FT f-space)
IDDPM
25.62
13.53
0.83
0.51
ADM
30.73
0.81
0.79
BigGAN
46.
15.93
0.82
0.31
StyleGAN2a
281.08
0.63
0.67
Ours
15.16
8.86
0.83
0.59
Ours
35.49
0.84
0.79
Table 8: Comparison of sample quality and diversity for generating minority samples. Various different criteria (other than AvgkNN which is covered in Tables and 7 are used for constructing real baseline minority data. "LOF" indicates the results derived with the real baseline data that yields the highest LOF values. Similarly, "RS" and "MS" are the evaluations based on the baseline data inducing the highest Rarity Score and minority score values, respectively. "ADM-ML" refers Minority Labels. "StyleGAN2a" indicates StyleGAN2-ADA (Karras et al., 2020). "FT f-space" refers to the evaluations made an MRI- adapted feature space constructed by fine-tuning the pretrained DINO (Caron et al. 202 brain MRI benchmark. The best results are marked in bold.
to classifier-guided ancestral sampler conditioning
34

---

Published as a conference paper at ICLR 2024
Figure 25: Sample comparison on CelebA. Generated samples by BigGAN Brock et al. 2019 (left), ADM (Dhariwal & Nichol, 202T) with ancestral sampling (middle), and minority guidance (right) are exhibited. We share the same random noise for the generations based on the DDPM-based samplers (i.e., middle and right).
35

---

Published as a conference paper at ICLR 2024
F
Figure 26: Sample comparison on CIFAR-10. 10. Generated samples by BigGAN Brock et al. 2019) (left), IDDPM (Nchol & Dhariwal 2021 with ancestral sampling (middle), the proposed sampler (right) are visualized herein. We share the same random noise for the generations based on the DDPM-based samplers (i.e., middle and right).
36

---

Published as a conference paper at ICLR 2024
lun
Figure 27: Additional images for comparison on LSUN-Bedrooms, Generated samples by Style- GAN (Karras al. 2019) (left), ADM (Dhariwal& & Nichol 2021) with ancestral sampling (middle), and minority guidance (right) are exhibited. We use the same random seed for the diffusion- samplers (i.e., middle and right).
37

---

Published as a conference paper at ICLR 2024
(a) Dhariwal & Nichol (2021
(b) Sehwag 2022
(c) Ours
Figure 28: Sample comparison on the class-conditional ImageNet. Generated samples from five classes are exhibited: (i) Water tower (top row); (ii) Bald eagle (top-middle row); (iii Joystick (middle row); (iv) Espresso maker (middle-bottom row); (v) Cloak (bottom row). For each row, share the same random seed across all three methods.
38

---

Published as a conference paper at ICLR 2024
Figure 29: Additional images for comparison on the brain MRI dataset. Generated samples by StyleGAN2-ADA (Karras et al. 2020) (left), ADM (Dhariwal& & Nichol, 2021) with ancestral sam- pling (middle). and minority guidance (right) are exhibited. We use the same random seeds for the diffusion- based samplers (i.e., middle and right).
39


Reviews:
[{"review_id": "fcJ2dRkcK5", "reviewer": "Reviewer_mz51", "summary": "The paper claims that the distribution that diffusion learns tends to lead to the underrepresentation of certain parts of latent space. They tried to show and provide guidance to produce the underrepresented classes.\nThey used CelebA 64×64, ImageNet 64x64, LSUN Bedrooms 256x256, and a proprietary Brain MR dataset to show the effectiveness of their approach.", "questions": "In Fig. 2, how the labels of small, moderate, and high minority scores are assigned in real data?\nIn Proposition 1, could you please clarify what you mean by \"Assume that a given noise-conditioned score network sθ(xt, t) have enough capacity?\" and how does it affect the proposed method?\nWhy did not you evaluate CIFAR-LT which is seemingly designed for such a task? (Cao, K., Wei, C., Gaidon, A., Arechiga, N. and Ma, T., 2019. Learning imbalanced datasets with label-distribution-aware margin loss. Advances in neural information processing systems, 3)", "limitations": "", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The introduction of metrics and the guidance to improve minority representation is interesting. The text is easy to follow and read, and the qualitative and quantitative results are intuitive. I enjoyed reading through the paper, despite some  questions raised through the arguments.", "weaknesses": "The paper interchangeably uses the general term of diffusion models and assumes that the Tweedy formula is a general form of any diffusion model.\n\nThe author cited the paper, Sehwag, V., Hazirbas, C., Gordo, A., Ozgenel, F. and Canton, C., 2022. Generating high fidelity data from low-density regions using diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 11492-11501).\nas a motivation \"The principle guarantees their generated samples to respect the (clean) data distribution, which naturally makes the sampler become majority-oriented, i.e., producing higher likelihood samples more frequently than lower likelihood ones (Sehwag et al., 2022),\"\nbut in the cited paper, they claimed that uniform sampling is the problem, and changing the sampling process could introduce novel, or as authors phrased samples from \"long-tail\" part of the distribution:\r\n\"We observe that uniform sampling from diffusion models predominantly samples from high-density regions of the data manifold. Therefore, we modify the sampling process to guide it towards low-density regions while simultaneously maintaining the fidelity of synthetic data.\"\n\nThe authors mentioned this baseline but only used it in one of the datasets, even though this work at least claimed that they tackled the problem of long-tailed distribution generation. In addition, in that dataset, the results are in 64x64, whereas Sehwag et al. mentioned that they produced 256x256 images after upscaling from 64x64.\n\nThis paper seems also doing similar approach, but has not been investgated.\nQin, Y., Zheng, H., Yao, J., Zhou, M. and Zhang, Y., 2023. Class-Balancing Diffusion Models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 18434-18443).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AcWDriV98X", "reviewer": "Reviewer_F8nE", "summary": "This work tackles the problem of minority data generation using diffusion models. First, they propose a metric that defines the uniqueness of a sample. Then, they propose a classifier guidance approach based on the proposed metric to generate minority samples. The proposed metric and guidance are based on Tweedie's formula for denoising. The method is evaluated on 4 datasets for the minority generation task.", "questions": "1. The method is evaluated quantitatively for the minority image generation task. Doesn't it lead to loss of performance for generating long-tail data?\n\n2. Were the minority samples only chosen based on the minority score? Did the authors perform any verification on whether they are all actually minorities? There are some samples shown for different minority scores, but I was wondering if there was a more systematic verification process or not.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The theoretical background is well explained and supports the claims.\n\n2. The paper is well-written and easy to follow.\n\n3. The idea of using Tweedie's denoising method for minority sample generation seems to be novel to the best of my knowledge.", "weaknesses": "1. Evaluation: The baselines are limited to a few outdated works (StyleGAN 2020, DDPM 2020). The only recent baseline is by Sehwag 2022 on ImageNet which outperforms the proposed method in two out of four metrics. All methods in Tab. 1 are evaluated for the minority generation task, which despite being old, outperform the proposed method in some metrics. More recent methods, such as LDM [a], which is not even cited, can generate minority samples to some extent.\n\n2. Literature Review: The literature review is limited. There are many related methods that are not cited, contrasted against, or compared to:\n\n[a] Rombach, Robin, et al. \"High-resolution image synthesis with latent diffusion models.\" CVPR 2022.\n\n[b] Huang, Gaofeng, and Amir Hossein Jafari. \"Enhanced balancing GAN: Minority-class image generation.\" Neural computing and applications 35.7 (2023): 5145-5154.\n\n[c] Samuel, Dvir, et al. \"Generating Images Of Rare Concepts Using Pretrained Diffusion Models. arXiv 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CGfS3XTo8E", "reviewer": "Reviewer_Xf5u", "summary": "In this work the authors proposed a guidance to diffusion models so it is encouraged to generate the minority data. In doing so, the author opposed the use of the distance between original data and the denoised version of the same data by a diffusion model as a minority score, inspired by Tweedie’s denosing formulae. Data with larger such score is used to train a auxiliary classifier, which is used to guide the generation with commonly used classifier-guidance approach. Substantial theoretical and empirical results are shown to demonstrate the superiority, especially quantitative, of the proposed method.", "questions": "See weakness", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Overall, this paper is okay to read. It is a neat combination of straightforward ideas that produces good results: identifying uniqueness with distance between clean and denoised data, marking the data points that are showing minority data, and training a classifier of minority data to guide the generation. \n\nParticularly, the problem of generating samples belonging to the minority part of the data domain is an interesting one with potential applications. Components in the proposed work are existing ideas (Tweedie’s formula inspired deonsing, training auxiliary classifier to guide the inference process), but the quantitative quality are consistent and should give readers some insights. Furthermore I’d like to praise the authors for detailed explanation of the method, theories behind and extra examples in appendix.\n\nHowever there are some issues (see weaknesses and questions below) in the manuscript. Overall my assessment of this manuscript is borderline, but if my concerns are addressed I’m happy to adjust accordingly.", "weaknesses": "Overall the technical contributions are somewhat limited, as ideas are from existing works. This may not be a issue on its own as long as the work presents an interesting approach in the whole, but a few missing details hurts, as follows:\n\nIt is unclear how the minority classifier guidance plays with other classifier guidance or classifier-free guidance. From an application perspective, it’s important that users can control the generation through guidance, and a mere generation of minority data, while still being useful for say data augmentation, may limit its value for the end-user if the user cannot control the generation effectively. However, now only in Appendix E some examples are shown for conditional generation and details/quantitative evaluates are omitted. This makes it hard to assess the broader impact of this work.\n\nAlso, it would be better to discuss the rationale behind minority score — the exact motivation of having a minority data generation remains not elaborated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Tutb4c9N7P", "reviewer": "Reviewer_xCB3", "summary": "This article primarily aims to address the issue of diffusion models' inability to generate data samples in low-density regions of the data manifold. Initially, they illustrate that Tweedie's denoising formula tends to generate the majority of samples, which lie in high-density regions of the manifold. Building upon this, they devise the minority score to quantify the uniqueness of the features of a given data instance. Drawing inspiration from classifier guidance, they incorporate minority guidance into the diffusion model to generate samples in low-density regions.", "questions": "Please refer to Weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is easy to read and generally well-written.\n2. The paper evaluates the proposed method on benchmark real datasets and demonstrates the effectiveness of their approach in demanding real-world scenarios, such as medical imaging.", "weaknesses": "1. Given the scarcity of real samples in low-density regions, how is it ensured that the proposed method doesn't suffer from overfitting? In other words, how can the authenticity and diversity of the generated samples in low-density regions be guaranteed?\n2. The authors should provide more experimental details. For instance, what is the size of the subset of real data used for validation? In the experimental section,what levels of minority score were chosen for generating the data?\n3. In Fig. 1(a), what perturbation timestep was used to generate the noise samples displayed in the middle, 0.9T or T? The choice of 't' is crucial in Section 3.1; the authors should consider showing specific images of $x_t$, $x_0$ and $\\hat x_0$ or provide theoretical evidence that  is indeed the reconstructed image.\n4. In the experimental section, the method proposed in this paper did not achieve satisfactory results in terms of both prec and rec. The authors should provide further explanations.\n5. The nomenclature in this paper should be more standardized. For instance, in Section 4.2, there is an error in the representation of $\\tilde l$.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)