PoGDiff: Product-of-Gaussians Diffusion Models for Imbalanced Text-to-Image Generation
Authors: Ziyan Wang
Sizhe Wei
Xiaoming Huo
Hao Wang
Venue: NeurIPS 2025 poster ()
Keywords: Diffusion Model
Probabilistic Methods
Primary area: probabilistic_methods
TL;DR: We propose a general fine-tuning approach to address the performance drops on the imbalanced text-to-image generation tasks.

Abstract: Diffusion models have made significant advancements in recent years. However, their performance often deteriorates when trained or fine-tuned on imbalanced datasets. This degradation is largely due to the disproportionate representation of majority and minority data in image-text pairs. In this paper, we propose a general fine-tuning approach, dubbed PoGDiff, to address this challenge. Rather than directly minimizing the KL divergence between the predicted and ground-truth distributions, PoGDiff replaces the ground-truth distribution with a Product of Gaussians (PoG), which is constructed by combining the original ground-truth targets with the predicted distribution conditioned on a neighboring text embedding. Experiments on real-world datasets demonstrate that our method effectively addresses the imbalance problem in diffusion models, improving both generation accuracy and quality.

PoGDiff: Product-of-Gausiian Diffusion Models for Imbalanced Text-to-Image Generation
Ziyan Wang Georgia Institute of Technology wzy@gatech. edu
Sizhe Wei Georgia Institute of Technology swei swei@gatech. edu
Xiaoming Huo Georgia Institute of Technology huo@gatech.ed
Hao Wang
Rutgers University
hw4880cs. gutgers.
Abstract
Diffusion models have made significant advancements in recent years. However, their performance often deteriorates when trained fine-tuned on imbalanced datasets. This degradation is largely due to the disproportionate representation majority and minority data in image-text pairs. In this paper, we propose a general fine-tuning approach, dubbed PoGDiff, to address this challenge. Rather than directly minimizing the KL divergence between the predicted and ground-truth distributions, PoGDiff replaces the ground-truth distribution with a Product of Gaussians (PoG), which constructed by combining the original ground- -tuth targets with the predicted distribution conditioned on neighboring text embed- ding. Experiments on real-world datasets demonstrate that our method effectively addresses the imbalance problem in diffusion models, improving both generation accuracy and quality.
Introduction
The development of diffusion models [1, 2] and their subsequent extensions [3-5] has significantly advanced the learning of complex probability distributions across various data types, including images [6-9], audio [10], and 3D biomedical imaging data [11-14], For these generative models, the amount of training data plays critical role in determining both the accuracy of probability estimation and the model's ability to generalize, which enables effective extrapolation within the probability
space.
Data diversity and abundance are key to improving the generative capabilities of large-scale models, enabling them to capture intricate details within a vast probability space [15-19]. However, many data-driven modeling tasks often rely small, imbalanced real-world datasets, leading to poor generation quality, particularly for minority groups. For example, when training and fine-tuning a diffusion model with imbalanced dataset of individuals, existing models often struggle to generate accurate images for those who appear less frequently the training data (Fig. 1). This challenge further compounded when accuracy is prioritized over simply high resolution. For example, generated images of individuals need to match the identity of at least one individual in the training set (Fig. 1). Addressing this gap crucial for deploying diffusion models in real-world applications where correctness is paramount, such as personalized content generation or medical imaging.
This limitation is true even for finetuning large diffusion models pretrained large-scale datasets like LAION-5B [21], e.g.. Stable Diffusion [7]. Imagine an imbalanced dataset consisting of employees in small company, senior employees might have more photos available, while new employees only
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Ground
Ground
PoG
Truth
Jesnette MacDonal
High Denuity
Ediward G. Rabmnsn
Deneity
Ground Truth
Stable Diftusion
CaDM
PoG (Ours)
Ground Truth
Stable Diffusion
CBDM
PoG-Uuf (Ours)
Ground Truth
Stable Diftution
CBDM
PoG-biff (Ours)
Willorrd Mamoee
Enosods
J.P. Morgan
Figure 1: PoGDiff for imbalanced text-to-image generation. Existing methods, e.g., Stable Diffusion [7] and CBDM [20], fall short for minority data (Low Density). In contrast, Our PoGDiff
successfully generates high-quality images even for minority data, outperforming all baselines.
have a very limited number of them. Since none of the employees appear the LAION-5B dataset, generating photos of them requires finetuning the Stable Diffusion model. Unfortunately, finetuming the model on such an imbalanced dataset might enable the model to generate accurate images for the majority group (i.e., senior employees), but it will perform poorly for the minority group (i.e., new employees).
To address this challenge, we propose general fine-tuning approach, dubbed PoGDiff. Rather than directly minimizing the KL divergence between the predicted and ground-truth distributions, PoGDiff replaces the ground-truth distribution with a Product of Gaussians (PoG), which is constructed by combining the original ground-truth targets with the predicted distribution conditioned on neighboring text embedding. Our contributions are as follows:
We identify the problem of imbalanced text-to-image generation (IT2I) and introduce first general diffusion model, dubbed Product-of-Gaussian Diffusion Models (PoGDiff), to address this problem.
Our thcoretical analysis shows that training of PoGDiff is equivalent to training a normal diffusion model while encouraging the model to generate the same image given similar text prompts (conditions).
We propose a new metric, "Generative Recall" (gRecall), which evaluates the generative diversity of model when generation accuracy is strictly enforced.
Our empirical results on real-world datasets demonstrate the effectiveness of our method, outperforming all state-of-the-ar baselines.
2
Related Work
Long-Tailed Recognition. Addressing the challenges posed by long-tailed data distributions has been a critical area of research in machine learning, for both classification and regression problems [22- 24). Traditional methods, such as re-sampling and re- eeweeggtting techniques, have been used to mitigate class imbalances by cither over-sampling minority classes or assigning higher weights to them during training [25-29]. Such algorithms fail measure the distance in continuous label space and fail to handle high-dimensional data (e.g., images, and text). Deep imbalanced regression methods [22, 30-32, 23] address this challenge by reweighting the data using the effective label density during representation learning. However, all methods above are designed for recognition tasks such classification and regression and therefore not applicable to our generation task.
Diffusion Models Related Long- Data. There are also works related to both diffusion models and long-tailed data [33]. They aim at improving generation robustness using feature engineering [34-36], feature augmentation [37], noisy label [38], improving fairness image generation [39-41], and improving classification accuracy using diffusion models [37]. More broadly,

---

there are also many works focusing on reweighting for general generative models [42-45]. However, these works have different goals and, therefore, are not applicable to our setting.
Most relevant to our work is the Class Balancing Diffusion Model (CBDM) [20], which uses a distribution adjustment regularizer that enhances tail-class generation based on the model's predictions for the head class, It improves the quality of long-tailed generation by assuming one-hot conditional labels (i.e., classification-bart setting where image generation is usually conditioned on free-form text prompts. As a result, when adapted the free-form setting, they often fail model the similarity among different text prompts, leading to suboptimal generation performance in minority data (as verified by empirical results in Sec. 4).
settings). However, this assumption does generalize the modern
3 Methods
3.1 Preliminaries
Diffusion models (DMs) [1] are probabilistic models that generate output image Xo from random noise vector Xy conditioned on text input c. DMs operate through two main processes: the forward diffusion process and the reverse denoising process. During the diffusion process, Gaussian noise is progressively added to data sample Xo over I steps. The forward process is defined as Markoy chain, where:
q (x, 11) = (xxx X /1 B,x1-1,
Here, 3r is the predefined diffusion rate at step t. By denoting = -1-3 and ā = II ag we can describe the entire diffusion process as:
' II 9 (x)),, q (xx(xx)
q (X1:7(X)) xn)
N (xx; Vaixo, (1 11-00 ā.)
The denoising process removes noise from the sample xy, eventually recovering Xo. denoising model (x(,,, 4, is trained to estimate the noise from x, and text-guided embedding ((c), where (b-)) is a pretrained text encoder. Formally:
po (x--1 X1, y) N (x of
The denoising process trained by maximizing the likelihood of the data under the model or, equivalently, by minimizing the variational lower bound on the negative log- likeeiidddood of the data, Ho al. [1] shows that this equivalent to minimizing the KL divergence between the predicted distribution 1x y) and the ground-truth distribution g(x;-1 [x,, Xo. y) at each time step I during the backward process. The training objective then becomes:
min DEL (q (x,-1 Xo. y) ||pe (x--1xx,.))
This can be simplified to (i.e., Eqn. (1) in Rombach et al. [7]):
LDM Ex,e~ [||e (x 4, y)
Latent diffusion models (LDMs) [7] are diffusion models that perform entire diffusion and denoising process in a dinensiinnni latent space. LDMs first learn encoder & and decoder D. which are then frozen during subsequent training of the diffusion models. The corresponding objective is then simplified to (i.e., Eqn. (2) in Rombach et al. [7]):
LLDS = e= [||c co(2, t.
In this paper, we use Stable Diffusion (SD) [7] as our backbone model. Since our method works for both the vanilla DMs and LDMs, for clarity, use the notation instead of z, the encoder & and decoder D are fixed during fine-tuning.
3.2 Peaduet-of-Gaussiann Difffusion Models (PoGDiff)
3.2.1 Main Idea
Method Overview. Given an image dataset D = (x((), c(0 where (+)) is the text description for
image (8), fixed CLIP encoder to produce e (O's corresponding text embedding y o(c).
3

---

Typical diffusion models minimize the KL divergence between the predicted distribution P
t,), and the ground-truth distribution q(x,-1 xx,, Xo. y) ' I) at each time step t during the backward denoising process. Here, Ay and X represent the precision. In contrast, our PoGDiff replaces the ground-truth target with Product of Gaussians (PoG) [46] and instead minimizes the following KL divergence (for each t)
1 X,
ChaDii DKL (4 (Xx-1|X4, Xo: y) po (x (') |po (xt-1 x),
where represents the product of two Gaus- sian distributions, y' is a selected neighbor- ing embedding from other samples in the training dataset (more details below), and po(X--1|xe: y') denotes the predicted distribu- tion when using y' as the input text embedding.
X-7
X₁
sampling
As shown in Fig. intuitively, PoGDiff's de- noising model co(X,, y) (or Po v)) optimized towards two target distributions, equivalently increasing the weights for minority instances (more details below). This enhances the text-lo-image mapping by leveraging the statistical strength of neighboring data points, thereby improving and quality of the generated images, especially for minority images.
Figure 2: Overview of our PoGDiff. During fine- tuning, PoGDiff collects & neighbors of the current text embedding and samples one y' from them based on Eqn. (8). Both y and y' will then em- ployed to denoise the current image x, to X
Intuition behind the Product of Gaussians (PoG). During fine-tuning, typical diffusion models "lock" the text conditional embedding y = (i(c) to the corresponding image x. Consequently, if the dataset follows a long-tailed distribution, the fine-tuned or post-trained diffusion model becomes heavily biased toward the majority data, performing poorly minority data. Fig. 3 demonstrates our intuition. When training using a text-image pair (y,x), our PoGDiff "borrows" information from neighboring text conditional embedding y', thereby effectively increasing the data density in the minority region and leading to smoother (less imbalanced) effective density, as shown in Fig. (right). However, since the text embedding is fixed during fine-tuning (i.e., is frozen), directly smoothing the text embedding space is not feasible. Instead, we rely on the properties of PoG.
By definition, given two Gaussian distributions, N(,1, 'I and N "I), their product is still a Gaussian distribution:
X11/1 + XzH22 > >> (A1 A ) A N (14P0::>pott)
N N )T') ((2,, =N
A1 D2
which can treated as a "composition" of two individual Gaussians, incorporating information from both. This intuition is key to developing our PoGDiff objective function.
3.2.2 Theoretical Analysis and Algorithmic Design
Based on Eqn. (I), we then derive concrete objective function following Proposition 3.1 below. Proposition 3.1. Assume by >roG A de + X we have our loss function
A (X)o(x).)) - -Mroolll + +C
CRUGDI C
E,g
(3)
Here, C is constant, and prea denotes the mean of the PoG, with the expression defined in Eqn. (2). Then, through derivations based on Gaussian properties, we obtain
PRGDDI S E.g A(2x) [1co(X(,)) - =11² A(Ay:) )leo((x,,)) -x) +
(4)
20, (+.) where the function A(A)= A X(1 11-aa) A(1-a,)
a2
The proof is available in Appendix Eqn. (4) in Proposition 3.1 provides a upper bound for the KI. divergence (Eqn. (1)) aim to minimize.

---

Diffusion Models In diffusion model literature [1, 7], one typically sets A(A,) = 1 to eliminate the dependency on the time step t, and thus Eqn. (4) can be written as';
C -xx- [1xo(x)))) - ell + Ay
simple
||co(x),)) co cx
(5)
For convenience, we rewrite A
Note that this weight still depends on the time step There-
of fore, to be consistent with the literature [1, 7]. we hypothetically define oy (i(x.,)). to cancel the term oA. thereby effectively removing the time step dependency; here ei [(x, y), ,),C (x', y0) de- notes the similarity between the two image-text pairs. By shortening the notation 1 [(x,)) y), (x, y') vb can further rewrite the objective function for PoGDiff as:
ePoGDiff cxx-,
[xea =ll² + sa (xxx))))) (6)
simple
3.2.3 Computing the Similarity
Next, we discuss the choice of vb in Eqn. (6). Given an image-text dataset D, the similarities between each image-text pair need to be considered in two parts:
PoGDiff (Ours)
v Vimg-sim (x, x') dimn--rt-den (y),
X
where Viimg-sim (x, x') is the similarity between images and x', and tlin--xt-don (y) is the prob- ability density of the text embedding y (more details below).
Image Similarity Ving-sim- For all D, we apply a pre- mre-trained image encoder to obtain the latent representations z. We then calculate the cosine similarities between each and select the nearest neighbors with the highest similarity values for all samples in the dataset D, denoted و where sj represents the cosine sim- ilarity scores between and other images in D sorted in descending order. These values are then normalized to produce the weights for each neighbor:
Effectiive Demity
:
-Origiaal Density
New Density (Ours)
Figure 3: Comparing denoising networks of typi- cal diffusion models [1, 7] and our PoGDiff. Left: In conditional text-to-image diffusion models, data point (i.e., x) is mainly affected by its text embedding (besides random latent codes). Right: In PoGDiff, neighbors participate modulate the final effective density. Here, y denotes the text prompts, which are the embeddings of the text descriptions of the images; x denotes the associ- ated images. The tightly packed circles the top indicate higher density, while the sparse circles indicate lower density.
8j W Σ,
(8)
For each data pair (x,y). y). we then randomly sam- ple a neighboring pair (x', y') from categori- cal distribution Cat([w;)f=)) ("Cat" is short for "Categorical"), i.e., with wy serving as the proba- bility weight, and compute their image similarity as:
V/img-sim (x, x') max (0, t1 (X,00)11===
where E {s/}/=1 denotes the cosine similarity sampled according to the weights {wy 1 in Eqn. (8), [-] denotes the indicator function, and I(.) retrieves the class/identity of the cur- rent input image; [I(x) I(x')] = 0
and x' are two photos of the same person (e.g., Albert Einstein), and [I(x) + Ixx') = 1 if x and x' are photos of two different persons (e.g., x is Einstein and x' Reagan). More details on I(.) can be found in Appendix K.7. d1, d2 are hyperparameters that control the scale of the similarities. For example, if the cosine similarity (s) between and x' is 0.4, and 0; = 02 1: if x and x' are of the same person, the image similarity will be 0.41. whereas if and z' are not of the same person, the image similarity will be 0.42, which is smaller. The intuition is
5

---

Algorithm Training Algorithm for PoGDiff
1: Inputs: A dataset D = (x!1) )
Ji=1
2: repeat
3: (Xo, c) ~ D
4: y == o(c)
5:
t~ Uniform(1,....
6:
(~,T)
7:
Sample y' and v from Eqn. (12)
8:
Calculate x Vā + vI açe
9:
Take gradient descent step
10:
Vo |le - eo (x,, y)llă - y') (x),,
11:
until converged
to compute the image similarity according to both the image content similarity, i.e., s, and identity similarity, ie., I(x) and I(x').
Winv--xt-den- Inspired by LDS in DIR [22] and the theoretical analysis in VIR [23], re-weighting the label distribution of an imbalanced dataset can increase the optimization scale for minority classes and reduce the emphasis on majority classes, resulting in better performance under imbalanced conditions. However, both DIR and VIR partition the label space into bins, treating it as a classification problem. This is not applicable to our setting because in text-to-image generation, the "label" actually text embeddings. Instead, train variational autoencoder (VAE) on this dataset and then approximate its likelihood p(y) through its evidence lower bound, or ELBO:
Inverse Text Densities
ply) jogp(r) HLBOvall))
(10)
The evidence for minority data will be lower than for majority classes. This then motivates inverse text densities defined as follows:
den--xt-den (y) A _ -LBBOws()))
(11)
where 0g is a hyperparameter that controls the scale of the inverse text densities. By combin- ing Eqn. (9) and Eqn. (11) to Eqn. (7), we can then compute &b as follows 2:
e = max (o.
ELBOvve(y),
(12)
3.2.4 Final Objective Function
By collecting all the components discussed above, we arrive at our final training objective:
final
CPoGDiff Bxx-
[leo((e,,)) -el2 + ta (xea ca cx x())]
(13)
where s is defined in Eqn. (12). Alg. summarizes our algorithm.
4 Experiments
4.1 Experimental Setup
Datasets. To demonstrate the effectiveness of PoGDiff in terms of both accuracy and quality, evaluate our method on four widely used imbalanced datasets, i.e., AgeDB [47], DigiFace [48], VGGFace2 [49] and CIFAR-100-LTT [50].
AgeDB-IT2I-L & AgeDB-IT2I-S: (geDB-IT2I-L(arge) is constructed from the AgeDB dataset [47]. It consists of 976 images across 223 identities, with each majority class containing 30 images and
Forr clarification, our A(Ax) ) equivalent in [I], with difference that our paper, refers to precision of the Gaussian distribution.
For simplicity, set 01, 02, og without tuning them

---

Table I: Performance based on FID score.
Table 2: Performance based on DINO Score.
Darasets IT20
AgeDD
Digiface
VGGGace (CIFAR
Datascts Caaascts<--T7 (-T12)
Ag
Digifarc VGGFFaee CIFAR-100
Size
|
Small
Large
Large
Large
Large
Size
Small
Large
Large
Large
Large
Matri
FID U
DINO
Shoc
| All
Few
All
Few
All
Few
AAI
Few
AAll
Few
Shot
All
Fow
All
Few]
All
Few
All
Few
All
Few
CRDM
VANTLA
14.88 13.72
14.13
7,67 11.67
7.38 11.13
6.96 12:72
1.18
7.59
12.08
11.91
7.19
11.94
11.46
VANILLA
0.37 0.25
0.42 0.36
0.41
0.48
T2H
POGiiie (QuRS)
14.85
13.66
6,83
7.61
11.44
7.14 12:22
1.34
10.97
12.02
7.30
11.39
CBDM
0.51 0.09 0.41 0.26 0.34 0.16
0.39 0.26
0.42
0.16 0.22
0.52
0.28
POGDue (Ouns) 0.77 0.73 1.66 0.52 0.64 0.49
0.69 0.55
0.73 1161
Table 3: Performance based on human evalu- ation. The evaluation is a binary decision: the image is either judged as representing the same individual (score 1.0) or not (score 0.0).
Table 4: Performance on A-TDR based
on GPT-4o evaluation. The scores are from 0 to 10, with higher scores indicating the individual resembles the well-known person.
Datasets (-I121)
AgeDB
VGGFace CIFAR- -000
Datasets (- (-I121)
AgeDB
VGGFace CIFAR- 100
Small
Large
Large
Large
Size
Small
Large
Large
Large
Metric
Human Score
Metric
Human Score
Shot
Few|
All
Few
All
Few
All
Few
Sbot
All
Few|
All
Few
Few
All
Few
VANILLA
00.50 0.60 0.20 0.62
0.30
VANILLA CBDM
5.20 3.20
4,90 3.60
4.50 2.90 6.00
3.20
CBDM
0.50 0.00
0.56 0.12 0.54 0.10 0.63
0.24
4.50 1.10
3.10
1.70 2.80 1.30 3.40
2.00
T2H
0.50 0.00
0.60 0.20
0.62 0.16
0.72
0.30
T2H
5.50 3.10
4.70
3.90
4.60
3.10
6.20
3.60
PoGDIFF (OURS)
0.84 0.68 0.78 0.64
0.72
PoGDDFF (OURS)
9.10 8.40
8.50 8.00
8.20
8.40
8.00
each minority class containing 2 images. We also Age (maaa) which contains 32
images across 2 identities, where each majority class consists of 30 images and each minority class consists of 2 images. Additionally, we construct (1ge)8--121-M((ddimm) and more details can be found in Appendix C.
DigiFace- II2I: DigiFace-IT2I is derived from the DigiFace dataset [48]. It contains 985 images across 179 identities, where each majority class consists of 30 images and each minority class consists of 2 images. We use process similar to AgeDB-IT21 to collect text-image pairs, forming this DigiFace--T21 dataset.
IGGFace-IT21: CGGFacc-IT21 is subset of VGGFace2 [49]. It contains 1933 images across 193 identities, where each majority class consists of 49 images and each minority class consists of 2 images.
100-IT2 is constructed from CIFAR-100-LT dataset [50]. We set the imbalance ratio to 250, where the largest class contains 500 images and the smallest class contains only 2 images. The dataset consists of 9502 images in total. During fine-tuning, to mitigate the dominance of majority classes, we downsample these classes in each epoch. Specifically, the most frequent class is reduced to 49 images per epoch out of its original 500 images.
CIFAR-100-IT21
Baselines. We employ Stable Diff'usion v1.5 [7] as the backbone diffusion model. As this is the first work to explore imbalanced text-to-image (IT2I) diffusion models with natural text prompts, we adapt the current state-of-the-an methods designed for long-tailed T2I diffusion models with one-hot text prompts to serve as baselines. The baselines are described below:
Vanilla: The Vanilla model simply fine-tunes a Stable Diffusion model without additional modifications.
CBDM: CBDM [20] is a Class Balancing Diffusion Model that incorporates a distribution adjust- ment regularizer during training. During fine-tuning, we sample an additional text embedding y' from the entire fine-tuning dataset and apply the CBDM objective function. All hyperparameters are kept the same as in the original paper, with further details available in Qin et al. [20].
T2H: T2H [37] is a Long- Tailed Diffusion Model with Oriented Calibration. It is feature augmentation method, but not directly applicable our setting. Specifically, T2H relies on the class frequency, which is not available in our experiments. We adapted this method to our settings by using the density for each text prompt embedding to serve as the class frequency in T2H [37].
Evaluation Protocols and Metrics. We use three types of evaluation metrics: generation diversity, generation accuracy, and generation quality.
For generation diversity, we propose a new metric, generative recall (gRecall), which evaluates the generative diversity of model when generation accuracy is strictly enforced.
7

---

SDv1.5
CBDM
T2H
GT
PoGDiff (Ours)
Few-Shot Many-Shot Samples)(110
Inaccurate Gender
Inaccurate Individual
( Samples)
Figure 4: Example generated images from different methods. Our PoGDiff outperforms the baselines in both generation accuracy and quality. Regarding the ground truth (GT), the training set for the minority class (left two columns) contains only 2 images per individual while the majority class has more than 8 samples per individual.
gRecall the Context of Image Generation: "Correct Image" and "Covered Image". For each generated image, classify it as a "correct image" if its distance to at least ground-truth (GT) image is below predefined threshold. For instance, suppose we have two training-se set images for Einstein, denoted I1 and 22. A generated image Tg is "correct image" if the cosine similarity between Tg and either æi or 22 is above some threshold (e.g., we set to 0.7 here). For example, if the cosine similarity Vig and is larger than 0.7, say that Vig "correct image", and that æi is a "covered image". Intuitively, a training-set image (e.g., v1) ( is covered a diffusion model is capable of generating similar image.
Cosine Similarity between Images. Note that in practice, compute the cosine similarity between DINO embeddings images rather than raw pixels.
Formal Definition for gRecall. Formally, for each model, compute the gRecall per ID as follows:
# of unique covered images for ID
gRecall
# of images for ID in the training
where c is the number of IDs in training set.
Analysis. This metric evaluates the generational diversity of a model. For example, if the training dataset contains two distinct images of Einstein, T1 and æ2, and model generates only images resembling I1, the gRecall, in this case, would be 0.5. While the model may achieve high accuracy in terms of facial identity (TTalee 3 and Table 4), it falls short in diversity because fails to generate images resembling T2. In contrast, a model generates images that cover both 22 and I2, the gRecall for this ID will be 1; for instance, the model generates 10 images for Einstein, where of them resemble 1, and 4 of them resemble 22, the gRecall would be 1, indicating high diversity and coverage.
To assess generation accuracy, for each minority class, we did sample 100 images rather than 10. For the human evaluation metric, we then sampled 10 images out of these 100 due to the high cost of human evaluation. We then gather feedback from both the GPT-4o model [51] and human evaluators to score the accuracy of identity recognition. Additionally, employ pre-trained DINO model [52] for calculating the DINO score for image similarities. More details about the evaluation process, including prompts used, are in Appendix E.
For general text-to-image generation quality, we report the widely used Fréchet Inception Distance (FID) score [53]. For all the facial datasets,
pre-trained face recognition model as the feature

---

extractor rather than traditional Inception-v3 [54]; since our goal is to evaluate the ability to recognize humans, we need to capture facial features rather than general features. For CIFAR-100- we employ the original feature extractor (i.e., Inception-v3 [54]). More details are in Appendix E.
4.2 Results
Generation Quality and Accuracy. We report the performance of different methods in terms of FID score, human evaluation score, GPT-4o score, and DINO score in Table 1, Table 2, Table 3 and Table 4, respectively'. Full results are included in Appendix G. Across all tables, we observe that our PoGDiff consistently outperforms all baselines. More results and discussions are in Appendix K.5. Notably, PoGDiff demonstrates significant improvements, especially in few-shot scenarios (i.e.. for minority classes). It is also worth noting that CBDM [20] performs extremely poorly on AgeDB-IT21- S dataset. This is because their method samples text conditions from the entire space, which may work in one-hot class settings, but in our context (natural text conditions), this sampling technique misguides the model during training.
Fig. 4 shows randomly sampled generated images on low-density classes (Column 1&:2) and high- density class (Column 3) in AgeDB-IT2I-L. for each method. Note that the ground-truth (GT) images the training images. For the high-density class, select 8 out of 24 total images in the training set to report in this figure. Across each column, the individual names are Albert Einstein, JW Marriott, and Luise Rainer, respectively. PoGDiff achieves significantly better accuracy and quality for both head and tail classes (see Appendix K for more comparisons and analysis). Specifically, both SDv1.5 and T2H fail to generate accurate individuals (green boxes), and CBDM even struggles generate images with correct genders (blue boxes). By contrast, our PoGDiff successfully generates accurate individuals, even when trained on dataset containing only two images.
Generation Diversity. Table and Fig. demonstrate our PoGDiff's promising generation diversity:
PoGDiff's Superior FID Performance. Table 1 shows that PoGDiff achieves a lower FID score, particularly in few-shot regions (i.e., minorities). This suggests that the images generated by our method capture broader range of variations present in the training dataset, such as backgrounds or facial angles.
PoGDifl's Qualitative Results. As shown in Fig. 4:
For Einstein (Column for each method), the training dataset (the GT section on the right) includes two face angles and two hairstyles. Our generated results successfully cover these attributes.
For JW Marriott (Column 2 for each method), the training dataset has only one face angle. Correspondingly our results focus on generating subtle variations in facial expressions with only one angle, as expected.
For the majority group (Column for each method), our PoGDiff's generated images cover wider range of diversity while maintaining ID consistency.
Results on gRecall. Table shows the gRecall for different methods on four datasets, Age----Tll-smal,, AgeDB--TTI-large, VGGFace-IT21
Table 5: Performance based on gRecall score. See the
detailed definition of gRecall in Sec. 4.1.
Datasets (-IT21)
AgeDB
VGGFace
CIFAR-I0K 100
and CIFAR- SETAR-100-ITY 100-IT21. These results show that our PoGDiff achieves much higher gRecall compared to all base- lines, demonstrating
Size
Small
Large
Large
Large
Metric
gRecall Score t
Shot
All
Few
All
Few
All
Few
All
Few
impressive di- versity and coverage of different at- tributes of the same individual in the training set (see Appendix I for more discussion and examples on gRecall).
VANILLA
0.017 0.000
0.196
0.200
0.133 0.167 0.200 0.160
CBDM T2H
0.267 0.000
0.138
0.100
0.120 0.100
0.086 0.067
0.017
0.000
0.196
0.200
0.133
0.167
0.200
0.160
POGDIFF (OURS)
0,800 1.000
0.435
0.540
0.400
0.533
0.433 0.567
*CLIP score is not applicable here. Our text prompts are predominantly human names, while CLIP primarily trained on common objects, not human names; therefore, CLIP score cannot measure the matching between images and human names.

---

Conclusions
In this paper, we propose general fine-tuning approach called PoGDiff to address the performance drop that occurs when fine-tuning imbalanced datasets. Instead of directly minimizing the KL divergence between the predicted and ground-truth distributions, PoGDiff replaces the ground-truth distribution with Product of Gaussians (PoG), constructed by combining the original ground-truth targets with the predicted distribution conditioned on neighboring text embedding. Looking ahead, an interesting avenue for future research would be to explore more innovative techniques for re- weighting minority classes (as limitations discussed Sec. H), particularly within the constraints of (1) tailed generation settings, as opposed to recognition tasks, and (2) natural text prompts rather than one-hot class labels. Exploring PoGDiff for other modalities (c.g., videos and time series) is also interesting future work.
Acknowledgement
We thank all reviewers, AC, and SAC for their valuable comments. ZW and XH are partially sponsored by a subcontract of NSF grant 2229876, the A. Russell Chandler III Professorship Georgia Institute of Technology, and NIH-sponsored Georgia Clinical & Translational Science Alliance. HW is supported by Amazon Faculty Research Award, Microsoft AI & Society Fellowship, NSF CAREER Award IIS 2340125, NIH grant RO1CA297832, and NSF grant IIS-2127918.
References
[1] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
[2] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13455 2020.
[3] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:20100025502 2020.
[4] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International conference on machine learning, pages 8162-8171. PMLR, 2021.
[5] Ding Huang, Jian Huang, Ting Li, and Guohao Shen. Conditional stochastic interpolation for generative learning. arXiv preprint arXiv:2312.05559, 2023.
[6] Jonathan Ho, Chitwan Saharia, William Chan, David Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. Journal of Machine Learning Research, 23(47):1 2022.
[7] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[8] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, al. Photorealistic text-to-image diffusion models with deep language understanding, Advances in neural information processing systems, 35:36479-36494, 2022.
[9] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv:2277.12598. 2022.
arXiv preprint
[10] Zhifeng Kong, Wei Ping. Jiaji Huang. Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. arXiv preprint arXiv:2009 09700, 2020.
[11] Shitong Luo and Wei Hu. Diffusion probabilistic models for 3d point cloud generation. In Proceedings of the IEFF/CVE conference on computer vision and pattern recognition, pages 2837-2845 1021.
10

---

[12] Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. XrViv preprint arXiv:2209. 14988, 2022.
[13] Yichun Shi, Peng Wang. Jianglong Ye, Mai Long, Kejie Li, and Xiao Yang. Mvdream: Multi- view diffusion for 3d generation. arXiv preprint arXiv:2308. 16512, 2023.
[14] Walter HL Pinaya, Petru-Daniel Tudosiu, Jessica Dafflon, Pedro F Da Costa, Virginia Fernandez, Parashkev Nachev, Sebastien Ourselin, and M Jorge Cardoso. Brain imaging generation with latent diffusion models. In MICCA/ Workshop on Deep Generative Models, pages 117-126. Springer, 2022.
[15] Hao Wang and Dit-Yan Yeung. Towards bayesian deep learning: A framework and some existing methods. TDKE, 28(12):3395-340 2016.
[16] Hao Wang and Dit- Yan Yeung. A survey on bayesian deep learning, CSUR, 53(5):1-37,
[17] Yibin Wang, Haizhou Shi, Ligong Han, Dimitris Metaxas, and Hao Wang. Blob: Bayesian low-rank adaptation by backpropagation for large language models. In NeurlPS, 2024.
[18] Haizhou Shi, Yibin Wang, Ligong Han, Huan Zhang, and Hao Wang. Training- free bayesian- ization for low-rank adapters of large language models. In NeurlPS, 2025.
[19] Zihao Xu, Guangyuan Hao, Hao He, and Hao Wang. Domain indexing variational bayes: Interpretable domain index for domain adaptation. In ICLR, 2023.
[20] Yiming Qin, Huangjie Zheng, Jiangchao Yao, Mingyuan Zhou, and Ya Zhang, Class-balancing diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18434 18434-18443, 2023.
[21] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion- 5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 01125288-22299 2022.
[22] Yuzhe Yang, Kaiwen Zha, Yingcong Chen, Hao Wang, and Dina Katabi. Delving into deep imbalanced regression. International conference on machine learning, pages 11842-11888 1151. PMLR, 2021.
[23] Ziyan Wang and Hao Wang. Variational imbalanced regression: Fair uncertainty quantification via probabilistic smoothing. Advances in Neural Information Processing Systems, 36, 2024.
[24] Yuzhe Yang, Hao Wang, and Dina Katabi. On momti-domain long-tailed recognition, imbalanced domain generalization and beyond. In ECCV, pages 57-75, 2022.
[25] Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer. Smote: synthetic minority over-sampling technique. Journal of artificial intelligence research, 16: 321-357, 2002.
[26] Haibo He and Edwardo A Garcia. Learning from imbalanced data. IEEE Transactions knowledge and data engineering, 1163):266112244 2009.
[27] Luis Torgo, Rita Ribeiro, Bernhard Pfahringer, and Paula Branco. Smote for regression, In Porluguese conference on artificial intelligence, pages 378-389 Springer, 2013.
[28] Paula Branco, Luis Torgo, and Rita P Ribeiro. Smogn: pre-procsssing approach for imbalanced regression. In First international workshop on learning with imbalanced domains: Theory and applications, pages 36-50. PMLR, 2017.
[29] Paula Branco, Luis Torgo, and Rita P Ribeiro. Rebagg: Resampled bagging for imbalanced regression. In Second International Workshop on Learning with Imbalanced Domains: Theory and Applications, pages 67-81. PMLR, 2018.
[30] Jiawei Ren, Mingyuan Zhang, Cunjun Yu, and Ziwei Liu. Balanced mse for imbalanced visual regression. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 7926-7935, 2022.
11

---

[31] Yu Gong, Greg Mori, and Frederick Tung. Ranksim: Ranking similarity regularization for deep imbalanced regression. arXiv preprint arXiv:2205.15236, 2022.
[32] Mahsa Keramati, Lili Meng, and R David Evans. Conr: Contrastive regularizer for deep imbalanced regression. arXiv preprint rrXiv:230906651, 2023.
[33] Ziyan Wang and Hao Wang. Variational imbalanced regression: Fair uncertainty quantification via probabilistic smoothing. In NeurIPS, 2023.
[34] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 22500-223 22510, 2023.
[35] Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun- Yan Zhu. Multi- concept customization of text-to-image diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1931-1941, 2023.
[36] Zhen Li, Mingdeng Cao, Xintao Wang, Zhongang Qi, Ming- Cheng, and Ying Shan. Photomaker: Customizing realistic human photos via stacked id embedding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8640-8650, 2024.
[37] Tianjiao Zhang, Huangjie Zheng, Jiangchao Yao, Xiangfeng Wang, Mingyuan Zhou, Ya Zhang, and Yanfeng Wang. Long-tailed diffusion models with oriented calibration. The Twelfth International Conference on Learning Representations, 2024.
[38] Byeonghu Na, Yeongmin Kim, HeeSun Bae, Jung Hyun Lee, Se Jung Kwon, Wanmo Kang, and I1-Chul Moon. Label-noise robust diffusion models. arXiv preprint arXiv:2402.17517, 2024.
[39] Xudong Shen, Chao Du, Tianyu Pang, Min Lin, Yongkang Wong, and Mohan Kankanhalli. Finetuning text-to-image diffusion models for fairness. arXiv preprint arXiv:231 2023.
[40] Yeongmin Kim, Byeonghu Na, Minsang Park, JoonHo Jang, Dongjun Kim, Wanmo Kang, and Il-Chul Moon. Training unbiased diffusion models from biased dataset. In The Twelfth International Conference on Learning Representations, 2024.
[41] Kristy Choi, Aditya Grover, Trisha Singh, Rui Shu, and Stefano Ermon. Fair generative modeling via weak supervision. In International Conference on Machine Learning, pages 1887-1898. PMIR,, 2020.
[42] Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy Liang, Quoe V Le, Tengyu Ma, and Adams Wei Yu. Doremi: Optimizing data mixtures speeds up language model pretraining. Advances in Neural Information Processing Systems, 36: 69798-69818 8, 2023.
[43] Simin Fan, Matteo Pagliardini, and Martin Jaggi. Doge: Domain reweighting with generalization estimation. arXiv preprint arXiv:2310.15393, 2023.
[44] Qian Liu, Xiaosen Zheng, Niklas Muennighoff, Guangtao Zeng, Longxu Dou, Tianyu Pang, Jing Jiang, and Min Lin. Regmix: Data mixture as regression for language model pre-training. arXiv preprini 24Xi::2407.01442..
[45] Yize Li, Yihua Zhang, Sijia Liu, and Xue Lin. Pruning then reweighting: Towards data-efficient training of diffusion models. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEEE, 2025.
[46] Hao Wang, Binyi Chen, and Wu-Jun Li. Collaborative topic regression with social regularization for tag recommendation. IJCAI, pages 2719-2725, 2013.
[47] Stylianos Moschoglou, Athanasios Papaioannou, Christos Sagonas, Jiankang Deng, Irene Kotsia, and Stefanos Zafeiiiou. Agedb: the first manually collected, in-the-wild age database. In proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 51-59, 2017.
12

---

[48] Gwangbin Bae, Martin de La Gorce, Tadas Baltrušaitis, Charlie Hewitt, Dong Chen, Julien Valentin, Roberto Cipolla, and Jingjing Shen. Digiface- Im: million digital face images for face recognition. In Proceedings of the IEEF/CVF Winter Conference on Applications of Computer Vision, pages 3526-3535, 2023.
[49] Qiong Cao, Li Shen, Weidi Xie, Omkar M Parkhi, and Andrew Zisserman. Vggface2: A dataset for recognising faces across pose and age. In 2018 13th IEEE international conference automatic face gesture recognition (FG 2018), pages 67-74. IEEE, 2018.
[50] Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with abbelldistribution-aaar margin loss. Advances neural information processing systems, 32, 2019.
[51] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, 1lge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint 2rX1v:2303.08774, 2023.
[52] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference computer vision, pages 9650-9660, 2021.
[53] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[54] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Re- thinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818-2826, 2016.
[55] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
13

---

A Proofs for Proposition 3.1
Proposition A.1. Assume Xy = Apou - de + Ay,, we have our loss function
Tyy CRODDI En
(1Ho(xe.)) - procil?
+
(14)
Here, C a constant, and paGg denotes the mean of the PoG, with the expression defined in Equ. (2).
Then, through derivations based Gaussian properties, we obtain
PoGDif 1-1
< Eng (((ce) ([ea(Xe; y) ell² + A(Ax)) [1ea(Xe, y) ea ex(x.,)1)1
+0 +C
(15)
where the function A(A) & A(1 - ax)
20, (xs)
Proof. To prove the above inequality, we need to prove the following lemma.
Lemma A.1. Assume by = APoG & d + Xyy, and for simplicity we shorten the notation from y) and po 5x( y) to coly) and poly), respectively. Then we have
1 2 No (wo (y) me)² by (no (y) so (y'))² 2 2>y (No (y) m2rcci2 (16)
Proof. By the definition of Gaussian property, we have
1
2 de (mo -1a)² + 2 Xy (uo(y) - ()))))
[uo(y)]² 21eeuo(y)) + uz [mo(y)] - 2u (yo + luo(y'))
22,1
22xd
uolx') de
poly) 2
[No(y')]² d
(x11 )33) (uo(y))3= -2
Ag.
Ayy
212:2y-1 111
[Meee + [nO ( Ay de + Ay
HGdee + (MO(y')) >>yy de + Ay
[mo 2x]))] -2
Mo(Y) +
2
d Ay.
Hide + ( (y'))")
Hide + [No(y')
de + Ay 2
A + Ayy 2
de Axy
de + Ayy
Me + (1))))))y
(o(())
d Ayy
(aE + (())))))))))({)) + Xx.) - (ur + [moey [[o((')))y.)'
2
2(A, + Axr))
X1 + Ay.
1 My (2o(Y) 22PG))
+ Xedy 2)
2(he + )x:))
> 1
/ ,y (Ho(y) 2
Thus complete the proof.
14

---

From Lemma A.1, we can derive
2 Ay (y - -MPoall
= yy (uo(y) - (LPoc))
2
1
_ 2
Ar (ms - -ma)² 2 + 2 Xy (Ho(y) - 2)²
1
2 2A (y) 1eel2 2 =2" > (y "y(y')1
A(A)) ()( - -1| A()y;) (1ro(y) cc(y''11?
where the function A(A) A X(1 22)
and the last equivalence is because the transform from
20x a+)
(-(-)) to c.).)
B Ablation Study
To verify the effectiveness of each component in the second term in our PoGDiff final objective function from Eqn. (12), we report the accu- racy of our proposed PoGDiff after re- moving the y' (i.e., same as Vanilla model), the Image Similarity term Wing-sim: and/or the Inverse Text Den- sities term Vnnv-txt-den
Table 6: Ablation Studies.
Datasets
BeeDB-IT2I-Lage
Size
FID +
Human
t GPT-4o DINO f
Shot
All
Few
All
Few
All
Few
Few
w/O y' (VANILLA)
7.67 11.67 0.60 0.20 4.90 3.60
0.34 0.25
w/O PIMGGSI
6.41 10.49
0.84 0.68 8.40 7.60
0.57 0.46
Table for AgeDB-IT21-2 The results show that removing either term leads to a per- formance drop, confirming the impor- tance of both terms in our PoGDiff.
w/O UINWTNSSON
6,35 10.43
0.84
0.68
8.20
7.80
0.64
0.51
PoGDIFF (OurS)
6.03
10.16
0.84
0.68
8.50
8.00
0.66 0.52
In addition, to verify the threshold calculating the gRecall score, we report the performance in Table 7 for AgeDB-IT2l-small, The results show that our method consistently outperforms the baselines. Notably, while lowering the threshold improves the gRecall scores of the baselines, our model still achieves higher scores and demonstrates clear advantages.
C Details for Datasets
Note that our method is designed for fine-tuning. Therefore our setup does not require large-scale, long-tailed human datasets. Instead, sample from these datasets, as long as they meet the following criteria: (1) the dataset must be long-tailed, (2) traditional methods must fail to recognize the minority classes, and (3) there must be distinguishable difference between the majority and minority classes (e.g., we prefer visual distinctions between the two groups to better highlight the impact of our method). Fig. 6 shows the label density distribution of these datasets, and their level of imbalance (see Appendix K.4 for details on data sparsity).
AgeDB-I721: ATTD is constructed from the AgeDB dataset [47]. For each image in AgeDB, we passed it through the pretrained LLaVA--16-77 model [55] to generate textual captions y. Since the identities in AgeDB are well-known individuals that the pretrained SDv1.5 [7] might have encountered during pre-training, we masked the true names and replaced them with generic, random names, leading to new caption y. For example, we replace "Albert Einstein" in the caption with random name "Lukas". Finally, we collect all (y, x) pairs to form our AgeDB-IT21 dataset.
Additionally, given that the identities (i.e., people individuals) in AgeDB are well-known figures, we sampled from AB to create three datasets for comprehensive analysis: AgeDB- (large), AgeDB-IT21a IT2I-M (medium), and AgeDB-IT2I-S (small). Specifically:
AgeDB-IT2I-L (large). This dataset consists of 976 images across 223 identities, with each majority class containing 30 images and each minority class containing 2 images.
AgeDB-Ir2I-M (medium). This dataset consists of 100 images across identities, with each majority class containing 30 images and each minority class containing 2 images.
15

---

Table 7: Ablation Studies for threshold in gRecall in AgeDB--I22Issaal
Shot
All
Few
All
Few
All
Few
All
Few
All
Few
Threshold
0.8
0.7
0.6
0.5
0.4
VANILLA
0.000
0.00
0.017
0.00
0.067
0.00
0.150
0.00
0.717
0.500
CBDM
0.100
0.000
0.267
0.000
0.283
0.000
0.817
0.500
0.883 0.500
T2H
0.000 0.000
0.017
0.000
0.067
0.000
0.150
0.000
0.717
0.500
POGDIFF
0.733 1.000
0.800
1.000
0.867 1.000 1.000 1.000
1.000 1.000
AgeDB-IT21-S (small). This dataset contains 32 images across 2 identities, where each majority class consists of 30 images and each minority class consists of 2 images.
DigiFace-I122 DigiFace-IT21 is derived from the DigiFace dataset [48]. contains 985 images across 179 identities, where each majority class consists of 30 images and each minority class consists of 2 images. We use process similar to AgeDB-IT21 to collect text-image pairs, forming this DigiFace-IT21 dataset.
VGGFace-IT21: VGGFace-IT2I is a subset from VGGFace2 [49]. It contains 1933 images across 193 identities, where each majority class consists of 49 images and each minority class consists of 2 images.
D Details for Baselines
We employ Stable Diffusion v1.5 [7] as the backbone diffusion model. As this is the first work to explore imbalanced text-to-image (IT2I) diffusion models with natural text prompts, we adapt the current state-of-thh-au methods designed for long-tailed T2I diffusion models with one-hot text prompts to serve as basclines. The basclines are described below:
Vanilla: We use term Vanilla to denote model that does not incorporate any techniques for handling imbalanced data, equivalent to fine-tuning a Stable Diffusion model without additional modifications.
CBDM: We use term CBDM to denote a model that incorporates the Class Balancing Diffusion Model (CBDM) [20] approach. During fine-tuning, we sample additional text embedding y' from the entire fine-tuning dataset and apply the CBDM objective function. All hyperparameters
kept the same in the original paper, with further details available in Qin et al. [20].
T2H: We use the term T2H to denote a model that uses Long- Tailed Diffusion Models with Oriented Calibration (T2H) [37]. T2H is a feature augmentation method, but is not directly applicable to our setting. Specifically. T2H [37] relies on the class frequency, which is not available in experiments. In this paper, adapt this method to our settings by using the density for each text prompt embedding to serve as the class frequency in T2H [37].
E Details for Evaluation
In this section, we provide details on evaluation procedures.
FID Score. For each identity, we collect all images from the original AgeDB or DigiFace datasets as the true image set. Then, In all-shot evaluation, for AgeDB-IT2I-S and AgeDB-IT2I-M, we generate 100 images per identity as the fake image set, and for AgeDB-IT2I-L. DigiFace-IT21, VGGFace-IT21 and 1IFAR-100-IT2L,
generate 20 images per identity the fake image set. In few-shot evaluation. we generate 500 images per identity as the fake image set. For all generations, we employ the DDIM sampling technique [3] with 50 steps. The prompt used during generation is "An image of (p)." where "p" is the name of the identity (e.g., Albert Einstein).
Human & GPT-4o Feedback. For each minority identity, we generate 5 images using DDIM sampling [3] with 50 steps. We then 10 people to evaluate whether the images depict the same person (scored as 1.0) not (scored as 0.0). Additionally, for each image, ask the GPT-4o model
16

---

Table 8: Performance based on FID score.
Datasets (-IT2I)
AgeDB
DigiFace
VGGFace
CIFAR-100
Size
Small
Medium
Large
Large
Large
Large
Metric
FID
Shot
All
Few
All
Few
All
Few
All
Few
All
Few
All
Few
VANILLA
14.88
13.72
12.87
12.56
7.67
11.67
7.18
12.23
7.59
12.08
7.19
11.46
CBDM
14.72
14.13
11.63
11.59
7.18
11.12
6.96
12.72
7.23
11.91
7.26
11.94
T2H
14.85
13.66
12.79
12.52
7.61
11.64
7.14
12.22
7.34
12.02
7.10
11.39
POGDIFF (OURS)
14.15
12.88
10.89
10.64
6.03
10.16
6.84
11.21
6.29
10.97
6.24
9.41
to rate the similarity scale from 1 to 10. The prompt used during generation is "An image of (p J." where "p" is the name of the identity. The text prompt using for GPT-4o model is "It is mandatory to give a score that how close the person in the image means they are exactly the same person, while score of 0.0 means they are definitely not the same person. How close you think the person in the image to 'p-true'." where "p-true" denotes the real name wwell-known name) in AgeDB. Note that the GPT-4o model might occasionally refuse to provide score, and you may need repeat and compel it to give rating. For each image, collect 10 scores from the GPT-40 model and report the average rating.
well-known individual. A score of 10.0
Evaluating Image Similarities. We collect samples that are outside our training dataset (e.g., AgeDB-T2I-4 but belong to the original dataset (e.g., AgeDB). Using the same prompt, we generate the corresponding images. A pre-trained DINOv2 model [52] is then applied to extract latent features, and cosine similarities are calculated.
F Details for Implementation
For both baselines and our model, we used the same hyper-parameter settings, specifically
AgeDB-I121-M & AgeDB-I12I-S. The learning
set to 1 10-5 with maximum of 6, 0000 training steps. The effective batch size per GPU was 8, calculated as (Batch Size) x 1 (Gradient Accumulation Step).
AgeDB-IT2I-L & DigiFace-1721. The learning rate was set to x 10-5, with a maximum of
12, 000 training steps. The effective batch size per GPU was 32, calculated (Batch Size) x
4 (Gradient Accumulation Steps).
VGGFace-IT2 & CIFAR- 10R--10--T21. The learning rate was set to 10-4, with a maximum of 24, 000 training steps. The effective batch size per GPU was 128, calculated as 32 (Batch Size) x 4 (Gradient Accumulation Steps).
G Full Results
We report the performance of different methods in terms of FID score, human evaluation score, GPT- 4o score, and DINO score in Table 8, Table Table 10 and Table 11, respectively. Across all tables, we observe that our PoGDiff consistently outperforms all baselines. More results and discussions are in Appendix K.5. Notably, PoGDiff demonstrates significant improvements, especially in few-shot scenarios (i.c., for minority classes). It is also worth noting that CBDM [20] performs extremely poorly on AgeDB-IT2I-S and AgeDB- --2--M datasets. This is because their method samples text conditions from the entire space, which may work in one-hot class settings, but in our context (natural text conditions). this sampling technique misguides the model during training. In addition, Table 12 shows the gRecall for different methods on three datasets, AgeDB-ITTI-Smee AgeDB-IT21-Mede
and A These results show that our PoGDiff achieves much higher gRecall compared to all baselines, demonstrating its impressive diversity and coverage of different attributes of the same individual in the training set (see Appendix for more discussion and examples on gRecall).
17

---

Table 9: Performance based on DINO score.
Datasets (-IT2I)
AgeDB
DigiFace
VGGFaae
CIFAR-I00
Size
Small
Medium
Large
Large
Large
Large
Metric
DINO (cosine similarity) scores f
Shot
All
Few
All
Few
All
Few
All
Few
All
Few
All
Few
VANILLA
0.42 0.37 0.39 0.28
0.34 0.25
0.42
0.36
0.41 0.29
0.48
0.32
CBDM
0.54 0.09 0.38 0.11
0.41 0.26
0.34
0.16
0.46 0.22
0.52
0.28
T2H
0.43 0.39 0.42 0.29
0.37
0.26
0.44
0.36
0.42
0.28
0345
0.30
POGDIFF (OURS)
0.77 0.73 0.69 0.56 0.66 0.52
0.64 0.49
0.69 0.55
0.73
0.61
Table 10: Performance based on human evaluation. The evaluation is binary decision: Image either judged as representing the same individual (score 1.0) or not (score 0.0).
Datasets (-IT2I)
AgeDB
VGGFace
CIFAR-100
Size
Small
Medium
Large
Large
Large
Metric
Human Score 1
Shot
All
Few]
All
Few
All
Few
All
Few
All
Few
VANILLA
0.50 0.00
0.66 0.32
0.60 0.20
0.62 0.16
0.72
0.30
CBDM
0.50 0.00
0.44 0.08
0.56 0.12
0.54 0.10
0.63
0.24
T2H
0.50 0.00
0.66 0.32
0.60 0.20
0.62
0.16
0.72
0.30
POGDIFF (OurS)
1.00 1.00
0.96 0.92
0.84 0.68
0.78 0.64
0.84
0.72
H
Limitations
Datasets. Our method relies heavily on "borrowing" the statistical strength neighboring samples from minority classes, making the results sensitive to the size of the minority class. (i.e., in assumption we require at least for each minority class). In addition, while our AgeDB-IT2I-smal and AgeDB--T2I-medium are actually the sparse dataset, the cardinality remains limited in our experiments. Therefore, how to address IT2I problem under this settings are interesting directions.
Models. Our method is general fine-tuning approach designed for datasets that the Stable Diffusion (SD) model has not encountered during pre-training. As shown in Fig. 1, color deviation is very common and is known issue when fine-tunes diffusion models (as also mentioned in [2]); for example, we can observe similar color deviation in both baselines (e.g.. CBDM and Stable Diffusion v1.5) and our PoGDiff. This be mitigated using the exponential moving average (EMA) technique [2]; however, this is orthogonal to our method and outside the scope of our paper. Moreover, shown in Fig. 5, the baseline Stable Diffusion also suffers from this issue. Besides, exploring PoGDiff's performance when training from scratch is also an interesting direction for future work.
Methodology. The distance between the current text embedding y and the sampled y' impacts the final generated results, therefore in paper, we introduced more sophisticated approach for computing the weight v,, which depends on the quality of the image pre-trained model and our trained VAE. These mechanisms ensure that data points with smaller distances are assigned higher effective weights. Effectively producing y for any new, arbitrary dataset remains an open question and an interesting avenue for future work, as it could further enhance the method's performance.
Evaluation. Our goal is to adapt the pretrained diffusion model to specific dataset; therefore the evaluation should focus on the target dataset rather than the original dataset used during pre-training. For example, when user fine-tunes a model on a dataset employee faces, s/he is not interested in how well the fine-tuned model can generate images of "tables" and "chairs". Evaluating the model's
18

---

Table 11: Performance based GPT-4o evaluation. The scores are from 0 to 10, with higher scores indicating the individual resembles the well-known person.
Datasets (-IT2I)
AgeDB
VGGFace
CIFAR-100
Size
Small
Medium
Large
Large
Large
Metric
GPT-4o Evaluation
t
Shot
All
Few
All
Few
All
Few
All
Few
All
Few
VANILLA
5.20
3.20
4.30
2.90
4.90
3.60
4.50
2.90
6.00
3.20
CBDM
4.50 1.10
1.30
1.00
3.10
1.70
2.80
1.30
3.40
2.00
T2H
5.50
3.10
4.60
3.00
4.70
3.90
4.60
3.10
6.20
3.60
PoGDIFF (OuRS)
9.10
8.40
8.80
8.20
8.50
8.00
8.20
7.60
8.40
8.00
Table 12: Performance based on gRecall score. See the detailed definition of gRecall in Sec. 4.1.
Datasets (-IT21)
AgeDB
VGGFace CIFAR-100
Size
Small
Medium
Large
Large
Large
Metric
gRecall Score
Shot
All
Few
All
Few
All
Few
All
Few
All
Few
VANILLA
0.017 0.000
00104 0.167 0.196 0.200 0.133 0.167 0.200 0. 160
CBDM
0.267 0.000
0.159 0.083 0.138 0. 0.100
0.120 0. 100
0.086 0.067
T2H
0,017 0.000
0.104
0.167
0.196 0.200
0.
0.167
0.200
0. 0.160
POGDIFF (OuRS)
0.800 1.000
0.517 0.642
0.435 0.540
0.400
0.533
0.433 0.567
performance on the original dataset used during pre-training would an intriguing direction for future work, but it is orthogonal to our proposed PoGDiff and out of the scope of our paper.
I Additional Details for AgeDB-IT2I-small in Table 5
For AgeDB-IT2I-small, there are two IDs, one "majority" ID with 30 images and one minority ID with 2 images.
For VANILLA and T2H, the gRecall for the majority ID and the minority ID is 1/30 and 0/2, respectively. Therefore, the average gRecall score is 0.5 * 1/30 + 0.5 * 0/2 z 0.0167.
For CBDM, the gRecall for the majority ID and the minority ID is 16/30 and 0/2, respectively. Therefore, the average gRecall score is 0.5 16/30 + 0.5 * 0/2 == 0.2667.
For PoGDiff (Ours), the gRecall for the majority ID and the minority ID is 18/30 and 2/2, respectively. Therefore, the average gRecall score is 0.5 * 18/30 + 0.5 * 2/2 = 0.8.
J Additional Results
J.1 Base Model
All our methods and baselines use SD 1.5 as the backbone model. Our preliminary results indicate that different SD variants affect only low-level image features (e.g., color, sharpness) but not accuracy. Therefore, for simplicity, we adopt the widely used SD 1.5 as our backbone. robustness, we also conduct additional experiments another SD variant, SD 2.1. As shown in Table 14, the results on AgeDB-small across five metrics demonstrate that our method remains robust across different SD variants.
further validate
19

---

Table 13: Results for additional baselines in AgeDB-IT2I-small
Metric
FID 4
|
DINO
Human t
GPT
gRecall f
Shot
All
Few
All
Few
All
Few
All
Few
All
Few
VANILLA
14.88 13.72
0.42 0.37
0.50 0.00
5.20
3.20
0.017 0.000
ITPI (0.2)
14.93 13.76
0.43 0.35
0.50 0.00
5.40
3.10
0.017 0.000
ITPI (0.5)
15.02 13.85
0.41
0.29
0.50
0.00
5.00
2.80
0.017 0.000
ITPI (0.8)
14.96 13.82
0.43
0.34
0.50
0.00
5.20
3.00
0.017 0.000
REB
14.72 13.32
0.49 0.39
0.50 0.00
5.40
3.60
0.050
0.000
POGDIFF (OuRS)
14.15 12.88
0.77 0.73
1.00 1.00
9.10
8.40
0.800 1.000
Table 14: Results across five metries in AgeDB-IT2I-small.
Metric
FID +
DINO +
Human 1
GPT +
gRecall 1
Shot
All
Few
All
Few
All
Few
All
Few
All
Few
SD1.5
14.88 13.72
0.42
0.37
0.50 0.00
5.20
3.20
0.017
0.000
POGDIFF (SD1.5)
14.15 12.88
0.77
0.73
1.00
1.00
9.10
8.40
0.800
1.000
SD2.1
15.02
13.97
0.40
0.32
0.50
0.00
5.60
3.40
0.017
0.000
PoGDIFF (SD2.1)
14.19
12.94
0.72
0.68
1.00
1.00
8.70
8.00
0.767 1.000
J.2 More Baselines
Inference-time Prompt Interpolation (ITPI). We incorporate an additional baseline, dubbed Inference-Time Prompt Interpolation (ITPI), into AgeDB-IT2I-small. Specifically, during inference, for a given text embedding y, we locate its nearest neighbor y' and perform prompt interpolation y Ty + (1 7)y/ where y [0, 1] is balancing factor. We report the performance for = 0.2, 0.5, and 0.8, respectively.
Rebalanced. We incorporate another baseline, dubbed Rebalancing (ReB), into AgeDB-IT2I-sme Specifically, we compute a similarity matrix and define the similarity score of each image as the sum of its corresponding row. Images with lower similarity scores are more likely to belong to minority groups During training, we resample images inversely proportional to their similarity scores, that is, the higher the score, the lower the sampling probability.
As shown Table 13, ITPI performs similarly to, or even worse than, the vanilla model, since directly manipulating the text embedding space with simple linear direction harms generation quality. ReB achieves comparable or occasionally slightly better performance than the vanilla baseline. In contrast, our PoGDiff significantly outperforms both, demonstrating its effectiveness.
K
Discussion
K.1
Problem Settings
We would like to clarify that our paper focuses setting different from works like DreamBooth [34], and our focus is not on diversity, but on finetuning diffusion model on an imbalanced dataset. Specifically:
Different Setting from Custom Techniques like DreamBooth [34], CustomDiffusion [35] and Mhoto [36]. Previous works like CustomDiffusinn and PhotoMaker focus on adjusting the model to generate images with single object, e.g. a specific dog. In contrast, our PoGDiff focuses finetuning the diffusion model on entire data with many different
20

---

SDv1.5
CBDM
T2H
PoGDiff (Ours)
Figure 5: Example generated images from different methods. Our PoGDiff outperforms the baselines terms of both generation accuracy and generation quality.
AgeDB-IT2M-S
AgeDB-IT2M-M
AgeDE-IT2MAL
DigiFace-IT2ILL
VGGFace-IT2I-L
CIFAR-100-IT2LL
30
30
30
30
50
500
25.
25
25
25
40-
400
of samples
30
300
15
20-
200
10
10
10
10
5
100
Identity
Identity
Identity
Identity
Identity
Identity
Figure 6: Overview of label distributions for six IT2I datasets in bar plots. The x-axis corresponds to the identities (i.e., people or class).
objects/persons simultaneously. They are very different settings and are complementary to each other.
Diversity. Note that while our PoG can naturally generate images with diversity, diversity is actually not our focus. Our goal is to fine-tune diffusion model an imbalanced dataset. For example, PoGDiff can fine-tune diffusion model on an imbalanced dataset of employee faces so that the diffusion model can generate new images that match each employee's identity. In this case, we are more interested in "faithfulness" rather than "diversity".
K.2 More Detailed Analysis: Understanding Fig. 5
Fig. 5 shows randomly sampled generated images on low-density classes in AgeDB-IT2I-L. Across each column, the individual names are Albert Einstein, JW Marriott, J.P. Morgan, Edward G. Robinson, Larry Ellison, and Luise Rainer, respectively. The results show that our PoGDiff achieves significantly better accuracy and quality for tail classes.
Note that one of our primary objectives is to generate accurate images of the same individual while ensuring facial consistency. Therefore diversity
sometimes be harmful. For example, given text input of "Finstein", generated images with high diversity would generate both male and females images; this is obviously incorrect. Therefore important to strike balance between diversity and accuracy, goal that our PoGDiff achieves.
Specifically, shown in Fig. 5:
First Three Columns of SDv1.5, CBDM, and PoGDiff: these cases, the training dataset contains only two images per person. With such limited data, it is impossible to introduce meaningful diversity.
SDv1.5 .5 fails to generate accurate images altogether in this scenario.
While CBDM might appear to produce the "diversity", does so incorrectly, as generates image of woman when the target is Einstein.
21

---

Intoon Dony Where
-2-
Figure TSNE visualization for all the methods for an example individual in the AgeDB-IT2M-large dataset.
In contrast, our PoGDiff can successfully generate accurate images (e.g.. Einstein images in Column 1) while still enjoying sufficient diversity.
Fourth and Fifth Columns: Here, the training dataset contains a medium number of images per person (5-7 images). Under these conditions:
SDv1.5 can generate accurate representations individuals, its outputs lack diversity.
CBDM, on the other hand, introduces "diversity" but consistently generates incorrect results.
In contrast, our method produces accurate images the target individual while demonstrating
greater diversity than SDv1.5.
Sixth Column: this case, the training dataset includes 30 images per person.
-SDv1.5 generates accurate images but with nearly identical expressions, i.c., poor diversity. - -CBDM still fails to generate accurate depictions of the individual.
In contrast, PoGDiff successfully generates accurate images while maintaining diversity.
In summary, typical diversity evaluation in diffusion model evaluations, such as generating multiple types of trees for a "tree" prompt, is not the focus of our setting and may even be mislcading. In our setting, the key is to balance accuracy and diversity.
K.3
Why Not Directly Smooth Text Embedding?
Preliminary results indicate that directly smoothing the text embeddings does not yield meaningful improvements. Below provide some insights into why this approach might fail. Suppose have a text embedding y and its corresponding neighboring embedding y'. Depending on their relationship. we are likely to encounter three cases:
Case 1: y' = y. this case, applying a reweighting method such as a linear combination results in no meaningful change, as the smoothing outcome is still y.
Case 2: y' is far from y. If y' is significantly distant from y, combining them becomes irrelevant and nonsensical, as y' no longer represents useful neighboring information.
Case y' is very close to y. When y' close to y, the reweighting can be approximated as: ory + (1 - oxyy' 2 y + (1 (y -y) Since y' is nearly identical to y. this effectively introduces a small weighted noise term (1 - a)(y' - y) into y. our preliminary experiments, this additional noise degraded the performance compared to the original baseline results.
Based on these observations, direct smoothing of embeddings appears ineffective and may even harm performance in some cases.
K.4 Our Dataset Covers Different Levels of Sparsity
Our BeeDB-IT2M-sman and BeeDB-IT2M-men for evaluate the sparse data. For example, the AgeDB-IT2M-smal only contains images from persons, it is therefore very sparse data setting, compared to B-LDD-II2M--BE 223 persons. Fig. shows the bar plot version for our datasets, while sparse settings are not our primary focus, we agree that addressing imbalanced image generation in such setting is an interesting and valuable direction, and we have included discussion about this in the limitations section of the paper.
datasets are actually very sparse and are meant
with images across
22

---

In method, our predicted variance is indeed conditioned th and eb is conditioned on K.5 Discussion on FID
is important to note that the FID score measures only the distance between Gaussian distributions of ground-truth and generated images, relying mainly on mean and variance. As a result, it does not fully capture the nuances of our task. This is why include additional evaluation metrics such as DINO Score, Human Score, and GPT-4o Score, to comprehensively verify our method's s superiority (as shown in Table 2, Table 3 and Table 4).
Additional Experiments: Limitation of FID. In addition, we have added figure showcasing a 1-SNE visualization for minority class example, as shown in Fig. 7, to further illustrate the limitation of FID we mentioned above. As shown in the figure:
There are two ground-truti IDs (i.e., two ground-truth individuals) in the training set. Our PoGDiff can successfully generate images similar to these two ground-truth ID while maintaining diversity.
All baselines, including CBDM, fail to generate accurate images according to the ground-truth IDs. In fact most generated images from the baselines are similar other IDs, i.e. generating the facial images wrong individuals.
These results show that:
Our PoGDiff significantly outperforms the baselines.
FID fails to capture such improvements because depends only on the mean and variance of the distribution, losing a lot of information during evaluation.
FID Measures Both ID Consistency and Diversity. Note that our FID computed for each ID separately, and the final FID score in the tables (e.g.. Table 1) is the average FID over all IDs. Therefore FID measures both ID consistency and diversity.
To see why, note that the FID score measures the distance between two Gaussian distributions, where the mean the Gaussian represents the identity (ID) and the variance represents the diversity. For example, the mean of the ground-truth distribution represents the embedding position of the ground- truth ID, while the variance of the ground-tres distribution represents the diversity of ground-truth images, and similarly for the generated images.
Therefore, a lower FID score indicates that the generated-image distribution better matches the ground-truth distribution in terms of both ID and diversity.
K.6
Which Distribution the Model Converges to After Training
Our PoG objective introduces an additional term that encourages consistency across semantically similar conditioning prompts, but does not fundamentally alter the underlying diffusion process. While full convergence guarantees would be an interesting work, such analyses are rare in conditional diffusion literature. Since our formulation preserves the standard denoising score matching structure, its convergence behavior broadly follows that of existing diffusion models.
K.7 The Variance of the Predicted Distribution
y. as mentioned Eqn. (6).
The definition of & relies on the assumption that data belongs to the same class or person (i.e., I()). Although such information might not implicitly available for some datasets, We can use a pretrained image classifier (e.g., ResNet and ViT) to obtain the label. Alternatively, we can also use clustering method and treat the cluster ID the class label.
Although the definition of & relies on the VAE training. In our experiment, we used simple VAE with three-layer MIPs for both the encoder and decoder. We found that variations in architecture, learning rate, and number of training epochs had little effect on the final fine-tuning performance since very efficient and easy to train, it only costs around few minutes for single GPU.
23

---

K.8
Computational Efficiency
For the computation efficiency of our method:
Since our denoising step requires two forward passes of the denoising model, the runtime is approximately twice that of Stable Diffusion (SD) [7]. However, we observe that under the same training time budget (e.g.. SD for 12k steps vs. PoGDiff for 6k steps), our method already able to generate accurate images for tail classes, while Stable Diffusion continues to produce samples biased toward head classes.
Training:
Inference:
During inference, our generation process remains identical to that of Stable Diffusion and thus incurs no computational overhead.
K.9 Discussions Failure Cases
The reason why some generated individuals do not match expectations lies in the selection of y/- Since our prompts are fully end-to-end generated from LLAVA-NEXT, different individuals may have higher similarity scores under &b than the true target. For example, Einstein at age 41 (y="The image is black and white photograph of man named Einstein, who has mustache, curly hair. and a pipe in his mouth.") might appear more similar based on textual semantics to CHARLIERUGGLES at age 48 GGGGGEEERRRGGG
man is a man with a mustache. holding pipe in his mouth." ") than to Einstein at age 24 (y/="The image is a black and white portrait of man named Einstein, who has mustache and wearing a suit.") "). As a result, PoGDiff may use of rather than 3/2 as the neighbor y for
In general, when the dataset includes many individuals with overlapping semantic traits, is possible that in some steps y' is not the same individual as leading to subtle deviations in facial details. These deviations typically small but beneficial; they help improve diversity while still preserving overall identity correctness.
L Impact Statement
Finetuning under imbalanced datasets in specific domain presents an inescapable challenge in generative AI. For example, when generating the counterfactual outcomes for users with specific actions, such "user(or patient)-action -ottomm" pairs are always imbalanced, as it is impossible for any company or any hospital to obtains all the pairs. As such, to save the budget, learning the mapping from "user(or patient)-action" (sentence description) to "outcome" (images) is where this challenge is particularly pronounced. Our proposed method, PoGDiff, represents an innovative and efficient solution to navigate this issue. We argue that the complexity and importance this problem warrant further research, given its profound implications across diverse fields. This exploration not only advances our understanding but also opens new avenues for significant impact, underscoring the need for continued investigation into training generative models under imbalanced datasets.
24

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] providedd proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA]" is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
It can be found in the Sec. 1.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
In Appendix H.
25

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: Although we do not have theoretical result like theory paper, but we provide the proof for our lemma in appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Yes, they are all discussed in main paper and appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
26

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g.. with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: They are public, and detailed information are in the appendix. For the code. we will release it once this paper is accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https:/ /tts:///npss../// cc/
public/ /guidegsssssseeeeeu CooeeSuodeiieeee ssionnii sionoooicyyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NewrIPS code and data submission guidelines (https: / /nips. cc/ httpppublccc//c wwt.c//guiieeeeeee Code Submi ss Mooooooiiii
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
27

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: Yes, they are all discussed in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [NA]
Justification: The error bars are not applicable in our settings.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
They are all discussed in the appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
28

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /nnuripu...... cc.ppublcc/.thccccu..eeee
Answer: [NA]
Justification: It is not applicable to settings.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: They discussed in the appendix.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: is not applicable to our settings.
Guidelines:
The answer NA means that the paper poses such risks.
29

---

Released models that have high risk for misuse or dual-use use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite all the references.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, pape rawe thoode. com da se has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
It is not applicable to our settings.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: It is not applicable to settings
30

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
It is not applicable to our settings.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the L.M used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: It is not applicable to settings.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy (https: / ht////ruipp...p// wwt.rriss..cccceeeeee wwc/onofeennceeeeeceeeeeeee 2025/ LLM) for what should or should not be described.
31


Reviews:
[{"review_id": "EOIzXAxyX0", "reviewer": "Reviewer_ZUaC", "summary": "The paper addresses the critical challenge of performance degradation in diffusion models when fine-tuned on imbalanced datasets. The proposed PoGDiff constructs a Product of Gaussians (PoG) by combining the original target with predictions from neighboring text embeddings to enhance minority-class generation. Empirical validations on four datasets (AgeDB, DigiFace, VGGFace2, CIFAR-100-LT) show gains in FID, DINO similarity, human/GPT-4o scores, and gRecall over other baselines (Stable Diffusion, CBDM, T2H).", "questions": "(1) The derivation in Section 3.1 relies on the key assumption that the denoising posteriors are Gaussian. This is a strong simplification for the complexity of the data manifolds learned by Stable Diffusion. Could authors provide more discussion or even an empirical analysis on the robustness of PoGDiff? For example, if two very disparate concepts are composed together, do you observe failure modes like concept bleeding or artifact generation? Furthermore, it might be interesting to see how much PoGDiff defends against adversarial attacks.\n\n(2) Could PoGDiff be extended to larger datasets, like ImageNet-LT? \n\n(3) gRecall depends on a cosine similarity threshold (e.g., 0.7) to define \"correct images\". The sensitivity of gRecall to this threshold requires investigation. Moreover, Eq. incorporates hyper-parameters $\\alpha_1$, $\\alpha_2$, and $\\alpha_3$. What criteria determine the optimal values for these hyper-parameters? \n\n(4) As mentioned in Section F, PoGDiff employs full-parameter tuning. However, LoRA-based PEFT is more commonly employed for fine-tuning SD models. Are any experiments conducted under LoRA settings for comparison?\n\n(5) It is necessary to consider citing some reweighting methods in generative models [1-4]. \n\nReference:\n\n[1] Xie et al. Doremi: Optimizing data mixtures speeds up language model pretraining. NeurIPS, 2023.\n\n[2] Fan et al. DoGE: Domain Reweighting with Generalization Estimation. ICML, 2024.\n\n[3] Liu et al. RegMix: Data Mixture as Regression for Language Model Pre-training. ICLR, 2025.\n\n[4] Li et al. Pruning then Reweighting: Towards Data-Efficient Training of Diffusion Models. ICASSP, 2025.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n(1) The paper is overall well-structured with clear methodological statements. The derivation of the PoGDiff objective at Eq. 4 and its upper bound at Eq. 6 is sound, which leverages Gaussian properties to connect KL minimization to semantic consistency.\n\n(2) PoGDiff as a distributional replacement seems novel. The proposed gRecall metric measures diversity coverage of training instances.\n\nWeaknesses:\n\n(1) The potential need to train or fine-tune separate models for each composable concept is a major practical limitation. \n\n(2) Although the authors claim SD 1.5 as the first work to explore imbalanced T2I diffusion models, the generalizability of PoGDiff to other SD variants—such as SD 2.1, SD 3, or ControlNet—remains unclear. \n\n(3) While PoG demonstrates improved generation for specific minority classes, the paper fails to discuss the general performance after fine-tuning. For instance, LLM evaluations typically assess broad capabilities after fine-tuning; in this context, sacrificing core capabilities for marginal gains in less frequent categories may not be justified.", "was_revised": true, "final_justification": "The authors have made satisfactory efforts to address the reviewers' questions and conducted additional experiments to provide further clarification. After reading the rebuttal and discussing with other reviewers, I believe PoGDiff makes a notable contribution to imbalanced T2I generation. Therefore, I recommend acceptance.", "rebuttal": "Thank you for your constructive comments and insightful questions. We are glad that you found our method \"`novel`\", our paper \"`well-structured`\", and our derivation \"`sound`\". Below, we address your comments one by one in detail. \n\n**Q1: The potential need to train or fine-tune separate models for each composable concept is a major practical limitation.**\n\nWe apologize for the confusion. Our approach uses a single model to handle all composable concepts, rather than separate models for each. We will clarify this explicitly in the revision to prevent any future misunderstandings.\n\n**Q2: ... SD 1.5 ... other SD variants—such as SD 2.1, SD 3 ...**\n\nThank you for mentioning this. Our preliminary results show that different SD variants affect only low-level image quality (e.g., color, sharpness) but not accuracy. For simplicity, we therefore adopt the widely used SD 1.5 as our backbone. \n\nFollowing your suggestion, we run additional experiments on another SD variant, SD 2.1. Table A.1 below shows the results on AgeDB-small across five metrics. We can see that our method is robust across different SD variants. \n\n\nTable A.1: Results across five metrics in AgeDB-IT2I-small.\n| Model | FID (All) | FID (Few) | DINO (All) | DINO (Few) | Human (All) | Human (Few) | GPT (All) | GPT (Few) | gRecall (All) | gRecall (Few) |\n| :--: | :--: | :--: | :--: | :--: | :--: | :--: | :--: | :--: | :---: | :---: |\n|SD1.5| 14.88 | 13.72 | 0.42 | 0.37 | 0.50 | 0.00 | 5.20 | 3.20 | 0.017 | 0.000 |\n|PoGDiff (SD1.5)| **14.15** | **12.88** | **0.77** | **0.73** | **1.00** | **1.00** | **9.10** | **8.40** | **0.800** | **1.000** |\n|SD2.1| 15.02 | 13.97 | 0.40 | 0.32 | 0.50 | 0.00 | 5.60 | 3.40 | 0.017 | 0.000 | \n|PoGDiff (SD2.1)| **14.19** | **12.94** | **0.72** | **0.68** | **1.00** | **1.00** | **8.70** | **8.00** | **0.767** | **1.000** |\n||||||||||||\n\n**Q3: ...  fails to discuss the general performance after fine-tuning. For instance, LLM evaluations typically assess broad capabilities after fine-tuning; in this context, sacrificing core capabilities for marginal gains in less frequent categories may not be justified.**\n\nWe apologize for the confusion. Actually our PoGDiff did not sacrifice core capabilities. Specifically:\n+ Tables 1–5 in the main paper also report the `all-shot` performance to validate our model’s effectiveness in **both minority and majority** categories.\n+ The gRecall scores in Table 5 provide strong evidence of our PoGDiff's improvements in both accuracy and diversity across majority and minority categories.\n+ Figure 4 further shows that our method does not \"sacrifice core capabilities for marginal gains in less frequent categories may not be justified\".\n\nWe will include the discussion above in the revision as suggested.\n\n**Q4: The derivation in Section 3.1 relies on the key assumption that the denoising posteriors are Gaussian. This is a strong simplification for the complexity of the data manifolds learned by Stable Diffusion. Could authors provide more discussion or even an empirical analysis on the robustness of PoGDiff? For example, if two very disparate concepts are composed together, do you observe failure modes like concept bleeding or artifact generation? Furthermore, it might be interesting to see how much PoGDiff defends against adversarial attacks.**\n\nThis is a good question. Actually our results on AgeDB-small happens to provide such empirical analysis. As stated in Lines 208–210 of the main paper, AgeDB-IT2I-S(mall) \"`contains 32 images across 2 identities, where each majority class consists of 30 images and each minority class consists of 2 images.`\" In our experiments, the majority corresponds to Jeanette MacDonald and the minority to Einstein, **two very disparate concepts**. Results in Tables 1–5 show that our model still achieves reasonable accuracy and diversity. \n\nWe will include this discussion in the revision as suggested.\n\n**Q5: ... to larger datasets, like ImageNet-LT?**\n\nThank you for mentioning this. Our method scales linearly with the number of images. \n\nFollowing your suggestion, we ran additional rexperiments on ImageNet-LT. Table A.2 below shows the results across five metrics, verifying our PoGDiff's improvement upon the baseline. \n\nTable A.2: Results across five metrics in ImageNet-LT.\n| Model | FID (All) | FID (Few) | DINO (All) | DINO (Few) | Human (All) | Human (Few) | GPT (All) | GPT (Few) | gRecall (All) | gRecall (Few) |\n| :--: | :--: | :--: | :--: | :--: | :--: | :--: | :--: | :--: | :--: | :--: |\n|Vanilla| 7.28 | 11.96 | 0.46 | 0.31 | 0.70 | 0.28 | 5.80 | 3.00 | 0.207 | 0.143 |\n|PoGDiff| **6.35** | **9.57** | **0.72** | **0.57** | **0.88** | **0.70** | **8.50** | **8.20** | **0.437** | **0.571** |\n||||||||||||\n\n**Q6.1: gRecall depends on a cosine similarity threshold (e.g., 0.7) to define \"correct images\". The sensitivity of gRecall to this threshold requires investigation.**\n\nThis is a good point. Following your suggestion, we ran additional experiments with different thresholds and report the results in Tables A.3.a and A.3.b below. They show that our method consistently outperforms the baselines. Notably, while lowering the threshold improves the gRecall scores of the baselines, our model still achieves higher scores and demonstrates clear advantages. \n\nTable A.3.a: Results for gRecall in AgeDB-IT2I-small for all-shot metric.\n| model | threshold=0.8 | threshold=0.7 | threshold=0.6 | threshold=0.5 | threshold=0.4 |\n| :--: | :--: | :--: | :--: | :--: | :--: |\n|Vanilla| 0.000 | 0.017 | 0.067 | 0.150 | 0.717 |\n|CBDM| 0.100 | 0.267 | 0.283 | 0.817 | 0.883 |\n|T2H| 0.000 | 0.017 | 0.067 | 0.150 | 0.717 |\n|PoGDiff| **0.733** | **0.800** | **0.867** | **1.000** | **1.000** |\n||||||\n\nTable A.3.b: Results for gRecall in AgeDB-IT2I-small for few-shot metric.\n| model | threshold=0.8 | threshold=0.7 | threshold=0.6 | threshold=0.5 | threshold=0.4 |\n| :--: | :--: | :--: | :--: | :--: | :--: |\n|Vanilla| 0.000 | 0.000 | 0.000 | 0.000 | 0.500 |\n|CBDM| 0.000 | 0.000 | 0.000 | 0.500 | 0.500 |\n|T2H| 0.000 | 0.000 | 0.000 | 0.000 | 0.500 |\n|PoGDiff| **1.000** | **1.000** | **1.000** | **1.000** | **1.000** |\n||||||\n\nWe will include the results above in the revision as suggested. \n\n**Q6.2: Moreover, Eq. incorporates hyper-parameters a1, a2, a3. What criteria determine the optimal values for these hyper-parameters?**\n\nThank you for mentioning this. In all experiments, we simply set $a_1, a_2, a_3$ to $1$ without tuning them. We will clarify this in the revision as suggested. \n\n**Q7: LoRA-based PEFT is more commonly employed for fine-tuning SD models. Are any experiments conducted under LoRA settings for comparison?**\n\nGood question. For fair comparison, we apply full-parameter tuning to all models, including the baselines. We will clarify this in the revision as suggested.\n\n**Q8: It is necessary to consider citing some reweighting methods in generative models [1-4].**\n\nThank you for pointing us to these interesting papers. We will be sure to cite and discuss them [1-4] in the revision as suggested.\n\n[1] Xie et al. Doremi: Optimizing data mixtures speeds up language model pretraining. NeurIPS, 2023.\n\n[2] Fan et al. DoGE: Domain Reweighting with Generalization Estimation. ICML, 2024.\n\n[3] Liu et al. RegMix: Data Mixture as Regression for Language Model Pre-training. ICLR, 2025.\n\n[4] Li et al. Pruning then Reweighting: Towards Data-Efficient Training of Diffusion Models. ICASSP, 2025."}, {"review_id": "mPoxOftHJ4", "reviewer": "Reviewer_5MYg", "summary": "The paper introduces PoGDiff, a text-to-image diffusion fine-tuning scheme that replaces the usual single-Gaussian posterior with a product of Gaussians.\nOne Gaussian conditions on the target caption $y$, the other on a neighbour caption $y’$ drawn from a visually similar training image.\nA scalar weight $\\psi$—the product of an image-similarity term and an inverse caption-density term—scales an additional MSE term so that the model’s predicted noise for $y$ stays close to the noise it would predict for $y’$.\nThe method needs only two forward passes per training example and leaves the UNet and text encoder unchanged.\nExperiments cover three long-tailed face datasets (AgeDB-IT2I, DigiFace-IT2I, VGGFace-IT2I) plus CIFAR-100-LT.", "questions": "* Did you consider an inference-time baseline that conditions on both $y$ and $y’$, for example via weighted classifier-free guidance or prompt interpolation? How close does it get to PoGDiff’s minority-class fidelity, and what is the runtime overhead?\n\nMinor notes:\n* Figure 1, the visual gains are subtle; consider annotating key differences.\n* Repeated “more details below” breaks flow; smoother transitions would help readability.\n* Line 172 has a typo\n* Algorithm 1, line 7: “Generate $y’$” might be clearer as “Sample (or Select) $y’$”.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n* Conceptually simple yet novel – a clean KL upper-bound derivation turns the idea into a two-term MSE loss.\n* Implementation simplicity, just one extra forward pass; no architectural tweaks.\n* Well-written, clear main text, figures and extensive appendices; theory is easy to follow.\n* Thorough empirical study (within scope), multiple datasets, four metrics (FID, DINO, GPT-4o, human) and both all-shot / few-shot regimes; ablations in App. J test every loss component.\n\nWeaknesses\n* Limited domain breadth, almost all data are faces with simple captions (right?); it’s unclear whether PoGDiff helps with richer language or open-domain scenes.\n* No confidence intervals – tables report single numbers; bootstrapped CIs or run-to-run stdevs (e.g. in Table 2) would strengthen significance claims.\n* Theory gap for Fig. 3 – the “effective density” intuition is appealing but unproven; a formal link between the neighbour term and density smoothing would add weight.", "was_revised": true, "final_justification": "I like the product of Gaussians approach and the neighbor caption approach, and see no reason not to accept.", "rebuttal": "Thank you for your constructive comments and insightful questions. We are glad that you found our method \"`simple yet novel`\" with \"`implementation simplicity`\", our paper \"`well-written`\", our theory \"`easy to follow`\", and our empirical study \"`thorough`\". Below, we address your comments one by one in detail. \n\n**Q1: Limited domain breadth, almost all data are faces with simple captions (right?); it’s unclear whether PoGDiff helps with richer language or open-domain scenes.**\n\nThank you for mentioning this. Actually our experiments are not limited to facial datasets; we also include general-domain data such as CIFAR-100-LT. The captions used are simple (e.g., one long sentence). We agree that extending our method to handle richer language or open-domain scenes would be interesting future work. \n\n\n**Q2: No confidence intervals – tables report single numbers; bootstrapped CIs or run-to-run stdevs (e.g. in Table 2) would strengthen significance claims.**\n\nThis is a good suggestion. We did not include confidence intervals because most baseline papers [1,2] include them.\n\nFollowing your suggestion, we compute and report these confidence intervals in Table B.1 below, verifying the significance of our improvement upon baselines. We will include them in the appendix of the revision as suggested. \n\nTable B.1: Results across metrics with stdev in AgeDB-IT2I-small.\n\n| Model | FID (All) | FID (Few) | DINO (All) | DINO (Few) |\n| :--: | :--: | :--: | :--: | :--: | \n|Vanilla| 14.88 $\\pm$ 0.1 | 13.72 $\\pm$ 0.14 | 0.42 $\\pm$ 0.06 | 0.37 $\\pm$ 0.04 |\n|PoGDiff| **14.15 $\\pm$ 0.12** | **12.88 $\\pm$ 0.15** | **0.77 $\\pm$ 0.04** | **0.73 $\\pm$ 0.03** | \n||||||\n\n[1] Qin et al. Class-Balancing Diffusion Models. CVPR 2023.\n\n[2] Zhang et al. Long-tailed diffusion models with oriented calibration. ICLR 2024\n\n**Q3: Theory gap for Fig. 3 – the “effective density” intuition is appealing but unproven; a formal link between the neighbour term and density smoothing would add weight.**\n\nWe are sorry for the confusion. Below we discuss the main idea of the theory behind effective density. \n\nIn general, we have that the generalization error is bounded with probability $1-\\eta$ by: test error $\\leq$ training error + bias + variance, where \n\nbias $=\\frac{\\Delta}{N} \\sum_{(x,y)} |1 - \\frac{P_{y}}{\\hat{P}_{y}}|$, \n\nvariance $=\\frac{\\Delta}{N} \\sqrt{\\frac{\\log ( 2 |{H}| / \\eta)}{2}} +  \\sqrt{\\sum_{(x,y)}(\\hat{P}_{y})^{-2}}$. \n\nHere \n\n(1) $\\hat{P}_{y}$ is the smoothed label (text) distribution used in our PoGDiff's objective function (e.g., it can be estimated by $E^{ELBO(y)$ in Eqn. 10 of the paper), \n\n(2) $P_{y}$ is the label (text) distribution,  \n\n(3) $N$ is the number of data points, \n\n(4) $\\Delta=y_{max}-y_{min}$,  where $y_{max}$ and $y_{min}$ are the maximum and minimum labels (in terms of text embeddings in PoGDiff) in the dataset, respectively, and \n\n(5) $\\mathcal{H}$ is the finite hypothesis space of prediction models. \n\nWe can see that if one directly uses the original label distribution in the training objective function, i.e., $\\hat{P}_{y}=P_y$: \n\n(a) The \"bias\" term will be $0$. \n\n(b) However, the \"variance\" term will be extremely large for minority data because $\\hat{P}_{y}$ is very close to $0$. \n\nIn contrast, after our reweighting, $\\hat{P}_{y}$ used in the training objective function will be smoothed. Therefore:\n\n(a) The minority data's label density $\\hat{P}_{y}$ will be smoothed out by its neighbors and becomes larger (compared to the original $P_y$), leading to smaller \"variance\" in the generalization error bound. \n\n(b) Note that $\\hat{P}_{y}\\neq P_y$, PoGDiff essentially increases bias, but significantly reduces its variance in the imbalanced setting, thereby leading to a lower generalization error. \n\nWe will include the discussion above and provide a rigorous proof in the revision as suggested.\n\n**Q4: Did you consider an inference-time baseline that conditions on both $y$ and $y'$, for example via weighted classifier-free guidance or prompt interpolation? How close does it get to PoGDiff’s minority-class fidelity, and what is the runtime overhead?**\n\nThis is a good suggestion. Folloing your suggestion, we added the suggested baseline, dubbed inference-time prompt interpolation (ITPI). \n\n**Baseline Description.** During inference, for a given text embedding $y$, we locate its neighbor $y'$, then perform prompt interpolation by $\\hat{y} = \\gamma y + (1 - \\gamma) y'$, where $\\gamma \\in [0, 1]$ is a balacing factor. Table B.2 reports results for $\\gamma = 0.2, 0.5, 0.8$, respectively. \n\n**Results.** This baseline performs similarly, or even worse, compared to the vanilla model, as directly manipulating the text embedding space with a simple linear direction harms generation quality. Furthermore, in AgeDB-IT2I-small, neighboring prompts are short and highly similar, leading to minimal differene after interpolation; we expect that using more complex prompts would further harm this baseline’s performance, which we consider an interesting direction for future work. \n\nTable B.2: Results for inference-time prompt interpolation (ITPI) in AgeDB-IT2I-small.\n\n| Model | FID (All) | FID (Few) | DINO (All) | DINO (Few) | Human (All) | Human (Few) | GPT (All) | GPT (Few) | gRecall (All) | gRecall (Few) |\n| :--: | :--: | :--: | :--: | :--: | :--: | :--: | :--: | :--: | :--: | :--: |\n|Vanilla| 14.88 | 13.72 | 0.42 | 0.37 | 0.50 | 0.00 | 5.20 | 3.20 | 0.017 | 0.000 |\n|ITPI (0.2)| 14.93 | 13.76 | 0.43 | 0.35 | 0.50 | 0.00 | 5.40 | 3.10 | 0.017 | 0.000 |\n|ITPI (0.5)| 15.02 | 13.85 | 0.41 | 0.29 | 0.50 | 0.00 | 5.00 | 2.80 | 0.017 | 0.000|\n|ITPI (0.8)| 14.96 | 13.82 | 0.43 | 0.34 | 0.50 | 0.00 | 5.20 | 3.00 | 0.017 | 0.000 |\n|PoGDiff| **14.15** | **12.88** | **0.77** | **0.73** | **1.00** | **1.00** | **9.10** | **8.40** | **0.800** | **1.000** |\n||||||||||||\n\nThe runtime of this baseline is approximately the same as that of the vanilla model and our PoGDiff during inference.\n\nWe will include the results and discussion above in the revision as suggested.\n\n**Q5: Minor notes**\n\nWe apologize for the typo and thank you for the suggestion. We will include all these changes in the revision as suggested."}, {"review_id": "LM0JzCYSNh", "reviewer": "Reviewer_fG9E", "summary": "This paper aims to enhance the performance of diffusion models when trained or fine-tuned on imbalanced datasets. Instead of relying on a single prompt, the authors align one image with multiple group prompts sampled from the training data. To achieve this, they employ a Product of Gaussians technique. Experimental results show that PoGDiff outperforms traditional methods like Stable Diffusion and Class Balancing Diffusion Model (CBDM) across various datasets, particularly enhancing generation for minority classes.", "questions": "1. The model faces the less diverse image given similar text prompts with the same image. How can this drawback be fixed?\n2. The paper states that in diffusion models, a data point's output is solely determined by its text embedding. However, this claim is complicated by the observation that even with identical text embeddings, different latent codes can produce images of varying quality. Furthermore, the influence of techniques like classifier-free guidance and the use of negative prompts undeniably plays a role in shaping the final image generation, suggesting that the process is not as singularly driven by text embeddings as initially presented.\n3. The paper highlights a common issue in diffusion models trained on imbalanced datasets: their struggle to accurately generate images for less frequent individuals. In contrast, personalized methods, such as PhotoMaker, effectively address this by learning a specific identity from just 3 to 5 images, allowing them to produce accurate representations of these underrepresented individuals. The key difference between PoGDiff and these personalized methods lies in whether PoGDiff also employs a similar small-sample identity learning approach, or if it tackles the imbalance problem through a different mechanism entirely.\n4. Upon reviewing Table 1, the proposed method does not demonstrate a significant advantage when compared to the existing baselines. This lack of substantial improvement leaves me unconvinced about the method's overall effectiveness. For a new approach to be truly compelling, it typically needs to show clear and measurable gains over established techniques, which is not evident from the presented data.\n5. The visualizations do not clearly support the proposed method. Could the author provide more results?", "limitations": "See above questions.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "1. PoGDiff introduces the use of Gaussian products for fine-tuning in imbalanced datasets, an original approach that improves minority class image generation.\n\n2. The paper provides theoretical analysis, showing that PoGDiff retains diffusion model properties while better representing minority classes.\n\n3. The authors provide theoretical support for their proposed method, which is intriguing.\n\n4. Extensive experiments are provided to support this idea.\n\n5. The paper presents good insights into the proposed method.", "was_revised": true, "final_justification": "Thanks for the rebuttal. Authors address most of my questions. I keep my score.", "rebuttal": "Thank you for your constructive comments and insightful questions. We are glad that you found our theoretical support \"`intriguing`\", our experiments \"`extensive`\", and that our paper presents \"`good insights`\". Below, we address your comments one by one in detail. \n\n**Q1: The model faces the less diverse image given similar text prompts with the same image. How can this drawback be fixed?**\n\nThis is a good question. \n\n**Qualitative Diversity Evaluation:** Our PoGDiff's genenerated images prioritize both **diversity** and **accuracy**. Note that in **Fig. 5**, while baselines like SDv1.5 and CBDM appear to generate diverse images, most images are **wrong**. For example, the Column 1 of CBDM contains diverse images, but **all images are wrong**; none of them are images of `Einstein` (Column 1 of the GT section in Fig. 5). In contrast, our PoGDiff generates images that are both **diverse** (both grayscale and colored images of Einstein of different ages) and **accurate**. \n\n**Quantitative Diversity Metric:** We also introduce a new metric **gRecall**, specifically for evaluating **diversity**. It measures a model’s ability to generate **diverse** yet **class-consistent** samples. gRecall will be higher if the method covers more images of the correct individual (e.g., Einstein) in the test set. Unlike traditional metrics such as FID, gRecall captures intra-class diversity while penalizing the wrong generation accuracy. Results in Table 5 verify that our PoGDiff's high diversity in terms of gRecall. \n\nWe will include the discussion above in the revision as suggested.\n\n**Q2: The paper states that in diffusion models, a data point's output is solely determined by its text embedding. However, this claim is complicated by the observation that even with identical text embeddings, different latent codes can produce images of varying quality. Furthermore, the influence of techniques like classifier-free guidance and the use of negative prompts undeniably plays a role in shaping the final image generation, suggesting that the process is not as singularly driven by text embeddings as initially presented.**\n\nWe apologize for the wording. We will revise “solely” to “mainly” to make the description more precise; this change does not affect our conclusions. To address varying image quality during evaluation, we generate multiple images per prompt and report the average performance. Moreover, we apply classifier-free guidance consistently across all methods and do not use negative prompt techniques, ensuring a fair comparison. We will add these clarifications in the revision as suggested.\n\n**Q3: The paper highlights a common issue in diffusion models trained on imbalanced datasets: their struggle to accurately generate images for less frequent individuals. In contrast, personalized methods, such as PhotoMaker, effectively address this by learning a specific identity from just 3 to 5 images, allowing them to produce accurate representations of these underrepresented individuals. The key difference between PoGDiff and these personalized methods lies in whether PoGDiff also employs a similar small-sample identity learning approach, or if it tackles the imbalance problem through a different mechanism entirely.**\n\nThank you for pointing us to personalized methods such as PhotoMaker. We will cite and discuss them in the revision. We would also like to clarify that our paper addresses a different setting from approaches like PhotoMaker. \n\n**Different Setting from Personalized Techniques like PhotoMaker.** Previous works like PhotoMaker focuses on adjusting the model to generate images with **a single object**, e.g., a specific dog. In contrast, our PoGDiff focuses finetuning the diffusion model on an entire data with **many different objects/persons simultaneously**. They are **very different settings** and are **complementary** to each other. \n\n**Q4: Upon reviewing Table 1, the proposed method does not demonstrate a significant advantage when compared to the existing baselines. This lack of substantial improvement leaves me unconvinced about the method's overall effectiveness. For a new approach to be truly compelling, it typically needs to show clear and measurable gains over established techniques, which is not evident from the presented data.**\n\nWe apologize for the confusion. Our PoGDiff significantly outperforms all baselines in the four out of five metrics in the paper. Even in terms the remaining metric, FID, PoGDiff can achieve relative improvements of up to $13\\%$. \n\nNote that FID is not a good metric in our setting, and we provided a detailed discussion in **Appendix J.5** on its limitations. \n\n**When our PoGDiff significantly outperforms the baselines, FID fails to capture such improvements because it depends only on the mean and variance of the distribution, losing a lot of information during evaluation.**\n\nThis is why, in addition to FID, we also report other metrics, such as DINO score, human evaluation, GPT score, and, most importantly, gRecall, to further demonstrate the effectiveness of our model.\n\n**Q5: The visualizations do not clearly support the proposed method. Could the author provide more results?**\n\nThank you for mentioning this. Additional results are provided in Figures 6 and 7 of the Appendix."}, {"review_id": "tw9In8vIMv", "reviewer": "Reviewer_CaQC", "summary": "The paper tackles the practical challenge of unbalanced datasets in diffusion models. It proposes a novel method called PoGDiff, designed to mitigate the distribution gap by minimizing the KL divergence between the predicted distribution and a Product of Gaussians—a formulation that combines ground-truth targets with the predicted distribution conditioned on a neighboring text embedding. Experimental results demonstrate the strong performance of the proposed approach.", "questions": "### Question 1\nCan you evaluate PoGDiff on diffusion models other than Stable Diffusion 1.5? How well does it generalize to other diffusion backbones?\n\n### Question 2\nIn L235, you mention that T2H is not applicable because it requires class frequency. However, the datasets described in L203–L218 have class labels. Although you generate text captions for these images, why not use the original class labels?\n\n### Question 3\nYou define image similarity in L160. I suggest adding a simple baseline that rebalances the training samples based on image similarity. Specifically, compute a similarity matrix and define the similarity score of each image by summing its corresponding row. Images with lower similarity scores are more likely to be minorities. During training, you can resample images inversely proportional to their similarity scores—i.e., the higher the score, the lower the sampling probability.\n\n### Question 4\nIn line 263, for each minority class, only 10 images are sampled. Please increase it to a larger number, like 100, to make the results more solid.\n\n### Question 5\nCan you do some ablation studies on the threshold of gRecall?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "### Strengths\n1. The paper addresses a practical and common issue in the text-to-image community.\n2. Figure 3 provides an intuitive illustration of the model's effectiveness.\n3. The theoretical analysis is insightful.\n4. The use of diverse evaluation metrics strengthens the validity of the work.\n\n### Weaknesses\nSee Questions.", "was_revised": true, "final_justification": "The authors propose a solid solution to imbalanced t2i generation, and give a theoretical analysis. As a result, I suggest acceptence.", "rebuttal": "Thank you for your constructive comments and insightful questions. We are glad that you found the problem we address \"`practical and common`\", our theoretical analysis \"`insightful`\", our illustration of the model \"`intuitive`\", and that our evaluation \"`strengthens the validity of the work`\". Below, we address your comments one by one in detail. \n\n**Q1: Can you evaluate PoGDiff on diffusion models other than Stable Diffusion 1.5? How well does it generalize to other diffusion backbones?**\n\nThank you for mentioning this. Our preliminary results show that different SD variants affect only low-level image quality (e.g., color, sharpness) but not accuracy. For simplicity, we therefore adopt the widely used SD 1.5 as our backbone. \n\nFollowing your suggestion, we run additional experiments on another SD variant, SD 2.1. Table D.1 below shows the results on AgeDB-small across five metrics. We can see that our method is robust across different SD variants. \n\nTable D.1: Results across five metrics in AgeDB-IT2I-small.\n| Model | FID (All) | FID (Few) | DINO (All) | DINO (Few) | Human (All) | Human (Few) | GPT (All) | GPT (Few) | gRecall (All) | gRecall (Few) |\n| :--: | :--: | :--: | :--: | :--: | :--: | :--: | :--: | :--: | :---: | :---: |\n|SD1.5| 14.88 | 13.72 | 0.42 | 0.37 | 0.50 | 0.00 | 5.20 | 3.20 | 0.017 | 0.000 |\n|PoGDiff (SD1.5)| **14.15** | **12.88** | **0.77** | **0.73** | **1.00** | **1.00** | **9.10** | **8.40** | **0.800** | **1.000** |\n|SD2.1| 15.02 | 13.97 | 0.40 | 0.32 | 0.50 | 0.00 | 5.60 | 3.40 | 0.017 | 0.000 | \n|PoGDiff (SD2.1)| **14.19** | **12.94** | **0.72** | **0.68** | **1.00** | **1.00** | **8.70** | **8.00** | **0.767** | **1.000** |\n||||||||||||\n\n**Q2: In L235, you mention that T2H is not applicable because it requires class frequency. However, the datasets described in L203–L218 have class labels. Although you generate text captions for these images, why not use the original class labels?**\n\nThis is a good question. In our setting, the input is free-form text and the output is an image; therefore, for fair comparison, class frequencies are not available. We will include the clarification above in the revision as suggested. \n\n**Q3: You define image similarity in L160. I suggest adding a simple baseline that rebalances the training samples based on image similarity. Specifically, compute a similarity matrix and define the similarity score of each image by summing its corresponding row. Images with lower similarity scores are more likely to be minorities. During training, you can resample images inversely proportional to their similarity scores—i.e., the higher the score, the lower the sampling probability.**\n\nThis is a good suggestion. Following your suggestion, we ran additional experiments with this baseline. Table D.2 shows the results on the AgeDB-small dataset. We can see that the new baseline has similar (sometimes slightly better) performance compared to the vanilla baseline. Our PoGDiff still significantly outperforms them both, demonstrating it effectiveness. \n\nTable D.2: Results for Rebalanced-Vanilla model in AgeDB-IT2I-small.\n| Model | FID (All) | FID (Few) | DINO (All) | DINO (Few) | Human (All) | Human (Few) | GPT (All) | GPT (Few) | gRecall (All) | gRecall (Few) |\n| :--: | :--: | :--: | :--: | :--: | :--: | :--: | :--: | :--: | :--: | :--: |\n|Vanilla| 14.88 | 13.72 | 0.42 | 0.37 | 0.50 | 0.00 | 5.20 | 3.20 | 0.017 | 0.000 |\n|Rebalanced| 14.72 | 13.32 | 0.49 | 0.39 | 0.50 | 0.00 | 5.40 | 3.60 | 0.050 | 0.000 |\n|PoGDiff| **14.15** | **12.88** | **0.77** | **0.73** | **1.00** | **1.00** | **9.10** | **8.40** | **0.800** | **1.000** |\n||||||||||||\n\n**Q4: In line 263, for each minority class, only 10 images are sampled. Please increase it to a larger number, like 100, to make the results more solid.**\n\nWe apologize for the confusion. For each minority class, we did sample 100 images rather than 10 ( \"`we use 10 different seeds to sample 10 images`\", as stated in Line 263). For the human evaluation metric, we then sampled 10 images out of these 100 due to the high cost of human evaluation. We will include the clarification above in the revision as suggested. \n\n**Q5: Can you do some ablation studies on the threshold of gRecall?**\n\nThis is a good point. Following your suggestion, we ran additional experiments with different thresholds and report the results in Tables D.3.a and D.3.b below. They show that our method consistently outperforms the baselines. Notably, while lowering the threshold improves the gRecall scores of the baselines, our model still achieves higher scores and demonstrates clear advantages. \n\n\nTable D.3.a: Results for gRecall in AgeDB-IT2I-small for all-shot metric.\n| model | threshold=0.8 | threshold=0.7 | threshold=0.6 | threshold=0.5 | threshold=0.4 |\n| :--: | :--: | :--: | :--: | :--: | :--: |\n|Vanilla| 0.000 | 0.017 | 0.067 | 0.150 | 0.717 |\n|CBDM| 0.100 | 0.267 | 0.283 | 0.817 | 0.883 |\n|T2H| 0.000 | 0.017 | 0.067 | 0.150 | 0.717 |\n|PoGDiff| **0.733** | **0.800** | **0.867** | **1.000** | **1.000** |\n||||||\n\nTable D.3.b: Results for gRecall in AgeDB-IT2I-small for few-shot metric.\n| model | threshold=0.8 | threshold=0.7 | threshold=0.6 | threshold=0.5 | threshold=0.4 |\n| :--: | :--: | :--: | :--: | :--: | :--: |\n|Vanilla| 0.000 | 0.000 | 0.000 | 0.000 | 0.500 |\n|CBDM| 0.000 | 0.000 | 0.000 | 0.500 | 0.500 |\n|T2H| 0.000 | 0.000 | 0.000 | 0.000 | 0.500 |\n|PoGDiff| **1.000** | **1.000** | **1.000** | **1.000** | **1.000** |\n||||||\n\nWe will include the results above in the revision as suggested."}]

Author rebuttal:

Meta-review: This paper proposes PoGDiff, a product-of-Gaussians diffusion model for imbalanced T2I generation. The main contribution is to address the performance degradation that arises when training or fine-tuning on imbalanced datasets. The authors minimize the KL divergence between the model and a target distribution that combines the ground-truth with the predictive distribution conditioned on neighboring text embeddings. The empirical results provide strong support for the core claim.

Overall, the reviewers are satisfied with the novelty, quality, and significance of the contribution. I share this view, as the motivation for adapting the PoG framework and the derivation of the PoGDiff objective are well-grounded and technically solid, ultimately leading to notable empirical improvements. While there were some initial concerns about the experimental setup—for instance, the method being demonstrated on older models—these were largely resolved during the rebuttal.

Hence, I recommend acceptance.
Decision: Accept (poster)