PoGDiff: Product-of-Gaussians Diffusion Models for Imbalanced Text-to-Image Generation

Authors: Ziyan Wang, Sizhe Wei, Xiaoming Huo, Hao Wang
Primary area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords: Diffusion Model, Probabilistic Methods

Abstract:
Diffusion models have made significant advancements in recent years. However, their performance often deteriorates when trained or fine-tuned on imbalanced datasets. This degradation is largely due to the disproportionate representation of majority and minority data in image-text pairs. In this paper, we propose a general fine-tuning approach, dubbed PoGDiff, to address this challenge. Rather than directly minimizing the KL divergence between the predicted and ground-truth distributions, PoGDiff replaces the ground-truth distribution with a Product of Gaussians (PoG), which is constructed by combining the original ground-truth targets with the predicted distribution conditioned on a neighboring text embedding. Experiments on real-world datasets demonstrate that our method effectively addresses the imbalance problem in diffusion models, improving both generation accuracy and quality.

Decision: reject
Reviewer scores: 5, 6, 5, 6

Reviews and discussion:
<thread>
creation date:
1730205791996

thread_type:
official_review

<thread_object>
creator: reviewer_zuxr

<content>
summary:
The paper proposes a novel method PoGDiff to address the long-tailed data distributions cause by the imbalanced datasets. This paper proposes a general fine-tuning approach, replacing the ground-truth distribution with a Product of Gaussians conditioned on a neighboring text embedding. Experiments are conducted on AgeDB-IT2I and DigiFace-IT2I using FID, DINO, Human Score and GPT-4o Evaluation evaluation metrics.

strengths:
1.	Selecting neighboring embeddings from other samples is an interesting approach, as it can help get a new density.
2.	Figure 3 is interesting, but it would benefit from a more detailed explanation.

weaknesses:
1.	“Encouraging the model to generate the same image given similar text prompts” may result in a loss of diversity in the generated images. How can this drawback be overcome?
2.	The paper mentions that in diffusion models, a data point is affected only by its text embedding. However, even with the same text embedding, different latent codes can produce images of varying quality. Additionally, classifier-free guidance and negative prompts also influence image generation. 
3.	Why is directly smoothing the text embedding not feasible?
4.	What is the basis for hypothetically defining $\sigma_{y'}^2 = \frac{\sigma_{t}^2}{\psi[(x,y), (x',y')]}$ ?
5.	What does ‘Cat’ refer to in line 249? It doesn’t seem to be explained in the paper.  The author should define or explain this term when it's first introduced
6.	What does the superscript of s in Equation 9 represent? The previous definition of s did not include a superscript (e.g., Equation 8).

questions:
1.	Artifacts from PoGDiff appear to be present in the images generated at low density (e.g., Figure 1, lower left corner, J. Willard Marriott), but not in those generated at high density. Is this a result of the model's limitations?
2.	The paper mentions that when training a diffusion model on an imbalanced dataset, existing models often struggle to generate accurate images for less frequent individuals. Personalized methods (e.g., CustomDiffusion, PhotoMaker) can use 3 to 5 images to learn an identity and generate accurate images for these less frequent individuals. What is the difference between PoGDiff and personalized methods that learn a specific identity?
CustomDiffusion: Multi-Concept Customization of Text-to-Image Diffusion
PhotoMaker: Customizing Realistic Human Photos via Stacked ID Embedding
3.	How to obtain the ground-truth distribution $q(x_{t-1}|x_t, x_0,y)$, when given $x_t$, $x_0$, and $y$ ?
4.	The $y^{'}$ in line 167 and the $y^{'}$in line 169 should be the same symbol.
5.	Fig. 3 is interesting, but the type and amount of data used in Fig. 3 is quite confusing to me.
6.	Equation 7 neglects $y^{'}$.
7.	Does the distance between the current text embedding $y$ and the sampled $y^{'}$ significantly affect the final generated results?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730282960997

thread_type:
official_review

<thread_object>
creator: reviewer_zjvi

<content>
summary:
This paper aims to enhance the performance of diffusion models when trained or fine-tuned on imbalanced datasets. Instead of relying on a single prompt, the authors align one image with multiple group prompts sampled from the training data. To achieve this, they employ a Product of Gaussians technique. The authors conduct various experiments to demonstrate that their method is effective.

strengths:
1. The authors introduce the Product of Gaussians technique to sample mixed prompts, which alleviates the need for numerous small image-text pairs in the training dataset. This technique allows one image to interact with more prompts, thereby increasing generative potential.
   
2. The approach of using multiple texts to represent a single image is a reasonable consideration.

3. The authors provide theoretical support for their proposed method, which is intriguing.

4. Extensive experiments are conducted to support this idea.

5. The paper presents valuable insights into the proposed methodology.

weaknesses:
1. The authors consider the use of non-target prompts for the current image, which may introduce noise and misalignment. This could result in generated images that do not align well with the prompts, potentially leading to lower CLIP scores, a metric that is not reported in the paper. Thus, there may be issues with text-image alignment.

2. The baseline comparisons are limited, focusing only on SD and CBDM, which may not be sufficient to fully validate the proposed idea.

3. As shown in Table 1, the proposed method does not demonstrate a significant advantage compared to the baselines, leaving me unconvinced about its effectiveness.

4. The visualizations do not clearly support the proposed method. For instance, Figure 5 reveals a lack of diversity in the generated outputs.

questions:
see Weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730377538236

thread_type:
official_review

<thread_object>
creator: reviewer_pys7

<content>
summary:
This paper presents Product-of-Gaussians Diffusion Models, a approach to fine-tuning diffusion models for imbalanced text-to-image generation. PoGDiff addresses the challenges of generating minority class images by using a product of Gaussians (PoG) to combine original target distributions with nearby text embeddings. Experimental results show that PoGDiff outperforms traditional methods like Stable Diffusion and Class Balancing Diffusion Model (CBDM) across various datasets, particularly enhancing generation for minority classes.

strengths:
1. PoGDiff introduces the use of Gaussian products for fine-tuning in imbalanced datasets, an original approach that improves minority class image generation.

2. The paper provides theoretical analysis, showing that PoGDiff retains diffusion model properties while better representing minority classes.

weaknesses:
1. Experiments are primarily conducted on AgeDB-IT2I and DigiFace-IT2I, which may not fully represent real-world, large-scale imbalanced datasets. Additional testing on broader datasets is necessary.

2. PoGDiff relies on neighboring samples for minority class improvement, which may be less effective in sparse data settings. There is a lack of discussion on how the model handles extremely sparse data.

questions:
see weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730776597440

thread_type:
official_review

<thread_object>
creator: reviewer_yw9m

<content>
summary:
This paper argues that current Diffusion models are trained on imbalanced datasets.
To solve this problem, they propose a fine-tuning framework, PoGDiff.
PoGDiff replaces the ground-truth distribution with a Product of Gaussians (PoG), which is constructed by combining the original ground-truth targets with the predicted distribution conditioned on a neighboring text embedding.
Experiments show that PoGDiff effectively addresses the imbalance problem in diffusion models, improving both generations' accuracy and quality.

strengths:
This paper addresses the issue of imbalanced training data in diffusion models and proposes a novel fine-tuning method. The core idea is to modify the ground-truth image supervision signal during training by incorporating neighboring text embeddings. 
1) The issue addressed in this paper is valuable and important. 
2) The proposed solution appears to be reasonable. 
3) The writing and analytical approach of the paper are clear. 
4) The experiments also demonstrate the effectiveness of the proposed method.

weaknesses:
Although the approach of PoGDiff is reasonable and effective, and I understand that the addition of text embeddings can increase the diversity of the supervision signal, I still have the following concerns: 1) From the results shown in Figure 1, some of the images generated by PoGDiff exhibit noticeable deviations in color and other aspects from the ground truth (GT). Does this modification align with the expected outcomes? 2) There are already several custom techniques that can achieve diversity with just a single or a few new style images, and in some cases, without any training. The fine-tuning method proposed by the authors might degrade the performance of the original model. How should we evaluate this? 3) The proposed method essentially resembles data re-weighting, yet the experiments lack comparisons and detailed analyses with similar methods.

questions:
see weakness
</content>
</thread_object>

</thread>
<thread>
creation date:
1732430746993

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Response

comment:
We thank all reviewers for their valuable comments.  

We are glad that the reviewers found our method ``"novel"``/``"original"``/``"reasonable"``/``"interesting"`` (yW9M, pyS7, zjVi, ZUxR), the problem we addressed ``"valuable and important"`` (yW9M), our theoretical analysis ``"valuable"``/``"intriguing"`` (zjVi, yW9M), our writing ``"clear"`` (yW9M), and that our experiments ``"demonstrate the effectiveness of the proposed method"`` (yW9M), are ``"extensive"`` (zjVi), and show that our method ``"outperforms traditional methods"`` (pyS7). 

Below we address the reviewers' questions one by one in detail. We have cited all related references and **included all discussions/results below in our revision** (with the changed part marked in blue).
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces PoGDiff, a fine-tuning method for improving the performance of diffusion models on imbalanced datasets. By aligning each image with multiple group prompts using a Product of Gaussians approach, the method replaces the ground-truth distribution with one conditioned on neighboring text embeddings. 

The introduction, related work, and technical sections of this manuscript all focus on addressing the general problem of imbalanced datasets. The proposed technique is sufficiently generic and can be applied to a wide range of scenarios, rather than being limited to the current experiments designed specifically for facial images. From this perspective, the current experimental setup restricts the applicability of the algorithm and limits the potential comparison methods and benchmark experiments for the proposed approach.