Title: Adapting Diffusion Models for Improved Prompt Compliance and Controllable Image Synthesis
Venue: NeurIPS 2024
Accepted: yes
Keywords: Image Synthesis; Controllable 2D/3D Synthesis; Diffusion
Abstract: Recent advances in generative modeling with diffusion processes (DPs) enabled breakthroughs in image synthesis. Despite impressive image quality, these models have various prompt compliance problems, including low recall in generating multiple objects, difficulty in generating text in images, and meeting constraints like object locations and pose. For fine-grained editing and manipulation, they also require fine-grained semantic or instance maps that are tedious to produce manually. While prompt compliance can be enhanced by addition of loss functions at inference, this is time consuming and does not scale to complex scenes. 
  To overcome these limitations, this work introduces a new family of  $\textit{Factor Graph Diffusion Models}$ (FG-DMs) that models the joint distribution of images and conditioning variables, such as semantic, sketch, depth or normal maps via a factor graph decomposition. This joint structure has several advantages, including support for efficient sampling based prompt compliance schemes, which produce images of high object recall, semi-automated fine-grained editing, explainability at intermediate levels, ability to produce labeled datasets for the training of downstream models such as segmentation or depth, training with missing data, and continual learning where new conditioning variables can be added with minimal or no modifications to the existing structure. We propose an implementation of FG-DMs by adapting a pre-trained Stable Diffusion (SD) model to implement all FG-DM factors, using only COCO dataset, and show that it is effective in generating images with 15\% higher recall than SD while retaining its generalization ability. We introduce an attention distillation loss that encourages consistency among the attention maps of all factors, improving the fidelity of the generated conditions and image. We also show that training FG-DMs from scratch on MM-CelebA-HQ, Cityscapes, ADE20K, and COCO produce images of high quality (FID) and diversity (LPIPS).


Peer reviews:
comments: SUMMARY:
This paper propose a new family of diffusion model: FG-DM for better prompt compliance. Unlike traditional diffusion model, FG-DM models the joint distribution of images and conditioning variables like semantic, sketch, depth or normal maps via a factor graph. These extra factors largely boost prompt compliance compared to use text only. Users can also easily edit target image by modifying intermediate conditions. This work shows better results compared to vanilla diffusion.

STRENGTHS:
By decomposing the text into different factors such as seg maps, depth maps, etc., and using these factors as conditions for generation, this method achieves better prompt compliance. Additionally, the introduction of intermediate factors makes this method easier to edit.

WEAKNESSES:
Since intermediate factors are generated sequentially, these method need more time for generation. The novelty of this method is limited.

QUESTIONS:
1) How much more time do we need for the conditional generation chains? (for example, how much more time do we need when we use one or two conditions compared to original SD?)
2) If we use an off-the-shelf condition generator e.g. a text-to-depth map generator, and we fed the generated depth map to controlnet for generation, what is the advantage of our method?

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-09

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a method for sequentially generating images using a frozen SD. Starting from a text prompt, the process iteratively generates several visual conditions and images, with each step depending on the previous ones. The model uses the VAE in SD to encode and decode visual conditions and employs an condition-specific adapter to introduce information into the SD for generating these conditions. This approach makes the image generation process more controllable, flexible, and easier to edit.\n\nSTRENGTHS:\n1. The iterative generation method using a stable diffusion model is impressive. It provides strong control over the output while maintaining flexibility, aided by classifier-free guidance training.\n2. The new method allows for flexible editing by modifying visual conditions and letting the model generate realistic content based on these changes.\n\nWEAKNESSES:\n1. The newly introduced iterative process adds extra loop to each generation step.\n2. Some components and experimental details are not well-explained.\n\nQUESTIONS:\n1. The assumption behind attention distillation suggests that word tokens and visual regions remain constant, even if the visual regions are non-semantic, like poses. Why does this happen? It seems odd that the attention score of a general word like \"person\" can help generate the poses of a person.\n2. Why do you use independent noise for different factors?\n3. The experimental details are unclear. Does the model run with a single visual condition for each result, or does it generate all conditions sequentially? It would help to clearly present the generation process for each experiment, including Tables 1-5 and Figures 5-7.\n4. How do you implement null conditions for different variables?\n5. You claim that after choosing the best segmentation factor, the image synthesis only needs to run once. Why is this? Does the segmentation factor determine the quality of the generated images, or is your goal just to ensure the object appears in the generated image?\n\nLIMITATIONS:\nThe authors have  adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-11"}

comments: SUMMARY:
The paper proposes unified generation framework of simultaneously generate image, segmap. depthmap, etc.

STRENGTHS:
The idea of simultaneous generation of images with various conditioning maps is very interesting.

WEAKNESSES:
1. Although the method shows good performance in image editing for generated images, it does not include any discussion of whether the framework can be applied to real image editing case. Does it can be applied to inverted real images? Does the framework can still predict the segmentation map and keypoints of given real images? If that is possible, there would be great impact and future usage.
If it is still applicable to real images, then please show some results which are similar to ControlNet.

2. The manuscript does not contain any qualitative comparison of ablation study. Please show some visual results on ablation study.

3. Please show the generated image results using handcrafted conditions, not the edited condition from generated conditions. For example, give the hand-drawn sketches (user-identified one) to the model and show the generated outputs.

QUESTIONS:
See weakness

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-11

comments: SUMMARY:
This paper introduces a Factor Graph Diffusion Models (FG-DMs) to address limitations in current generative models. FG-DMs model the joint distribution of images and conditioning variables using a factor graph decomposition, offering advantages like efficient sampling for better prompt compliance, fine-grained editing, and explainability. The method's effectiveness is validated on several datasets.

STRENGTHS:
This paper introduces a method to model the joint distribution of images and conditioning variables, enhancing prompt compliance and control over image synthesis.

The Attention Distillation Loss is proposed to enhance the consistency and quality of generated images and conditions.

WEAKNESSES:
The proposed method seems promising, however, it's very important to compare with current SOTA works to justify its effectiveness. The author only put Stable Diffusion in Table 5, which is not fair. Please include  more recent works especially sine 2023 such as  StyleGAN-T: Unlocking the Power of GANs for Fast Large-Scale Text-to-Image Synthesis,  Make a Scene.

QUESTIONS:
What's the training cost of your model? Is there a comparison with other methods?

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12