TweedieMix: Improving Multi-Concept Fusion for Diffusion-based Image/Video Generation

Authors: Gihyun Kwon, Jong Chul Ye
Primary area: generative models
Keywords: Diffusion Models, Custom Concepts, Text-to-Image, Video

Abstract:
Despite significant advancements in customizing text-to-image and video generation models, generating images and videos that effectively integrate multiple personalized concepts remains challenging. To address this, we present TweedieMix, a novel method for composing customized diffusion models during the inference phase. By analyzing the properties of reverse diffusion sampling, our approach divides the sampling process into two stages. During the initial steps, we apply a multiple object-aware sampling technique to ensure the inclusion of the desired target objects. In the later steps, we blend the appearances of the custom concepts in the de-noised image space using Tweedie's formula. Our results demonstrate that TweedieMix can generate multiple personalized concepts with higher fidelity than existing methods. Moreover, our framework can be effortlessly extended to image-to-video diffusion models by extending the residual layer's features across frames}, enabling the generation of videos that feature multiple personalized concepts.

Decision: accept (poster)
Reviewer scores: 8, 6, 6

Reviews and discussion:
<thread>
creation date:
1729159637467

thread_type:
official_review

<thread_object>
creator: reviewer_y5wz

<content>
summary:
The paper introduces TweedieMix, a novel method designed to customize diffusion models during the inference phase, particularly for the challenge of integrating multiple personalized concepts in image and video generation. It operates by dividing the reverse diffusion sampling process into two distinct stages: an initial stage that uses multiple object-aware sampling to ensure the inclusion of desired target objects, and a later stage that employs Tweedie’s formula to blend the appearances of custom concepts in the de-noised image space. The results indicate that TweedieMix outperforms existing methods in generating images and videos with multiple personalized concepts with higher fidelity.

strengths:
1、The multi-concept generation results looks good；
2、The authors illustrate the framework details and experimental setting well.

weaknesses:
1、The whole pipeline seems naive and lack of novelty.
2、Some problems exist in experimental settings.
See questions for more details.

questions:
1、Making use of Text-SAM and Tweedie’s formula do not contribute to the novelty.
2、Each concept a model. I suspect the performance gain may derived from the overfitting of specific concept. And the hard mask composition seems like a naive post processing. The whole framework is more like a system which is designed under substantial ablation experiments and lack of novelty and essential understanding towards this task.
3、The extension to video, if I do not misunderstand, is a technical trick on a off-the-shell I2VGen-XL. I2VGen-XL is finetuned from ModelScopeT2V with a large dataset. While DreamVideo use ModelScopeT2V only for subject injection with a little amount of custom videos. Maybe this need further discussion and I'd like to see opinions from other reviews.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730552894771

thread_type:
official_review

<thread_object>
creator: reviewer_yjkf

<content>
summary:
This paper introduces a new approach called TWEEDIEMIX for multi-concept text-to-image generation. The author introduce a tuning-free approach during the inference stage and divides the process into two main stages. The key contribution of this paper is using a resampling strategy and multi-concept fusion sampling enabling generate multiple personalized concepts with higher fidelity than existing methods.

strengths:
1. The idea is easy to follow and the paper is well written.
2. The results on Custom Concept 101 dataset outperforms the previous baselines, demonstrating better generation qualities compared to the concept personalization methods. 
3. The author's code has been open-sourced, which is somewhat helpful to the community.

weaknesses:
1. My main concern lies in the technical contribution of this paper. It seems like an incremental work of ConceptWeaver[1] which is also a training-free method that combines multiple concepts during inference.
2. In terms of qualitative results, there doesn't seem to be a significant improvement compared to ConceptWeaver which is also can handle more than two concepts.


[1] Gihyun Kwon, Simon Jenni, Dingzeyu Li, Joon-Young Lee, Jong Chul Ye, and Fabian Caba Heilbron. Concept weaver: Enabling multi-concept fusion in text-to-image models. In Proceedings of the IEEE/CVF  (CVPR), pp. 8880–8889, June 2024.

questions:
Please refer to the weaknesses above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730651932249

thread_type:
official_review

<thread_object>
creator: reviewer_jzyv

<content>
summary:
This paper proposes a method to fuse multiple concepts for personalized T2I diffusion-based models that are generalizable to both image and video generation. The method mainly involves two parts: (1) concept-aware sampling that coarsely localizes the concepts in the latent space, and (2) multi-concept fusion that generates the personalized concepts based on the extracted regional masks. The qualitative and quantitative results both demonstrate the effectiveness of the proposed method.

strengths:
+ The proposed method is reasonable and intuitive.
+ The generated results of multi-concept fusion are impressive.
+ The model is properly extended from image generation to video generation.
+ From the quantitative results, the proposed method outperforms previous models.

weaknesses:
- $t_con$ and $P$ are two important hyperparameters. Can the authors provide generated results with varying $t_con$ and $P$, while keeping all other settings fixed?
- DINO [1] can extract subtle visual features that are not specific to any particular category. The metric of DINO image similarity should also be reported.
- The evaluation is conducted on combinations of more than three concepts; however, the paper does not present any results of combinations involving more than three concepts.
- It is recommended to compare the sampling time of the proposed method with that of other models.
- Missing reference on unsupervised multi-concept extraction [2].
 
[1] Caron etal. Emerging Properties in Self-Supervised Vision Transformers. ICCV 2021.
[2] Hao et al. ConceptExpress: Harnessing Diffusion Models for Single-image Unsupervised Concept Extraction. ECCV 2024.

questions:
- Please see the weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732020687519

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Reply

comment:
We sincerely thank all reviewers for the constructive feedback. Per all the reviewers' comments, we revised the manuscript. Here is the summary of changes to our paper.

●	We included DINO image similarity score for further comparison.

●	We included new quantitative scores which are calculated using extended concept dataset (15 concepts -> 32 concepts)

●	We included additional experiment with varying the hyperparameter t_cond and P (Appendix B, Figure 9)

●	We included inference time comparison between the baselines. (Appendix C)

●	We included qualitative results on combining 4 different concepts (Fig 11, Appendix E)

●	We included more qualitative comparison between Concept Weaver and ours in Figure 11,13,14,15.

●	To evaluate the versatility of our proposed method, we included qualitative results of multiple style composition in Figure 10 and Appendix D.

●	We included missing references in related work.

●	We included video generation results on Stable Video Diffusion in Figure 12.

●	We corrected some typos.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces a novel method, TweedieMix, for personalized multi-concept text-to-image and text-to-video generation using diffusion-based models. The method mainly has two key stages: concept-aware sampling for localization in latent space, and fusing appearances of personalized concepts. The results demonstrate convincing performance in combining multiple concepts, outperforming prior models in fidelity and quality on image and video generations. There are some concerns about limited novelty and incremental contributions compared to prior work like ConceptWeaver. Given the method’s practicality and effectiveness from quantitative and qualitative results, the method has potential value to the research community. The authors are encouraged to address concerns about experimental settings, novelty, and additional evaluations in their revisions.