FABRIC: Personalizing Diffusion Models with Iterative Feedback
Authors: name: Dimitri von Rütte; name: Elisabetta Fedele; name: Jonathan Thomm; name: Lukas Wolf
Published: 2023-07-19 17:39:39

Abstract: In an era where visual content generation is increasingly driven by machine
learning, the integration of human feedback into generative models presents
significant opportunities for enhancing user experience and output quality.
This study explores strategies for incorporating iterative human feedback into
the generative process of diffusion-based text-to-image models. We propose
FABRIC, a training-free approach applicable to a wide range of popular
diffusion models, which exploits the self-attention layer present in the most
widely used architectures to condition the diffusion process on a set of
feedback images. To ensure a rigorous assessment of our approach, we introduce
a comprehensive evaluation methodology, offering a robust mechanism to quantify
the performance of generative visual models that integrate human feedback. We
show that generation results improve over multiple rounds of iterative feedback
through exhaustive analysis, implicitly optimizing arbitrary user preferences.
The potential applications of these findings extend to fields such as
personalized content creation and customization.