Title: DIVIDE-AND-DENOISE: A GAME THEORETIC METHOD FOR FAIRLY COMPOSING DIFFUSION MODELS
Primary area: generative models
Keywords: Diffusion Models, Fair Composition, Game-Theoretic, Text-to-Image

Abstract: The widespread availability of large-scale pre-trained generative models raises a
question: how can we best leverage them beyond their original training distribu-
tions? Two strategies provide partial answers. Composition combines multiple
diffusion models, typically through linear averaging of their predictions, to pro-
duce out-of-distribution samples. Guidance steers a single model by biasing its
generation with rewards or classifier scores. We unify these perspectives with
Divide-and-Denoise, a game-theoretic approach to compositional sampling from
multiple pre-trained diffusion models, coordinated through an allocation flow. At
each denoising step, we alternate between (i) partitioning the sample into regions
assigned to distinct models for denoising (composition) and (ii) aligning the sam-
ple with this division (guidance). The partition is determined by solving a fair al-
location problem under a shared alignment objective. We evaluate our method on
text-to-image generation. Using models conditioned on different prompts, Divide-
and-Denoise reliably generates images that capture the semantics of each prompt,
even surpassing joint-prompt conditioning. On the GenEval benchmark, it further
outperforms energy-based composition and joint prompting baselines, resolving
common issues such as missing objects and attribute mismatches.

TL;DR: a game-theoretic approach to compositional sampling from multiple pre-trained diffusion models