Title: Fixing Model-Fitting: Compressing Guidance for Better Sampling
Primary area: generative models
Keywords: Diffusion, guidance

Abstract: Model-fitting, a phenomenon where generated samples are overly adjusted to the model used for guidance rather than the intended conditions, is a key drawback and often leads to suboptimal outcomes. The root cause of this problem is the consecutiveness of guidance timesteps throughout the diffusion sampling process. In this work, We quantify this effect and show that breaking the consecutiveness of standard guidance alleviates the problem. Based on this insight, our method, Compress Guidance, distributes a small number of guidance steps across the full sampling process, yielding substantial improvements in image quality and diversity while cutting guidance cost by over 80\%. Experiments on both label-conditional and text-to-image generation, across multiple datasets and models, confirm that Compress Guidance consistently surpasses baselines in image quality with significantly lower computational overhead.

TL;DR: The paper analyses model-fitting problem when using guidance, and propose a method to solve that