Title: DPG: Exploiting Data and Process Knowledge for Diffusion Guidance
Primary area: generative models
Keywords: Deep learning, image generation, style transfer, image restoration

Abstract: Diffusion models have excelled in imperfect-label guidance tasks, including weak-label guidance (e.g., style transfer) and degraded-label guidance (e.g., image super-resolution and deblurring), where supervision is incomplete or compromised. Current methods are either tailored to specific tasks, limiting generalization and transferability, or rely solely on loss-guided methods for imperfect-label tasks, imposing insufficient constraints and overlooking valuable domain priors, leading to suboptimal results. We argue that these limitations stem from the failure to uncover the universally applicable latent knowledge in these tasks and methods. To address this, we propose DPG, a universal framework that exploits data knowledge inherent to the tasks and process knowledge derived from reverse diffusion in imperfect-label guidance tasks. Explicitly, we diffuse imperfect-label data or its variants and then incorporate them into the initial stages of reverse diffusion, elevating the precision and efficiency of outputs in aligning with labels. As well, we capitalize on the temporal progression of the diffusion model’s denoising path, ensuring that each step surpasses its predecessor in satisfying label constraints to refine optimization choice, thus improving guidance fidelity and accelerating convergence. By integrating this knowledge, DPG can achieve generalization and optimal performance in imperfect-label tasks, paving the way for future innovations in unified frameworks. Extensive experiments demonstrate DPG's effectiveness, consistently generating high-quality outputs. The code will be public.

TL;DR: We propose a unified framework, termed DPG, for addressing imperfect-label guidance tasks by incorporating both data and process knowledge.