Title: Discrepancy-aware Score Learningfor Diffusion Training
Primary area: generative models
Keywords: diffusion model, text-to-image generation, generative adversarial network, adversarial training

Abstract: Diffusion models excel in stable training and distribution coverage, achieving remarkable results in various generative tasks. However, especially in high-resolution or structurally complex settings, their reliance on denoising score matching (DSM) leads to overly smoothed textures and limited perceptual detail. This limitation arises from DSM's propensity to reduce average reconstruction error rather than addressing challenging perceptual features in the data distribution. We propose Discrepancy-aware Score Learning (DSL), a novel adversarial training framework that incorporates a margin-based energy regularizer to score matching in order to address this challenge. In the noise space, DSL introduces an energy-based discriminator that adaptively highlights samples with high generation discrepancies. Our approach retains the denoising formulation while guiding the generator to prioritize difficult cases. We theoretically connect DSL to Wasserstein gradient flows, interpreting it as functional gradient descent regularized by the discriminator's energy surface. Moreover, we demonstrate that DSL is compatible with the underlying probabilistic model by establishing an equilibrium consistent with the true score function. Compared to baseline diffusion models and recent adversarial approaches, DSL significantly improves sample fidelity, perceptual sharpness, and semantic alignment, according to extensive experiments conducted across text-to-image generation, conditional synthesis, super-resolution, and 2D-to-3D reconstruction.