Title: Sparse-Compression Diffusion Models
Primary area: generative models
Keywords: Sparse Coding, Diffusion Process, Representation Learning

Abstract: Diffusion models have demonstrated capabilities in successful generating synthetic data, especially the pictural ones. However, the conventional reverse diffusion process merely overfits in data denoising. Empirical studies indicate that the denoising mechanism obstructs the diffusion models to generate robust physical synthetic data. In this work, we propose the Sparse Compression-Diffusion Model (SCDM), with a novel reverse diffusion process that enforces compression. The compression attempts to recover the low-dimensional and sparse representations, potentially enabling the ability to capture the intrinsic factors of some hidden physical laws. Experiments prove that the sparse latent representations learned by SCDM align well with top factors identified in physical theories.