Diff-2-in-1: Bridging Generation and Dense Perception with Diffusion
  Models
Authors: name: Shuhong Zheng; name: Zhipeng Bao; name: Ruoyu Zhao; name: Martial Hebert; name: Yu-Xiong Wang
Published: 2024-11-07 18:59:53

Abstract: Beyond high-fidelity image synthesis, diffusion models have recently
exhibited promising results in dense visual perception tasks. However, most
existing work treats diffusion models as a standalone component for perception
tasks, employing them either solely for off-the-shelf data augmentation or as
mere feature extractors. In contrast to these isolated and thus sub-optimal
efforts, we introduce a unified, versatile, diffusion-based framework,
Diff-2-in-1, that can simultaneously handle both multi-modal data generation
and dense visual perception, through a unique exploitation of the
diffusion-denoising process. Within this framework, we further enhance
discriminative visual perception via multi-modal generation, by utilizing the
denoising network to create multi-modal data that mirror the distribution of
the original training set. Importantly, Diff-2-in-1 optimizes the utilization
of the created diverse and faithful data by leveraging a novel self-improving
learning mechanism. Comprehensive experimental evaluations validate the
effectiveness of our framework, showcasing consistent performance improvements
across various discriminative backbones and high-quality multi-modal data
generation characterized by both realism and usefulness.