EmerDiff: Emerging Pixel-level Semantic Knowledge in Diffusion Models
Authors: name: Koichi Namekata; name: Amirmojtaba Sabour; name: Sanja Fidler; name: Seung Wook Kim
Published: 2024-01-22 07:34:06

Abstract: Diffusion models have recently received increasing research attention for
their remarkable transfer abilities in semantic segmentation tasks. However,
generating fine-grained segmentation masks with diffusion models often requires
additional training on annotated datasets, leaving it unclear to what extent
pre-trained diffusion models alone understand the semantic relations of their
generated images. To address this question, we leverage the semantic knowledge
extracted from Stable Diffusion (SD) and aim to develop an image segmentor
capable of generating fine-grained segmentation maps without any additional
training. The primary difficulty stems from the fact that semantically
meaningful feature maps typically exist only in the spatially lower-dimensional
layers, which poses a challenge in directly extracting pixel-level semantic
relations from these feature maps. To overcome this issue, our framework
identifies semantic correspondences between image pixels and spatial locations
of low-dimensional feature maps by exploiting SD's generation process and
utilizes them for constructing image-resolution segmentation maps. In extensive
experiments, the produced segmentation maps are demonstrated to be well
delineated and capture detailed parts of the images, indicating the existence
of highly accurate pixel-level semantic knowledge in diffusion models.