CleanDIFT: Diffusion Features without Noise
Authors: name: Nick Stracke, name: Stefan Andreas Baumann, name: Kolja Bauer, name: Frank Fundel, name: Björn Ommer

Internal features from large-scale pre-trained diffusion models have recently
been established as powerful semantic descriptors for a wide range of
downstream tasks. Works that use these features generally need to add noise to
images before passing them through the model to obtain the semantic features,
as the models do not offer the most useful features when given images with
little to no noise. We show that this noise has a critical impact on the
usefulness of these features that cannot be remedied by ensembling with
different random noises. We address this issue by introducing a lightweight,
unsupervised fine-tuning method that enables diffusion backbones to provide
high-quality, noise-free semantic features. We show that these features readily
outperform previous diffusion features by a wide margin in a wide variety of
extraction setups and downstream tasks, offering better performance than even
ensemble-based methods at a fraction of the cost.