Lotus: Diffusion-based Visual Foundation Model for High-quality Dense
  Prediction
Authors: name: Jing He; name: Haodong Li; name: Wei Yin; name: Yixun Liang; name: Leheng Li; name: Kaiqiang Zhou; name: Hongbo Liu; name: Bingbing Liu; name: Ying-Cong Chen
Published: 2024-09-26 17:58:55

Abstract: Leveraging the visual priors of pre-trained text-to-image diffusion models
offers a promising solution to enhance zero-shot generalization in dense
prediction tasks. However, existing methods often uncritically use the original
diffusion formulation, which may not be optimal due to the fundamental
differences between dense prediction and image generation. In this paper, we
provide a systemic analysis of the diffusion formulation for the dense
prediction, focusing on both quality and efficiency. And we find that the
original parameterization type for image generation, which learns to predict
noise, is harmful for dense prediction; the multi-step noising/denoising
diffusion process is also unnecessary and challenging to optimize. Based on
these insights, we introduce Lotus, a diffusion-based visual foundation model
with a simple yet effective adaptation protocol for dense prediction.
Specifically, Lotus is trained to directly predict annotations instead of
noise, thereby avoiding harmful variance. We also reformulate the diffusion
process into a single-step procedure, simplifying optimization and
significantly boosting inference speed. Additionally, we introduce a novel
tuning strategy called detail preserver, which achieves more accurate and
fine-grained predictions. Without scaling up the training data or model
capacity, Lotus achieves SoTA performance in zero-shot depth and normal
estimation across various datasets. It also significantly enhances efficiency,
being hundreds of times faster than most existing diffusion-based methods.