Lotus: Diffusion-based Visual Foundation Model for High-quality Dense Prediction
Authors: Jing He
Haodong Li
Wei Yin
Yixun Liang
Leheng Li
Kaiqiang Zhou
Hongbo Zhang
Bingbing Liu
Ying-Cong Chen
Venue: ICLR 2025 Poster (2025)
Keywords: Diffusion Models; Dense Prediction; Monocular Depth Estimation; Monocular Normal Estimation
Primary area: generative models
TL;DR: Based on pre-trained Stable Diffusion, Lotus delivers SoTA performance on monocular depth & normal estimation with simple yet effective fine-tuning protocol.

Abstract: Leveraging the visual priors of pre-trained text-to-image diffusion models offers a promising solution to enhance zero-shot generalization in dense prediction tasks. However, existing methods often uncritically use the original diffusion formulation, which may not be optimal due to the fundamental differences between dense prediction and image generation. In this paper, we provide a systemic analysis of the diffusion formulation for the dense prediction, focusing on both quality and efficiency. And we find that the original parameterization type for image generation, which learns to predict noise, is harmful for dense prediction; the multi-step noising/denoising diffusion process is also unnecessary and challenging to optimize. Based on these insights, we introduce $\textbf{Lotus}$, a diffusion-based visual foundation model with a simple yet effective adaptation protocol for dense prediction. Specifically, Lotus is trained to directly predict annotations instead of noise, thereby avoiding harmful variance. We also reformulate the diffusion process into a single-step procedure, simplifying optimization and significantly boosting inference speed. Additionally, we introduce a novel tuning strategy called detail preserver, which achieves more accurate and fine-grained predictions. Without scaling up the training data or model capacity, Lotus achieves SoTA performance in zero-shot depth and normal estimation across various datasets. It also enhances efficiency, being significantly faster than most existing diffusion-based methods. Lotus' superior quality and efficiency also enable a wide range of practical applications, such as joint estimation, single/multi-view 3D reconstruction, etc.

Published as a conference paper at ICLR 2025
LOTUS: DIFFUSION-BASED VISUAL FOUNDATION MODEL FOR HIGH-QUALITY DENSE PREDICTION
Jing He 1# Haodong Li Wei Yin Yixun Liang' Leheng Li' Kaiqiang Zhou' Hongbo Zhang' Binghing Liu Yingcong Chen!.22 (GZ University of Adelaide "Noah's Ark Lab 'HKUST (jhe812, h11736) @connect. hkust gz. edu. cn; yingcongchenOust..
Repliskyyhing W
Es NE
brvil
Lotus (Ours)
Letus KOurs)
Lobus (Ours)
Lotus (Ours)
Rank
Discriiiintive
Avp. Rank
Discriminative
7.5
Omeidata
12.2M
7.0
D6515
110K
5.8 DPT
1.4M
5.9
12. 2M
HDN
300K
EESNNU
4.9
3.6
4.4
Omevnata
3.5
Depeninything V?
02. GM
3.1
DSINE
160K
59K
Lotus-D
74% Diflsion-EEE-FT
Rank
Generative Depth Est.
Generative Normal Est.
3.3
Beowloard
280K
3.6
/4K Marigoid
2.9
3.1
SranieGormai
59K
Lotus-G
5SK
Lotus-G
Training
Figure 1: We present Lotus, diffusion- based visual foundation model for dense geometry pre- diction. With minimal training data, Lotus achieves promising performance in zero-shot depth and normal estimation. "Avg. Rank" indicates the average ranking across all metrics, where lower values are better. Bar length represents the amount of training data used.
ABSTRACT
Leveraging the visual priors of pre-trained text-to-image diffusion models offers a promising solution enhance zero-shot generalization in dense prediction tasks. However, existing methods often uncritically use the original diffusion formula- tion, which may not be optimal due to the fundamental differences between dense prediction and image generation. In this paper, we provide a systemic analysis the diffusion formulation for the dense prediction, focusing both quality and efficiency. And we find that the original parameterization type for image gener- ation, which learns predict noise, is harmful for dense prediction; the multi- step noising/denoising diffusion process is also unnecessary and challenging to optimize. Based on these insights, we introduce Lotus, diffusiin-based visual foundation model with simple yet effective adaptation protocol for dense predic- tion. Specifically, Lotus trained directly predict annotations instead of noise. thereby avoiding harmful variance. We also reformulate the diffusion process into single-step procedure, simplifying optimization and significantly boosting in- ference speed. Additionally, we introduce a novel tuning strategy called detail preserver, which achieves more accurate and fine-grained predictions. Without scaling up the training data or model capacity, Lotus achieves promising perfor- mance zero-shot depth and normal estimation across various datasets. also enhances efficiency, being significantly faster than most existing diffusion-basee methods. Lotus' superior quality and efficiency enables wide range practical applications, such joint estimation, single/multi-view 3D reconstruction, etc.

---

Published as a conference paper at ICLR 2025
INTRODUCTION
Dense prediction is a fundamental task in computer vision, benefiting a wide range of applications. such as 3D/4D reconstruction (Huang et al. 2024 Long et al.] 2024 Wang et al. 22044 Lei et al.
2024), tracking (Xiao et al. 2024 Song et al, 2024 and autonomous driving (Yurtsever et al. 2020, Hu et al. 2023). Estimating pixel-level geometric attributes from single image requires compre- hensive scene understanding. Although deep learning has advanced dense prediction progress is limited by the quality, diversity, and scale of training data, leading to poor zero-shot generalization. Instead of merely scaling data and model size, recent works (Lee et al.] 2024; Ke et al.] 2024 Fu
et al. 2024 Xu et al.] 2024) leverage diffusion priors for zero-shot dense prediction. These stud- les demonstrate that text-to-image diffusion models like Stable Diffusion (Rombach et al.) 2022).
pretrained on billions of images, possess powerful and comprehensive visual priors to elevate dense prediction performance. However, most of these methods directly inherit the pre-trained diffusion models for dense prediction tasks, without exploring more suitable diffusion formulations. This oversight often leads to challenging issues. For example, Marigold (Ke et al.] (2024) directly fine- tunes Stable Diffusion for image-conditioned depth generation. While it significantly improves depth estimation, its performance is still constrained by overlooking the fundamental differences between dense prediction and image generation. Especially, efficiency is also severely limited by standard iterative denoising processes and ensemble inferences.
Motivated by these concerns, we systematically analyze the diffusion formulation, trying to find a better formulation to fit the pre-trained diffusion model into dense prediction. Our analysis yields several important findings: ID The widely used parameterization, i.e., noise prediction, for diffusion- based image generation is ill-suited for dense prediction. It results in large prediction errors due to harmful prediction variance at initial denoising steps. which are subsequently propagated and magnified throughout the entire denoising process (Sec.4..) 4.1. @ Multi-step diffusion formulation is cmpputatiioninteniiie and is prone to sub-optimal with limited data and resources. These factors significantly hinder the adaptation of diffusion priors dense prediction tasks, leading to decreased accuracy and efficiency (Sec. 4.2) @ Though remarkable performance achieved, we observed that the model usually outputs vague predictions in highly-detailes areas (Fig. [8]. This vagueness is attributed to catastrophic forgetting: the pre-trained diffusion models gradually lose their ability to generate detailed regions during fine-tuning (Sec. 4.3).
Following our analysis, we propose Lotus, a diffusion-based visual foundation model for dense prediction, featuring a simple yet effective fine-tuning protocol (see Fig. [ First, Lotus is trained to directly predict annotations, thereby avoiding the harmful variance associated with stan- dard noise prediction. Next, we introduce a one-step formulation, i.e., one step between pure noise and clean output, to facilitate model convergence and achieve better optimization perfor- mance with limited high-quality data. It also considerably boosts both training and inference implement novel detail preserver through task switcher, allow-
efficiency. Moreover, ing the model either to generate annotations or reconstruct the input images. It can better pre- serve the fine-grained details in the input image during dense annotation generation, achieving
single-step T
Xg preddccctionn
Image
x
Training Objective:
concat. add
|| 2z s.) II' (imago reconstruction)
denoiser U-Net fo
E
|| - fo (Z (z, tt, s,) (predict annotation)
Annotation
z
z
switcher $
2
dctail preserver
single-step
Adaptation protocol of Lotus. After the pre-trained VAE encoder & encodes the im- age x and annotation to the latent space: ① the denoiser U-Net model fo is fine-tuned using ro-prediction; @ we employ single-step diffusion formulation time-step L T for better conver- gence; @ we propose novel detail preserver, to switch the model either to reconstruct the image or generate the dense prediction via a switcher s, ensuring a more fine- tnneggrained prediction. The noise zy. in bracket is used for our generative Lotus-G and is omitted for the discriminative Lotus-D.
Figure 2:
2

---

Published as a conference paper at ICLR 2025
Inference Time Image Recolution Depth Est.
higher performance without compromising ef-
102
Marigold
322.
ficiency, requiring additional parameters, or be- ing affected by surface textures.
Inferencce Mitrencee Time (on NVIDIA A800)
Marigolducw
377,440
10
Lotus (Ours)
Depthhnythhng V2
107.773
To validate Lotus, we conduct extensive exper- iments two primary geometrie dense predic- tion tasks: zero-shot monocular depth and nor- mal estimation. The results demonstrate that Lotus achieves promising, and even superior, performance on these tasks across wide range of evaluation datasets. Compared to traditional discriminative methods, Lotus delivers remark- able results with only 59K training samples. Among generative approaches, Lotus also out- performs previous methods in both accuracy and efficiency. being significantly faster than methods like Marigold (Ke et al., 2024) (Fig. 3). Beyond these improvements. Lotus seamlessly supports various applications, e-g., joint estima- tion, single/multi-view 3D reconstruction, etc.
3.0
12 12
40,620
657x
63+
2. 1666
2,470
1.712
10 g0a
0.602
0.g13
0.100-
0.065 100
2504250012512
1024x1024
1536x1530
2048x
Image
Figure 3: Inference time comparison in depth estimation between Lotus and SoTA methods. Lotus is hundreds of times faster than Marigold and slightly faster than DepthAnything V2 at high resolutions. DepthAnything V2's inference time at 2048 x 2048 is not plotted because it requires 80GB graphic memory.
In conclusion, key contributions
follows:
We systematically analyze the diffusion formulation and find their parameterization type, designed for image generation, is unsuitable for dense prediction and the computation- intensive multi-step diffussin process is also unnecessary and challenging to optimize.
We propose novel detail preserver that ensures more accurate dense predictions especially in detail-rich areas, without compromising efficiency, introducing additional network pa- rameters, or being affected by surface textures.
Based on our insights, we introduce Lotus, a diffusion- visual foundation model for dense prediction with simple yet effective fine-tuning protocol. Lotus achieves promising performance on both zero-shot monocular depth and surface normal estimation. It also enables wide range of applications.
RELATED WORKS
2.1
TEXT-TO-IMAGE GENERATIVE MODELS
In the field of text-to-inn immee generation, the evolution of methodologies has transitioned from gen-
erative adversarial networks (GANs) (Goodfellow et al.)
2014 Zhang et al., 2017: 2018 2021 He
let al. 2022 Karras et al. 2019 2020 2021
al. 2021 to advanced diffusion models (oo et al. 2020 Ramesh et al. 2022] Saharia et al., 2022
al.
2017
2018 Xu al. 20088)/Zan
Ramesh et al. |02 Nichol al.) 2021 Chen et al. 2023 Rombach et al.
2022 Ramesh et al.
2021). A series of iestussoonbaaseee methods such as GLIDE (Nichol et al. 2022)) DALL-E2 (Ramesh
al. 2022), and Imagen (Saharia al. 2022) have been introduced, offering enhanced image qual- ity and textual coherence. The Stable Diffusion (SD) (Rombach et al. 2022). trained on large-scale
LAION-5B dataset (Schuhmann et al., 2022), further enhances the generative quality, becoming the community standard. In our paper, we aim to leverage the comprehensive and encyclopedic visual priors of SD to facilitate zero-shot generalization for dense prediction tasks.
2.2 GENERATIVE MODELS FOR DENSE PERCEPTION
Currently, notable trend involves adopting pre-trained generative models, particularly diffusion models, into dense prediction tasks. Marigold (Ke et al. (024) and GeoWizard (Fu et al.) 2024) directly apply the standard diffusion formulation and the pre-trained parameters, without address- ing the inherent differences between image generation and dense prediction, leading constrained performance. Their efficiency is also severely limited by standard iterative denoising processes and ensemble inferences. In this paper, we propose novel diffusion formulation tailored to the dense prediction. Aiming to fully leveraging the pre-trained diffusion's 's powerful visual priors, Lotus en- ables more accurate and efficient predictions, finally achieving promising performance.
More recent works, GenPercept (Xu et al. 2024) and StableNormal YYe et al. 2024), also adopted single-step diffusion. However, GenPercept (Xu al. 2024 first removes noise input for deter-

---

Published as a conference paper at ICLR 2025
ministic characteristic based DMP (Lee et al. 2024). and then adopts one-step strategy to avoid surface texture interference. It lacks systematic analysis of the diffusion formulation, only treats the U-Net deterministic backbone and still falls short in performance. In contrast, Lotus systemati- cally analyzes the standard stochastic diffusion formulation for dense prediction and proposes inno- vations such the detail preserver improve accuracy especially in detailed area, finally delivering much better results (Tab. D. Additionally, Lotus is a stochastic model. In contrast to GenPercept's s deterministic nature, Lotus enables uncertainty predictions. StableNormal (Ye et al.) 2024) predicts normal maps through a two-stage process. While the first stage produces coarse normal maps with single-step diffusion, the second stage performs refinement still with iterative diffusion which is computation-intensiye
In comparison, Lotus not only achieves fine-grained predictions thanks to our novel detail preserver without extra stages or parameters, but also delivers much superior re- sults (Tab. 2 thanks to our designed diffusion formulation that better fits the pre-trained diffusion for dense prediction. Recently, a concurrent work, Diffusion-E221 -EEssonnnEE--FTT-- (Garcia et al., 2024), has also achieved promising results in a single step. Its main contribution lies in addressing the issue where Marigold (Ke et al. (2024) and similar models (Fu et al. 2024) use inconsistent pairings of time-step and noise, resulting in poor predictions. By setting the "time-step spacing" to "trailing" mode in schedulers, it prevents "GT" signal leakage during inference, improving accuracy. While the per- formance of Lotus-D and FE2BFFF is similar, Lotus is based on a systematic analysis of stochastic diffusion for dense prediction. with innovations like the detail preserver to enhance accu- racy, particularly in detailed areas. Additionally, unlike the deterministic (Lotus-G) G) is a stochastic model that enables uncertainty predictions.
Lotus
2.3
MONOCULAR DEPTH AND NORMAL. PREDICTION
Monocular depth and normal prediction are two crucial dense prediction tasks. Solving them typ- ically demands comprehensive scene understanding capability. Starting from (Eigen et 2014). early CNN-based methods for depth prediction, such as (Fu et al.) 2018). (Lee et al. 2019), Yuan al. 2022 focus only on specific domains. Subsequently, in pursuit of a generalizable depth es- timator, many methods expand model capacity and train on larger and more diverse datasets, such as DiverseDepth (Yin et al., 222 and MiDaS (Ranftl et al. 2020). DPT (Ranftl et al. (02T) and Omnidata (Eftekhar et al. 2021) are further proposed based on vision transformer (Ranfil et al. 2021. significantly enhancing performance. LeRes (Yin et al. 2021b) and HDN (Zhagg et al.
2022 further introduce novel training strategies and multi-scale depth normalization to improve predictions in detailed areas. More recently, the DepthAnything series (Yang et al. 2024a b) and Metric3D series (Yin et al. 2023 Hu al. 2024) collect and leverage millions of training data to develop more powerful estimators. Normal prediction follows the same trend. Starting with the early CNN-based methods like OASIS (Chen et 2020), EESNU (Bae & Davison (021)) and Omnidata
series (Eftekhar et al. 2021: Kar el al. (0022) ) expand the model capacity and scale up the training
data. Recently, DSINE [Bae& & Davison 2024) achieves SoTA performance by rethinking inductive biases for surface normal estimation. In our paper, we focus on leveraging pre- trairri diffusion priors to enhance zero-shot dense predictions, rather than expanding model capacity or relying on large training data, which avoids the need for intensive resources and computation.
PRELIMINARIES
Diffusion Formulation for Dense Prediction. Following Ke et al. (024)) and Fu al. (2024), also formulate dense prediction as an condie-condiiooned annotation generation task based on Stable Diffusion (Rombach al.] 2022). which performs the diffusion process in low-dimensional latent space for computational efficiency. First, the auto-encoder, which consists an encoder E(-) and a decoder D(-). is trained to map between RGB space and latent space, i.e., E(x) z², D(z*) x. The auto-encoder also maps between dense annotations and latent space effectively, i..., (le.,(()== z", D(z)) y (Ke et al. 2024, Fu et al., 2024. Xu et al. 2024: Ye et al.) (024)). Following Ho et al.) 2020). Stable Diffusion establishes a pair of forward nosing and reversal denoising processes in latent space. In forward process, Gaussian noise is gradually added at levels E [1, T into sample z to obtain the noisy sample zY:
2 = -V + V1 DCC.
B.). and ( 82...., Br) is the noise schedule with T steps. At time-step T, the sample zd is degraded to pure Gaussian noise. In the reversal process, a neural network fer usually U-Net model (Ronneberger et al.) 2015). is trained to iteratively remove noise from 2 to predict the clean sample z². The network is trained by sampling a random t E [1, T) and minimizing the loss function L,p.
where a10,,1)a,
4

---

Published as a conference paper at ICLR 2025
Image
z
multi-stop
E [
concat
e-prediction Training Objective: || 2z, (predict noise)
3
denoiscr U-Net fo
add
Annotation
z
z
[ELLT]]
multi-step
Figure 4: Adaptation protocol of Direct Adaptation. Starting with a pre-trained Stable Diffusion model, image and annotation are encoded using the pre-trained VAE. The noisy annotation z is obtained by adding noise at level E [1. 77. The U-Net input layer is coupled to accommodate the concatenated inputs and then fine-tuned using the standard diffusion objective, e-prediction, under the original multi-step formulation.
Parameterization Types. To enable gradient computation for network training, there are two basic parameterizations of the loss function Lt ① e-prediction (Ho et al. 2020): the model fe learns to predict the added noise E; 2 zo-prediction (Ho et al. 2020): the model Jo learns to directly predict the clean sample z². The loss functions for these parameterizations are formulated as:
e-prediction: L; = ||e
f/ (z) 2x t)]??
(2)
-ooprediction: I = ||z² f/ 2² t)]1
where fă is the denoiser model be learnt, E {e,z). e-prediction is commonly chosen as the standard for parameterizing the denoising model, as it empirically achieves high-quality image gen- eration with fine details and realism.
Denoising Process. DDIM (Song et al. 2020) is key technique for multi-step diffusion models to achieve fast sampling, which implements an implicit probabilistic model that can significantly reduce the number of denoising steps while maintaining output quality. Formally, the denoising process from 2 to 2²-1 is:
= = 1 direction (2)) o.Cr,
where 2 is the predicted clean sample at the denoising step direction(z)) represents the direction pointing to z and o, can be 0 if deterministic denoising is needed. And T E { .., Ts), an increasing sub-sequence of the time-step set [1, 7'. is used for fast sampling. During inference, DDIM iteratively denoises the sample from Ts Ti obtain the clean one.
METHODOLOGY
We start our analysis by directly adapting the original diffusion formulation with minimal modifica- tions as illustrated in Fig. 4 We call this starting point "Direct Adaptation"] ||] Direct Adaptation optimized using the standard diffusion objective as formulated in Eq. (first row) and inferred by standard multi-step DDIM sampler. As shown in Tab. Direct Adaptation fails to achieve sat- isfactory performance. In following sections, we will systematically analyze the key factors that affect adaptation performance step by step: parameterization types (Sec. 4.1 number of time-steps (Sec. 4.2); ; and the novel detail preserver (Sec. 4.3)
4.1 PARAMETERIZATTON TYPES
The type of parameterization is crucial, it not only determines the loss function discussed in Sec. 3 but also influences the inference process (Eq. 3). During inference, the predicted clean sample 2y a key component in Eq. is calculated according to different parameterizations
1 Va,
e-prediction: 2.
(4)
To xo-prediction: 2. (2,,,2*,7)
In the community, c-prediction is chosen as the standard for image generation. However, it is not effective for dense prediction task. In the following, we will discuss the impact of different parame- terization types in denoising inference process for dense prediction task.
Details of Direct Adaptation he provided in the supplementary materials. Thhe latest parameterization, r-prediction, combines e-prediction and Predrriic,,on, producing results that intermediate between the two. Please the supplementary materials for more details.
5

---

Published as a conference paper at ICLR 2025
A-prodiction
E-pediationn seed
syppoaliction saal
Xy poodiction sead 2
lxpui Image
1000 600
200
Figure 5: Comparisons among different parameterizations using various seeds. All models trained on Hypersim (Roberts et al., (2221) and tested on the input image for depth estimation. The standard DDIM sampler is used with 50 denoising steps. Four steps are selected for clear illustration. From left (larger 7) to right (smaller r) is the iterative denoising process.
NYUv2 AbsRel vs. Dencising Step
Insights from the literature (Benny & Wolf 2022 SSlliaan & Ho 2022) reveal that
Pred. Type-Xo
12
Pred. Type-r
e-prediction introduces larger pixel variance compared to proprddiition, especially at the ini- tial denoising steps (large 7). This variance
NYU22 AboRel (%)
11
10
mainly originates from the noise input. Specifi- cally, for c-prediction in Eq. [ ing step, r + T, the value va, + +00. Thus, the prediction variance from fo (z ,,,,,, will be amplified significantly, resulting in large variance of predicted 2%. In contrast, there no coefficient for zu-prediction re-seale the model output, achieving more stable pre- dictions of 2/ initial denoising steps. Subse- quently, the predicted 2. is used in Eq. 3 where its coefficient v vā,-1 are same across the two parameterizations, and other terms are of the same order of magnitude. Therefore, the 27 predicted by e-prediction, which has larger variance, exerts more significant influence denoising process. Since the process is iterative, this influence continually preserved and maybe amplified.
at initial denois-
1000
600
400
200
Denoising Step
Figure 6: Quantitative evaluation of the pre- dicted depth maps 2% along the denoising pro- cess. The experimental settings are same Fig. 5l Six steps are selected for illustration. The banded regions around each line indicate the variance, wider areas representing larger variance.
Abs vs. Training
We take the depth estimation During the inference process, we compute the predicted depth map 2 each denoising step T. As illustrated in Fig. []] the depth maps predicted by e-prediction significantly vary un- der different seeds while those predicted by zo-prediction are more consistent. Although the large variance enhances diversity for im- age generation, lead to unstable predictions in dense prediction tasks, potentially resulting in significant errors. For example Fig. [ the "dark gray cabinet" (highlighted in red cir- cles) maybe wrongly considered as an "opened door" with significantly larger depth. While the predicted depth map looks more and more plausible, the error gradually propagates to the final prediction (r = 1) along the denoising process, indicating the persistent influence of the large variance. We further quantitatively measure the predicted depth maps by the ab- solute mean relative error (A (AbsRel) on NYUv2 dataset (Silberman et al.) 2012). As shown in Fig. 6, e-prediction exhibits higher error with much larger variance compared to Tg eo-prediction at the initial denoising steps (T - T), and the prediction error propagates with higher slope. In contrast, 2g predictiin,, directly predicting ω without any coefficients to amplify the prediction variance, yields more stable and correct dense predictions than e-prediction. In conclusion, to mitigate the errors from large variance that ad-
example.
11
T=1
T-2
.00
T-55
NYUU2 AboRel (B))
T=
9
10-100
1=1000
8
5K
10K 19K
39K
Training Data
Figure 7: Comparisons among various training time-steps and data scales evaluated on NYUv2 in depth estimation. All models are fine-tuned on Hypersim using xo-prediction. During infer- ence, if > 50, the DDIM sampler is used with 50 denoising steps; otherwise, the number of de- noising steps is equal to T". The results demon- strate improved performance with decreased train- ing time-steps. The single-step diffusion formula- tion (T' = 1) exhibits best performance across different data volumes.

---

Published as a conference paper at ICLR 2025
Input Image
Reconstruction
Input Image
Reconstruction
w/o Preserver
Preserver
w/o Preserver
w/ Preserver
Figure 8: Depth maps w/ and w/o the detail preserver and reconstruction outputs. Fine-tuning the diffusion model for dense prediction tasks can potentially degrade its ability to generate highly detailed images, resulting in blurred predictions in regions with rich detail. To preserve these fine- grained details, introduce detail preserver that incorporates additional reconstruction task, enhancing the model's capacity to produce more accurate dense annotations.
versely affect the performance of dense prediction, we replace the standard e-prediction with the more tailored ro-prediction.
4.2 NUMBER OF TIME-STEPS
Although 2n oroediiiiioon can improve the prediction quality, the multi-step diffusion formulation still leads to the propagation of predicted errors during the denoising process (Fig. 5. 6 Furthermore, utilizing multiple time-steps enhances the model's capacity, typically requiring large-scale training data to optimize and is beneficial for complex tasks such as image generation. However, for simpler tasks like dense prediction, where large-scale, high-quality training data is also scarce, employing multiple time-steps can make the model difficult to optimize. Additionally, training/inferring multi-step diffusion model is slow and
-nntenttion, hindering its practical application.
Therefore, to address these challenges, we propose fine-tuning the pre-trained diffusion model with fewer training time steps. Specifically, the original set of training time-steps is defined as [1,T] {1, 2, ... T). where T denotes the total number of original training time-steps. We fine-tune the pre-trained diffusion model using a sub-sequee sub-sequence derived from this set. We define the length of this sub-sequence as T", where & T and T' is divisible by T". This sub-sequence is obtained by evenly sampling the original set intervals, defined as:
{t. = i- k 1,2..... Th
(5)
where k = T/T' is the sampling interval. During inference, the DDIM denoises the sample from noise annotation using the same sub-sequence if T' & 30, otherwise use 50 denoising steps.
As illustrated in Fig. 7 we conduct experiments by varying the number of time-steps T' under co-prediction. The results clearly show that the performance gradually improves the number of time-steps is reduced, no matter the training data scales, culminating the best result when re- duced only single step. We further consider more strict scenarios with more limited training data to assess its impact on model optimization. As depicted in Fig. 7 these experiments reveal that the multi-step formulation is more sensitive to increases in training data scales compared with single-step. Notably, the single-step formulation consistently yields lower prediction errors and demonstrates greater stability. Although it is conceivable that multi-step and single-step formula- tions might achieve comparable performance with unlimited high-quality data, it's expensive and sometimes impractical dense prediction.
Decreasing the number of denoising steps can reduce the optimization space of the diffusion model, leading to more effective and efficient adaption, as suggested by the above phenomenon. There- fore, for better adaptation performance under limited resource, we reduce the number of training time-steps of diffusion formulation to only one, and fixing the only time-step t to T. Addition- ally, the single-step formulation is much more computationally efficient. It also naturally prevents the harmful error propagation as discussed in Sec. 4.1 further enhancing the diffusion's 's adaptation performance in dense prediction.
7

---

Published as a conference paper at ICLR 2025
123
Iaput Image
Seed
Seed
Seed 2
Seed 3
Uncestainty Map
Figure 9: Depth maps of multiple inferences and uncertainty maps. Areas like the sky, object edges, and intricate details (e.g., cat whiskers) typically exhibit high uncertainty.
4.3 DETAIL PRESERVER
Despite the effectiveness of the above designs, the model still struggles with processing detailed w// Preserver). The original diffusion model excels at generating detailed images. However, when adapted to predict dense annotations, it can lose such detailed generation ability, due to unexpected catastrophic forgetting (Zhai et al. 2023: Du al. 2024). This leads to challenges in predicting dense annotations in intricate regions.
areas (Fig. g
To preserve the rich details of the input images, we introduce novel regularization strategy called Detail Preserver. Inspired by previous works (Long al., 2024, Fu et al.) 2024). we utilize task switcher s E {s2, enabling the denoiser model fo to either generate annotation or reconstruct the input image. When activated by 8y, the model focuses when 8 is selected, it reconstructs the input image. The switcher s is a one-dimensional vector encoded by the positional encoder and then added with the time embeddings of diffusion model, ensuring seamless domain switching without mutual interference. This dual capability enables the diffusion model to make detailed predictions and thus leading better performance. Overall, the loss function L, is:
predicting annotation. Conversely,
L. = - fo(z) xxx s,)) + || - folzz z² ,,,,t,t sy)l1
(6)
where f T and thus 2. is pure Gaussian noise.
4.4 STOCHASTIC NATURE OF DIFFUSION MODEL.
One major characteristic of generative models is their stochastic nature, which, in image generation, enables the production of diverse outputs. In per- ception tasks like dense prediction, this stochasticity has the potential to allow the model generating pre- dictions with uncertainty maps. Specifically, for any input image, we can conduct multiple inferences ing different initialization noises and aggregate these predictions to calculate its uncertainty map. Thanks to our systematic analysis and tailored fine-tuning protocol, our method effectively reduces excessive flickering (large variance), only allowing for more accurate uncertainty calculations naturally uncertain details (e.g. cat whiskers), shown in Fig. 9
Image
denoiser
COONS
switcher sy
D
2
Prediction
Figure 10: Inference Pipeline of Lotus. The noise 2) in bracket is used for Lotus-G and omitted for Lotus-D. areas, such as the sky, object edges, and fine
Most existing perception models are deterministic. To align with these, we can remove the noise input zi and only input the encoded image features zx to the U-Net denoiser. The model still performs well. In this paper, we finally present two versions of Lotus: Lotus-G (generative) with noise input and Lotus-D (discriminative) without noise input, catering to different needs.
8

---

Published as a conference paper at ICLR 2025
4.5 INFERENCE
We initialize the annotation map with standard Gaussian noise zy. and encode the imput image into its latent code z=. The noise z and the image z" are concatenated and fed into the denoiser U-Net model. In our single-step formulation, set = and the switcher to Sy. The denoiser U-Net model then predicts the latent code of the annotation map. The final annotation map is decoded from the predicted latent code via the VAE decoder. For deterministic prediction, we eliminate the Gaussian noise z and only feed the latent code of the input image into U-Net.
The inference pipeline is illustrated in Fig. 10
EXPERIMENTS
5.1 EXPERIMENTAL SETTINGS
Implementation details. We implement Lotus based on Stable Diffusion V2 (Rombach et al., 2022). without text conditioning. During training, fix the time-step 1000. For depth estima- tion, we predict in disparity space, i.e., d = 1/d', where d represents the values in disparity space and d' denotes the true depth. For more details, please see the supplementary materials.
Training Datasets. Both depth and normal estimation are trained on two synthetic dataset covering indoor and outdoor scenes: T Hypersim (Roberts et al., 202T) is a photorealistic synthetic dataset featuring 461 indoor scenes. We use the official training split, which contains approximately 54K samples. After filtering out incomplete samples, around 39K samples remain, all resized to 576 768 for training. @ Virtual KITTI (Cabon et al. (020)) is synthetic street-scene dataset with five urban scenes under various imaging and weather conditions. We utilize four of these scenes for training, comprising about 20K samples. All samples cropped 352 x 1216, with the far plane at 80m.
Following Marigold (Ke et al. 2024), we probabilistically choose one the two datasets and then draw samples from it for each batch (Hypersim 90% and Virtual KITTI 10%).
Evaluation Datasets and Metrics. ① For zero-shot invir-nnv depth estimation, we evaluate Lotus NYUv2 (Silberman et al. 2012), ScanNet (Dai et al., 2017). KITTI (Geiger et al., 2013), ETH3D (Schops et al. 2017). and DIODE (Vasiljevic et al. 2019) using absolute mean relative error (AbsRel), and also report ø1 and 82 values. 2 For surface normal prediction, we employ NYUv2, ScanNet, iBims-1 (Koch et al., 2018), Sintel (Butler et al. (012)) and OASIS (Chen et al.) 2020) datasets, reporting mean angular error (m.) well as the percentage of pixels with an angular error below 11.25' and 30% Please see supplementary materials for further details on the evaluation datasets and metrics.
5.2 QUANTITATIVE COMPARISONS
① For depth estimation (Tab. D. Lotus-G demonstrates promising performance across all evaluation datasets, achieving the overall best rank compared to other generative baselines. Notice that we only require single step denoising process, significantly boosting the inference speed as shown in Fig. B Lotus-D) also performs well, achieving comparable results to DepthAnything series. It is worthy to notice that Lotus trained on only 0.059M images compared to DepthAnything' 's 62.6M images. @ For normal estimation (Tab. 2). both Lotus-G and Lotus-D outperform all other generative and
discriminative methods in terms of average ranking. Please see the supplementary materials for Qualitative Comparisons.
5.3 ABLATION STUDY
As shown in Tab. 3 we conduct ablation studies to validate our designs. Starting with "Direct Adaptation", we incrementally test the effects of different components, such as parameterization types, the single-step diffusion process, and the detail preserver. Initially, we train the model using only the Hypersim dataset to establish baseline. We then expand the training dataset using a mixture dataset strategy by including Virtual KITTI, aiming to enhance the model's generalization ability across different domains. For depth estimation, we further train the model in the disparity space to improve the accuracy. The findings from these ablations validate the effectiveness of our proposed adaptation protocol, demonstrating that each design plays a vital role in optimizing the diffusion models for dense prediction tasks.

---

Published as a conference paper at ICLR 2025
6 CONCLUSION
In this paper, we introduce Lotus, a diffusion-based visual foundation model for dense prediction. Through systematic analysis and specifically tailored diffusion formulation, Lotus finds a way to better fit the rich visual prior from pre-trained diffusion models into dense prediction. Extensive experiments demonstrate that Lotus achieves promising performance on zero-shot depth and normal estimation with minimal training data, paving the way of various practical applications.
Table 1: Quantitative comparison on zero-shot affine-invariant depth estimation between Lotus and SOTA methods. The upper section lists discriminative methods, the lower lists generative ones. The best and second best performances are highlighted. Lotus-G outperforms all others methods while Lotus-D is only slightly inferior to DepthAnything. inddicates results revised by ourselves, following Marigold (Ke al. 2024). *denotes the method relies on pre-trained Stable Diffusion.
Method
Training Datal
AbsRelj flf 621
NYLx2 (Indoor)
AbsRell 61f 62t
KITTI (Outdnor)
ETHED (Various) AbsRell
61f 62f
AhhRRel] 51 62t
ScunNet (Indour)
Absitel 617 82f
DIODE (Various)
Avg Rank
Deptr
320K
11.7
87.5
19.0
70.4
22.8
69.4
10.9
88.2
37.6
63.1
10.6
MiDaS
2M
11.1
88.5
23.6
63.0
18.4
75.2
12.1
84.6
33.2
71.5
10.2
LeRes
354K
9.0
91.6
14.9
78.4
17.1
77.7
9.1
91.7
27.1
76.6
7.8
Omnidata
12.2M
7.4
94.5
14.9
83.5
16.6
77,8
7.5
93.6
33.9
74.2
7.5
DPT
14M
9.8
90.3
10.0
90.1
7.8
94.6
8.2
9344
18.2
75.8
5.8
HDN
6.9
94.8
11.5
12.1
8.0
93.9
24.6
78.0
5.3
GenPercept"
74K
5.6
96.0
99.2
13.0
84.2
97.2
7.0
95.6
98.8
6.2
96.1
99.1
35.7
75.6
86.6
4.9
Diffusion-EEEF EIE-FT
74K
5.4
96.5
99.1
9.6
92.1
98.0
6.4
95.9
98.7
5.8
96.5
98.8
30.3
77.66
157.9
3.6
thpthAnythin
62 6M
4.5
97.9
99.3
7.4
94.6
98.6
13.1
86.5
97.5
4.2
97.8
99.3
26.5
73.4
87.1
3.5
Latus-D (Ours'
59K
5.1
97.2
99.2
8.1
93.1
98..
6.1
97.0
99.1
5.5
96.5
99.0
22,8
73.8
86.2
3,0
DepthAnythig
98.1 99.6
7.6
99.2
88.2
98.3
99.6
26.0
75.9
2.4
GeoWizard
280K
5.6
96.3
14.4
82.0
96.6
95.8
98.4
6.4
95.0
98.4
33.5
72.3
3.3
Marigoló
74K
6.1
95.8
99.0
9.8
91.8
98.7
6.8
95.6
19..
6.9
94.6
98.6
34.7
77.5
89.3
2.9
Marigold
5.5
96.4
99.1
9.9
91.6
98.7
95.9
99.0
6.4
95.2
30.8
77.3
2.1
Lotus-G (Ours))
59K
5.4
96.8
99.2
8.5
92.2
98.4
5.9
97.0
99.2
5.9
95.7
98.8
22.9
72.9
$6.0
1.3
Table 2: Quantitative comparison on zero-shot surface normal estimation between Lotus and SoTA methods. Diseriminative methods shown in the upper section, generative methods in the lower. Both Lotus-D and Lotus-G outperform all other methods. 'refers the Marigold normal model as detailed in this link denotes the method relies on pre-trained Stable Diffusion.
Method
Tmining Data
m.j
NYU12 (Indoor)
11.25" 1 30'1
m.
ScanNet (Indoor)
11.25"
30
m. 11.25" 1 30
[Bims-1 (Indaor)
m.]
11.25"
(Outdoor)
30"1
m.]
OASIS (Various)
11.25't
30"
Avg. Rank
OASIS
110K
29.2
23.8
60.7
33.8
15.4
$2.6
32.6
23.5
57.4
43.1
35.7
7.8
Omnidatn
12.2M
25.1
45.8
73.6
22.9
47.4
73.2
19.0
62.1
80.1
41.5
11.4
42.0
24.9
31.0
71.4
5.9
BESNU
2.5
16.2
58.6
83.5
20.0
58.5
78.2
42.1
11.5
41.2
27.7
24.0
66.6
5.8
74K
18.2
56.3
81.4
17.7
58.3
83.7
64.00
82.0
26.3
Omnidann V2
12.2M
17.2
55.5
83.0
16.2
60.2
84.7
18.2
63.9
81.1
40.5
14.7
43.5
24.2
27.7
74.2
4.4
DSINE
160K
16.4
59.6
83.5
16,2
61.0
84.4
17.1
67.4
82.3
34.9
21.5
52.7
24.4
78.8
72.0
3.1
Diflusion.EBB.FT1
74K
16.5
14.7
66.1
85.1
16.1
69.7
K3.0
22.3
25.2
20.4
74.5
Lotas-D (Ours)*
59K
16.2
59.8
83.9
14.7
64.0
86.1
17.1
66.4
83.0
32.3
22.4
57.0
22.3
31.8
76.1
1.4
Marigald
74K
20.9
50.5
21.3
45.6
18.5
64.7
3.6
GeoWizad'
280K
18.9
81.5
53.8
63.0
80.3
25.2
68.1
SubheeNormal*+
250K
18.6
51.5
81.7
17.1
57.4
84.1
18.2
65.0
82.4
36.7
14.1
50.7
26.5
23.5
68.7
2.1
Lolus-G (Ours)"
39K
16.5
39.4
83,5
15.1
63.9
85,3
17.2
66.2
82.7
33.6
21.0
53.8
22,7
29.4
75.8
1.0
Table 3: Ablation studies on the step-by-step design of our adaptation protocol for fitting pre-trained diffusion models into dense prediction. Here we show the results in monocular depth estimation.
Training Data
NYUv2 (ladoor)
KITTI (Outdoor)
ETH3D (Various)
ScanNet (Indoor)
Method
AbsRee.
61f
527
AbsRet] 81f
821
AbsRel]
511
621
AbsRel] 811
821
Direct Adaptation
39K
11.551
87.692
96.161..2 122
10.
70.403 90.996
19.894
76.464
87.960
15.726
78.885 93.651
+ Po-prrdiction
39K
8.332
92.769
97. 941
17.008
74.969 93.611
11.075
87.952
94.978
10:212
89. 130 97.18)
Single Time-step
39K
5,587
96.272
13.262
83.210
97 237
7,586
94.143
97. 676
6.262
95.394 98,791
+ Detail Preserver
39K
5.555
96.303
99.118
13.170
83.657
47.454
7.147
95.000
98.05%
6,201
95.470 98.814
Mixture Datasel
59K
5.425
96.597
95.156
11.324
87.692
97.70)
6.172
96.077
98, 980
6.024
96.026
99,730
Noise Trput
59K
5.334
96.729
729
99. 198
9.334
92.813
98.795
6.846
95.290
98, 899
5,982
96 287
99.087
Disparity Space (Lotus-G)
59K
5,379
96.736
155
8.521
92,206
98 98.374
5.878
97.024
233
5,925
95.727 98.839
- Noise Lnput (Lotas-)) D)
59K
5.123
97.182
99.15 134
8.117
93.097
98.654
6.147
96.964
99.077
5.494
96.534 99.039
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Gilwon Bae and Andrew J Davison. Aleatoric uncertainty in monocular surface normal estimation.
IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), pp. 1472-1485, 2021.
Gilwon Bae and Andrew Davison. Rethinking inductive biases for surface normal estimation.
IEEE Conference Computer Vision and Pattern Recognition ( (CVPR), 2024.
Yaniv Benny and Lior Wolf. Dynamic dual-output diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11482-11491, 2022.
Daniel Butler, Jonas Wulff, Garrett B Stanley, and Michael J Black. A naturalistic open source
movie for optical flow evaluation. In Computer Vision-ECCV 2012: 12th European Conference on Computer Vision, Florence, Italy, October 7-13, 2012, Proceedings, Part VI 12, pp. 6-152... Springer, 2012.
Yohann Cabon, Naila Murray, and Martin Humenberger. 10Xi::2001.10773, 2020.
Virtual kitti 2. arXiv preprint
Junsong Chen, Jincheng Yu, Chongjian Ge. Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, et al. Pixart-a: Fast training of diffusion transformer for photore- alistic text-to-image synthesis. arXiv preprint arXiv:2310.00426, 2023.
Weifeng Chen, Shengyi Qian, David Fan, Noriyuki Kojima, Max Hamilton, and Jia Deng. Oasis: A large-scale dataset for single image 3d in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 779-688. 2020.
Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nicfsner. Scannet: Richly-annotate 3d reconstructions of indoor scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5828-5839, 2017.
Wenyu Du, Shuang Cheng, Tongxu Luo, Zihan Qiu, Zeyu Huang, Ka Chun Cheung, Reynold Cheng, and Jie Fu. Unlocking continual learning abilities in language models. arXiv preprint arXiv:2406. 17245, 2024.
Ainaz Eftekhar, Alexander Sax, Jitendra Malik, and Amir Zamir. Omnidata: A scalable pipeline for making multi-task mid-level vision datasets from 3d scans. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 10786-10796, 2021.
David Eigen, Christian Puhrsch, and Rob Fergus. Depth map prediction from single image using a multi-scale deep network. Advances in neural information processing systems, 27, 2014.
Huan Fu, Mingming Gong, Chaohui Wang, Kayhan Batmanghelich, and Dacheng Tao. Deep ordinal regression network for monocular depth estimation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2002-2011, 2018.
Xiao Fu, Wei Yin, Mu Hu, Kaixuan Wang, Yuexin Ma, Ping Tan, Shaojie Shen, Dahua Lin, and Xiaoxiao Long. Geowizard: Unleashing the diffusion priors for 3d geometry estimation from single image. arXiv preprint arXiv:2403. (tfi:2203312011 12013, 2024.
Gonzalo Martin Garcia, Karim Abou Zeid, Christian Schmidt, Daan de Geus, Alexander Hermans, and Bastian Leibe. Fine-tuning mage-conniitonal diffusion models is easier than you think. arXiv preprint arXiv:2409.1 1155, 2024.
kitti dataset. The International Journal of Robotics Research, 32(11):1231-1237, 2013. Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances neural information processing systems, 27, 2014.
Jing He, Yiyi Zhou, Qi Zhang, Jun Peng, Yunhang Shen, Xiaoshuai Sun, Chao Chen, and Rongrong Ji. Pixelfolder: An efficient progressive pixel synthesis network for image generation. arXiv preprint ar arXiv:2204.000 00833, 2022.
11

---

Published as a conference paper at ICLR 2025
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-666 6851, 2020.
Mu Hu, Wei Yin, Chi Zhang, Zhipeng Cai, Xiaoxiao Long, Hao Chen, Kaixuan Wang, Gang Yu, Chunhua Shen, and Shaojie Shen. Metric3d v2: A versatile monocular geometric foundation model for zero-shot metric depth and surface normal estimation. arXiv preprint arXiv:2404 15506, 2024.
Yihan Hu, Jiazhi Yang. Li Chen, Keyu Li, Chonghao Sima, Xizhou Zhu, Siqi Chai, Senyao Du, Tianwei Lin, Wenhai Wang, et al. Planning-oriented autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17853-17862, 2023.
Binbin Huang, Zehao Yu, Anpei Chen, Andreas Geiger, and Shenghua Gao. 2d gaussian splatting for geometrically accurate radiance fields. In SIGGRAPH 2024 Conference Papers. Association for Computing Machinery, 2024. doi: 10. 10.114556441111.365774
Oguzhan Fatih Kar, Teresa Yeo, Andrei Atanov, and Amir Zamir. 3d common corruptions and data augmentation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 18963-18974, 2022.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedingy of the TEEE/CVF conference on computer vision and pattern recognition, pp. 4401-4410, 2019.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyz- ing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8110-8 2020.
Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-free generative adversarial networks. Advances in Neural Information Process- ing Systems, 34:852 863, 2021.
Bingxin Ke, Anton Obukhov, Shengyu Huang, Nando Metzger, Rodrigo Caye Daudt, and Kon- rad Schindler. Repurposing diffusion-based image generators for monocular depth estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 9492-9502. 2024.
Tobias Koch, Lukas Liebel, Friedrich Fraundorfer, and Marco Korner. Evaluation of cnn-based single-image depth estimation methods, In Proceedings of the European Conference on Computer Vision (ECCV) Workshops, pp. 0-0, 2018.
Sing Lee, Hung- Yu Tseng, and Ming-Hsuan Yang. Exploiting diffusion prior for generalizable dense prediction. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 7861-7871, 2024.
Jin Han Lee, Byyung-Kyu Han, Dong Wook Ko, and Il Hong Suh. From big to small: Multi-scale local planar guidance for monocular depth estimation. arXiv preprint arXiv: 10326, 2019.
Jiahui Lei, Yijia Weng, Adam Harley, Leonidas Guibas, and Kostas Daniilidis. Mosca: Dynamic gaussian fusion from casual videos via 4d motion scaffolds. arXiv preprint arXiv:2405.17421, 2024.
Xiaoxiao Long, Yuan-Chen Guo, Cheng Lin, Yuan Liu, Zhiyang Dou, Lingjie Liu, Yuexin Ma, Song-Hai Zhang, Mare Habermann, Christian Theobalt, et al. Wonder3d: Single image to 3d using cross-domain diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9970-9980, 2024.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorcalistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv: /2t.::///22/00// 10741 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pp. 8821-8831 PMLR, 2021.
12

---

Published as a conference paper at ICLR 2025
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 1(2):3, 2022.
René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-datased transfer. IEEE transac- tions on pattern analysis and machine intelligence, 44(3):1623-1637 2020.
René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. In Proceedings of the IEEE/CVF international conference computer vision, pp.
2021.
Mike Roberts, Jason Ramapuram, Anurag Ranjan, Atulit Kumar, Miguel Angel Bautista, Nathan Paczan, Russ Webb, and Joshua M Susskind. Hypersim: A photorealistic synthetic dataset for holistic indoor scene understanding. In Proceedings of the IEEF/CVF international conference on computer vision, pp. 10912-10922, 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEF/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomed- ical image segmentation. In Medical image computing and computer-assisted intervention- MICCAI 2015; 18th international conference, Munich, Germany, October 5-9, 2015, proceed- ings, part III 18, pp. 234-241. Springer, 2015.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Informa- lion Processing Systems, 35:36479-36499 2022.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.0055/1 2022.
Thomas Schops, Johannes L. Schonberger, Silvano Galliani, Torsten Sattler, Konrad Schindler, Marc Pollefeys, and Andreas Geiger. A multi-view stereo benchmark with high-resolution images and multi-camera videos. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3260-3269, 2017.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278-25294 2022.
Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and sup- port inference from rgbd images. In Computer Vision- 2012: 12th European Conference Computer Vision, Florence, Italy, October 7-13, 2012, Proceedings, Part V 12, pp. 746-760. Springer, 2012.
Jiaming Song. Chenlin Meng, and Stefano Ermon. preprint arXiv:2010.02255 02502, 2020.
Denoising diffusion implicit models. arXiv
Yunzhou Song, Jiahui Lei, Ziyun Wang, Lingjie Liu, and Kostas Daniilidis. Track everything every- where fast and robustly, 2024.
Igor Vasiljevic, Nick Kolkin, Shanyi Zhang, Ruotian Luo, Haochen Wang, Falcon Z Dai, Andrea Daniele, Mohammadreza Mostajabi, Steven Basart, Matthew R Walter, et al. Diode: A dense 2019.
indoor and outdoor depth dataset. arXiv preprint arXiv:/908.000
Qianqian Wang, Vickie Ye, Hang Gao, Jake Austin, Zhengqi Li, and Angjoo Kanazawa. Shape of motion: 4d reconstruction from single video. arXiv preprint arX.vv2407.1133 2024.
Yuxi Xiao, Qianqian Wang, Shangzhan Zhang, Nan Xue, Sida Peng, Yujun Shen, and Xiaowei Zhou. Spatialtracker: Tracking any 2d pixels in 3d space. In Proceedings of the IEEE/CVE Conference Computer Vision and Pattern Recognition (CVPR), 2024.
13

---

Published as a conference paper at ICLR 2025
Guangkai Xu, Yongtao Ge, Mingyu Liu, Chengxiang Fan, Kangyang Xie, Zhiyue Zhao, Hao Chen, and Chunhua Shen. Diffusion models trained with large data are transferable visual models. arXiv preprint ar arXhv:2403.06090 2024.
Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He. Attngan: Fine-grained text to image generation with attentional generative adversarial net- works. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.
1316-1324, 2018.
Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10371-10381, 2024a.
Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything v2. arXiv preprint arXiv:2406.0940 09444, 2024b.
Chongjie Ye, Lingteng Qiu, Xiaodong Gu. Qi Zuo, Yushuang Wu, Zilong Dong, Liefeng Bo, Yuliang Xiu, and Xiaoguang Han. Stablenormal: Reducing diffusion variance for stable and sharp normal. arXiv preprint arXiv:2406. 0122446666888 16864, 2024.
Wei Yin, Yifan Liu, and Chunhua Shen. Virtual normal: Enforcing geometric constraints for curate and robust depth prediction. IEEE Transactions on Pattern Analysis and Machine Intelli- gence, 44(10):7282-722 7295, 2021a.
Wei Yin, Jianming Zhang, Oliver Wang, Simon Niklaus, Long Mai, Simon Chen, and Chunhua Shen. Learning to recover 3d scene shape from a single image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 204-213, 2021b.
Wei Yin, Chi Zhang, Hao Chen, Zhipeng Cai, Gang Yu, Kaixuan Wang. Xiaozhi Chen, and Chunhua Shen. Metric3d: Towards zero-shot metric 3d prediction from single image. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9043-9053, 2023.
Weihao Yuan, Xiaodong Gu, Zuozhuo Dai, Siyu Zhu, and Ping Tan. Neural window fully- fully-connected crfs for monoeular depth estimation. In Proceedings of the IEEF/CVF conference computer vision and pattern recognition, pp. 3916-3925, 2022.
Ekim Yurtsever, Jacob Lambert, Alexander Carballo, and Kazuya Takeda. A survey of autonomous driving: Common practices and emerging technologies. IEEE access, 1584433-58699 2020.
Yuexiang Zhai, Shengbang Tong, Xiao Li, Mu Cai, Qing Qu, Yong Jae Lee, and Yi Ma. In- vestigating the catastrophic forgetting in multimodal large language models. arXiv preprint 1rXi::2309.10313. 2023.
Chi Zhang, Wei Yin, Billzb Wang, Gang Yu, Bin Fu, and Chunhua Shen. Hierarchical normalization for robust monocular depth estimation. Advances in Neural Information Processing Systems, 35: 14128-14139, 2022.
Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dim- itris N Metaxas. Stackgan: Text to chalo-realistic image synthesis with stacked generative ad- versarial networks. In Proceedings of the IEEE international conference computer vision, pp. 5907-5915, 2017.
Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dim- itris N Metaxas. Stackgan++: Realistic image synthesis with stacked generative adversarial net- works. JEEE transactions on pattern analysis and machine intelligence, 41(8):1947-196) 2018.
Han Zhang, Jing Yu Koh, Jason Baldridge, Honglak Lee, and Yinfei Yang. Cross-modal contrastive learning for text-to-image generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 833 842, 2021.
14


Reviews:
[{"review_id": "ELhIuaihJQ", "reviewer": "Reviewer_kYir", "summary": "The authors simply told us diffusion-based method is not suitable, not necessary, and even harm dense prediction quality. They form a story that they reduce the diffusion step to 1, while adding a regularization of detail preserver, and finetune from a large-scale UNet pretrained on T2I task, to achieve good dense prediction performance.", "questions": "- Do we need to finetune the VAE or decoder for dense representation given the original VAE can be only trained on image data?\n- How did the author evaluate the generative methods given the randomness?\n- Does the conclusions from this paper also apply to other simpler vision tasks? How do the author feel about segmentation task? Also whether diffusion is just not suitable for any tasks requiring pixel-level preservation?\n\nOverall I feel the authors proposed a good observation for dense prediction, but a couple of statements are over-claimed, like foundation model, 1-step 'diffusion' is enough, or detail preserver is necessary.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The authors presented an interesting story of removing 'diffusion' from dense prediction to make things less complicated. \n- If the proposed method and claims are proved to be effective, it shows using a diffusion-based dense prediction method is overkill compared to the efficiency of a 1-step direct prediction. It can be a nice finding.", "weaknesses": "- What is the fundamental difference between the proposed method with a model trained with UNet using latent-space? If there is no diffusion process and the model is doing x0 prediction, why it is still called diffusion method, especially when the authors proposed a discriminator version, does it just fall back to a simple UNet-based method?\n- If the authors claim the powerful prediction quality can come from the strong prior of the pretrained model, but the currently proposed detail preserver, is just learning the reconstruction, and can be lazily learned via the skip connection within the UNet structure, it does not seem to preserve the original generator capability. The reviewer believe after longer training or large batch size training, the performance will degrade more. It is interesting that the detail preserver can enhance the detail by a large margin, given the reconstruction can be easily learned by the residual connection in each layer. \n- If the authors claim it can be served as a visual foundation model, it needs to show more capability of generalization or potentials of other tasks, either through finetuning or adaptation. \n- The qualitative results seem still not strong enough compared with other baselines, and are very close to Marigold.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YRcVZPlb3L", "reviewer": "Reviewer_forA", "summary": "This paper introduces a novel method for dense prediction tasks by fine-tuning a pre-trained diffusion model. To achieve stable and consistent predictions, the method shifts the model’s prediction from epsilon to x0 (clean image) prediction, significantly reducing the variance in outputs. This transition allows for accurate single-step inference, which simplifies the process while maintaining high-quality results.\n\nTo avoid generating excessively smoothed outputs—a common issue in dense prediction with diffusion models—the approach incorporates input frame reconstruction into the training process. This addition serves to preserve finer details and structure in the output predictions, balancing accuracy with realistic visual representation.\n\nThe proposed method is evaluated in a challenging zero-shot scenario, where its performance is benchmarked against both discriminative and generative approaches. Despite being fine-tuned on limited data, the model demonstrates competitive, promising performance, highlighting its robustness and adaptability for dense prediction tasks.", "questions": "I’m curious about the mention of “removing noise” on line 430. In the figure, the noise appears to be concatenated with the input frame—how exactly is this term removed? What value, if any, is set in its place? Does this modification affect any underlying assumptions of diffusion models?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "The paper presents its motivation clearly, leading readers through a coherent explanation of the research objectives and the rationale behind the proposed method. Supported by carefully crafted figures, the narrative flows in a way that helps readers follow the progression of ideas and understand the purpose of each methodological contribution. Each figure complements the text by visually illustrating the transition from epsilon prediction to x0 prediction, the integration of input frame reconstruction, and the steps involved in single-step inference. This combination of clear motivation and visual aids effectively demonstrates how the approach stabilizes predictions and reduces variance, helping readers grasp the value and impact of the proposed method for dense prediction tasks, even in zero-shot settings with limited data.", "weaknesses": "The proposed method, while effective in its application, appears relatively simplistic and may lack substantial innovation. Both the use of  \\mathbf{x}_0  prediction and few-step inference have been previously explored in the diffusion model literature, which limits the originality of these aspects. The  \\mathbf{x}_0  prediction has been extensively discussed in seminal works such as “Denoising Diffusion Probabilistic Models” and Stable Diffusion. Similarly, few-step inference has been considered in studies like “UFOGen: You Forward Once Large Scale Text-to-Image Generation via Diffusion GANs” and “One-step Diffusion with Distribution Matching Distillation.” Consequently, these elements alone may not constitute a significant contribution to the field.\n\nFurthermore, the paper’s evaluation and analysis are narrowly focused on the zero-shot scenario, which, while valuable, provides a limited perspective on the method’s potential. To fully illustrate the strengths of this approach for generative dense prediction, a broader evaluation across diverse scenarios beyond zero-shot would enhance the analysis. Expanding the experiments to include intra-dataset inference, or testing in varying contexts, such as semantic segmentation or optical flow, could highlight the model’s generalization abilities and its versatility in handling different levels of supervision.\n\nI also acknowledge the authors’ claims regarding the model’s performance in minimal-data contexts. Exploring the scalability of the model could offer significant insights, especially given the inherent scalability of diffusion-based methods. Demonstrating the model’s performance with larger, more complex training datasets, similar to “DepthAnything,” would facilitate a deeper understanding of its robustness and computational efficiency. This could be achieved by training the model with datasets of varying sizes and evaluating the performance gains of such a diffusion-based approach. An expanded evaluation would provide a more balanced perspective, showcasing the model’s adaptability beyond minimal-data contexts and emphasizing its potential applicability in real-world, data-rich environments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nedcus0kZO", "reviewer": "Reviewer_myC3", "summary": "This paper introduces a novel approach to dense prediction tasks in computer vision. The authors analyze existing diffusion formulations and identify that traditional noise prediction methods are suboptimal for dense prediction, leading to significant prediction errors. They propose Lotus, a diffusion-based model that directly predicts annotations instead of noise, simplifying the optimization process and enhancing inference speed. The model employs a single-step diffusion process and introduces a detail preserver mechanism to improve accuracy in detail-rich areas. Experimental results demonstrate that Lotus achieves state-of-the-art performance in zero-shot depth and normal estimation tasks, significantly outperforming existing methods while requiring minimal training data.", "questions": "What is the relationship between the proposed network and existing conditional restoration methods? Could this design be extended to other dense prediction tasks, such as segmentation?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "+ This paper introduces a dense prediction method that directly predicts annotations rather than using traditional noise prediction, enhancing prediction stability and enabling one-step inference.\n+ The paper provides an in-depth analysis of existing diffusion models, highlighting instability issues in dense prediction and examining the relationship between prediction variance and time steps.\n+ The proposed method achieves competitive performance across multiple benchmarks, rivaling state-of-the-art approaches.", "weaknesses": "- In this work, the diffusion model functions more like a single-step restoration network, employing a one-step strategy for both training and inference. It may be beneficial to explore using ZtZ_tZt with different ttt values in one-step training and inference.\n- For the detail-preserving component, a task switcher s is selected during each training iteration. Since predicting annotations and reconstructing images are mutually exclusive tasks, the loss function should be adjusted to reflect this \"either-or\" relationship.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cpeGGAPEMB", "reviewer": "Reviewer_d5L4", "summary": "The paper follows a recent trend to formulate dense per-pixel regression tasks (in particular, monocular depth and normal estimation) as conditional image generation, so as to benefit from the strong prior of (latent) denoising diffusion models. Building on Marigold [Ke et al., 2024] and its derivative GeoWizard [Fu et al., 2024], the paper proposes the following technical improvements.\n1) Training the diffusion process to predict the denoised image, rather than the noise, as commonly done for image generation; Arguing that the increased variability when predicting the noise is undesirable for regression tasks.\n2) Faster and less variable one-step inference, including a deterministic variant without initial random noise.\n3) Additional regularisation with the image generation task to enhance the preservation of details.", "questions": "Most important for the rebuttal: please clarify the performance gap of \"direct adaptation\" in Table 3 compared to other recent conditional diffusion approaches. Why do they almost match your best performance despite not using x0-prediction and single-step inference? Or, conversely, why does Lotus not perform decently without those extensions? The answer to this question is likely to significantly influence my final rating (either way).\n\nWhy did you drop DIODE from the test datasets? I am aware that it has certain issues, but since many papers (e.g., Marigold, GeoWizard, BetterDepth, etc.) show DIODE results in addition to exactly the datasets used in the paper, it would be better to also include it, for completeness. Once more, it is not a problem if, for once, you wouldn't get the bold numbers.\n\nSimilarly, why not also include DIODE and OASIS for normals, given that it does not require any training effort and makes the comparisons more complete? Also, some numbers in Table 2 differ from those I have seen floating around (although for some methods, e.g. Marigold, there do not seem to be author-approved or peer-reviewed numbers available at the time of writing). I would recommend to double-check the literature and available code bases for the rebuttal, and again for a possible final submission, to avoid swamping the literature with preliminary and potentially contradictory numbers.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The analysis of predicting the clean image vs. predicting the noise in conditional regression tasks is interesting. It is a relevant observation that, empirically, predicting the clean output directly may perform better in terms of mean prediction error. This could contribute to a better understanding of the mechanisms in diffusion models, and may potentially be interesting also for other conditional uses of those models.\n\nThe regularisation, by switching between predicting the regression target and reconstructing the input image, is plausible. While it is really a simple implementation trick with a fairly hand-wavy explanation, it might be useful for a whole range of applications that repurpose generative models for image analysis tasks.\n\nThe paper provides further evidence for the confluence of conditional generation and discriminative prediction - the evidence is thickening in the last few months that one can move to one-step prediction and even drop the random initialisation (or replace it with a constant zero-input), effectively using a pre-trained multi-step generative model as a “training scheme” or “teacher model” for a discriminative one.", "weaknesses": "The paper does not go very deep in its analysis of the difference between predicting the clean target or the noise. It is a plausible and important message that clean image prediction yields lower errors for certain conditional regression tasks. But the slightly higher mean error of noise prediction could, to some degree, also be an inevitable price to pay for the benefit of proper probabilistic modelling. Is the larger variability really just an algorithmic artefact, or is it perhaps a correct rendition of the underlying predictive uncertainty in the ill-posed task? The findings in the paper are an interesting starting point, but a deeper and more neutral analysis would be helpful.\n\n\nThe findings regarding one-step inference, as well as deterministic prediction without initial noise, are credible and important for the further development of the field. That being said, they should be discussed in the context of other recent literature, in particular [Garcia et al., 2024]. Yes, that paper only came out as a preprint 2 weeks before the ICLR deadline. I am aware that it does not qualify as \"prior art\", and I am not suggesting that there would be any novelty issue. But I would opine that, in this specific case, the work of Garcia et al. should be discussed and not ignored altogether. The findings - one-step training and inference, zero-noise initialisation - are very similar to one of the present paper's main messages. To maximally benefit the research community it will be important to paint a complete picture. Importantly, Garcia et al. show that these improvements are equally possible with the standard formulation (with noise prediction), after fixing a small but impactful bug in the DDIM scheduler of the most popular implementation (HuggingFace Diffusers). Again: I applaud the authors of the present paper for independently confirming that 1-step and deterministic inference are possible based on a pre-trained diffusion model. Still, since [Garcia et al.] is in some sense a bugfix and shows that these improvements apply to pretty much all of the conditional diffusion schemes of the past year if it were not for an incorrect implementation of the scheduler, I really think the authors should relate, compare and discuss that work in the final version of the paper.\n- [Garcia et al., 2024] Garcia, GM, Abou Zeid, K, Schmidt, C, de Geus, D, Hermans, A, Leibe, B (2024). Fine-tuning image-conditional diffusion models is easier than you think. arXiv:2409.11355\n(for the record: I am not an author of it)\n\nMy main technical criticism, where I really hope the authors could provide some insight during the discussion, comes from the ablation study in Table 3. According to that table, the “direct adaptation” of the diffusion model works a lot worse (2-3 times higher AbsRel), and both x0-prediction and single-timestep inference are needed to reach good performance. But the baselines Marigold and GeoWizard both reach comparable performance (NYU ~5.5, KITTI ~11, ETH3D ~6.5, ScanNet ~6.5) without those modifications, just doing “direct adaptation”. Why is the Lotus version of “direct adaptation” so much worse than prior art, and would the modifications have the same impact if one started from one of the competitors that already achieves low errors with direct adaptation?\n\nI also find the \"league table mentality\" in the experiments (which the paper shares with many recent ones) really irritating. I would urge the authors to take a more objective, scientific stance and interpret the experimental results in a sensible and candid manner instead of desperately looking for a “win”. My interpretation of the bottom 4 lines in Table 1 is not “Lotus-G is the best”, but rather “all methods based on Stable Diffusion have comparable performance”. And it is totally ok to say that - your method is still fast, elegant and efficient in terms of training data and brings interesting insights, there is no need to over-interpret differences of mostly <0.5 percent points (mischievously, one could say that the only significant differences are on KITTI, where Lotus does not win - but then again that dataset is problematic anyway and should not be taken too serious).\n\nThe language and formulations are sometimes slightly imprecise or awkward, perhaps this can still be improved. Nothing serious that would seriously impair understanding, but still. Random examples:\n- “first, their is a pair of autoencoders E(.), D(.)”. I would argue that there is only a single auto-encoder: encoder and decoder parts together form one auto-encoder, not a pair.\n- “the type of parameterization is a vital configuration”. What exactly does this mean? The type of parametrisation may be important, or the configuration of the diffusion loop may be important, but something seems to be redundant in this sentence.\n\n(there are more such small language and style issues, please check)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0RGFkWtnXX", "reviewer": "Reviewer_vkmx", "summary": "This paper introduces a new way of using pre-trained text-to-image diffusion model for dense prediction tasks, including monocular depth estimation and surface normal estimation. Different from existing methods that use the pre-trained diffusion model without considering the dense prediction tasks, authors present several meaningful modifications. First, the parameterization type is changed in a way of predicting original image or annotation ($x_o$) instead of estimating noise. Second, the number of time-steps is reduced (even to one). Third, to enhance the capability of dealing with image details, authors utilize a task switcher that enables the proposed denoiser to generate annotation or reconstruct an input image.", "questions": "Refer to the comments in the weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) The paper was written clearly and is easy to understand.\n2) The three modifications tailored to dense prediction tasks seem to be effective.", "weaknesses": "1) The two tasks (depth and surface normal estimation) presented in this paper are rather insufficient. It would be better to show the possibility of using this framework in other tasks such as segmentation and detection.\n\n2) More explanations are necessary for the detail preserver, as it is hard to understand why the image reconstruction task improves the details of estimated annotations. The image reconstruction and dense prediction tasks share the U-net denoiser, so it would be harmful to perform the two heterogeneous tasks within the single network due to some unexpected interferences. This part needs to be clarified with more experiments and detailed analysis.\n\n3) It was shown experimentally that decreasing the number of denoising steps is more effective in the dense prediction tasks, but no theoretic analysis is given. For instance, what attributes of dense prediction tasks make it possible to reduce the number of denoising steps, unlike the image generation task? Also, does it work for other prediction tasks such as segmentation and detection?\n\nOverall, this work introduces an interesting framework for dense prediction tasks based on pre-trained text-to-image diffusion model, demonstrating the effectiveness in the monocular depth estimation and surface normal estimation tasks. However, the performance should also be validated in different tasks (segmentation and detection), and more analysis on the detail preserver and the number of denoising steps are necessary.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)