What Matters When Repurposing Diffusion Models for General Dense Perception Tasks?

## Abstract

Abstract Extensive pre-training with large data is indispensable for downstream geometry and semantic visual perception tasks. Thanks to large-scale text-to-image (T2I) pretraining, recent works show promising results by simply fine-tuning T2I diffusion models for dense perception tasks. However, several crucial design decisions in this process still lack comprehensive justification, encompassing the necessity of the multi-step stochastic diffusion mechanism, training strategy, inference ensemble strategy, and fine-tuning data quality. In this work, we conduct a thorough investigation into critical factors that affect transfer efficiency and performance when using diffusion priors. Our key findings are: 1) High-quality fine-tuning data is paramount for both semantic and geometry perception tasks. 2) The stochastic nature of diffusion models has a slightly negative impact on deterministic visual perception tasks. 3) Apart from fine-tuning the diffusion model with only latent space supervision, task-specific image-level supervision is beneficial to enhance fine-grained details.
These observations culminate in the development of GenPercept, an effective deterministic one-step fine-tuning paradigm tailed for dense visual perception tasks.
Different from the previous multi-step methods, our paradigm has a much faster inference speed, and can be seamlessly integrated with customized perception decoders and loss functions for image-level supervision, which is critical to improving the fine-grained details of predictions.
Comprehensive experiments on diverse dense visual perceptual tasks, including monocular depth estimation, surface normal estimation, image segmentation, and matting, are performed to demonstrate the remarkable adaptability and effectiveness of our proposed method.
Code is at: https://github.com/aim-uofa/GenPercept/

## 1 Introduction

Recent studies have explored the transferability of text-to-image (T2I) diffusion models to dense visual perception tasks, such as geometry estimation (Ke et al., ; Lee et al., ; Fu et al., ; Gui et al., ; Ye et al., ), image segmentation (Van Gansbeke & De Brabandere, ; Lee et al., ), and inverse rendering (Chen et al., ; Kocsis et al., ; Zeng et al., ). While these works have demonstrated impressive results by repurposing diffusion models for estimating fine-grained geometric and semantic dense prediction maps, the critical design choices made in transferring diffusion models to other dense perception tasks still lack comprehensive justification. This makes it challenging to determine the optimal strategy for achieving the best performance.

For example, Ke et al. () align the visual perception process with the denoising process of Stable Diffusion by fine-tuning all U-Net parameters. They highlight the significance of ”multi-resolution noise” in the forward diffusion process during training, aiming to obtain clean predictions by gradually removing Gaussian noise. On the other hand, (Lee et al., ) modify the forward diffusion process by interpolating perception annotations with RGB images instead of using Gaussian noise, and only train the low-rank adaptation (LoRA) Hu et al. () parameters while keeping the U-Net frozen. To the best of our knowledge, the effective components of these approaches have not been thoroughly investigated, leading to uncertainty about which aspects are most crucial for success.

In this work, we examine the design space of repurposing diffusion models for dense visual perception tasks, and explore the key question: What factors are important when adapting diffusion models for general dense perception tasks?

To answer this question, we rethink the importance of both fine-tuning protocols and fine-tuning data. From the perspective of fine-tuning protocols, we categorize recent methods into two main groups: stochastic multi-step generation and deterministic multi-step generation. We explore several critical design spaces, including the diffusion mechanism, key architectural components, training methodologies, and data quality. Our key observations are as follows: 1) The stochastic nature of diffusion models somewhat conflicts with the deterministic requirements of perception tasks, and the amount of noise used significantly impacts perception performance. 2) Strict adherence to traditional diffusion processes is unnecessary. Single-step inference provides similar performance with significantly faster execution. 3) High-quality synthetic fine-tuning data is crucial for geometry perception tasks. From the perspective of fine-tuning data quality, we conduct comprehensive dataset ablation studies on both synthetic datasets and real-world datasets.

Based on the aforementioned observations, we propose GenPercept (see fig. 1), a deterministic fine-tuning paradigm featuring a remarkably simple one-step inference pipeline, an optional customized decoder, and an easily adaptable pixel-specific customized loss. We conduct extensive quantitative and qualitative experiments on a wide range of fundamental visual dense perception tasks, including monocular depth estimation, surface normal estimation, image segmentation, and matting to demonstrate the effectiveness and universality of our method.

In conclusion, our contributions can be summarized as follows: 1) We systematically analyze the design space of current fine-tuning protocols, considering both model architecture and dataset selection, through comprehensive ablation studies. 2) Based on these insights, we propose GenPercept, a simple paradigm that harnesses the power of the pre-trained UNet from diffusion models for generalizable dense visual perception tasks.

Figure: Figure 1: Comparisons of three different pipelines. Our GenPercept enables one-step inference and supports pixel-wise losses and customized decoders to replace the cumbersome VAE decoder. We also extend GenPercept to five dense perception tasks including monocular depth estimation, surface normal estimation, dichotomous image segmentation, semantic segmentation, and image matting.
Refer to caption: x1.png

## 2 preliminary

Standard diffusion models (Rombach et al., ; Chen et al., ; Song et al., ; Ho et al., ) define a forward process to inject Gaussian noise $\bm{\epsilon}$ into the input data and a reverse denoising process to estimate noises and generate clean images with a denoiser $\bm{v}_{\theta}$. While diffusion models generate images from noise, the visual perception tasks take RGB images $\mathbf{x}$ as input and estimate the visual perception labels $\mathbf{y}$. Aiming at transferring the prior knowledge of pre-trained diffusion models to computer vision tasks, previous works have proposed to align with the diffusion process and reformulate visual perception tasks as a multi-step denoising process, especially on the monocular depth estimation task. We review the main high-level ideas of previous algorithms in this section and offer detailed formulations in the supplementary material.

Recent works mainly explore two alignment choices. For the stochastic multi-step generation (Ke et al., ; Fu et al., ; Gui et al., ), the perception tasks are reframed as an RGB image-conditioned denoising process. For training, they add random Gaussian noise $\bm{\epsilon}$ to the ground truth latent $\mathbf{z}^{(\mathbf{y})}$ in the forward diffusion process. For inference, the RGB latent $\mathbf{z}^{(\mathbf{x})}$ is concatenated with the noisy depth latent $\hat{\mathbf{z}}^{(\mathbf{y})}_{t}$ before being sent to U-Net $\bm{v}_{\theta}$ for the $t$-th denoising timestep. Clean depth latent $\hat{\mathbf{z}}^{(\mathbf{y})}_{0}$ will be available after denoising T times. The inference denoising process and forward diffusion process are shown in fig. 1 (a) and fig. 2 (a) separately. These methods perform well but face two significant drawbacks. Firstly, its stochastic nature conflicts with the deterministic nature of perception tasks, which may have a negative impact on performance. Secondly, the high-computation ensemble strategy poses challenges across various perception tasks.

To integrate the deterministic requirements of visual perception tasks with the stochastic diffusion models, deterministic multi-step generation methods such as DMP (Lee et al., ) reformulate the forward diffusion process as a blending interpolation between RGB images and ground-truth labels. The inference denoising process and training forward diffusion process are shown in fig. 1 (b) and fig. 2 (a), respectively. Although it addresses the uncertainty inherent in the process, the persistence of RGB texture in the generated perception predictions can detrimentally impact accuracy when visualizing the estimated outcomes. This phenomenon is particularly evident in tasks such as surface normal estimation, as exemplified in fig. 2 (c).

Figure: Figure 2: Visualization of different noise forms and proportions in the forward diffusion process. (a) Compared to Gaussian Noise, interpolating RGB images with ground-truth labels can bring less image nature loss. (b) The relationship between the proportion of noise $\sqrt{\bar{\alpha}_{t}}$ and the ($\beta_{start}$, $\beta_{end}$) hyperparameters. (Please see the supplementary for detailed formulations.) (c) Low $\beta$ values during the training of deterministic multi-step generation will lead to the retention of RGB information. Increasing $\beta$ enhances the training difficulty and addresses this issue.
Refer to caption: x2.png

## 3 Diffusion Models for Visual Perception Tasks

In this section, we explore the necessity and highlight the findings of the multi-step stochastic diffusion mechanism, prior knowledge hides behind the architectural components, training strategy, and fine-tuning data quality. The default experimental setting is similar to Ke et al. () and can be found in supplementary material. We select stochastic method Marigold (Ke et al., ) and the deterministic method DMP (Lee et al., ) as our baseline.

### 3.1 The Form and Proportion of Noise in the Forward Diffusion Process

In our initial attempt to integrate the core concepts of Marigold and DMP within a unified codebase, we observed that the performance of the ”baseline DMP” was significantly inferior to that of the ”baseline Marigold.” This discrepancy led us to uncover two notable phenomena: 1) Although deterministic methods are theoretically expected to outperform stochastic, multi-step generation processes for visual perception tasks due to their reduced dependency on random Gaussian noise, our findings contradicted this assumption, with deterministic methods showing inferior performance. 2) The ”multi-resolution noise” strategy introduced by Marigold substantially enhances accuracy.

To further investigate these phenomena, we hypothesize that the challenge associated with denoising from Gaussian noisy latents is substantially more difficult than that from RGB-blended noisy latents. These two different blending paradigms are illustrated in fig. 2(a). We argue that the Gaussian noisy latent less closely approximates natural images and potentially increases the complexity of training, which can enhance the training difficulty. To prove it, we modified the noise blending proportions for both methods by adjusting the $\beta$ values within the DDPM scheduler, effectively increasing the noise component during the forward diffusion process, as depicted in fig. 2(b).

Our quantitative and qualitative analyses, presented in table 1 and fig. 2(c), indicate that increasing training challenges lead to improved model performance. As the $\beta$ value increases, the impact of ”multi-resolution noise” diminishes. However, unlike DMP, which continues to show performance improvements, Marigold’s performance begins to be slightly unstable but shows a rough improvement trend when the $\beta$ value is sufficiently high. We consider it brought by the randomness of the Gaussian Noise, which dominates most components of the input latent.

When $\beta$ reaches 1, the noisy label latent input effectively becomes a pure noise latent, akin to the RGB latent in DMP. Consequently, the target output of the ”v-prediction” diffusion model shifts to the negative value of the ground truth latent, eliminating any intermediate state of blending between RGB noise latent and ground-truth latent. The denoising process becomes repeats of “predict target label $\hat{\mathbf{z}}^{(\mathbf{y})}_{0}$ from RGB image $\mathbf{z}^{(\mathbf{x})}$ and add (RGB) noise back to the RGB image $\mathbf{z}^{(\mathbf{x})}$”, whose estimation results in any timestep $t$ are exactly identical. Therefore, we propose to reduce the DDIM steps to one and call it ”deterministic one-step perception”. The resulting inference speeds are significantly faster, with performance remaining largely consistent. We consider it as “our baseline” for the subsequent analyses.

**Table 1: Comprehensive quantitative comparisons about the impact of noise forms and proportions in the forward diffusion process on monocular depth estimation. Visualizations of different noise forms and the effect of $\beta$ values are shown in fig. 2. The performance of DMP improves steadily, while Marigold shows initial improvements followed by a decline. When $\beta$ reaches 1, the inference process can be reduced to one step without compromising performance. “Rank” means the average rank of ten evaluation performance (smaller is better).**
| Type | Noise | Multi-res | Steps | ($\beta_{start}$, $\beta_{end}$) | KITTI | NYU | ScanNet | DIODE | ETH3D | Rank |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Form | Noise | AbsRel$\downarrow$ | $\delta_{1}$$\uparrow$ | AbsRel$\downarrow$ | $\delta_{1}$$\uparrow$ | AbsRel$\downarrow$ | $\delta_{1}$$\uparrow$ | AbsRel$\downarrow$ | $\delta_{1}$$\uparrow$ | AbsRel$\downarrow$ | $\delta_{1}$$\uparrow$ |  |  |  |  |
| Marigold | Gaussian | $\checkmark$ | 10 | (0.0002125, 0.003) | 0.358 | 0.462 | 0.297 | 0.555 | 0.246 | 0.625 | 0.494 | 0.565 | 0.267 | 0.640 | 7.0 |
| Marigold | Gaussian | $\checkmark$ | 10 | (0.000425, 0.006) | 0.122 | 0.854 | 0.106 | 0.887 | 0.136 | 0.829 | 0.345 | 0.716 | 0.086 | 0.927 | 5.8 |
| baseline Marigold | Gaussian | $\checkmark$ | 10 | (0.00085, 0.012) | 0.099 | 0.909 | 0.063 | 0.956 | 0.075 | 0.937 | 0.316 | 0.764 | 0.075 | 0.947 | 3.6 |
| Marigold | Gaussian | $\checkmark$ | 10 | (0.0034, 0.048) | 0.100 | 0.906 | 0.057 | 0.963 | 0.063 | 0.957 | 0.308 | 0.768 | 0.074 | 0.948 | 2.3 |
| Marigold | Gaussian | $\checkmark$ | 10 | (0.1360, 0.192) | 0.119 | 0.861 | 0.058 | 0.963 | 0.061 | 0.961 | 0.315 | 0.760 | 0.073 | 0.950 | 2.8 |
| Marigold | Gaussian | $\checkmark$ | 10 | (0.5440, 0.768) | 0.124 | 0.852 | 0.060 | 0.961 | 0.064 | 0.958 | 0.322 | 0.749 | 0.079 | 0.943 | 4.7 |
| Marigold | Gaussian | $\checkmark$ | 10 | (1.0, 1.0) | 0.104 | 0.897 | 0.055 | 0.965 | 0.059 | 0.962 | 0.312 | 0.762 | 0.069 | 0.955 | 1.7 |
| Marigold | Gaussian | $\checkmark$ | 1 | (1.0, 1.0) | 0.104 | 0.897 | 0.055 | 0.965 | 0.059 | 0.962 | 0.312 | 0.762 | 0.069 | 0.955 | - |
| Marigold | Gaussian | $\times$ | 10 | (0.0002125, 0.003) | 0.587 | 0.255 | 0.337 | 0.490 | 0.257 | 0.604 | 0.600 | 0.469 | 0.372 | 0.503 | 7.0 |
| Marigold | Gaussian | $\times$ | 10 | (0.000425, 0.006) | 0.536 | 0.289 | 0.313 | 0.527 | 0.248 | 0.621 | 0.565 | 0.499 | 0.328 | 0.575 | 6.0 |
| baseline Marigold | Gaussian | $\times$ | 10 | (0.00085, 0.012) | 0.153 | 0.807 | 0.162 | 0.802 | 0.187 | 0.737 | 0.411 | 0.641 | 0.157 | 0.826 | 5.0 |
| Marigold | Gaussian | $\times$ | 10 | (0.0034, 0.048) | 0.101 | 0.907 | 0.058 | 0.963 | 0.066 | 0.954 | 0.309 | 0.765 | 0.074 | 0.950 | 2.4 |
| Marigold | Gaussian | $\times$ | 10 | (0.1360, 0.192) | 0.115 | 0.870 | 0.056 | 0.965 | 0.060 | 0.961 | 0.313 | 0.763 | 0.072 | 0.953 | 2.3 |
| Marigold | Gaussian | $\times$ | 10 | (0.5440, 0.768) | 0.124 | 0.848 | 0.059 | 0.963 | 0.063 | 0.958 | 0.318 | 0.752 | 0.077 | 0.946 | 3.7 |
| Marigold | Gaussian | $\times$ | 10 | (1.0, 1.0) | 0.102 | 0.901 | 0.054 | 0.966 | 0.059 | 0.962 | 0.312 | 0.762 | 0.071 | 0.955 | 1.5 |
| Marigold | Gaussian | $\times$ | 1 | (1.0, 1.0) | 0.102 | 0.901 | 0.054 | 0.966 | 0.059 | 0.962 | 0.312 | 0.762 | 0.071 | 0.955 | - |
| DMP | RGB | $\times$ | 10 | (0.0002125, 0.003) | 0.476 | 0.336 | 0.267 | 0.601 | 0.216 | 0.677 | 0.457 | 0.588 | 0.185 | 0.757 | 6.9 |
| DMP | RGB | $\times$ | 10 | (0.000425, 0.006) | 0.265 | 0.630 | 0.201 | 0.072 | 0.195 | 0.717 | 0.386 | 0.674 | 0.116 | 0.880 | 6.1 |
| baseline DMP | RGB | $\times$ | 10 | (0.00085, 0.012) | 0.134 | 0.837 | 0.117 | 0.871 | 0.147 | 0.808 | 0.353 | 0.721 | 0.093 | 0.919 | 5.0 |
| DMP | RGB | $\times$ | 10 | (0.0034, 0.048) | 0.107 | 0.890 | 0.077 | 0.939 | 0.087 | 0.923 | 0.318 | 0.766 | 0.078 | 0.940 | 3.8 |
| DMP | RGB | $\times$ | 10 | (0.1360, 0.192) | 0.107 | 0.890 | 0.063 | 0.959 | 0.068 | 0.955 | 0.305 | 0.773 | 0.073 | 0.948 | 2.2 |
| DMP | RGB | $\times$ | 10 | (0.5440, 0.768) | 0.106 | 0.897 | 0.061 | 0.959 | 0.066 | 0.952 | 0.309 | 0.768 | 0.075 | 0.945 | 2.3 |
| DMP | RGB | $\times$ | 10 | (1.0, 1.0) | 0.100 | 0.902 | 0.053 | 0.966 | 0.059 | 0.961 | 0.309 | 0.768 | 0.068 | 0.956 | 1.2 |
| Our baseline | RGB | $\times$ | 1 | (1.0, 1.0) | 0.100 | 0.902 | 0.053 | 0.966 | 0.059 | 0.961 | 0.309 | 0.768 | 0.068 | 0.956 | - |

#### Finding 1.

#### Finding 1.

### 3.2 Where Does the Rich Visual Knowledge Reside in Diffusion Models?

Based on the baseline we proposed in §3.1, we conduct detailed ablation studies to thoroughly investigate the necessity of each component of Stable Diffusion. Results are reported in table 2.

**Table 2: Explorations on the impact of the Stable Diffusion components on depth estimation. Customized decoders and losses can also enable inference acceleration and performance improvement.**
| Setting | Loss | KITTI | NYU | ScanNet | DIODE | ETH3D |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| AbsRel$\downarrow$ | $\delta_{1}$$\uparrow$ | AbsRel$\downarrow$ | $\delta_{1}$$\uparrow$ | AbsRel$\downarrow$ | $\delta_{1}$$\uparrow$ | AbsRel$\downarrow$ | $\delta_{1}$$\uparrow$ | AbsRel$\downarrow$ | $\delta_{1}$$\uparrow$ |  |  |
| Our baseline | MSE (Latent) | 0.100 | 0.902 | 0.053 | 0.966 | 0.059 | 0.961 | 0.309 | 0.768 | 0.068 | 0.956 |
| Train U-Net from scratch | MSE (Latent) | 0.219 | 0.650 | 0.186 | 0.736 | 0.183 | 0.729 | 0.426 | 0.614 | 0.185 | 0.741 |
| Train VAE decoder from scratch | MSE (Image) | 0.096 | 0.916 | 0.055 | 0.964 | 0.058 | 0.964 | 0.302 | 0.759 | 0.071 | 0.950 |
| Baseline + Image MSE loss | MSE (Image) | 0.097 | 0.915 | 0.054 | 0.964 | 0.059 | 0.964 | 0.305 | 0.760 | 0.071 | 0.953 |
| Baseline + Image customized loss | MSE + SSI + | 0.094 | 0.923 | 0.052 | 0.966 | 0.056 | 0.965 | 0.302 | 0.767 | 0.066 | 0.967 |
| Grad. (Image) |  |  |  |  |  |  |  |  |  |  |  |
| Train DPT decoder from scratch | MSE (Image) | 0.099 | 0.912 | 0.055 | 0.964 | 0.058 | 0.963 | 0.302 | 0.759 | 0.069 | 0.956 |

Denoiser.
We reinitialize the U-Net parameter and train it from scratch on the same datasets. Without prior knowledge of large data from LAION-5B, the network performs poorly and loses the generalization. This indicates that most of the prior knowledge is stored in the denoiser module.

VAE AutoEncoder.
The VAE encoder’s original architecture is kept intact to maintain the consistency of the encoding process. For the VAE decoder, we train it from scratch with image pixel MSE loss. Without pre-trained parameters of the VAE decoder, it still performs well.

Customized Head and Loss.
The deterministic one-step perception pipeline enables customized heads and loss functions for training. By utilizing a DPT decoder or the loss functions of DepthAnythingv2 (Yang et al., ), we can implement a lightweight decoder that supervises pixel-wise information at a higher resolution rather than latent features at 1/8 resolution. This approach can not only accelerate inference times but also enhance the acquisition of fine-grained details.

#### Finding 2.

#### Finding 2.

### 3.3 What About the Timesteps and Text Prompts?

The timesteps and text prompts are crucial elements in utilizing the Stable Diffusion model to generate diverse images. We conducted ablation studies to investigate their significance. The results reported in table 3 indicate a negligible difference between various settings. Owing to the inherent certainty associated with visual perception tasks, the diversity typically offered by the textual inputs appears to be unnecessary. Similarly, the utility of timesteps is reduced, as the single-step paradigm does not require progressive denoising.

**Table 3: Quantitative comparisons among different timesteps and text prompts of diffusion models on depth estimation.**
| Setting | Text Prompt | Train / Infer | KITTI | NYU | ScanNet | DIODE | ETH3D |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Timesteps | AbsRel$\downarrow$ | $\delta_{1}$$\uparrow$ | AbsRel$\downarrow$ | $\delta_{1}$$\uparrow$ | AbsRel$\downarrow$ | $\delta_{1}$$\uparrow$ | AbsRel$\downarrow$ | $\delta_{1}$$\uparrow$ | AbsRel$\downarrow$ | $\delta_{1}$$\uparrow$ |  |  |
| Our baseline | “” | Random / 1 | 0.100 | 0.902 | 0.053 | 0.966 | 0.059 | 0.961 | 0.309 | 0.768 | 0.068 | 0.956 |
| Valid text input | “A high quality RGB image” | Random / 1 | 0.101 | 0.900 | 0.053 | 0.967 | 0.058 | 0.964 | 0.312 | 0.762 | 0.070 | 0.954 |
| Random text input | “F3@qV!k2*#Zp^n%1Lz” | Random / 1 | 0.099 | 0.904 | 0.054 | 0.965 | 0.059 | 0.963 | 0.311 | 0.763 | 0.069 | 0.955 |
| Timestep1 | “” | 1 / 1 | 0.100 | 0.906 | 0.054 | 0.965 | 0.060 | 0.961 | 0.304 | 0.769 | 0.069 | 0.956 |
| Timestep500 | “” | 500 / 500 | 0.102 | 0.897 | 0.053 | 0.966 | 0.059 | 0.961 | 0.307 | 0.765 | 0.068 | 0.956 |
| Timestep900 | “” | 900 / 900 | 0.105 | 0.891 | 0.054 | 0.966 | 0.058 | 0.964 | 0.309 | 0.762 | 0.068 | 0.953 |

#### Finding 3.

#### Finding 3.

### 3.4 How to Leverage the U-Net’s Prior Knowledge?

The significance of the denoiser cannot be overstated. However, the strategies for its utilization are worth careful consideration. Should we freeze the denoiser, utilize its intermediate features, and merely fine-tune the decoder for specific tasks? Alternatively, could we employ LoRA (Hu et al., ) instead of extensively fine-tuning the entire denoiser? Unfortunately, the evidence suggests that neither approach is ideal. As illustrated in table 4, freezing the denoiser significantly compromises performance. Although incorporating LoRA offers some advantages, it can not fully leverage the potential of denoiser, especially with regular LoRA ranks of 4 and 16. This limitation likely stems from the substantial differences between the noise-to-image denoising process and the image-to-perception prediction task.

**Table 4: Explorations on the paradigms to leverage U-Net’s prior knowledge on depth estimation.**
| Setting | LoRA | KITTI | NYU | ScanNet | DIODE | ETH3D |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Rank | AbsRel$\downarrow$ | $\delta_{1}$$\uparrow$ | AbsRel$\downarrow$ | $\delta_{1}$$\uparrow$ | AbsRel$\downarrow$ | $\delta_{1}$$\uparrow$ | AbsRel$\downarrow$ | $\delta_{1}$$\uparrow$ | AbsRel$\downarrow$ | $\delta_{1}$$\uparrow$ |  |
| Our baseline | - | 0.100 | 0.902 | 0.053 | 0.966 | 0.059 | 0.961 | 0.309 | 0.768 | 0.068 | 0.956 |
| Freeze U-Net + Train DPT decoder | - | 0.144 | 0.803 | 0.086 | 0.931 | 0.097 | 0.911 | 0.309 | 0.768 | 0.068 | 0.956 |
| Train U-Net with LoRA | 4 | 0.211 | 0.644 | 0.095 | 0.914 | 0.100 | 0.902 | 0.372 | 0.689 | 0.121 | 0.864 |
| Train U-Net with LoRA | 16 | 0.166 | 0.746 | 0.085 | 0.931 | 0.087 | 0.927 | 0.352 | 0.712 | 0.104 | 0.901 |
| Train U-Net with LoRA | 64 | 0.138 | 0.817 | 0.077 | 0.944 | 0.079 | 0.940 | 0.336 | 0.734 | 0.089 | 0.930 |
| Train U-Net with LoRA | 256 | 0.133 | 0.827 | 0.069 | 0.952 | 0.073 | 0.947 | 0.325 | 0.745 | 0.088 | 0.933 |
| Train U-Net with LoRA | 1024 | 0.125 | 0.849 | 0.067 | 0.955 | 0.074 | 0.947 | 0.324 | 0.747 | 0.084 | 0.939 |

#### Finding 4.

#### Finding 4.

### 3.5 Is the Training Data Quality Essential?

The quality of annotations in real datasets is often lower compared to synthetic datasets, where data is precisely rendered via simulators. In table 5, we explore the impact of data quality on the fine-tuning process. We sample the same distribution of real data, consisting of 90% from approximately 50K indoor images from the Taskonomy dataset (Zamir et al., ) and 10% from about 40K outdoor images from the Cityscapes dataset (Cordts et al., ). With lower annotation quality, the model achieves slightly worse performance. Also, the visualization in the supplementary material indicates that noisy data significantly influences detailed predictions in visual perception tasks.

**Table 5: Investigations into the impact of training data quality on depth estimation.**
| Data Quality | Datasets | KITTI | NYU | ScanNet | DIODE | ETH3D |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| AbsRel$\downarrow$ | $\delta_{1}$$\uparrow$ | AbsRel$\downarrow$ | $\delta_{1}$$\uparrow$ | AbsRel$\downarrow$ | $\delta_{1}$$\uparrow$ | AbsRel$\downarrow$ | $\delta_{1}$$\uparrow$ | AbsRel$\downarrow$ | $\delta_{1}$$\uparrow$ |  |  |
| Synthetic Data | Hypersim (50K) + Virtual KITTI (40K) | 0.100 | 0.902 | 0.053 | 0.966 | 0.059 | 0.961 | 0.309 | 0.768 | 0.068 | 0.956 |
| Real Data | Taskonomy (50K) + Cityscapes (40K) | 0.123 | 0.857 | 0.055 | 0.966 | 0.062 | 0.958 | 0.293 | 0.762 | 0.074 | 0.947 |

#### Finding 5.

#### Finding 5.

### 3.6 Summary of the obervations

Based on the preceding analysis, an effective approach to leveraging the prior knowledge of diffusion models is to use them as single-step deterministic perception estimators. This can be done with either a VAE decoder or a customized lightweight decoder. Additionally, employing pixel-specific customized loss functions can further enhance detail and overall performance. We compare our deterministic single-step perception method with previous multi-step paradigms in fig. 1. In the following section, we extend these findings to a broader set of visual perception tasks, including surface normal estimation, semantic image segmentation, dichotomous image segmentation, and image matting.

## 4 Experiments on Various Dense Visual Perceptual Tasks

In this section, we empirically show
the robust transfer ability of our GenPercept on diverse visual tasks.
Unless specified otherwise, we freeze the VAE AutoEncoder and fine-tune the U-Net of Stable Diffusion v2.1 to estimate the ground-truth label latent for 30000 iterations, with a resolution of (768, 768), a batch size of 32, and a learning rate of 3e-5. Different customized loss functions are utilized to improve the performance further on dense visual perception tasks.

### 4.1 Geometric Estimation

Monocular Depth Estimation. The monocular depth estimation aims to predict the vertical distance between the observed object and the camera from an RGB image. The estimated depth is formulated as affine-invariant depth (Yin et al., ; Ranftl et al., ; ), and should be recovered by performing least square regression with the ground truth. The evaluation is performed on five zero-shoft datasets including KITTI (Geiger et al., ), NYU (Silberman et al., ), ScanNet (Dai et al., ), DIODE (Vasiljevic et al., ), and ETH3D (Schops et al., ). We compute the absolute relative error (AbsRel$\downarrow$) and percentage of accurate valid depth pixels ($\delta_{1}$$\uparrow$). Invalid regions are filtered out and the metrics are averaged on all images.

**Table 6: Quantitative comparison of affine-invariant depth estimation on five zero-shot datasets.**
| Method | Training | KITTI | NYU | ScanNet | DIODE | ETH3D |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Samples | AbsRel$\downarrow$ | $\delta_{1}$$\uparrow$ | AbsRel$\downarrow$ | $\delta_{1}$$\uparrow$ | AbsRel$\downarrow$ | $\delta_{1}$$\uparrow$ | AbsRel$\downarrow$ | $\delta_{1}$$\uparrow$ | AbsRel$\downarrow$ | $\delta_{1}$$\uparrow$ |  |
| MiDaS (Ranftl et al., ) | 2M | 0.236 | 0.630 | 0.111 | 0.885 | 0.121 | 0.846 | 0.332 | 0.715 | 0.184 | 0.752 |
| Omnidata (Eftekhar et al., ) | 12.2M | 0.149 | 0.835 | 0.074 | 0.945 | 0.075 | 0.936 | 0.339 | 0.742 | 0.166 | 0.778 |
| DPT-large (Ranftl et al., ) | 1.4M | 0.100 | 0.901 | 0.098 | 0.903 | 0.082 | 0.934 | 0.182 | 0.758 | 0.078 | 0.946 |
| DepthAnything (Yang et al., ) | 63.5M | 0.080 | 0.946 | 0.043 | 0.980 | 0.043 | 0.981 | 0.261 | 0.759 | 0.058 | 0.984 |
| DepthAnything v2 (Yang et al., ) | 62.6M | 0.080 | 0.943 | 0.043 | 0.979 | 0.042 | 0.979 | 0.321 | 0.758 | 0.066 | 0.983 |
| Metric3D v2 (Hu et al., ) | 16M | 0.052 | 0.979 | 0.039 | 0.979 | 0.023 | 0.989 | 0.147 | 0.892 | 0.040 | 0.983 |
| DiverseDepth (Yin et al., ) | 320K | 0.190 | 0.704 | 0.117 | 0.875 | 0.109 | 0.882 | 0.376 | 0.631 | 0.228 | 0.694 |
| LeReS (Yin et al., ) | 354K | 0.149 | 0.784 | 0.090 | 0.916 | 0.091 | 0.917 | 0.271 | 0.766 | 0.171 | 0.777 |
| HDN (Zhang et al., ) | 300K | 0.115 | 0.867 | 0.069 | 0.948 | 0.080 | 0.939 | 0.246 | 0.780 | 0.121 | 0.833 |
| Marigold (Ke et al., ) | 74K | 0.099 | 0.916 | 0.055 | 0.964 | 0.064 | 0.951 | 0.308 | 0.773 | 0.065 | 0.960 |
| GeoWizard (Fu et al., ) | 280K | 0.129 | 0.851 | 0.059 | 0.959 | 0.066 | 0.953 | 0.328 | 0.753 | 0.077 | 0.940 |
| DepthFM (Gui et al., ) | 63K | 0.174 | 0.718 | 0.082 | 0.932 | 0.095 | 0.903 | 0.334 | 0.729 | 0.101 | 0.902 |
| Our GenPercept (Depth) | 90K | 0.094 | 0.923 | 0.052 | 0.966 | 0.056 | 0.965 | 0.302 | 0.767 | 0.066 | 0.957 |
| Our GenPercept (Disparity) | 90K | 0.080 | 0.934 | 0.058 | 0.969 | 0.063 | 0.960 | 0.226 | 0.741 | 0.096 | 0.959 |
| Our GenPercept (Disparity + DPT head) | 90K | 0.078 | 0.935 | 0.059 | 0.967 | 0.064 | 0.961 | 0.228 | 0.740 | 0.094 | 0.961 |

**Table 7: Quantitative comparison of surface normal estimation on three zero-shot datasets. We evaluate mean error$\downarrow$, median error $\downarrow$ (med.), and the percentages of pixels $\uparrow$ with five thresholds.**
| Method | Training | NYU v2 | ScanNet | Sintel |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Samples | mean | med. | $5.0^{\circ}$ | $7.5^{\circ}$ | $11.25^{\circ}$ | $22.5^{\circ}$ | $30^{\circ}$ | mean | med. | $5.0^{\circ}$ | $7.5^{\circ}$ | $11.25^{\circ}$ | $22.5^{\circ}$ | $30^{\circ}$ | mean | med. | $5.0^{\circ}$ | $7.5^{\circ}$ | $11.25^{\circ}$ | $22.5^{\circ}$ | $30^{\circ}$ |  |
| Omnidata v1 (Eftekhar et al., ) | 12.2M | 23.1 | 12.9 | 21.6 | 33.4 | 45.8 | 66.3 | 73.6 | 22.9 | 12.3 | 21.5 | 34.5 | 47.4 | 66.1 | 73.2 | 41.5 | 35.7 | 3.0 | 5.8 | 11.4 | 30.4 | 42.0 |
| Ominidata v2 (Kar et al., ) | 12.2M | 17.2 | 9.7 | 25.3 | 40.2 | 55.5 | 76.5 | 83.0 | 16.2 | 8.5 | 29.1 | 44.9 | 60.2 | 79.5 | 84.7 | 40.5 | 35.1 | 4.6 | 7.9 | 14.7 | 33.0 | 43.5 |
| Metric3D v2 (Hu et al., ) | 8.8M | 13.5 | 6.7 | 40.1 | 53.5 | 65.9 | 82.6 | 87.7 | 11.8 | 5.5 | 46.6 | 60.7 | 71.6 | 85.4 | 89.7 | 22.8 | 14.2 | 18.4 | 28.5 | 41.6 | 66.7 | 75.8 |
| DINSE (Bae & Davison, ) | 160K | 16.4 | 8.4 | 32.8 | 46.3 | 59.6 | 77.7 | 83.5 | 16.2 | 8.3 | 29.8 | 45.9 | 61.0 | 78.7 | 84.4 | 34.9 | 28.1 | 8.9 | 14.1 | 21.5 | 41.5 | 52.7 |
| Geowizard (Fu et al., ) | 280K | 19.8 | 11.2 | 18.0 | 32.7 | 50.2 | 73.0 | 79.9 | 21.1 | 11.9 | 15.9 | 29.7 | 47.4 | 70.7 | 77.8 | 36.1 | 28.4 | 4.1 | 8.6 | 16.9 | 39.8 | 52.5 |
| Our GenPercept (Latent MSE loss) | 90K | 17.4 | 9.5 | 23.3 | 40.0 | 56.3 | 76.8 | 83.0 | 16.3 | 8.9 | 25.8 | 42.7 | 59.6 | 79.4 | 84.8 | 44.4 | 31.6 | 3.4 | 7.5 | 15.0 | 37.0 | 48.0 |
| Our GenPercept (Image angular loss) | 90K | 16.4 | 8.0 | 33.3 | 47.8 | 60.9 | 78.3 | 83.7 | 15.2 | 7.4 | 33.9 | 50.7 | 65.0 | 80.9 | 85.7 | 34.6 | 26.2 | 5.2 | 9.8 | 18.4 | 43.8 | 55.8 |

Surface Normal Estimation.
The surface normal estimation aims to predict a vector perpendicular to tangent plane of the surface at each point P, which represents the orientation of the object’s surface. For evaluation, we compute the angular error on three zero-shot datasets including NYU (Silberman et al., ), ScanNet (Dai et al., ), and Sintel (Butler et al., ). The mean $\downarrow$, median $\downarrow$, and the percentages of pixels $\uparrow$ with error below thresholds [5^∘, 7.5^∘, 11.25^∘, 22.5^∘, 30^∘] are reported. Invalid regions are filtered out and the metrics are averaged on all images.

Quantitative Evaluation. Quantitative results on monocular depth estimation and surface normal estimation are shown in table 6 and table 7, respectively.
Even trained on limited synthetic datasets only, our GenPercept shows much robustness and achieves promising performance on diverse unseen scenes. For monocular depth models, we train them with pixel-wise MSE loss, scale-shift-invariant loss (Ranftl et al., ), and gradient loss (Ranftl et al., ). Furthermore, our disparity model (inverse of the depth) shows much better performance on datasets with outdoor scenes, such as KITTI and DIODE, but less performance on indoor datasets. By replacing the VAE decoder with a lightweight DPT head, GenPercept can infer faster without bearing the performance penalty. For surface normal estimation, the image angular loss brings significant performance improvement thanks to our one-step estimation paradigm.

Qualitative Results. Qualitative visualizations are shown in fig. 3. We observe excellent generalization of our models in that they can estimate accurate geometric information and promising details not only on diverse real scenes and synthetic scenes, but also on comics, color drafts, and even some sketches.

Figure: Figure 3: Qualitative results of monocular depth and surface normal estimation. The model works surprisingly well on out-of-domain images (sketch and cartoon images).
Refer to caption: x3.png

### 4.2 Image Segmentation

**Table 8: Quantitative results of dichotomous image segmentation on DIS5K validation and testing sets. Additional cross-dataset evaluation is provided in the supplementary material.**
| Dataset | DIS-VD | DIS-TE4 | Overall DIS-TE (1-4) |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Metric | $maxF_{\beta}\uparrow$ | $F^{w}_{\beta}\uparrow$ | $~{}M\downarrow$ | $S_{\alpha}\uparrow$ | $E_{\phi}^{m}\uparrow$ | ${\rm HCE}_{\gamma}\downarrow$ | $maxF_{\beta}\uparrow$ | $F^{w}_{\beta}\uparrow$ | $~{}M\downarrow$ | $S_{\alpha}\uparrow$ | $E_{\phi}^{m}\uparrow$ | ${\rm HCE}_{\gamma}\downarrow$ | $maxF_{\beta}\uparrow$ | $F^{w}_{\beta}\uparrow$ | $~{}M\downarrow$ | $S_{\alpha}\uparrow$ | $E_{\phi}^{m}\uparrow$ | ${\rm HCE}_{\gamma}\downarrow$ |
| U^2Net (Qin et al., ) | 0.748 | 0.656 | 0.090 | 0.781 | 0.823 | 1413 | 0.795 | 0.705 | 0.087 | 0.807 | 0.847 | 3653 | 0.761 | 0.670 | 0.083 | 0.791 | 0.835 | 1333 |
| SINetV2 (Fan et al., ) | 0.665 | 0.584 | 0.110 | 0.727 | 0.798 | 1568 | 0.699 | 0.616 | 0.113 | 0.744 | 0.824 | 3683 | 0.693 | 0.608 | 0.101 | 0.747 | 0.822 | 1411 |
| HySM (Nirkin et al., ) | 0.734 | 0.640 | 0.096 | 0.773 | 0.814 | 1324 | 0.782 | 0.693 | 0.091 | 0.802 | 0.842 | 3331 | 0.757 | 0.665 | 0.084 | 0.792 | 0.834 | 1218 |
| IS-Net (Qin et al., ) | 0.791 | 0.717 | 0.074 | 0.813 | 0.856 | 1116 | 0.827 | 0.753 | 0.072 | 0.830 | 0.870 | 2888 | 0.799 | 0.726 | 0.070 | 0.819 | 0.858 | 1016 |
| Our Genpercept | 0.857 | 0.835 | 0.04 | 0.87 | 0.934 | 1511 | 0.848 | 0.829 | 0.049 | 0.854 | 0.938 | 3799 | 0.863 | 0.839 | 0.039 | 0.872 | 0.936 | 1364 |

**Table 9: Quantitative results of semantic segmentation performance on Hypersim and ADE20k datasets.**
| Method | Fine-tune | mIoU$\uparrow$ | mIoU$\uparrow$ |
| --- | --- | --- | --- |
| (Hypersim) | (ADE20K) |  |  |
| Our GenPercept |  | 46.0 | 34.1 |
| Our GenPercept | ✓ | 52.9 | 38.3 |

Dichotomous Image Segmentation. This is a category-agnostic, high-quality object segmentation task that accurately separates the object from the background in an image.
Consistent with previous methods, we use the six evaluation metrics specified in the DIS task, which include maximal F-measure ($maxF_{\beta}\uparrow$) (Achanta et al., ), weighted F-measure ($F^{w}_{\beta}\uparrow$) (Margolin et al., ), mean absolute error ($~{}M\downarrow$) (Perazzi et al., ), structural measure ($S_{\alpha}\uparrow$) (Fan et al., ), mean enhanced alignment measure ($E_{\phi}^{m}\uparrow$) (Fan et al., ; ) and human correction efforts (${\rm HCE}_{\gamma}\downarrow$) (Qin et al., ).
We choose DIS5K (Qin et al., ) as the training and testing dataset.
We utilize DIS-TR for training and evaluate our model on DIS-VD and DIS-TE subsets. The pixel-wise MSE loss is utilized during training.

Quantitative results of dichotomous image segmentation are shown in table 8, respectively. We only show partial results due to paper page limitations, full comparisons are accessible in the supplementary material. Our proposed model outperforms all other methods by a significant margin on this challenging dataset across most evaluation metrics. As shown in Fig. 5, the segmentation results of our approach are more refined, providing a more detailed foreground mask. For thin lines and meticulous objects that are difficult for previous methods to process, our method can also output accurate segmentation results.

Semantic Image Segmentation. This is a fundamental computer vision task that involves classifying each pixel in an image into a specific category or class. For training, we utilized the indoor synthetic dataset, HyperSim (Roberts et al., ), which comprises 40 semantic segmentation class labels. We encode different classes into 3-channel colormaps, treat the task as a regression problem, and fine-tune the original Stable Diffusion with the pixel-wise MSE loss. As demonstrated in fig. 4, the model generalizes well to classes within the HyperSim annotations, such as chairs and desks, but struggles with unrecognized categories such as cats and cars.

Another choice involves using a customized segmentation head. We incorporate a custom segmentation head, namely UperNet (Xiao et al., ), onto the multi-level features extracted by UNet, and the quantitative results are presented in table 10. In addition to evaluating on Hypersim (Roberts et al., ), we also test the model’s zero-shot performance on a subset of the ADE20k (Zhou et al., ) validation set, which contains overlapping classes. The results demonstrate the model’s capacity for zero-shot generalization.

Figure: Figure 4: Quanlitative results of semantic segmentation in the wild. Trained on the synthetic indoor Hypersim dataset, GenPercept shows much robustness on the trained categories of complex in-the-wild images, e.g., yellow chairs, green floor, and light blue wall. Due to the limited annotation categories and little negative label of “unknown category”, it sometimes fails in outdoor scenes and unseen categories such as cats and cars.
Refer to caption: x4.png

Figure: Figure 5: Qualitative comparison of dichotomous image segmentation.
Refer to caption: x5.png

### 4.3 Image Matting

Task Definition.
Image matting aims to extract the foreground, background, and alpha mask from an image. Traditional approaches depend on supplementary inputs that delineate foreground, background, and ambiguous areas to reduce uncertainty. Automatic image matting seeks to remove this dependency by directly estimating these components from the image alone.

Implementation Details.
We utilize P3M10K (Li et al., ), the largest portrait matting dataset with high-resolution portrait images along with high-quality alpha masks to train our model. The training set contains 9,421 high-quality images and annotations and the test set P3M-500-NP is composed of 500 public images from the Internet. We train our GenPercept with pixel-wise MSE loss to further improve the final performance.

Quantitative and Qualitative Results. We evaluate metrics including the sum of absolute differences (SAD), mean squared error (MSE), mean absolute difference (MAD), gradient (Grad.), and Connectivity (Conn.) on the P3M-500-NP test set. SAD and MAD measure the mean L1 distance between predictions and ground truth labels. MSE and CONN focus on L2 distance and connectivity that better reflects human intuition. As shown in table 10, our GenPercept is comparable with the state-of-the-art methods. Quantitative results of image matting are shown in fig. 6.

Figure: Figure 6: Visualization of image matting on the P3M-500-NP test set.
Refer to caption: x6.png

## 5 Related work

Vision Pre-Training.
Models pretrained on large-scale datasets possess powerful feature extraction capabilities, enabling them to be effectively transferred to a wide range of visual tasks. For instance, the ResNet (He et al., ) model pretrained on ImageNet (Russakovsky et al., ) can be fine-tuned and applied to perception tasks. By means of contrastive learning, MoCo (He et al., ) and CLIP (Radford et al., ) acquire rich visual and semantic representations, leveraging their advantages in joint visual and semantic modeling to enhance the performance of multimodal tasks. DINO (Zhang et al., ), through self-distillation, endows Vision Transformer and convolutional networks with comparable visual representation quality and demonstrates that self-supervised ViT representations contain explicit semantic segmentation information. In our work, we leverage Stable Diffusion (Rombach et al., ) as a prior for scene understanding and transfer it to various perception tasks.

Diffusion Priors for Dense Prediction.
Several works explore to use the priors of generative models for perceptual tasks. Some works (Bhattad et al., ; Du et al., ) demonstrate that generative models encode property maps of the scene. By finding latent variable offsets, using LoRA (Hu et al., ), etc., generative models can directly produce intrinsic images like surface normals, depth, albedo, etc.
LDMSeg (Van Gansbeke & De Brabandere, ) devises an image-conditioned sampling process, enabling diffusion models to directly output panoptic segmentation. UniGS (Qi et al., ) proposes location-aware color encoding and decoding strategies, allowing diffusion models to support referring segmentation and entity segmentation. Marigold (Ke et al., ) fine-tunes diffusion model on limited synthetic data, enabling it to support affine-invariant monocular depth estimation and exhibit strong generalization performance. However, Marigold is time-consuming due to the need for multiple iterations of denoising. Additionally, the Gaussian noise leads to inconsistent results across inferences, requiring aggregation over multiple inferences.
Xiang et al. () train a denoising auto-encoder for image classification. The difference of their method compared with traditional denoising auto-encoder is that input images are encoded into a latent code and denoising is performed in the latent space rather than the pixel space. They show good results on very small-scale datasets (CIFAR and ImageNet-tiny) to prove the concept and no results were reported on larger datasets. Furthermore, GeoWizard (Fu et al., ) extends the generative capabilities of Marigold, achieving better performance in joint depth and normal estimation, which enhances applications like 3D reconstruction and novel view synthesis. Moreover, DepthFM (Gui et al., ) addresses the speed challenge of Marigold by employing flow matching, offering a fast and efficient monocular depth estimation model.

## 6 Conclusion

In this work, we introduce GenPercept, an embarrassingly straightforward yet powerful approach to re-use the off-the-shelf UNet trained using diffusion processes.
In essence, we initialize our image perception networks using pre-trained diffusion models trained on the extensive LAION dataset, without making many modifications.
GenPercept demonstrates the capability to effectively leverage pre-trained diffusion models across a range of downstream dense perception tasks. We contend that our proposed methodology provides an efficient and potent paradigm for harnessing the capabilities of pre-trained diffusion models in dense visual perception tasks.

## Appendix A The formulation of Three Different Pipelines

### A.1 Stochastic Multi-Step Generation

For the training process, the RGB image $\mathbf{x}$ and ground-truth label $\mathbf{y}$ are encoded into the latent space with the VAE encoder $\mathbf{z}^{(\mathbf{x})}=\mathcal{E}(\mathbf{x})$, $\mathbf{z}^{(\mathbf{y})}=\mathcal{E}(\mathbf{y})$.
The Gaussian noise $\bm{\epsilon}$ is added to the ground-truth label latent $\mathbf{z}^{(\mathbf{y})}$, and the noisy label latent $\mathbf{z}^{(\mathbf{y})}_{t}$ is concatenated with the clean image latent $\mathbf{z}^{(\mathbf{x})}$ as U-Net input
$\mathbf{z}_{t}$ for each timestep:

$$ $\begin{gathered}\mathbf{z}^{(\mathbf{y})}_{t}=\sqrt{\bar{\alpha}_{t}}\mathbf{z }^{(\mathbf{y})}+\sqrt{1-\bar{\alpha}_{t}}\bm{\epsilon},~{}~{}t=[1,...,T],\\ \mathbf{z}_{t}=\text{concat}(\mathbf{z}^{(\mathbf{y})}_{t},\mathbf{z}^{( \mathbf{x})}),\end{gathered}$ (1) $$

where $\bar{\alpha}_{t}=\prod_{s=1}^{t}{(1-\beta_{s})}$, and $\beta_{s}$ is sampled from a variance schedule $\{\beta_{t}\in(0,1)\}_{t=1}^{T}$. Then, the denoiser $\bm{v}_{\theta}(\cdot,\cdot)$ is enforced to learn the velocity of “v-prediction” Salimans & Ho () from a timestep $t$ and the corresponding input $\mathbf{z}_{t}$. During training, the parameters of VAE are frozen, and only the denoiser $\bm{v}_{\theta}$ is fine-tuned. The timestep is uniformly sampled from 1 to $T$.

$$ $\begin{gathered}\mathcal{L}=\mathbb{E}_{\mathbf{z}^{(\mathbf{y})},\bm{\epsilon }\sim\mathcal{N}(0,I),t\sim\mathcal{U}(T)}\left\|(\sqrt{\bar{\alpha}_{t}}\bm{ \epsilon}-\sqrt{1-\bar{\alpha}_{t}}\mathbf{z}^{(\mathbf{y})})-\bm{v}_{\theta}( \mathbf{z}_{t},t)\right\|^{2}_{2}.\end{gathered}$ (2) $$

For inference, a Gaussian noise $\bm{\epsilon}_{t}$ is randomly sampled and denoised step by step with the denoiser $\bm{v}_{\theta}(\cdot,\cdot)$.

$$ $\begin{gathered}\hat{\mathbf{z}}^{(\mathbf{y})}_{T}=\bm{\epsilon},~{}~{} \mathbf{z}_{t}=\text{concat}(\hat{\mathbf{z}}^{(\mathbf{y})}_{t},\mathbf{z}^{( \mathbf{x})}),~{}~{}t=[T,...,1],\\ \hat{\mathbf{z}}^{(\mathbf{y})}_{t\rightarrow 0}=\sqrt{\bar{\alpha}_{t}}\cdot \hat{\mathbf{z}}^{(\mathbf{y})}_{t}-\sqrt{1-\bar{\alpha}_{t}}\cdot\bm{v}_{ \theta}(\mathbf{z}_{t},t),~{}~{}\hat{\bm{\bm{\epsilon}}}_{t}=\sqrt{\bar{\alpha }_{t}}\cdot\bm{v}_{\theta}(\mathbf{z}_{t},t)-\sqrt{1-\bar{\alpha}_{t}}\cdot \hat{\mathbf{z}}^{(\mathbf{y})}_{t},\\ \hat{\mathbf{z}}^{(\mathbf{y})}_{t-1}=\sqrt{\bar{\alpha}_{t-1}}\cdot\hat{ \mathbf{z}}^{(\mathbf{y})}_{t\rightarrow 0}+\sqrt{1-\bar{\alpha}_{t-1}}\cdot \hat{\bm{\bm{\epsilon}}}_{t},~{}~{}\hat{\mathbf{y}}=\mathcal{D}(\hat{\mathbf{z }}^{(\mathbf{y})}_{1\rightarrow 0}).\end{gathered}$ (3) $$

where the denoising process first computes the estimated noise $\hat{\bm{\bm{\epsilon}}}_{t}$ and the predicted clean latent code $\hat{\mathbf{z}}^{(\mathbf{y})}_{t\rightarrow 0}$ of timestep $t$ from the current latent code $\hat{\mathbf{z}}^{(\mathbf{y})}_{t}$ and the predicted velocity $\bm{v}_{\theta}(\mathbf{z}_{t},t)$. Then, it adds the computed noise $\hat{\bm{\bm{\epsilon}}}_{t}$ back to $\hat{\mathbf{z}}^{(\mathbf{y})}_{t\rightarrow 0}$ to get the latent code of timestep $t-1$. After repeating it for $T$ times, the predicted clean latent code $\hat{\mathbf{z}}^{(\mathbf{y})}_{1\rightarrow 0}$ is computed and sent to the VAE decoder $\mathcal{D}$ and estimate the target label $\hat{\mathbf{y}}$.
During inference, $m$ randomly sampled noises are introduced to estimate $m$ different predictions, and they are averaged with an ensemble process to reduce the randomness of prediction and improve performance.

### A.2 Deterministic Multi-Step Generation

The inherent random nature of diffusion models makes them challenging to apply to perceptual tasks, which typically aim for accurate results. As a result, existing works have replaced the original Gaussian noise with the target image as RGB noise (Bansal et al., ; Lee et al., ).
Technically,
rather than
introducing
the random Gaussian noise $\bm{\epsilon}$ ,
we blend
the ground-truth label latent $\mathbf{z}^{(\mathbf{y})}=\mathcal{E}(\mathbf{y})$ with the RGB image latent $\mathbf{z}^{(\mathbf{x})}=\mathcal{E}(\mathbf{x})$,
which is formulated as:

$$ $\begin{gathered}\mathbf{z}_{t}=\mathbf{z}^{(\mathbf{y})}_{t}=\sqrt{\bar{\alpha }_{t}}\mathbf{z}^{(\mathbf{y})}+\sqrt{1-\bar{\alpha}_{t}}\mathbf{z}^{(\mathbf{ x})},~{}~{}t=[1,...,T],\\ \end{gathered}$ (4) $$

Figure: Figure 7: Qualitative visualization of ablation study.
Refer to caption: x7.png

Figure: Figure 8: Visualization of different noise forms and proportions in the forward diffusion process.
Refer to caption: x8.png

Furthermore, the input latent has been modified to the latent code of input image $\mathbf{z}^{(\mathbf{x})}$ instead of random Gaussian.
Consequently,
the learning objective function and the inference process are reformulated as follows.

$$ $\begin{gathered}\mathcal{L}=\mathbb{E}_{(\mathbf{z}^{(\mathbf{x})},\mathbf{z}^ {(\mathbf{y})}),t\sim\mathcal{U}(T)}\left\|(\sqrt{\bar{\alpha}_{t}}\mathbf{z}^ {(\mathbf{x})}-\sqrt{1-\bar{\alpha}_{t}}\mathbf{z}^{(\mathbf{y})})-\bm{v}_{ \theta}(\mathbf{z}_{t},t)\right\|^{2}_{2}.\end{gathered}$ (5) $$

$$ $\begin{gathered}\hat{\mathbf{z}}^{(\mathbf{y})}_{T}=\mathbf{z}^{(\mathbf{x})}, ~{}~{}t=[T,...,1],\\ \hat{\mathbf{z}}^{(\mathbf{y})}_{t\rightarrow 0}=\sqrt{\bar{\alpha}_{t}}\cdot \hat{\mathbf{z}}^{(\mathbf{y})}_{t}-\sqrt{1-\bar{\alpha}_{t}}\cdot\bm{v}_{ \theta}(\hat{\mathbf{z}}^{(\mathbf{y})}_{t},t),\hat{\mathbf{z}}^{(\mathbf{x})} _{t}=\sqrt{\bar{\alpha}_{t}}\cdot\bm{v}_{\theta}(\hat{\mathbf{z}}^{(\mathbf{y} )}_{t},t)-\sqrt{1-\bar{\alpha}_{t}}\cdot\hat{\mathbf{z}}^{(\mathbf{y})}_{t},\\ \hat{\mathbf{z}}^{(\mathbf{y})}_{t-1}=\sqrt{\bar{\alpha}_{t-1}}\cdot\hat{ \mathbf{z}}^{(\mathbf{y})}_{t\rightarrow 0}+\sqrt{1-\bar{\alpha}_{t-1}}\cdot \hat{\mathbf{z}}^{(\mathbf{x})}_{t},~{}~{}\hat{\mathbf{y}}=\mathcal{D}(\hat{ \mathbf{z}}^{(\mathbf{y})}_{1\rightarrow 0}).\end{gathered}$ (6) $$

where $\hat{\mathbf{z}}^{(\mathbf{x})}_{t}$
denotes the predicted RGB noise of timestep $t$.

### A.3 GenPercept: Deterministic One-step Perception

In the main paper, we suppose that blending with the RGB image is much simpler compared to Gaussian noise, and we increase the blending proportion of the RGB image to make it more difficult. This is realized by increasing the initial value $\beta_{0}$ and final values $\beta_{T}$ of the DDPM schedule. When all the beta values $\{\beta_{t}\}_{t=1}^{T}$ are increased to 1, and $\bar{\alpha}_{t}=\prod_{s=1}^{t}{(1-\beta_{s})}=0$, the formulation of eq. 4 to eq. 6 can be greatly simplified as follows.

$$ $\begin{gathered}\mathbf{z}_{t}=\mathbf{z}^{(\mathbf{y})}_{t}=\mathbf{z}^{( \mathbf{x})},~{}~{}\mathcal{L}=\mathbb{E}_{(\mathbf{z}^{(\mathbf{x})},\mathbf{ z}^{(\mathbf{y})}),t\sim\mathcal{U}(T)}\left\|-\mathbf{z}^{(\mathbf{y})}-\bm{v }_{\theta}(\mathbf{z}_{t},t)\right\|^{2}_{2},\\ \hat{\mathbf{z}}^{(\mathbf{y})}_{T}=\mathbf{z}^{(\mathbf{x})},~{}~{}\hat{ \mathbf{z}}^{(\mathbf{y})}_{t-1}=-\hat{\mathbf{z}}^{(\mathbf{y})}_{t},~{}~{} \hat{\mathbf{z}}^{(\mathbf{y})}_{1\rightarrow 0}=-\bm{v}_{\theta}(\hat{\mathbf {z}}^{(\mathbf{y})}_{1},t=1),~{}~{}\hat{\mathbf{y}}=\mathcal{D}(\hat{\mathbf{z }}^{(\mathbf{y})}_{1\rightarrow 0}).\end{gathered}$ (7) $$

One-step prediction.
The input of the U-Net is an RGB latent code, and the output becomes the negative value of the ground-truth latent code, with no relationship to the timestep $t$. Therefore, we set the number of timesteps $T$ to 1 while preserving the same performance.

$$ $\begin{gathered}\mathcal{L}=\mathbb{E}_{(\mathbf{z}^{(\mathbf{x})},\mathbf{z}^ {(\mathbf{y})})}\left\|-\mathbf{z}^{(\mathbf{y})}-\bm{v}_{\theta}(\mathbf{z}^{ (\mathbf{x})},t=1)\right\|^{2}_{2},\\ \hat{\mathbf{z}}^{(\mathbf{y})}_{1\rightarrow 0}=-\bm{v}_{\theta}(\mathbf{z}^{ (\mathbf{x})},t=1),~{}~{}\hat{\mathbf{y}}=\mathcal{D}(\hat{\mathbf{z}}^{( \mathbf{y})}_{1\rightarrow 0}).\end{gathered}$ (8) $$

## Appendix B More Visualization Analysis

Quantitative Comparisons of Ablations. Visualization of the ablation study experiments in the main paper §2 is shown in fig. 7. The estimated depth detail remains comparable with a customized “DPT head”. Models trained with lower-quality data like “Taskonomy + Cityscapes” or without the pre-trained VAE decoder parameters suffer from a decline in the ability to predict details. We attribute them to the low-quality annotation and the oversized decoder, respectively.

Forward Diffusion Process. More detailed visualization of different noise forms and proportions in the forward diffusion process is shown in fig. 8. With larger $\beta$ values of the DDIM scheduler, the proportion of noise will be much higher during the forward diffusion process. When $\beta_{start},\beta_{end}$ reaches (1.0, 1.0), the noisy latent will be pure noise latent, which is Gaussian noise and RGB latent for (a) and (b), respectively.

## Appendix C An Extra Attempt on Human Pose Estimation

Task Definition
Human Pose Estimation is a task aimed at determining the spatial configuration of a person or object in a given image or video. This involves identifying and predicting the coordinates of particular keypoints.

Implementation Details
For human keypoint detection, we use Simple Baseline (Xiao et al., ) for person detection and conduct training on the COCO training set with 15K training samples. Performance is evaluated on the COCO (Lin et al., ) validation set. As shown in fig. 9, it is generalizable to unseen objects in the training set. To evaluate the performance on COCO, we use the customized keypoint head of ViTPose (Xu et al., ) for decoding the output. The quantitative results compared with a generalist model, Painter(Wang et al., ) is shown in table 11.

**Table 11: Pose estimation on COCO.**
| Metrics | AP $\uparrow$ | AP .5 $\uparrow$ | AP .75 $\uparrow$ | AP (M) $\uparrow$ | AP (L) $\uparrow$ |
| --- | --- | --- | --- | --- | --- |
| Painter (Wang et al., ) | 0.721 | 0.900 | 0.781 | 0.686 | 0.786 |
| GenPercept | 0.752 | 0.907 | 0.824 | 0.691 | 0.778 |

Figure: Figure 9: Generalized test results of keypoint detection.
Refer to caption: extracted/5950807/fig/new.png

## Appendix D Implementation Details

Implementation Details of Preliminary Section in the Main Paper.
Unless specified otherwise, we train for 30000 iterations to estimate the affine-invariant monocular depth on a mixture of the Hypersim (Roberts et al., ) (90%) and Virtual KITTI (Cabon et al., ) (10%) datasets. We freeze the VAE AutoEncoder and fine-tune the U-Net of Stable Diffusion v2.1 to estimate the ground-truth label latent, with a resolution of (480, 640), a batch size of 32, and a learning rate of 3e-5. The multi-resolution noise (Ke et al., ) is employed for Gaussian noise and not used for RGB noise (Lee et al., ) by default. For inference, the ensemble size and denoising steps are set to 1 and 10, respectively. Evaluation results of absolute relative error (AbsRel) and $\delta_{1}$ are reported on five unseen monocular depth datasets, including KITTI (Geiger et al., ), NYU (Silberman et al., ), ScanNet (Dai et al., ), DIODE (Vasiljevic et al., ), and ETH3D (Schops et al., ).

## Appendix E More Quantitative and Qualitive Experiments

### E.1 Monocular Depth Estimation

More Qualitative Evaluation.
We show the robustness of the monocular depth estimation model in diverse scenes in Fig. 11. Compared to DPT (Ranftl et al., ), our Genpercept performs better on estimating details and shows much better robustness even on some sketches. Compared to Marigold (Ke et al., ), our method achieves better relative depth visualization.

### E.2 Surface Normal Estimation

More Qualitative Evaluation. In Fig. 12, we showcase more qualitative results for the surface normal estimation. DSINE (Bae & Davison, ) is trained on 160K images of 10 datasets, including both real and synthetic data. Our GenPercept is only trained on one synthetic dataset (Hypersim (Roberts et al., )), and can estimate much more detailed surface normal maps.

### E.3 Dichotomous Image Segmentation

More Quantitative Evaluation.
To conduct a comprehensive evaluation, we compare our approach with numerous previous models including models for medical image segmentation (Ronneberger et al., ), salient object detection (Qin et al., ; Zhao et al., ; Wei et al., ; Chen et al., ; Qin et al., ), camouflaged object detection (Fan et al., ; Mei et al., ), semantic segmentation (Zhao et al., ; Chen et al., ; Wang et al., ; Yu et al., ; Zhao et al., ; Howard et al., ; Fan et al., ; Nirkin et al., ) and the model IS-Net (Qin et al., ) specifically trained for DIS. As shown in table 12, our proposed model significantly outperforms other methods across most evaluation metrics on this challenging dataset. The results highlight the effectiveness of our approach for DIS.

**Table 12: Quantitative results on DIS5K validation and testing sets.**
| Dataset | Metric | UNet | BASNet | GateNet | F^3Net | GCPANet | U^2Net | SINetV2 | PFNet | PSPNet | DLV3+ | HRNet | BSV1 | ICNet | MBV3 | STDC | HySM | IS-Net | Ours |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| DIS-VD | $maxF_{\beta}\uparrow$ | 0.692 | 0.731 | 0.678 | 0.685 | 0.648 | 0.748 | 0.665 | 0.691 | 0.691 | 0.660 | 0.726 | 0.662 | 0.697 | 0.714 | 0.696 | 0.734 | 0.791 | 0.857 |
| $F^{w}_{\beta}\uparrow$ | 0.586 | 0.641 | 0.574 | 0.595 | 0.542 | 0.656 | 0.584 | 0.604 | 0.603 | 0.568 | 0.641 | 0.548 | 0.609 | 0.642 | 0.613 | 0.640 | 0.717 | 0.835 |  |
| $~{}M~{}\downarrow$ | 0.113 | 0.094 | 0.110 | 0.107 | 0.118 | 0.090 | 0.110 | 0.106 | 0.102 | 0.114 | 0.095 | 0.116 | 0.102 | 0.092 | 0.103 | 0.096 | 0.074 | 0.040 |  |
| $S_{\alpha}\uparrow$ | 0.745 | 0.768 | 0.723 | 0.733 | 0.718 | 0.781 | 0.727 | 0.740 | 0.744 | 0.716 | 0.767 | 0.728 | 0.747 | 0.758 | 0.740 | 0.773 | 0.813 | 0.870 |  |
| $E_{\phi}^{m}\uparrow$ | 0.785 | 0.816 | 0.783 | 0.800 | 0.765 | 0.823 | 0.798 | 0.811 | 0.802 | 0.796 | 0.824 | 0.767 | 0.811 | 0.841 | 0.817 | 0.814 | 0.856 | 0.934 |  |
| ${\rm HCE}_{\gamma}\downarrow$ | 1337 | 1402 | 1493 | 1567 | 1555 | 1413 | 1568 | 1606 | 1588 | 1520 | 1560 | 1660 | 1503 | 1625 | 1598 | 1324 | 1116 | 1511 |  |
| DIS-TE1 | $maxF_{\beta}\uparrow$ | 0.625 | 0.688 | 0.620 | 0.640 | 0.598 | 0.694 | 0.644 | 0.646 | 0.645 | 0.601 | 0.668 | 0.595 | 0.631 | 0.669 | 0.648 | 0.695 | 0.740 | 0.841 |
| $F^{w}_{\beta}\uparrow$ | 0.514 | 0.595 | 0.517 | 0.549 | 0.495 | 0.601 | 0.558 | 0.552 | 0.557 | 0.506 | 0.579 | 0.474 | 0.535 | 0.595 | 0.562 | 0.597 | 0.662 | 0.814 |  |
| $~{}M~{}\downarrow$ | 0.106 | 0.084 | 0.099 | 0.095 | 0.103 | 0.083 | 0.094 | 0.094 | 0.089 | 0.102 | 0.088 | 0.108 | 0.095 | 0.083 | 0.090 | 0.082 | 0.074 | 0.038 |  |
| $S_{\alpha}\uparrow$ | 0.716 | 0.754 | 0.701 | 0.721 | 0.705 | 0.760 | 0.727 | 0.722 | 0.725 | 0.694 | 0.742 | 0.695 | 0.716 | 0.740 | 0.723 | 0.761 | 0.787 | 0.868 |  |
| $E_{\phi}^{m}\uparrow$ | 0.750 | 0.801 | 0.766 | 0.783 | 0.750 | 0.801 | 0.791 | 0.786 | 0.791 | 0.772 | 0.797 | 0.741 | 0.784 | 0.818 | 0.798 | 0.803 | 0.820 | 0.918 |  |
| ${\rm HCE}_{\gamma}\downarrow$ | 233 | 220 | 230 | 244 | 271 | 224 | 274 | 253 | 267 | 234 | 262 | 288 | 234 | 274 | 249 | 205 | 149 | 204 |  |
| DIS-TE2 | $maxF_{\beta}\uparrow$ | 0.703 | 0.755 | 0.702 | 0.712 | 0.673 | 0.756 | 0.700 | 0.720 | 0.724 | 0.681 | 0.747 | 0.680 | 0.716 | 0.743 | 0.720 | 0.759 | 0.799 | 0.876 |
| $F^{w}_{\beta}\uparrow$ | 0.597 | 0.668 | 0.598 | 0.620 | 0.570 | 0.668 | 0.618 | 0.633 | 0.636 | 0.587 | 0.664 | 0.564 | 0.627 | 0.672 | 0.636 | 0.667 | 0.728 | 0.852 |  |
| $~{}M~{}\downarrow$ | 0.107 | 0.084 | 0.102 | 0.097 | 0.109 | 0.085 | 0.099 | 0.096 | 0.092 | 0.105 | 0.087 | 0.111 | 0.095 | 0.083 | 0.092 | 0.085 | 0.070 | 0.035 |  |
| $S_{\alpha}\uparrow$ | 0.755 | 0.786 | 0.737 | 0.755 | 0.735 | 0.788 | 0.753 | 0.761 | 0.763 | 0.729 | 0.784 | 0.740 | 0.759 | 0.777 | 0.759 | 0.794 | 0.823 | 0.884 |  |
| $E_{\phi}^{m}\uparrow$ | 0.796 | 0.836 | 0.804 | 0.820 | 0.786 | 0.833 | 0.823 | 0.829 | 0.828 | 0.813 | 0.840 | 0.781 | 0.826 | 0.856 | 0.834 | 0.832 | 0.858 | 0.938 |  |
| ${\rm HCE}_{\gamma}\downarrow$ | 474 | 480 | 501 | 542 | 574 | 490 | 593 | 567 | 586 | 516 | 555 | 621 | 512 | 600 | 556 | 451 | 340 | 480 |  |
| DIS-TE3 | $maxF_{\beta}\uparrow$ | 0.748 | 0.785 | 0.726 | 0.743 | 0.699 | 0.798 | 0.730 | 0.751 | 0.747 | 0.717 | 0.784 | 0.710 | 0.752 | 0.772 | 0.745 | 0.792 | 0.830 | 0.885 |
| $F^{w}_{\beta}\uparrow$ | 0.644 | 0.696 | 0.620 | 0.656 | 0.590 | 0.707 | 0.641 | 0.664 | 0.657 | 0.623 | 0.700 | 0.595 | 0.664 | 0.702 | 0.662 | 0.701 | 0.758 | 0.862 |  |
| $~{}M~{}\downarrow$ | 0.098 | 0.083 | 0.103 | 0.092 | 0.109 | 0.079 | 0.096 | 0.092 | 0.092 | 0.102 | 0.080 | 0.109 | 0.091 | 0.078 | 0.090 | 0.079 | 0.064 | 0.035 |  |
| $S_{\alpha}\uparrow$ | 0.780 | 0.798 | 0.747 | 0.773 | 0.748 | 0.809 | 0.766 | 0.777 | 0.774 | 0.749 | 0.805 | 0.757 | 0.780 | 0.794 | 0.771 | 0.811 | 0.836 | 0.883 |  |
| $E_{\phi}^{m}\uparrow$ | 0.827 | 0.856 | 0.815 | 0.848 | 0.801 | 0.858 | 0.849 | 0.854 | 0.843 | 0.833 | 0.869 | 0.801 | 0.852 | 0.880 | 0.855 | 0.857 | 0.883 | 0.951 |  |
| ${\rm HCE}_{\gamma}\downarrow$ | 883 | 948 | 972 | 1059 | 1058 | 965 | 1096 | 1082 | 1111 | 999 | 1049 | 1146 | 1001 | 1136 | 1081 | 887 | 687 | 973 |  |
| DIS-TE4 | $maxF_{\beta}\uparrow$ | 0.759 | 0.780 | 0.729 | 0.721 | 0.670 | 0.795 | 0.699 | 0.731 | 0.725 | 0.715 | 0.772 | 0.710 | 0.749 | 0.736 | 0.731 | 0.782 | 0.827 | 0.848 |
| $F^{w}_{\beta}\uparrow$ | 0.659 | 0.693 | 0.625 | 0.633 | 0.559 | 0.705 | 0.616 | 0.647 | 0.630 | 0.621 | 0.687 | 0.598 | 0.663 | 0.664 | 0.652 | 0.693 | 0.753 | 0.829 |  |
| $~{}M~{}\downarrow$ | 0.102 | 0.091 | 0.109 | 0.107 | 0.127 | 0.087 | 0.113 | 0.107 | 0.107 | 0.111 | 0.092 | 0.114 | 0.099 | 0.098 | 0.102 | 0.091 | 0.072 | 0.049 |  |
| $S_{\alpha}\uparrow$ | 0.784 | 0.794 | 0.743 | 0.752 | 0.723 | 0.807 | 0.744 | 0.763 | 0.758 | 0.744 | 0.792 | 0.755 | 0.776 | 0.770 | 0.762 | 0.802 | 0.830 | 0.854 |  |
| $E_{\phi}^{m}\uparrow$ | 0.821 | 0.848 | 0.803 | 0.825 | 0.767 | 0.847 | 0.824 | 0.838 | 0.815 | 0.820 | 0.854 | 0.788 | 0.837 | 0.848 | 0.841 | 0.842 | 0.870 | 0.938 |  |
| ${\rm HCE}_{\gamma}\downarrow$ | 3218 | 3601 | 3654 | 3760 | 3678 | 3653 | 3683 | 3803 | 3806 | 3709 | 3864 | 3999 | 3690 | 3817 | 3819 | 3331 | 2888 | 3799 |  |
| Overall<br>DIS-TE (1-4) | $maxF_{\beta}\uparrow$ | 0.708 | 0.752 | 0.694 | 0.704 | 0.660 | 0.761 | 0.693 | 0.712 | 0.710 | 0.678 | 0.743 | 0.674 | 0.711 | 0.729 | 0.710 | 0.757 | 0.799 | 0.863 |
| $F^{w}_{\beta}\uparrow$ | 0.603 | 0.663 | 0.590 | 0.614 | 0.554 | 0.670 | 0.608 | 0.624 | 0.620 | 0.584 | 0.658 | 0.558 | 0.622 | 0.658 | 0.628 | 0.665 | 0.726 | 0.839 |  |
| $~{}M~{}\downarrow$ | 0.103 | 0.086 | 0.103 | 0.098 | 0.112 | 0.083 | 0.101 | 0.097 | 0.095 | 0.105 | 0.087 | 0.110 | 0.095 | 0.085 | 0.094 | 0.084 | 0.070 | 0.039 |  |
| $S_{\alpha}\uparrow$ | 0.759 | 0.783 | 0.732 | 0.750 | 0.728 | 0.791 | 0.747 | 0.756 | 0.755 | 0.729 | 0.781 | 0.737 | 0.758 | 0.770 | 0.754 | 0.792 | 0.819 | 0.872 |  |
| $E_{\phi}^{m}\uparrow$ | 0.798 | 0.835 | 0.797 | 0.819 | 0.776 | 0.835 | 0.822 | 0.827 | 0.819 | 0.810 | 0.840 | 0.778 | 0.825 | 0.850 | 0.832 | 0.834 | 0.858 | 0.936 |  |
| ${\rm HCE}_{\gamma}\downarrow$ | 1202 | 1313 | 1339 | 1401 | 1395 | 1333 | 1411 | 1427 | 1442 | 1365 | 1432 | 1513 | 1359 | 1457 | 1426 | 1218 | 1016 | 1364 |  |

More Qualitative Evaluation.
As shown in Fig. 13, our method yields more refined segmentation results, providing cleaner foreground masks. It also produces precise outputs for intricate lines that are challenging for previous approaches to handle.

Cross Dataset Evaluation.
To test the generalization ability of our model, we randomly select some images from other datasets(Agustsson & Timofte, ; Lin et al., ; Shao et al., ) for experiments. As shown in Fig. 10, compared to IS-Net and IS-Net-General-UseQin et al. (), our approach exhibits finer segmentation quality across diverse images, providing cleaner foreground masks. It is noteworthy that IS-Net-General-Use is fine-tuned on extra datasets to enhance generalization, which indicates that our method has a stronger generalization ability.

Figure: Figure 10: Cross dataset comparison of our model and other models.
Refer to caption: x9.png

### E.4 Image Matting

More Qualitative Evaluation. In Fig. 14, we showcase more qualitative results for the image matting task. It is worth noting that our model works well in various resolutions, light environments, human poses, and human orientations. Please zoom in for better visualization and more details.

### E.5 Image Segmentation

More Qualitative Evaluation. In Fig. 15, we showcase more qualitative results for the image segmentation task. Our method shows much robustness on the trained categories of complex in-the-wild images. Due to the limited annotation categories and little negative label of “unknown category”, it sometimes fails in outdoor scenes and unseen categories such as cats and cars. Please zoom in for better visualization and more details.

### E.6 Human Pose Estimation

More Qualitative Evaluation. In Fig.16, we showcase more qualitative results for the human pose estimation. We conduct experiments on MHP datasetLi et al. (), and we use mmposeContributors () to render the human pose following the setting of Bai et al. ().

[Uncaptioned image]: x10.png

Figure: Figure 11: More qualitative results for monocular depth estimation.
Refer to caption: x11.png

[Uncaptioned image]: x12.png

Figure: Figure 12: More qualitative comparisons of surface normal estimation. Our GenPercept can achieve more detailed results, even compared to the competitive CVPR2024 method DSINE Bae & Davison (). Note that these two visualization coordinate systems are slightly different.
Refer to caption: x13.png

Figure: Figure 13: Qualitative results of dichotomous image segmentation.
Refer to caption: x14.png

Figure: Figure 14: More qualitative results for image matting.
Refer to caption: extracted/5950807/fig/matting_supplementary.jpg

Figure: Figure 15: More qualitative results for image matting.
Refer to caption: x15.png

Figure: Figure 16: More qualitative results for human pose estimation. (Left: original Image, Mid: prediction, Right: ground truth)
Refer to caption: extracted/5950807/fig/human_on.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: 1. What training set was used for Table 1? Are you following the training data of DMP or Marigold? If so, why is the first row of Table 5 identical to the baseline? If not, why did you begin directly with a synthetic dataset?
2. For monocular depth estimation, why is DMP not included as a baseline?
3. The authors mention that using a customized head and loss could accelerate inference time. Could you provide a comparison to demonstrate this improvement?
soundness: 3
strengths: 1. The exploration of key factors influencing the transferability of text-to-image (T2I) diffusion models to dense visual perception tasks is intriguing and relevant.
2. The experiments are thorough and comprehensive, providing strong support for the findings.
3. The proposed deterministic one-step perception approach effectively integrates these findings and demonstrates comparable performance with minimal fine-tuning.
summary: This paper investigates key factors affecting the transfer performance of pretrained diffusion models repurposed for dense visual perception tasks, emphasizing the importance of fine-tuning data quality, training strategies, and task-specific supervision. It introduces GenPercept, a one-step fine-tuning paradigm that enhances inference speed and improves fine-grained details in predictions across various perception tasks.
weaknesses: There are some confusing aspects in the experimental setup and comparisons for the downstream tasks. See questions.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: What would be the comparison of generalization for OOD data for models trained using synthetic and real data?
soundness: 3
strengths: - The authors conduct extensive ablation studies from different aspects and finally figure out the key factors affecting transfer efficiency and performance when using pre-trained diffusion models.
- Starting from the inconsistency of the results of two existing methods, the authors dig deeper into the influencing factors and then propose their GenPercept. This approach can provide some new ideas for doing research.
- These findings are inspiring and can provide constructive insights into model design when adapting pre-trained models for downstream tasks.
summary: This paper reveals key elements of diffusion models for downstream dense perception tasks. The authors did a full range of validation from the perspectives of model design and training data, unveiled some IMPORTANT factors, and proposed a new model named GenPercept,
The authors have conducted extensive experiments on five dense perception tasks, including monocular depth estimation, surface normal estimation, image segmentation, and matting. This extensive experimentation serves as a testament to the effectiveness and universality of their method.
weaknesses: - The biggest weakness is that all conclusions are based on experiments using large synthetic datasets trained on deep estimation tasks. I have two concerns. First, would these conclusions still hold if the datasets were small? Second, would these conclusions still hold if the dataset was completely real? 
- It is not appropriate to use the P3M 10K dataset to verify the robustness of GenPercept in image matting. This is because the human matting dataset is extremely similar to the dichotomous image segmentation task if only the boundary regions are considered and the deterministic foreground regions are ignored. Instead, if the authors wish to validate its robustness in the image matting task, more types of objects (e.g., semi-transparent objects) should be included and then its performance should be observed.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 2
questions: The main reason that I currently give a slightly low rating is because of the incorrect numbers of existing models, the missing quantitative comparisons to some other existing models, the missing efficiency experiments, and some missing explanations (see the ‘weaknesses’ section). I would like to ask the authors to carefully address my concerns, answer the questions posed in the ‘weaknesses’ section, and revise the manuscript accordingly.

Additionally, I have some other questions/suggestions:

1.	From the text in L411-L423, it appears like the classes for semantic segmentation are always encoded into 3-channel colormaps, even when UPerNet is used. Is this really the case? If so, why isn’t the ‘regular’ semantic segmentation format used, with one channel for each individual class? If not, please clarify this in the text.
2.	In the related work section, it seems appropriate to also mention DINOv2, because it has shown to be very suitable for downstream visual perception tasks like depth estimation (e.g., with Depth Anything [g]) and semantic segmentation.

[f] Oquab et al., “DINOv2: Learning Robust Visual Features without Supervision,” TMLR 2024.

[g] Yang et al., “Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data,” CVPR 2024.

---

**Update after author discussion.** After reading the different reviews, the authors' response, and the revised manuscript, and having a discussion with the authors, I have decided to upgrade my rating from 5 to 6. 

By answering my questions and revising the manuscript based on my review and follow-up questions, the authors have convincingly addressed the majority of my concerns. Importantly, the paper now compares the proposed method more fairly to existing methods, provides insightful inference speed results, better explains the operation of the method, and discusses the cause of the impact of different $\beta$ values in a more nuanced way. As such, the I believe the paper is stronger than the initial submission, and I upgrade my rating.
soundness: 3
strengths: 1.	Most importantly, the analysis of the design choices that play a role when fine-tuning diffusion models for monocular depth estimation, described in Sec. 3, leads to various relevant and useful insights. 

    a)	The finding that fine-tuning can be done accurately with a one-step, deterministic approach (Tab. 1) is useful because it allows for significantly more efficient inference. 

    b)	Moreover, by formulating the task as one-step estimation, the model can now te trained with task-specific losses, such as the image angular loss for surface normal estimation. With experiments (Tab. 7), this is shown to improve the results.

    c)	The finding that most of the useful prior knowledge is contained in the U-Net of the pre-trained diffusion model, and not in the VAE decoder (Tab. 2), is useful because it means that the VAE decoder can be replaced with task-specific decoders. This is shown to be significantly boost the semantic segmentation performance in Tab. 9 (per L418-L423). 

    d)	The finding that the use of multiple timesteps during training is not necessary (Tab. 3) is useful because it allows the training procedure to be simplified to single-timestep training.

    e)	The finding that the U-Net denoiser should be fine-tuned fully, without low-rank adaptation (Tab. 4), is useful because it provides clear guidelines for future methods that aim to fine-tune diffusion models for depth estimation and other dense perception tasks.

2.	The presentation of the main findings of the paper in the text boxes (e.g., L283 and L296) is very helpful and useful. These text boxes clearly summarize the impact of the results that have been discussed thus far, and they help the reader to see the most interesting results of the paper at a glance.

3.	The presented GenPercept model is valuable because it takes advantage of the aforementioned findings of the design-choice analysis. The application and evaluation of this model on multiple tasks and datasets provides the reader with insights into the performance that can be obtained when combining the best individual design choices. Moreover, these results can serve as a baseline for future works that aim to fine-tune diffusion models for multiple downstream perception tasks.
summary: This paper researches the process of fine-tuning pre-trained text-to-image diffusion models for dense perception tasks like monocular depth estimation and semantic segmentation, and analyzes several design choices in that process. In this analysis, it considers the model architecture, training procedure, model initialization, dataset selection, and fine-tuning protocol. Based on this analysis, five findings are made:

1.	Diffusion models can be fine-tuned accurately and efficiently for dense perception tasks with a one-step, deterministic approach.
2.	The most useful prior knowledge of the pre-trained diffusion model is contained in the U-Net denoiser. The VAE decoder can be replaced with other components without problems.
3.	With the one-step model, multi-timestep training is irrelevant, and additional text inputs do not significantly impact performance.
4.	Fine-tuning the U-Net denoiser leads to better results than freezing it or applying low-rank adaptation.
5.	Training data quality affects the prediction quality.

Based on these findings, the paper presents GenPercept, a paradigm for fine-tuning diffusion models for dense prediction tasks. Experiments show that GenPercept is an effective method to fine-tune Stable Diffusion for various dense prediction tasks. On several benchmarks, GenPercept achieves a competitive performance.
weaknesses: 1.	The scores for DepthFM in Tab. 6 and GeoWizard in both Tab. 6 and Tab. 7 do not correspond to the scores originally reported in the respective papers. The numbers in these tables should be altered to reflect the originally reported numbers, or the text should explain why the numbers differ from these original numbers. Some examples:

    a)	GeoWizard: 9.7 AbsRel and 92.1 $\delta_{1}$ on KITTI in original paper [a], but 12.9 AbsRel and 85.1 $\delta_{1}$ in this submitted manuscript. 

    b)	DepthFM: 8.3 AbsRel and 93.4 $\delta_{1}$ on KITTI in original paper [b], but 17.4 AbsRel and 71.8 $\delta_{1}$ in this submitted manuscript.

2.	The results in Tab. 8 (and Tab. 2 of the supp) make it seem like GenPercept achieves state-of-the-art results in dichotomous image segmentation. However, this table does not contain the results of top-performing model MVANet [c]. This model achieves a max$F_{\beta}$ score of 0.916 on Overall DIS-TE (1-4), compared to 0.863 by GenPercept. Even with results that are inferior to MVANet, GenPercept has value, but it should be clear to the reader that GenPercept does not achieve state-of-the-art results, so these results should be added to the tables.
3.	For the image matting task on the P3M-500-NP dataset, GenPercept is only compared to the ResNet-34 variant of P3M [d], not the Swin-T or ViTAE-S variants which achieve much better results, e.g., 7.59 SAD for ViTAE-S compared to 11.23 SAD for ResNet-34. By only reporting the results for the ResNet-34 variant, it seems like GenPercept performs similarly to the state of the art, whereas this is not the case. The results of P3M for ViTAE-S and/or Swin-T should be added to the paper, or the text should clearly explain why such a comparison is not necessary.
4.	One of the main benefits of the one-step inference procedure of GenPercept is the improved efficiency compared to Marigold and other diffusion-based methods. The efficiency is also mentioned as a key characteristic of the method (L539). However, the paper does not explicitly evaluate the efficiency of the model. Therefore, it is not clear what the exact speedup is over existing diffusion-based models, or how the efficiency compares to the other methods reported in Tab. 6, 7, 8, and 10. The paper would be significantly stronger if the efficiency of GenPercept was shown quantitatively, by reporting the inference time of GenPercept and other methods.
5.	L187-L188 states that the results indicate that increasing the ‘training challenges’ leads to improved model performance. However, the paper does not provide any explanation for this. Why does the performance increase when the training task becomes more challenging? This seems counterintuitive. The value of the paper would improve if the authors provided an explanation (or hypothesis) for this phenomenon, as this could provide insights into the way these diffusion models learn dense perception tasks.
6.	The paper does not clearly define $\beta_{start}$ and $\beta_{end}$, even though first experiment and finding (Tab. 1 & L204) are fully focused around changing the values of these parameters. Fig. 2(b) illustrates that they impact the proportion of noise that is added to the latents in different time steps, but an exact definition is not provided. In L107, the paper refers to the supplementary material for more details, but the supplementary material only mentions $\beta_{s}$ and $\beta_{t}$. The clarity of the paper would improve if a clear definition of $\beta_{start}$ and $\beta_{end}$ was provided.
7.	The paper does not evaluate if the multi-class image segmentation performance of GenPercept is competitive with existing image segmentation methods. Currently, the paper only conducts an experiment where GenPercept is trained on HyperSim images with 40 semantic classes, and evaluated on ADE20K images with 40 classes. As this is a newly proposed setting, these results are not comparable with existing segmentation methods. To see if fine-tuning diffusion models is truly valuable for image segmentation, GenPercept should be fine-tuned and evaluated on standard segmentation benchmarks like ADE20K, and compared to existing segmentation models like Mask2Former [e]. 
8.	A minor weakness of the paper is that there is not a version of GenPercept that clearly works better than the other. As discussed in L357-L359 and shown in Tab. 6, the GenPercept model trained for depth estimation scores better on NYU, ScanNet and ETH3D, while the model trained for disparity estimation scores significantly better on KITTI and DIODE. As a result, two different models are necessary for different situations, and it is not always clear which of the models works best for an unknown application domain. 

Some minor weaknesses, which do not significantly impact my rating:

9.	L230: The acronym “DPT” is not defined, nor is there a reference to a work where it is presented. As a result, it’s not clear what a “DPT encoder” is.
10.	The meaning of the “fine-tune” column in Tab. 9 is not clear. From the text in L418-L423, it appears that it refers to the use of the UPerNet decoder. If so, the text “UPerNet” seems more appropriate than “fine-tune”. If it means something else than this, this should be better specified in the paper.
11.	The ordering of the labels in the legend of Fig. 2 (b) is confusing, as there appears to be no logical ordering. The legend, and the figure as whole, would be easier to interpret if the labels were ordered in descending or ascending manner, e.g. starting with (0.0002125, 0.003) and ending with (1.0, 1.0), or vice versa.

12.	There are a few errors in the text:

    a) L024: "tailed for" - Do the authors mean "tailored for"?

    b)	L187-L188: "increasing training challenges lead to" => "increasing training challenges leads to"

[a] Fu et al., “GeoWizard: Unleashing the Diffusion Priors for 3D Geometry Estimation from a Single Image,” ECCV 2024.

[b] Gui et al., “DepthFM: Fast Monocular Depth Estimation with Flow Matching,” arXiv:2403.13788, 2024.

[c] Yu et al., "Multi-view Aggregation Network for Dichotomous Image Segmentation," CVPR 2024.

[d] Ma et al., "Rethinking Portrait Matting with Privacy Preserving," IJCV 2023.

[e] Cheng et al., “Masked-attention Mask Transformer for Universal Image Segmentation,” CVPR 2022.

---

confidence: 4
contribution: 4
initial_rating: 6
presentation: 4
questions: 1. Trained with synthetic data, why does GenPercept not suffer from the sim2real domain gap?
2. In the supplementary materials, line 22 should utilize the citation format 'citep'.
3. Can GenPercept be applied to general perception tasks, such as detection?
soundness: 3
strengths: 1. This paper provides systematical analysis of the design space of diffusion model for dense perception task, as shown in the 5 findings in paper. The core design of this paper is to employ a deterministic one step perception.
2. By using the proposed training protocol, the proposed GenPercept harnesses the power of the pre-trained UNet from diffusion models for dense visual perception tasks, including monocular depth estimation, surface normal estimation, dichotomous image segmentation and semantic segmentation. 
3. This paper presents a solid analysis accompanied by extensive experiments. The results are not only intuitive but also promising, demonstrating a strong potential for downstream application.
summary: This paper presents a novel approach to harnessing pretrained diffusion models for general dense perception tasks, including monocular depth estimation, surface normal estimation, dichotomous image segmentation, semantic segmentation, and human pose estimation. The central idea of this work is to utilize the diffusion model as a robust pretrained backbone, subsequently fine-tuning it for a variety of downstream applications.
weaknesses: 1. Given that GenPercept is currently trained on a relatively small dataset, it tends to lag behind models that benefit from extensive data training. To enhance its performance, it would be beneficial to scale GenPercept's training with a larger volume of data in the future.
2. Given the robust prior established by training on the extensive LAION dataset, the question arises: what would be the outcome of employing alternative self-supervised methods, such as MAE or CLIP, using the same LAION dataset? A comparative analysis of these approaches against the diffusion pretrain would provide valuable insights into their relative efficacy and potential advantages.

Meta-review: The paper introduces GenPercept, a novel approach leveraging diffusion models to enhance dense visual perception tasks. The work claims improvements in inference speed and detail of predictions, substantiated through a series of experiments across multiple tasks.

The paper stands out for its rigorous experimental validation and the clear demonstration of the versatility and efficiency of diffusion models in dense perception contexts. The design space analysis and the results presented are commendable and represent the paper's key strengths.

Concerns are raised about the heavy reliance on extensive synthetic datasets, leading to questions about the model's performance with smaller or real-world data. The initial comparison with existing methods was not entirely convincing, and the paper lacked a quantitative evaluation of model efficiency. Additionally, some theoretical claims, such as "ground truth leakage," were not sufficiently clear or substantiated.

The decision leans toward acceptance due to the paper's robust experimental work and the novelty of applying diffusion models in this manner. Despite this, the unresolved issues around the real-world applicability and some theoretical ambiguities slightly detract from the paper's impact.

Decision: Accept (Poster)