What Matters When Repurposing Diffusion Models for General Dense Perception Tasks?
Authors: Guangkai Xu
Yongtao Ge
Mingyu Liu
Chengxiang Fan
Kangyang Xie
Zhiyue Zhao
Hao Chen
Chunhua Shen
Venue: ICLR 2025 Poster (2025)
Keywords: Transfer Learning
Diffusion Models
Visual Perception
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We comprehensively investigate the variours potential ways for reformulating diffusion models as general visual perception predictors, including monocular depth, surface normal, image segmentation, matting, and human pose estimation.

Abstract: Extensive pre-training with large data is indispensable for downstream geometry and semantic visual perception tasks. Thanks to large-scale text-to-image (T2I) pretraining, recent works show promising results by simply fine-tuning T2I diffusion models for a few dense perception tasks. However, several crucial design decisions in this process still lack comprehensive justification, encompassing the necessity of the multi-step diffusion mechanism, training strategy, inference ensemble strategy, and fine-tuning data quality. In this work, we conduct a thorough investigation into critical factors that affect transfer efficiency and performance when using diffusion priors. Our key findings are: 1) High-quality fine-tuning data is paramount for both semantic and geometry perception tasks. 2) As a special case of the diffusion scheduler by setting its hyper-parameters, the multi-step generation can be simplified to a one-step fine-tuning paradigm without any loss of performance, while significantly speeding up inference. 3) Apart from fine-tuning the diffusion model with only latent space supervision, task-specific supervision can be beneficial to enhance fine-grained details. These observations culminate in the development of GenPercept, an effective deterministic one-step fine-tuning paradigm tailored for dense visual perception tasks exploiting diffusion priors. Different from the previous multi-step methods, our paradigm offers a much faster inference speed, and can be seamlessly integrated with customized perception decoders and loss functions for task-specific supervision, which can be critical for improving the fine-grained details of predictions. Comprehensive experiments on a diverse set of dense visual perceptual tasks, including monocular depth estimation, surface normal estimation, image segmentation, and matting, are performed to demonstrate the remarkable adaptability and effectiveness of our proposed method. Code: https://github.com/aim-uofa/GenPercept

Published as a conference paper at ICLR 2025
WHAT MATTERS WHEN REPURPOSING DIFFUSION MODELS FOR GENERAL DENSE PERCEPTION TASKS?
Guangkai Xu
Yongtao Ge Zhiyue Zhao'
Mingyu Liut
Chengxiang Fan Chunhua Shen 1.2
Kangyang Xie
Hao Chen'
Zhejiang University, China
Ant Group
ABSTRACT
Extensive pre-training with large data is indispensable for downstream geometry and semantic visual perception tasks. Thanks to large-scale text-to-image (T2I) pretraining, recent works show promising results by simply fine-tuning T2I dif- fusion models for a few dense perception tasks. However, several crucial design decisions in this process still lack comprehensive justification, encompassing the necessity of the multi-step diffusion mechanism, training strategy, inference en- semble strategy, and fine-tuning data quality. In this work, we conduct thorough investigation into critical factors that affect transfer efficiency and performance when using diffusion priors. Our key findings are: 1) High-quality fine-tuning data is paramount for both semantic and geometry perception tasks. 2) As spe- cial case of the diffusion scheduler by setting its syyper-parametet the multi-step generation can be simplified to one-step fine-tuning paradigm without any loss of performance, while significantly speeding up inference. 3) Apart from fine-tuning the diffusion model with only latent space supervision, task task-specific supervision can be beneficial to enhance fine-grained details. These observations culminate the development of GenPercept, an effective deterministic one-step fine-tuning paradigm tailored for dense visual perception tasks exploiting diffusion priors. Different from the previous multi-step methods, our paradigm offers much faster inference speed, and can be seamlessly integrated with customized perception de- coders and loss functions for task-specific supervision, which can be critical for improving the fine-gratd grained details of predictions. Comprehensive experiments on a diverse set of dense visual perceptual tasks, including monocular depth es- timation, surface normal estimation, image segmentation, and matting, are per- formed demonstrate the remarkable adaptability and effectiveness of our pro- posed method. Code: https: /ttp//gtttt////// com/ Geeee/eeeeeeeeeeeteeeeeeeeeeeeee
INTRODUCTION
Recent studies have explored the transferability of text-to-image (T2I) diffusion models to dense
visual perception tasks, such as geometry estimation (Ke et al. 2024 Lee et al.) 2024 Fu et al.
2024b
Gui al.] 2024: Ye et al.) 2024), image segmentation Van Gansbeke De Brabandere
2024
Lee et al. 2024). and inverse rendering (Chen et al. 2024 Kocsis et al.) 2024 Zeng et al.
2024 While these works have demonstrated impressive results by repurposing diffusion models
for estimating geometric and semantic dense prediction maps, the critical design choices made in transferring diffusion models to other dense perception tasks still lack comprehensive justification. This makes it challenging to determine the optimal strategy for achieving optimal performance.
For example, Ke et (2024) align the visual perception process with the denoising process of Stable Diffusion by fine-tuning all U-Net parameters. They highlight the significance of "multi-resolution noise" in the forward diffusion process during training, aiming to obtain clean predictions by grad- ually removing Gaussian noise. On the other hand, Lee et al. (2024) modify the forward diffusion process by interpolating perception annotations with RGB images instead of using Gaussian noise, and only train the low-rank adaptation (LoRA) (Hu al. 2022) parameters while keeping the U-Net frozen. To our knowledge, the effective components of these approaches have not been thoroughly investigated, and it is unclear which design choices contribute most to the success.

---

Published as a conference paper at ICLR 2025
Encoder
2
y-
2
U-Net A
U-Not A
Decoder
(a) Stochastic malti-step generation.
Repeat
Times
x
2(3)
VAE
U-Net
U-Net
Decoder
Repeat for Times
Deterministic multi-stop generation.
Forward Once!
Encoder
U-Net A
①
Decoder
Customized
Decoder
or
(c) Our GenPercept: Deterministic unc-step perception.
Figure I: Comparisons three different pipelines. Our GenPercept enables one-step inference and supports pixel-wise losses and customized decoders to replace the cumbersome VAE decoder. We also extend GenPercept five dense perception tasks including monocular depth estimation, surface normal estimation, dichotomous image segmentation, semantic segmentation, and image matting.
In this work, we examine the design space of repurposing diffusion models for dense visual percep- tion tasks, and attempt to answer the key question: What are the important design choices when adapting diffusion models for general dense perception tasks?
To answer this question, we rethink the importance of both fine-tuning protocols and fine-tuning data. From the perspective of fine-tuning protocols, we categorize recent methods into two main groups: stochastic multi-step generation and deterministic multi-step generation. We explore several critical design choices, including the diffusion mechanism, key architectural components, training methodologies, and data quality. Our key observations are as follows: 1) By setting the hyperparam- eters of the diffusion scheduler to particular values, the multi-step generation can be simplified to a one-step fine-tuning paradigm without any loss of performance. 2) Strict adherence to traditional diffusion processes appears to be unnecessary. Single-step inference provides similar performance with significantly faster execution. 3) High-quality synthetic fine-tuning data is crucial for sev- eral perception tasks. From the perspective of fine-tuning data quality, conduct comprehensive dataset ablation studies on both synthetic datasets and real-world datasets.
Based on the aforementioned observations, propose GenPercept (see Fig. D deterministic fine-tuning paradigm featuring remarkably simple one-step inference pipeline, an optional cus- tomized decoder, and an easily adaptable pixel-wise loss. We conduct extensive quantitative and qualitative experiments on wide range of visual dense perception tasks, including monocular depth estimation, surface normal estimation, image segmentation, and matting to demonstrate the effec- tiveness and generalization capability of our method.
In conclusion, our contributions can be summarized as follows: We systematically analyze the design space of fine-tuning protocols, considering both model architecture and dataset selection, through comprehensive ablation studies. 2) Based on these insights, we propose GenPercept, simple paradigm that harnesses the power of the pre-trained UNet from diffusion models for gener- alizable dense visual perception tasks.
2 PRELIMINARY
We take the latent diffusion model as an example, To model the data distribution, idea of the diffusion model (Rombach et al. 2022 Chen et al., (2023] [Song et al. 2020; Ho et al.) 2020) is to randomly sample a noise z N(O. I) and sequentially denoise into z which is distributed
according the data. In the forward diffusion process, z(*) is sampled by x (r, +
vI- â,e, where N), and I.' (1 3.). The variance schedule {,3, (0. 1))T_ is interpolated between Bytart and Bend with steps, where larger values (B Bend) correspond
smaller on values, iee, smaller proportion of noise. In the reverse process, Salimans & Ho (2021)
2

---

Published as a conference paper at ICLR 2025
Noise
Normal
(t= 1000) t= 999 t= 800
600
400
t= 200
[t= 0)
Goussion
Noise
(n)
RGB
Noise
0.0
RGB
00,002222, 0.005) (0,000425, 0.005)
(3,00055, (.11))
(0 (0,00340, 0,048)
(0.1360, 0,192)
10.5460, 0,765)
Predictions trained with schodalers of different (eearr Boxs)
Smaller (fstarts Pena)
Larger Ustares Pend)
Figure 2: Illustration of different noise forms and proportions in the forward diffusion process, (a) Visualization of interpolating ground-truth labels with Gaussian noise and RGB noise. (b) The re- lationship between the noise proportion vā, and the (Sstart, Bend) hyperparameters. (c) Small (Bstart, Bend) values during the training of deterministic multi-step generation tend to lead to un- clean estimation, which contains some RGB information. Enlarging them may alleviate this issue.
use a "v-prediction" objective, where a denoiser Dpp minimizes the following:
= Egtr) (xw,))).exx10.....u ||(aaee - v1 - -azza - vo(ze, 1) 1
To fully leverage the pre-trained prior of the diffusion models for dense prediction tasks, previous works have reformulated these tasks multi-step denoising process, especially monocular depth estimation. Given a data pair (z(x), z(>)) where 2xx) is the observation and z(y) is the pre- diction target, stochastic multi-step generation methods (Ke et al. 2024 Fu et al. 2024b Gui al.) 2024 such as Marigold (Ke al. (024) add xxx) as an additional input to the denoiser vy. and use (2
z (x) to predict z(x), By contrast, deterministic multi-step generation methods such as DMP Lee al. 2024) take the observation 2xx) deterministic noise and compose x.
blend between z (x) and (x));
2, 2, vo + v1 (z, 1= (1,,,, T)
The denoising process and forward diffusion process these two categories are illustrated in Fig. and Fig. (a). We offer detailed formulations in the supplementary material.
DIFFUSION MODELS FOR VISUAL PERCEPTION TASKS
In this section, we explore the necessity and highlight the findings of the multi-step diffusion mech- anism, the architectural components, training strategy, and fine-tuning data quality. We select the stochastic method Marigold (Ke et al., 2024) and the deterministic method DMP (Lee et 2024) as our baseline methods. The default experimental setting here is similar to Ke et al. (2024) and can be found in the supplementary material.
3.1 THE FORM AND PROPORTION OF NOISE IN THE FORWARD DIFFUSION PROCESS
For the training process of Marigold and DMP, the timestep sampled control the proportion of noise added to the ground truth, and the network is trained to recover a clean ground truth from noisy latent. For smaller timesteps such as "t 200", as illustrated Fig. a)) the input to the network retains significant ground truth information (e.g., the purple color of the normal). Therefore, we hypothesize that a certain level of ground-truth label information being part of the input makes the network comparatively be easier recover the clean ground truth latent than the case of the absence of any ground truth information during training. This can limit the network's 's capacity to learn comprehensive knowledge and lead to unsatisfactory performance, it known that networks can become lazy that tend to exploit "shortcuts" " the ground truth labels the input in our case. On the other hand, for inference there is such ground truth available, causing disparity of input
3

---

Published as a conference paper at ICLR 2025
Table 1: Comprehensive quantitative comparisons about the impact of noise forms and proportions in the forward diffusion process on monocular depth estimation. Visualizations of different noise forms and the effect of (Astart: (end) values are shown in Fig. 2 The performance of DMP improves steadily, while Marigold shows initial improvements followed by a decline. When Satart and Send are equal to I, the inference process can be reduced to one step without compromising performance. "Rank" means the average rank of ten evaluation performance (smaller is better).
Type
Nose
Fonn
Muttires Naise
Step
AbaReai
KITTI
8.
AlsRcl
NYU
4g
AbnRci:
ScanNet
4.1
AlsRRcl
DIODE
AlaRol
ETIDD
51
Rank
Marigok
Cazseen
15
0,355
0.577
0,353
17:34
05625
0,255
0.355
0.251
0/540
7.0
Manigild
Grsseon
10
00.005575 CL0RTO)
0.127
0884
0.106
0.887
0.156
0.870
0.345
0.716
0.085
0.227
5.8
basiine
Marigoi
Gusssin
10
(0L00SS. (.012)
0.099
0.909
0.053
0.958
0.075
0.937
0.316
0.764
0.075
0.547
3.6
Mariold
Guussin
10
(0,0034, (L044)
0.100
0.900
0.877
0.63
0.03
0.957
0.308
0.788
0.01
0.543
Manisll
Gaascan
10
dB 10.136
0.119
0.861
0.858
0.63
0.61
0.961
6.315
0.073
Marigold
Cousson
10
20.440,0023 0.768)
0.124
0352
0.930
0.61
0.66
0.322
Mantood
Causson
10
(1.0)
0.104
0.055
0359
0.312
Mangpod
Causeon
(1.0
0150
Marigol
Guzssian
10
(0,000425, 0.006)
0.536
0.313
0.527
0.248
0.565
haadine Manigski
Gassco
10
ELRDORS, (.012)
0.153
0.807
0.62
0.302
0.667
0.731
0411
0.64
0.157
0.836
50
Marigsl
Gasscin
10
(0.0554, 0.048)
0.101
0.907
0.058
0.963
0.066
0.954
0,309
0.765
0.074
0.550
2.4
Marigold
Gassson
TO
3DI 13.1200, U. 0.19
0.115
0.770
0.056
0.565
0390
0.961
0.313
0.363
0.72
0.953
Marigoli
Grassia
10
(0 (0.5440, 0.768)
0.124
0.848
0.859
0.963
0.063
0.958
0.318
0.752
0.077
0.946
Marisold
Grssein
10
(L.S 1.0
0.102
0.901
0.854
0.966
0.059
0.962
0.312
0.76
0001
0:955
1-5
Marigok
FLA L.
0000
0.366
DX29
0.11
0.704
4.0
0.557
1700
00755
111
0225
1750
17100
03571
03477
DMP
RGB
10
10.000425, 0.005)
0.365
0.630
0.201
0.19
6.386
8116
baselne
RGB
(ULO0IIS (1012)
0117
0.811
0.21
DMP
0.107
0.850
0.087
0.318
DMP
RGB
10
1360,
0.107
0.850
0.063
0.959
0.068
0.305
DMP
RGB
10
5460, 017680
0.105
01897
0061
0.59
01866
0.952
0.009
0105
0.965
DMP
RGB
10
(L.O, 1.00
0.100
0
0.055
0.966
0.059
0.961
0.309
0.768
0.068
0.556
1.2
Our bascline
RGB
(1.0) 1.39
0.100
0.002
30055
0.965
0.259
0390
0.309
0.788
0.556
signals between inference and training. We note that T2I tasks may suffer less due to their stochastic nature, namely, converting a text prompt to a generated image is one-to-many mapping process,
To alleviate this issue, we attempt to control the blending proportion by changing the (Bstart, Bend) values of the diffusion model's DDPM scheduler. As shown in Fig. bb) and Fig. 2 of the supplemen- tary. training with (Bstort, Bend) value of (1.0, 1.0) achieves the best rank performance for both Gaussian noise and RGB noise, which is demonstrated in Table [ Rather than achieving consis- tent performance improvement while increasing the noise proportion, we observed that Marigold's performance begins to be slightly unstable when the (Bstart, Bend) values are sufficiently high. Ex- periments of varying the random seed during both the training and inference process are conducted to rule out the influence of randomness. The results show that we may not be able to exactly find set of unique values for (Astart, Bena) to achieve the best accuracy, the final accuracy can be affected by a few other factors besides the aforementioned one.
Additionally, when (Bstort, Bend) are equal to 1, the noise proportion Vag is equal to O, and the formulation of DMP can be derived from Eq. ① and Eq. (2) as follows.
24 z Varz(() + vI āzz(()) (x) =x)x t [1,..., TT
(3)
C = (e0.
|2(0x) - vo (0 t)
In this case, the output of the denoiser vo(,,)) is enforced to learn the negative value of the ground truth latent for each step. and the multi-step denoising is equivalent to the single-step denoising. We propose to reduce the DDIM steps of DMP to one and call it "deterministic one-step perception". The resulting inference can be significantly faster, with performance remaining almost unchanged. We name this as "our baseline" for the subsequent analysis.
Finding 1. By setting the (Astarts Bend) values to I, the multi-step generation is simplified to a one-step fine-tuning paradigm without any loss of performance in both stochastic and de- terministic methods, e.g., Marigold (Ke et al. 2024) and DMP (Lee et al. 2024) respectively.
3.2 WHERE DOES THE RICH VISUAL KNOWLEDGE RESIDE IN DIFFUSION MODELS?
Based on the baseline proposed in 3.1 we conduct detailed ablation studies to thoroughly investi- gate the necessity of each component of Stable Diffusion. Results are reported in Table 2
Denoiser. We reinitialize the U-Net parameters and train the network from scratch on the same datasets. Without prior knowledge of large data from LAION-5B, the network performs poorly and
4

---

Published as a conference paper at ICLR 2025
Table 2: Explorations on the impact of the Stable Diffusion components depth estimation. Cus-
tomized decoders and losses can also enable inference accelcration and performance improvement.
Setting
Loss
AbsRal
Krrn
6
Abwaa
NYU
0.
AbsRel
SeanNet
01
AmsRel
DIODE
61
Abskol:
ETH3D
Our baseline
MSE (Latent)
0,100
0,002
0,83
0.966
0,059
0.961
0.309
0.768
0065
0.950
scratch
MSE (Latent)
B219
0.186
0426
Train VAE decoder from scratch
MSE (Image)
00066
0916
0.055
0.964
0.058
0.964
0.30
0.759
0071
0350
Baseline I Image MSE loss
MSE((mage)
D.00
0.15
0,054
0.64
0.889
0.964
0.308
0.760
0071
0.93
Baseline Inage customized loss
Grad. (Image)
0.094
0.923
0.052
0.966
0.056
0.965
0.302
0.767
0.066
0.967
Trin DPT Uccoder from sctatcc
MSE (Inage))
0,059
0.912
0.035
0.961
0.058
0.963
0.302
0.759
0.09
0,95
loses the generalization capability. This indicates that most of the prior knowledge is stored in the U-Net module.
VAE AutoEncoder: The VAE encoder's original architecture is kept intact to maintain the consis- tency of the encoding process. For the VAE decoder, train it from scratch with image pixel MSE loss. Without pre-trained parameters of the VAE decoder, it still performs well.
Customized Head and Loss. The deterministic one-step perception pipeline enables customized heads and loss functions. By utilizing DPT decoder (Ranftl et al. 2021) and the loss functions of DepthAnythingy2 (Yang et al.) 2024b), we can implement a lightweight decoder that supervises pixel-wise information at a higher resolution rather than latent features at 1/8 resolution. This approach can accelerate inference times and enhance the acquisition of fine-grained details.
The primary perceptual prior knowledge of diffusion models is encapsulated within the U-Net of the diffusion model. Customized heads and loss functions offers flexibil- ity and may lead to faster inference speed and improved results.
Finding 2.
3.3 WHAT ABOUT THE TIMESTEPS AND TEXT PROMPTS?
The timesteps and text prompts are crucial elements in utilizing the Stable Diffusion model to gen- erate diverse images. We conducted ablation studies to investigate their significance. The results reported in Table 3 indicate negligible difference between various settings. Owing to the inherent certainty associated with visual perception tasks, the diversity typically offered by the textual inputs appears to be unnecessary. Similarly, the utility of timesteps reduced, as the single-step paradigm does not require progressive denoising.
Table 3: Quantitative comparisons among different timesteps and text prompts on depth estimation.
Setting
Promp:
Timestops
Asakali
KITTI
NYU
AbuRall
Scom
DIDDE
Abakalj
ETJ13D
Our baseline
Random
0,100
03582
0,053
0.966
0,08%
0361
0.30
0.768
0,063
01966
Volid lest input
"A high quality KGB imagee "V3 La""
Kandom/
0.101
0.500
0.033
0.967
0.05S
0.964
0,312
0.762
0,070
0.55
Kandom test inpur
Random/
0399
0.984
0.04
01965
B3099
01983
0,313
0.763
0.099
01955
Timooepl
171
0.100
15006
0.4
13835
E.R0
11361
0.20
0.700
Timestep500
500 500
0.102
0.897
0,053
0.966
0,059
0.961
0.307
0.765
0.068
0.956
Timestepfi00
9001
0.105
0.054
Finding 3.
The timesteps and text prompts of diffusion models are negligible for the per- formance of visual perception tasks.
3.4 How TO LEVERAGE THE U-NET'S ' PRIOR KNOWLEDGE?
The significance of the denoiser cannot be overstated. However, the strategies for its utilization are worth careful study. Should we freeze the denoiser, utilize its intermediate features, and merely fine-tune the decoder for specific tasks? Alternatively, can we employ LoRA (Hu et al.) 2022) instead of extensively fine-tuning the entire denoiser? Unfortunately, the evidence suggests that neither approach is ideal. As illustrated in Table 4 freezing the denoiser significantly compromises performance. Although incorporating LoRA offers some advantages, may not fully leverage the potential of denoiser, especially with regular LoRA ranks of 4 and 16. This limitation likely stems from the substantial differences between the noise-to tosse-toiing tasse-t--mmae denoising process and the image-to- perception prediction task.
5

---

Published as a conference paper at ICLR 2025
Table 4: Explorations the paradigms leverage U-Net's prior knowledge depth estimation.
Setting
LoRA Rank
KITTI
NYU
ScanNet
DIODE
ETH3D
AbsRel.
61
AbaRell
81
AbsRel]
AbsRelj
5.
AbsRel,
Our baseline
0.100
0.502
0.053
0.356
0.039
0.961
0.309
0.765
0.066
0.956
Frocze U-Net Train DPT decoder
0.144
0.803
0.086
0.931
0.097
0.911
0.309
0.768
0.068
0.956
U--er
0.902
0.121
Train U-Not with LoRA
16
0.166
0,746
0.085
0.931
0.087
0.927
0.352
0.712
0.104
0.901
Train U-Net with LoRA
0.138
0.817
0.077
0.944
0.079
0.940
0.336
0.734
0.069
0.930
Train U-Net with LoRA
256
0.133
0.827
0.069
0.952
0.073
0.947
0.325
0.745
0.088
0.935
Train U-Net with LoRA
1024
0.125
0.849
0.067
0.955
0.074
0.947
0.324
0.747
0.084
0,939
Table 5: Investigations into the impact of training data quality on depth estimation.
DIODE
Data Quality
AhwRal|
6,7
AhoRelj
AboRelj
6,1
AbsRel,
41
AbsRel
61
Synietic Data
Hypersim (50K) + Virtual (30K)
0.100
0502
0.053
0.966
0,059
0.961
0,309
0.768
0.068
0.956
Real Data
Taskonoryy (50K) + Citysaaee (40K)
0.123
0.857
0,055
0.966
0,062
0.958
0.293
0.762
0.074
0,947
Finding 4. Fine-tuming the denoiser appears be preferable for achieving better results, compared to either merely utilizing its intermediate features or training a LoRA.
3.5 IS THE TRAINING DATA QUALITY ESSENTIAL?
The quality of annotations in real datasets is often lower compared to synthetic datasets, where data precisely rendered via simulators. In Table 5 we explore the impact of data quality on the fine- tuning process. We sample the same distribution real data, consisting of 90% from approximately 50K indoor images from the Taskonomy dataset (Zamir al. 2018) and 10% from about 40K outdoor images from the Cityscapes dataset (Cordts et al. (0166). With lower annotation quality, the model achieves slightly worse performance. Also, the visualization in the supplementary material indicates that noisy data significantly influences detailed predictions in visual perception tasks.
Finding
Data quality affects the fine-grained details of dense predictions significantly.
3.6 SUMMARY OF THE OBERVATIONS
Based the preceding analysis, effective approach to leveraging the prior knowledge of diffu- sion models is to use them as single-step deterministic perception estimators. This can be done with either VAE decoder customized losses can further enhance detail and overall performance. We compare our determin- istic single-step step perception method with previous multi-step paradigms in Fig. [ In the following section, we extend these findings to a broader set of visual perception tasks, including surface normal estimation, semantic image segmentation, dichotomous image segmentation, and image matting.
customized lightweight decoder. Additionally, employing pixcl-specific
EXPERIMENTS ON VARIOUS DENSE VISUAL PERCEPTUAL TASKS
In this section, we empirically show the robust transfer ability of our GenPercept on diverse visual tasks. Unless specified otherwise,
freeze the VAE AutoEncoder and fine-tune the U-Net of Stable Diffussin v2.1 1 to estimate the ground-truth label latent for 30000 iterations, with a resolution of (768, 768), a batch size of 32, and learning rate of 3e-5. Different customized loss functions are utilized to improve the performance further on dense visual perception tasks.
4.1 GEOMETRIC ESTIMATION
For geometry evaluation, the ensemble size, inference resolution, valid evaluation depth range (spe- cific for depth estimation), and evaluation average paradigm (average by pixels or average by the number of images) can be different for each method. To compare these approaches fairly, we follow the open- evaluation code of Marigold (Ke et al. (024) for depth and DSINE (Bae & Davison, 2024) for surface normal, and evaluate the performance of partial existing SOTA methods with their officially released model weights. They are labeled with 1 in the Table.

---

Published as a conference paper at ICLR 2025
Table 6: Quantitative comparison of affine-invariant depth estimation five zero-shot datasets.
Results marked with are evaluated following the evaluation protocol of Marigold by ourselves.
Method
Training Samples
Abakell
KITTI
6,"
AbaRell
NYU
51"
AbsRal
ScanNer
5,1
AbsRel,
DIODE
4,
AhsRel,
ETHN
djT
MilaS
(Ranfi et al.
2020
2M
0,236
0.630
0.111
0.885
0.121
0.846
0.332
0.715
0.184
0.752
Omnidata
Beeeearr
12.2M
0.149
0.835
0.074
0.945
0.075
0.936
0.339
0.742
0.166
0.778
Kanii 33
1.4M
0.100
0.501
0.098
0.903
0.082
0.934
0.182
0.758
0.078
0.946
DepthAnything
Yane m1.
91443
63.5M
0,080
D.946
0.043
0.980
0.043
0.981
0.261
0.759
0.058
0.984
DepthAmpthing
62.6M
0,080
0.943
0,013
0.979
0.321
0.758
0.066
Metri:31>>
Hu.d d
2024
16M
0.052
0.979
0.0399
0,979
0.023
0,989
0.147
0.892
0.040
0.983
Depll
2020
320K
0.190
0.704
0.317
0.875
0.109
0.882
0.376
0.631
0.228
0.694
Lokes's
Tin R.
20241
354K
0.149
0.784
0.090
0.916
03091
0.917
0:271
0.766
0.171
0.777
HDN (Thing 2
21222
500K
0.115
0.867
0.069
0.948
0.080
0.939
0.246
0.780
0.121
0,833
GooWizrd
28IK
0.097
0
0.953
0:297
0.792
0.064
DepthFM
CRa
2074
63K
0.083
D.934
0.065
0,956
0.800
Marigold
Kc
2021
74K
0.099
0.916
0.055
0.964
0.064
0.951
0.308
0.773
0.065
0.960
Official!
Lxe 8
0.240
0.622
0.109
0.146
0.814
0.361
0.706
0.128
GeoWizard!
Furet 21
202
280K
0.129
0.851
0.059
0.959
0.066
0.953
0.328
0.753
0.077
0,910
DepthFM
0.903
0.729
0.101
Our Cene' (Liepit)
90K
0.099
0.523
0.052
0.366
0.056
0.965
0.302
0.767
0.066
0397
Our GcnPPencopi (Disparity)
90K
0.080
0.934
0.058
0.969
0.063
0.960
0.226
0.741
0.096
0.959
Our GenPercept (Disparity
DPT head)
90K
0,078
0,935
0.059
0.967
0.064
0.961
0.228
0.740
0.094
0,961
Table Quantitative comparison of surface normal estimation on three zero-shot datasets. We eval- uate mean error,, median error (med.), and the percentages of pixels 1 with five thresholds. Part of the reported results (1) are evaluated following the evaluation protocol of DSINE by ourselves.
Mathod Traising
NYU
ScanNer
Sintel
Samples mecn med.
[.0"
7.5"
11.05*
20.5"
aur
mein
med
6.0"
7:5"
11.25"
22.0"
50"
moon
med
[,00
7.5"
13:50"
20.6
50"
Chumdlats v1
N0.
12755
2XT
120
11.6
35.4
45.8
66.3
757
12.3
21.3
14.5
27.
06.1
15.2
41.3
35.7
300
58
11.4
504
4X
Ominidad
12:2M
17.2
9.7
25.3
60.1
55,5
76.5
83.0
16.2
8.5
29.1
44.9
60.
79.5
84.7
40.5
35.1
7.9
14.7
33.0
43.5
MetricD
6.7
40.1
87.7
41.6
75.8
Giow zand Pa al
281
170
36.5
154
61.6
DINSE Hae
14NK
84
328 46.3
26.1
21.5
52.7
Gowwrrd
280K
198
11.
32.7
50.2
75.0
79.9
21.1
11.9
15.9
20.7
70.7
77.8
28.4
169
50.8
53.5
Our GailPerepp
Lall MSE ko0)
174
93
23.3
2010
56.3
763
830
16.3
89
258.22
427
5016
7924
24.3
44
31.0
34
73
150
570
43
Our GenPercept (Image angular Joaa
90K
16.4
8,0
33.3
47.8
68.9
78.3
R3.7
15.2
7.4
33,9
50.7
65.0
80,9
85.7
34.6
26.2
5.2
98
18.4
43.8
55.8
Monocular Depth Estimation. The monocular depth estimation aims to predict the vertical dis- tance between the observed object and the camera from an RGB image. The estimated depth is formulated as invariatt depth (Yin et al. [021: Ranftl et al.) 2020) 202I), and should be re- covered by performing least square regression with the ground truth. The evaluation is performed on five zero-shoft datasets including KITTI (Geiger et al.) 2013), NYU (Silberman et al. 2012). ScanNet (Dai et al. 2017). DIODE (aaiijeevic et al.] 2019). and ETH3D (Schops et al., 2017). We compute the absolute relative error (AbsRell) and percentage of accurate valid depth pixels (61 f). Invalid regions are filtered out and the metrics are averaged on all images.
Surface Normal Estimation. The surface normal estimation aims to predict vector perpendicular to tangent plane of the surface at each point P, which represents the orientation of the object's 's surface. For evaluation, et al. (0122), ScanNet (Dai et al. [017]. and Sintel (Butler et al.) 2012). The mean 4. median J. and the percentages of pixels with error below thresholds [5 7.5". 11.25%, 22.5", 30"] are reported. Invalid regions are filtered out and the metrics are averaged on all images.
compute the angular error three zero-shot datasets including NYU (Silberman)
Quantitative Evaluation. Quantitative results on monocular depth estimation and surface normal estimation are shown in Table[Bland Table respectively. Even trained on limited synthetic datasets only, our GenPercept shows much robustness and achieves promising performance on diverse un- seen scenes. For monocular depth models, we train them with pixel-wise MSE loss, scale-shift- invariant loss (Ranful et al.. 2020), and gradient loss (Ranfil et al., 2020). Furthermore, our disparity model (inverse of the depth) shows much better performance on datasets with outdoor scenes, such as KITTI and DIODE, but less performance on indoor datasets. Therefore, we suggest adopting the depth model for indoor scenes and the disparity model for outdoor scenes experimentally. By replacing the VAE decoder with a lightweight DPT head (Ranftl et al., 2021). GenPercept can infer faster without bearing the performance penalty. For surface normal estimation, the image angular loss brings significant performance improvement thanks to our one-step estimation paradigm.
Qualitative Results. Qualitative visualizations shown in Fig. We observe excellent general- ization of our models in that they can estimate accurate geometric information and promising details not only diverse real and synthetic scenes, but also on comics, color drafts, and even sketches.
7

---

Published as a conference paper at ICLR 2025
Depeh
Prod, Deph
Frod. Depth
Prod.
Figure 3: Qualitative results of monocular depth and surface normal estimation. The model works surprisingly well on out-of-domain images (sketch and cartoon images).
4.2 IMAGE SEGMENTATION
Dichotomous Image Segmentation. This is category-agnostic, high-quality object segmenta- tion task that accurately separates the object from the background in an image. Consistent with previous methods, we use the six evaluation metrics specified in the DIS task, which include max- imal F-measure (marF's t) (Achanta et al. 2009), weighted F-measure (F f) (Margolin et al. 2014), mean absolute error (M4) (Perazzi et al. 2012). structural measure (S,1) (Fan et al. 2017). mean enhanced alignment measure (ET") (Fan et al. 2018 (022b) and human correction efforts (HCF,,)) (Qin al. 2022). We choose DIS5K (Qin et al. (022) as the training and testing dataset. We utilize DIS-TR for training and evaluate our model on DIS-VD and DIS-TE subsets. The pixel-
wise MSE loss utilized during training.
Quantitative results of dichotomous image segmentation are shown in Table g We only show partial results due paper page limitations, full comparisons are accessible in the supplementary material. GenPercept outperforms methods like HySM (Nirkin et al. 2021) and IS-Net (Qin et al. (022) on this challenging dataset across most evaluation metrics, but there exists room for further improve- ment compared to SoTA methods like MVANet Yu et al. 2024). As shown in Fig. [ our approach provides detailed foreground mask. For thin lines and meticulous objects that are difficult for previous methods to process, our method can also output accurate segmentation results.
Semantic Image Segmentation. This a fundamental computer vision task that involves classify- ing each pixel in image into specific category class (Zhao et al., 2024 Ji et al. (024). For training, we utilized the indoor synthetic dataset, HyperSim (Roberts et 20211) which comprises 40 semantic segmentation class labels. We encode different classes into 3-channel colormaps, treat the task as regression problem, and fine-tune the original Stable Diffusion with the pixel-wise MSE loss. As demonstrated in Fig. 4 the model generalizes well to classes within the HyperSim annotations, such as chairs and desks, but struggles with unrecognized categories such as cats and cars.
Another choice involves using a customized segmentation head. We incorporate a custom segmen- tation head, namely UperNet (Xiao et al., 2018), onto the multi-level features extracted by UNet. For the UperNet segmentation head, we follow the traditional semantic segmentation format to use n-channel output, where the number of categories. The quantitative results are presented in Table 9 we test the model's performance on Hypersim (Roberts et al., 2021) and zero-shot ability on subset of the ADE20k (Zhou et al. (017) validation set, which contains overlapping classes.
Table 8: Quantitative results of dichotomous image segmentation on DISSK validation and testing sets. Additional cross-datase evaluation is provided in the supplementary material.
Datsel
DIS-VD
D3S-7E4
Chouul DAS- LOS-TEIR (1-4)
MER:
1F1
3.1
BCE.
11277
3/1
S.1
BCE
11377
S,1
E.1
JCC,
11:10
3750
0.15
0.315
11:37
03377
3.50
10270
01887
01701
01731
0.400
0773
11814
1526
0.993
0/95
11381
0.00
01842
1131
0.55
05665
0188
0.70
0.874
1318
MIVAN
0961
0361
0.877
01:4
6.6
0191
0.16
0.885
0.00
09005
0.96
1330
3320
210
11311
1311
0311
0.20
1138
2330
11011
1700
0323
0179
0105
03772
0106
1351
Guspanup Cafce (024)))
0.59
0.34
8

---

Published as a conference paper at ICLR 2025
Dataser GT
In-lhe-wild Example
Failed on unseen categorics
Dataset RGB
GT In-the-wild RGB
Pred. Seg.
In-the-willd RGB
Pred. In-the-wild RGB
Pred.
Figure 4: Quanlitative results semantic segmentation the wild. Trained on the synthetic indoor Hypersim dataset, GenPercept shows much robustness the trained categories of complex in-the- wild images, e.g., yellow chairs, green floor, and light blue wall. Due to the limited annotation categories and little negative label of "unknown category", it sometimes fails in outdoor scenes and unseen categories such as cats and cars.
Imago
HYSM
Figure 5: Qualitative comparison of dichotomous image segmentation.
Besides, we compare with Mask2Former (Cheng et al. 2022) by training on ADE20K, GenPercept outperforms ResNet50 (He et 2016) and Swin-T (Liu 2021) of Farkerormer but achieves lower performance than Swin-L (iuu et al. 2021).
4.3
IMAGE MATTING
Task Definition. Image matting aims to extract the foreground, background, and alpha mask from an image. Traditional approaches depend on supplementary inputs that delineate foreground, back- ground, and ambiguous areas to reduce uncertainty. Automatic image matting seeks to remove this dependency by directly estimating these components from the image alone. The implementation details can be found in the supplementary material.
Quantitative and Qualitative Results. We evaluate metrics including the sum of absolute differ- ences (SAD), mean squared етrоr (MSE). mean absolute difference (MAD), gradient (Grad.), and Connectivity (Conn.) on the P3M-500-NP test set. SAD and MAD measure the mean LI distance between predictions and ground truth labels. MSE and CONN focus L2 distance and connec- tivity that better reflects human intuition. As shown Table TO our GenPercept less accurate compared with the state-of- the-art methods. However, when transferring the human image matting ability to general image matting tasks, GenPercept achieves much better performance. It proves the robustness brought the prior knowledge of diffusion models pre-trained the LAION dataset. Quantitative results of image matting are shown in the supplementary.
Table 9: Quantitative results of semantic segmentation on Hypersim and ADE20k.
Table 10: Quantitative comparisons of image matting on the P3M-500-NP and AIM500.
Methed
Tritiix
Paianel
(lypenami
mlot (ATECNIC
HATT
Qua ctal
Meduol
20221
Test Dusseet
SAD 1
X0.3
MAD
100176
MSE
00072
CONN
27.42
GenPescope UpecNiei
Kenron Fleain UpezNec
Hyperits
MODNET
SHM
P3M-500-NP
20.77
11.23
0.0065
0.0122
0.093
17.09
Makclwwor Somn-
Mak-Feema
ADEZOK
GenPenospt (Tain NNN UjecNen)
30.2
019444
36.74

---

Published as a conference paper at ICLR 2025
5 RELATED WORK
Vision Pre- Training. Models pretrained on large-scale datasets possess powerful feature extraction capabilities, enabling them to be effectively transferred to wide range of visual tasks. For instance, the ResNet (He et al. 2016) model pretrained on ImageNet (Russakovsky et al. 2015) can be fine-
tuned and applied to perception tasks. By means of contrastive learning, MoCo (He al. 2020)
and CLIP (Radford et al., (021) acquire rich visual and semantic representations, leveraging their advantages in joint visual and semantic modeling enhance the performance of multimodal tasks.
DINO CCaro et al. 202T), through self-distillation, endows Vision Transformer and convolutional networks with comparable visual representation quality and demonstrates that self-supervised ViT representations contain explicit semantic segmentation information. DINOv2 (Oquab et al., (024) leverages self-supervised learning on large curated dataset and cxhibits remarkable zero-shot gen- eralization capabilities across computer vision tasks at both image and pixel levels, including clas- sification, semantic segmentation, and depth estimation. In our work, we leverage Stable Diffusion RRomaach et al. 2022) as a prior for scene understanding and transfer it various perception tasks.
Diffusion Priors for Dense Prediction. Several works explore to use the priors of generative models for perceptual tasks. Some works (Bhattad et al., 2024 Du et al. (023) demonstrate that generative models encode property maps of the scene. By finding latent variable offsets, using LoRA (Hu et al., 2022), etc., generative models can directly produce intrinsic images like surface normals, depth, albedo, etc. LDMSeg (Van Gansbeke & De Brabandere 2024) devises an image-conditioned sam- pling process, enabling diffusion models directly output panoptic segmentation. UniGS (Qi al. 2023) proposes location- location-aware aware color encoding and decoding strategies. allowing diffusion models to support referring segmentation and entity segmentation. Marigold (Ke et al., 2024) fine-tunes diffusion model on limited synthetic data, enabling it to support affine-invariant monocular depth estimation and exhibit strong generalization performance. However, Marigold is cine-conuuning due to the need for multiple iterations of denoising. Additionally, the Gaussian noise leads to incon- sistent results across inferences, requiring aggregation over multiple inferences, Xiang et al. (2023) train denoising auto-encoder for image classification. The difference of their method compared with traditional denoising auto-encoder that input images are encoded into a latent code and de- noising is performed in the latent space rather than the pixel space. They show good results on very small-scale datasets (CIFAR and ImageNet-tiny) to prove the concept and no results were reported on larger datasets. Furthermore, Geo Gezai (Fu al. 2024a) extends the generative capabilities of Marigold, achieving better performance in joint depth and normal estimation, which enhances ap- plications like 3D reconstruction and novel view synthesis. Moreover, DepthFM (Gui et al.) (024) addresses the speed challenge of Marigold by employing flow matching, offering a fast and efficient monocular depth estimation model.
CONCLUSION
In this work, we introduce GenPercept, an embarrassingly straightforward yet powerful approach to re-use the off-the-shelf UNet trained using diffusion processes. GenPercept demonstrates the capa- bility to effectively leverage pre-trained diffusion models across a range of downstream dense per- ception tasks. We contend that proposed methodology provides efficient and potent paradigm for harnessing the capabilities of ppe--trained diffusion models in dense visual perception tasks.
For future research, suggest investigating the impact of scaling up the volume of fine-tuning data and exploring the key components of pre-training by applying alternative self-supervised pre- training methods the LAION dataset, such Masked Autoencoders (MAE) or Contrastive Language-Image Pretraining (CLIP). It will be helpful to clarify whether the highly detailed visual predictions produced by existing diffusion models are primaríly driven by the extensive LAION dataset or the diffusion pretraining paradigm itself.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work is partially supported by the National Key R&D Program of China 00022222201100000000 Ningbo Science and Technology Bureau (Grant Number 20242291) and the National Natural Science Foundation of China (No. 62206244). Y. Ge was with The University of Adelaide and his contribution was made when visiting Zhejiang University. C. Shen is the corresponding author.
REFERENCES
Radhakrishna Achanta, Sheila Hemami, Francisco Estrada, and Sabine Susstrunk. Frequency-tunde salient region detection. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2009.
Gwangbin Bae and Andrew J Davison. Rethinking inductive biases for surface normal estimation. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2024.
Anand Bhattad, Daniel McKee, Derek Hoiem, and David Forsyth. Stylegan knows normal, depth, albedo, and more. Proc. Advances in Neural Inf. Process. Syst. 2024.
D. Butler, J. Wulff, G. B. Stanley, and M. J. Black. naturalistic open source movie for optical flow evaluation. In A. Fitzgibbon et al. (Eds.) (ed.), Proc. Eur. Conf. Comp. Vis., Part IV, LNCS 7577, pp. 61 611-625. Springer-Verlag, Verlag, October 2012.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in sulf-supervised vision transformers. In Proc. IEEE Int. Conf. Comp. Vis., pp. 9650-9660, 2021.
Junsong Chen, YU Jincheng, GE Chongjian, Lewei Yao, Enze Xie, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-o: Fast training of diffusion transformer for photorealistic text-to-image synthesis. In Proc. Int. Conf. Learn. Representations, 2023.
Quan Chen, Tiezheng Ge, Yanyu Xu, Zhiqiang Zhang, Xinxin Yang, and Kun Gai. Semantic human matting. arXiv: Comp. Res. Repository, 2018.
Xi Chen, Sida Peng, Dongchen Yang, Yuan Liu, Bowen Pan, Chengfei Lv, and Xiaowei Zhou. Intrinsicanything: Learning diffusion priors for inverse rendering under unknown illumination. arXiv: Comp. Res. Repository, 2024.
Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked- attention mask transformer for universal image segmentation. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pp. 1290-1299, 2022.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pp. 3213-3223, 2016.
Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2017.
Xiaodan Du, Nicholas Kolkin, Greg Shakhnarovich, and Anand Bhattad. Generative models: What do they know? do they know things? let's find out! arXiv: Comp. Res. Repository, 2023.
Ainaz Eftekhar, Alexander Sax, Jitendra Malik, and Amir Zamir. Omnidata: scalable pipeline for making multi-task mid-level vision datasets from 3d scans. In Proc. IEEE Int. Conf. Comp. Vis., 2021.
D Fan, M Cheng, Liu, Li, and A Borji. A new way to evaluate foreground maps. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2017.
Deng-Ping Fan, Cheng Gong, Yang Cao, Bo Ren, Ming-Ming Cheng, and Ali Borji. Enhanced- alignment measure for binary foreground map evaluation. In Proc. International Joint Conf. Artificial Intelligence, 2018.
11

---

Published as a conference paper at ICLR 2025
Deng-Ping Fan, Ge-Peng Ji, Ming-Ming Cheng, and Ling Shao. Concealed object detection. IEEE Trans. Pattern Anal. Mach. Intell., 2021a.
Deng-Ping Fan, Ge-Peng Ji, Xuebin Qin, and Ming-Ming Cheng. Cognitive vision inspired object segmentation metric and loss function. Scientia Sinica Informationis, 2021b.
Xiao Fu, Wei Yin. Mu Hu, Kaixuan Wang, Yuexin Ma, Ping Tan, Shaojie Shen, Dahua Lin, and Xiaoxiao Long. Geowizard: Unleashing the diffusion priors for 3d geometry estimation from a single image. In Proc. Eur. Conf. Comp. Vis., 2024a.
Xiao Fu, Wei Yin. Mu Hu, Kaixuan Wang, Yuexin Ma, Ping Tan, Shaojie Shen, Dahua Lin, and Xiaoxiao Long. Geowizard: Unleashing the diffusion priors for 3d geometry estimation from a single image. In Proc. Eur. Conf. Comp. Vis., 2024b.
Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti dataset. Int. J. Robotics Research, 2013.
Ming Gui, Johannes Fischer, Ulrich Prestel, Pingchuan Ma, Dmytro Kotovenko, Olga Grebenkova, Stefan Andreas Baumann, Vincent Tao Hu, and Bjöm Ommer. Depthfm: Fast monocular depth estimation with flow matching. arXiv: Comp. Res. Repository, 2024.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2020.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Proc. Ad- vances in Neural Inf. Process. Syst., 2020.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. Proc. Int. Conf. Learn. Representations, 2022.
Mu Hu, Wei Yin, Chi Zhang. Zhipeng Cai, Xiaoxiao Long, Hao Chen, Kaixuan Wang, Gang Yu, Chunhua Shen, and Shaojie Shen. Metric3d v2: A versatile monocular geometric foundation model for zero-shot metric depth and surface normal estimation. arXiv: Comp. Res. Repository, 2024.
Wei Ji, Jingjing Li, Qi Bi, Tingwei Liu, Wenbo Li, and Li Cheng. Segment anything is not always perfect: An investigation of SAM on different real-world applications. Machine Intelligence Research, 21:617-630, 2024.
Oguzhan Fatih Kar, Teresa Yeo, Andrei Atanov, and Amir Zamir. 3d common corruptions and data augmentation. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2022.
Bingxin Ke, Anton Obukhov, Shengyu Huang, Nando Metzger, Rodrigo Caye Daudt, and Konrad Schindler. Repurposing diffusion-based image generators for monocular depth estimation. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2024.
Zhanghan Ke, Jiayu Sun, Kaican Li, Qiong Yan, and Rynson W.H. Lau. Modnet: Real-time trimap- free portrait matting via objective decomposition. In Proc. AAAI Conf. Artificial Intell., 2022.
Peter Kocsis, Vincent Sitzmann, and Matthias Nießner. Intrinsic image diffusion for indoor single- view material estimation. In Proc. IEEE Conf. Comp. Vis. Patl. Recogn., pp. 5198-5208, 2024.
Hsin- Hsin-Ying Ying Lee, Hung- Yu Tseng, Hsin- Ying Lee, and Ming-Hsuan Yang. Exploiting diffusion prior for generalizable dense prediction. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2024.
Jizhizi Li, Sihan Ma, Jing Zhang, and Dacheng Tao. ACM Int. Conf. Multimedia, 2021.
portrait matting. In Proc.
Ze Liu, Yutong Lin, Yue Cao, Han Hu. Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proc. IEEE Int. Conf. Comp. Vis., pp. 10012-10022, 2021.
12

---

Published as a conference paper at ICLR 2025
Sihan Ma, Jizhizi Li, Jing Zhang, He Zhang, and Dacheng Tao. Rethinking portrait matting with privacy preserving. Int. J. Comput. Vision (3(8))2172-2197, 2023.
Ran Margolin, Lihi Zelnik-Manor, and Ayellet Tal. How to evaluate foreground maps? In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2014.
Yuval Nirkin, Lior Wolf, and Tal Hassner. Hyperseg: Patch-wise hypernetwork for real-time seman- tic segmentation. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2021.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, and Mare Szafraniec et al. DINOv2: Learning robust visual features without supervision. Trans. Mach. Learn. Research, 2024.
Federico Perazzi, Philipp Krähenbühl, Yael Pritch, and Alexander Horuung Saliency filters: Con- trast based filtering for salient region detection. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2012.
Lu Qi, Lehan Yang, Weidong Guo, Yu Xu, Bo Du, Varun Jampani, and Ming- Haunn Yang. Unigs: Unified representation for image generation and segmentation. arXiv: Comp. Res. Repository, 2023.
Yu Qiao, Yuhao Liu, Xin Yang, Dongsheng Zhou, Mingliang Xu, Qiang Zhang, and Xiaopeng Wei. Attention-guided hierarchical structure aggregation for image matting. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., June 2020.
Xuebin Qin, Zichen Zhang, Chenyang Huang, Masood Dehghan, Osmar Zaiane, and Martin Jagersand. U2-net: Going deeper with nested u-structure for salient object detection. Pattern Recogn., 2020.
Xuebin Qin, Hang Dai, Xiaobin Hu, Deng-Ping Fan, Ling Shao, and Luc Van Gool. Highly accurate dichotomous image segmentation. In Proc. Eur. Conf. Comp. Vis., 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In Proc. Int. Conf. Mach. Learn., 2021.
René Ranfil, Katrin Lasinger. David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE Trans. Pattern Anal. Mach. Intell., 2020.
René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. In Proc. IEEE Int. Conf. Comp. Vis., 2021.
Mike Roberts, Jason Ramapuram, Anurag Ranjan, Atulit Kumar, Miguel Angel Bautista, Nathan Paczan, Russ Webb, and Joshua M. Susskind. Hypersim: A photorealistic synthetic dataset for holistic indoor scene understanding. Proc. IEEE Conf. Comp. Vis., 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjom Ommer. High- resolution image synthesis with latent diffusion models. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2022.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. Int. J. Comput. Vision, 115, 2015.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In Proc. Int. Conf. Learn. Representations, 2021.
Thomas Schops, Johannes L. Schonberger, Silvano Galliani, Torsten Sattler, Konrad Schindler, Marc Pollefeys, and Andreas Geiger. A multi-view stereo benchmark with high-resolution images and multi-camera videos. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2017.
Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and sup- port inference from rghd images. In Proc. Fur. Conf. Comp. Vis., 2012.
13

---

Published as a conference paper at ICLR 2025
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In Proc.
Int. Conf. Learn. Representations, 2020.
Wouter Van Gansbeke and Bert De Brabandere. A simple latent diffusion approach for panoptic segmentation and mask inpainting. arXiv: Comp. Res. Repository, 2024.
Igor Vasiljevie, Nick Kolkin, Shanyi Zhang. Ruotian Luo, Haochen Wang. Falcon Z. Dai, An- drea F. Daniele, Mohammadreza Mostajabi, Steven Basart, Matthew R. Walter, and Gregory Shakhnarovich. DIODE: Dense Indoor and Outdoor DEpth Dataset. arXiv: Comp. Res. Repos- itory, 2019.
Weilai Xiang, Hongyu Yang, Di Huang, and Yunhong Wang. Denoising diffusion autoencoders unified self-supervised learners. In Proc. IEEE Int. Conf. Comp. Vis., 2023.
Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In Proc. Eur. Conf. Comp. Vis., 2018.
Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn. pp. 10371-10381, 2024a.
Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything v2. arXiv: Comp. Res. Repository, 2024b.
Chongjie Ye, Lingteng Qiu, Xiaodong Gu, Zuo, Yushuang Wu, Zilong Dong, Liefeng Bo, Yuliang Xiu, and Xiaoguang Han. Stablenormal: Reducing diffusion variance for stable and sharp normal. arXiv: Comp. Res. Repository, 2024.
Wei Yin, Xinlong Wang, Chunhua Shen, Yifan Liu, Zhi Tian, Songcen Xu, Changming Sun, and Dou Renyin. Diversedepth: Affine-invariant depth prediction using diverse data. arXiv: Comp. Res. Repository, 2020.
Wei Yin, Jianming Zhang, Oliver Wang, Simon Niklaus, Long Mai, Simon Chen, and Chunhua Shen. Learning to recover 3d scene shape from single image. In Proc. IEEE Conf. Comp. Vis.
Patt. Recogn. 2021.
Qian Yu, Xiaoqi Zhao, Youwei Pang, Lihe Zhang, and Huchuan Lu. Multi-view aggregation network for dichotomous image segmentation. Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pp. 3921- 3930, 2024.
Amir R Zamir, Alexander Sax, William Shen, Leonidas J Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn,, pp. 3712-3722, 2018.
Zheng Zeng, Valentin Deschaintre, Iliyan Georgiev, Yannick Hold-Geoffroy, Yiwei Hu, Fujun Luan, Ling-Qi Yan, and Miloš Hašan. RGB ++ X: Image decomposition and synthesis using material- and awwre diffusion models. In ACM SIGGRAPH 2024 Conf. Papers, pp. 1-11, 2024.
Chi Zhang, Wei Yin, Billzb Wang, Gang Yu, Bin Fu, and Chunhua Shen. Hierarchical normalization for robust monocular depth estimation. Proc. Advances in Neural Inf. Process. Syst. 2022.
Yifan Zhao, Jia Li, and Yonghong Tian. Parsing objects at finer granularity: A survey. Machine Intelligence Research, 21:431 2024.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2017.
14


Reviews:
[{"review_id": "1ailxNLGYH", "reviewer": "Reviewer_nfdP", "summary": "This paper investigates key factors affecting the transfer performance of pretrained diffusion models repurposed for dense visual perception tasks, emphasizing the importance of fine-tuning data quality, training strategies, and task-specific supervision. It introduces GenPercept, a one-step fine-tuning paradigm that enhances inference speed and improves fine-grained details in predictions across various perception tasks.", "questions": "1. What training set was used for Table 1? Are you following the training data of DMP or Marigold? If so, why is the first row of Table 5 identical to the baseline? If not, why did you begin directly with a synthetic dataset?\n2. For monocular depth estimation, why is DMP not included as a baseline?\n3. The authors mention that using a customized head and loss could accelerate inference time. Could you provide a comparison to demonstrate this improvement?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The exploration of key factors influencing the transferability of text-to-image (T2I) diffusion models to dense visual perception tasks is intriguing and relevant.\n2. The experiments are thorough and comprehensive, providing strong support for the findings.\n3. The proposed deterministic one-step perception approach effectively integrates these findings and demonstrates comparable performance with minimal fine-tuning.", "weaknesses": "There are some confusing aspects in the experimental setup and comparisons for the downstream tasks. See questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cKWpnBbcnr", "reviewer": "Reviewer_6WHQ", "summary": "This paper reveals key elements of diffusion models for downstream dense perception tasks. The authors did a full range of validation from the perspectives of model design and training data, unveiled some IMPORTANT factors, and proposed a new model named GenPercept,\nThe authors have conducted extensive experiments on five dense perception tasks, including monocular depth estimation, surface normal estimation, image segmentation, and matting. This extensive experimentation serves as a testament to the effectiveness and universality of their method.", "questions": "What would be the comparison of generalization for OOD data for models trained using synthetic and real data?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The authors conduct extensive ablation studies from different aspects and finally figure out the key factors affecting transfer efficiency and performance when using pre-trained diffusion models.\n- Starting from the inconsistency of the results of two existing methods, the authors dig deeper into the influencing factors and then propose their GenPercept. This approach can provide some new ideas for doing research.\n- These findings are inspiring and can provide constructive insights into model design when adapting pre-trained models for downstream tasks.", "weaknesses": "- The biggest weakness is that all conclusions are based on experiments using large synthetic datasets trained on deep estimation tasks. I have two concerns. First, would these conclusions still hold if the datasets were small? Second, would these conclusions still hold if the dataset was completely real? \n- It is not appropriate to use the P3M 10K dataset to verify the robustness of GenPercept in image matting. This is because the human matting dataset is extremely similar to the dichotomous image segmentation task if only the boundary regions are considered and the deterministic foreground regions are ignored. Instead, if the authors wish to validate its robustness in the image matting task, more types of objects (e.g., semi-transparent objects) should be included and then its performance should be observed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dfRM9MlENz", "reviewer": "Reviewer_7LHd", "summary": "This paper researches the process of fine-tuning pre-trained text-to-image diffusion models for dense perception tasks like monocular depth estimation and semantic segmentation, and analyzes several design choices in that process. In this analysis, it considers the model architecture, training procedure, model initialization, dataset selection, and fine-tuning protocol. Based on this analysis, five findings are made:\n\n1.\tDiffusion models can be fine-tuned accurately and efficiently for dense perception tasks with a one-step, deterministic approach.\n2.\tThe most useful prior knowledge of the pre-trained diffusion model is contained in the U-Net denoiser. The VAE decoder can be replaced with other components without problems.\n3.\tWith the one-step model, multi-timestep training is irrelevant, and additional text inputs do not significantly impact performance.\n4.\tFine-tuning the U-Net denoiser leads to better results than freezing it or applying low-rank adaptation.\n5.\tTraining data quality affects the prediction quality.\n\nBased on these findings, the paper presents GenPercept, a paradigm for fine-tuning diffusion models for dense prediction tasks. Experiments show that GenPercept is an effective method to fine-tune Stable Diffusion for various dense prediction tasks. On several benchmarks, GenPercept achieves a competitive performance.", "questions": "The main reason that I currently give a slightly low rating is because of the incorrect numbers of existing models, the missing quantitative comparisons to some other existing models, the missing efficiency experiments, and some missing explanations (see the ‘weaknesses’ section). I would like to ask the authors to carefully address my concerns, answer the questions posed in the ‘weaknesses’ section, and revise the manuscript accordingly.\n\nAdditionally, I have some other questions/suggestions:\n\n1.\tFrom the text in L411-L423, it appears like the classes for semantic segmentation are always encoded into 3-channel colormaps, even when UPerNet is used. Is this really the case? If so, why isn’t the ‘regular’ semantic segmentation format used, with one channel for each individual class? If not, please clarify this in the text.\n2.\tIn the related work section, it seems appropriate to also mention DINOv2, because it has shown to be very suitable for downstream visual perception tasks like depth estimation (e.g., with Depth Anything [g]) and semantic segmentation.\n\n[f] Oquab et al., “DINOv2: Learning Robust Visual Features without Supervision,” TMLR 2024.\n\n[g] Yang et al., “Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data,” CVPR 2024.\n\n---\n\n**Update after author discussion.** After reading the different reviews, the authors' response, and the revised manuscript, and having a discussion with the authors, I have decided to upgrade my rating from 5 to 6. \n\nBy answering my questions and revising the manuscript based on my review and follow-up questions, the authors have convincingly addressed the majority of my concerns. Importantly, the paper now compares the proposed method more fairly to existing methods, provides insightful inference speed results, better explains the operation of the method, and discusses the cause of the impact of different $\\beta$ values in a more nuanced way. As such, the I believe the paper is stronger than the initial submission, and I upgrade my rating.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1.\tMost importantly, the analysis of the design choices that play a role when fine-tuning diffusion models for monocular depth estimation, described in Sec. 3, leads to various relevant and useful insights. \n\n    a)\tThe finding that fine-tuning can be done accurately with a one-step, deterministic approach (Tab. 1) is useful because it allows for significantly more efficient inference. \n\n    b)\tMoreover, by formulating the task as one-step estimation, the model can now te trained with task-specific losses, such as the image angular loss for surface normal estimation. With experiments (Tab. 7), this is shown to improve the results.\n\n    c)\tThe finding that most of the useful prior knowledge is contained in the U-Net of the pre-trained diffusion model, and not in the VAE decoder (Tab. 2), is useful because it means that the VAE decoder can be replaced with task-specific decoders. This is shown to be significantly boost the semantic segmentation performance in Tab. 9 (per L418-L423). \n\n    d)\tThe finding that the use of multiple timesteps during training is not necessary (Tab. 3) is useful because it allows the training procedure to be simplified to single-timestep training.\n\n    e)\tThe finding that the U-Net denoiser should be fine-tuned fully, without low-rank adaptation (Tab. 4), is useful because it provides clear guidelines for future methods that aim to fine-tune diffusion models for depth estimation and other dense perception tasks.\n\n2.\tThe presentation of the main findings of the paper in the text boxes (e.g., L283 and L296) is very helpful and useful. These text boxes clearly summarize the impact of the results that have been discussed thus far, and they help the reader to see the most interesting results of the paper at a glance.\n\n3.\tThe presented GenPercept model is valuable because it takes advantage of the aforementioned findings of the design-choice analysis. The application and evaluation of this model on multiple tasks and datasets provides the reader with insights into the performance that can be obtained when combining the best individual design choices. Moreover, these results can serve as a baseline for future works that aim to fine-tune diffusion models for multiple downstream perception tasks.", "weaknesses": "1.\tThe scores for DepthFM in Tab. 6 and GeoWizard in both Tab. 6 and Tab. 7 do not correspond to the scores originally reported in the respective papers. The numbers in these tables should be altered to reflect the originally reported numbers, or the text should explain why the numbers differ from these original numbers. Some examples:\n\n    a)\tGeoWizard: 9.7 AbsRel and 92.1 $\\delta_{1}$ on KITTI in original paper [a], but 12.9 AbsRel and 85.1 $\\delta_{1}$ in this submitted manuscript. \n\n    b)\tDepthFM: 8.3 AbsRel and 93.4 $\\delta_{1}$ on KITTI in original paper [b], but 17.4 AbsRel and 71.8 $\\delta_{1}$ in this submitted manuscript.\n\n2.\tThe results in Tab. 8 (and Tab. 2 of the supp) make it seem like GenPercept achieves state-of-the-art results in dichotomous image segmentation. However, this table does not contain the results of top-performing model MVANet [c]. This model achieves a max$F_{\\beta}$ score of 0.916 on Overall DIS-TE (1-4), compared to 0.863 by GenPercept. Even with results that are inferior to MVANet, GenPercept has value, but it should be clear to the reader that GenPercept does not achieve state-of-the-art results, so these results should be added to the tables.\n3.\tFor the image matting task on the P3M-500-NP dataset, GenPercept is only compared to the ResNet-34 variant of P3M [d], not the Swin-T or ViTAE-S variants which achieve much better results, e.g., 7.59 SAD for ViTAE-S compared to 11.23 SAD for ResNet-34. By only reporting the results for the ResNet-34 variant, it seems like GenPercept performs similarly to the state of the art, whereas this is not the case. The results of P3M for ViTAE-S and/or Swin-T should be added to the paper, or the text should clearly explain why such a comparison is not necessary.\n4.\tOne of the main benefits of the one-step inference procedure of GenPercept is the improved efficiency compared to Marigold and other diffusion-based methods. The efficiency is also mentioned as a key characteristic of the method (L539). However, the paper does not explicitly evaluate the efficiency of the model. Therefore, it is not clear what the exact speedup is over existing diffusion-based models, or how the efficiency compares to the other methods reported in Tab. 6, 7, 8, and 10. The paper would be significantly stronger if the efficiency of GenPercept was shown quantitatively, by reporting the inference time of GenPercept and other methods.\n5.\tL187-L188 states that the results indicate that increasing the ‘training challenges’ leads to improved model performance. However, the paper does not provide any explanation for this. Why does the performance increase when the training task becomes more challenging? This seems counterintuitive. The value of the paper would improve if the authors provided an explanation (or hypothesis) for this phenomenon, as this could provide insights into the way these diffusion models learn dense perception tasks.\n6.\tThe paper does not clearly define $\\beta_{start}$ and $\\beta_{end}$, even though first experiment and finding (Tab. 1 & L204) are fully focused around changing the values of these parameters. Fig. 2(b) illustrates that they impact the proportion of noise that is added to the latents in different time steps, but an exact definition is not provided. In L107, the paper refers to the supplementary material for more details, but the supplementary material only mentions $\\beta_{s}$ and $\\beta_{t}$. The clarity of the paper would improve if a clear definition of $\\beta_{start}$ and $\\beta_{end}$ was provided.\n7.\tThe paper does not evaluate if the multi-class image segmentation performance of GenPercept is competitive with existing image segmentation methods. Currently, the paper only conducts an experiment where GenPercept is trained on HyperSim images with 40 semantic classes, and evaluated on ADE20K images with 40 classes. As this is a newly proposed setting, these results are not comparable with existing segmentation methods. To see if fine-tuning diffusion models is truly valuable for image segmentation, GenPercept should be fine-tuned and evaluated on standard segmentation benchmarks like ADE20K, and compared to existing segmentation models like Mask2Former [e]. \n8.\tA minor weakness of the paper is that there is not a version of GenPercept that clearly works better than the other. As discussed in L357-L359 and shown in Tab. 6, the GenPercept model trained for depth estimation scores better on NYU, ScanNet and ETH3D, while the model trained for disparity estimation scores significantly better on KITTI and DIODE. As a result, two different models are necessary for different situations, and it is not always clear which of the models works best for an unknown application domain. \n\nSome minor weaknesses, which do not significantly impact my rating:\n\n9.\tL230: The acronym “DPT” is not defined, nor is there a reference to a work where it is presented. As a result, it’s not clear what a “DPT encoder” is.\n10.\tThe meaning of the “fine-tune” column in Tab. 9 is not clear. From the text in L418-L423, it appears that it refers to the use of the UPerNet decoder. If so, the text “UPerNet” seems more appropriate than “fine-tune”. If it means something else than this, this should be better specified in the paper.\n11.\tThe ordering of the labels in the legend of Fig. 2 (b) is confusing, as there appears to be no logical ordering. The legend, and the figure as whole, would be easier to interpret if the labels were ordered in descending or ascending manner, e.g. starting with (0.0002125, 0.003) and ending with (1.0, 1.0), or vice versa.\n\n12.\tThere are a few errors in the text:\n\n    a) L024: \"tailed for\" - Do the authors mean \"tailored for\"?\n\n    b)\tL187-L188: \"increasing training challenges lead to\" => \"increasing training challenges leads to\"\n\n[a] Fu et al., “GeoWizard: Unleashing the Diffusion Priors for 3D Geometry Estimation from a Single Image,” ECCV 2024.\n\n[b] Gui et al., “DepthFM: Fast Monocular Depth Estimation with Flow Matching,” arXiv:2403.13788, 2024.\n\n[c] Yu et al., \"Multi-view Aggregation Network for Dichotomous Image Segmentation,\" CVPR 2024.\n\n[d] Ma et al., \"Rethinking Portrait Matting with Privacy Preserving,\" IJCV 2023.\n\n[e] Cheng et al., “Masked-attention Mask Transformer for Universal Image Segmentation,” CVPR 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VYYdyk3jiK", "reviewer": "Reviewer_QgJw", "summary": "This paper presents a novel approach to harnessing pretrained diffusion models for general dense perception tasks, including monocular depth estimation, surface normal estimation, dichotomous image segmentation, semantic segmentation, and human pose estimation. The central idea of this work is to utilize the diffusion model as a robust pretrained backbone, subsequently fine-tuning it for a variety of downstream applications.", "questions": "1. Trained with synthetic data, why does GenPercept not suffer from the sim2real domain gap?\n2. In the supplementary materials, line 22 should utilize the citation format 'citep'.\n3. Can GenPercept be applied to general perception tasks, such as detection?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1. This paper provides systematical analysis of the design space of diffusion model for dense perception task, as shown in the 5 findings in paper. The core design of this paper is to employ a deterministic one step perception.\n2. By using the proposed training protocol, the proposed GenPercept harnesses the power of the pre-trained UNet from diffusion models for dense visual perception tasks, including monocular depth estimation, surface normal estimation, dichotomous image segmentation and semantic segmentation. \n3. This paper presents a solid analysis accompanied by extensive experiments. The results are not only intuitive but also promising, demonstrating a strong potential for downstream application.", "weaknesses": "1. Given that GenPercept is currently trained on a relatively small dataset, it tends to lag behind models that benefit from extensive data training. To enhance its performance, it would be beneficial to scale GenPercept's training with a larger volume of data in the future.\n2. Given the robust prior established by training on the extensive LAION dataset, the question arises: what would be the outcome of employing alternative self-supervised methods, such as MAE or CLIP, using the same LAION dataset? A comparative analysis of these approaches against the diffusion pretrain would provide valuable insights into their relative efficacy and potential advantages.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)