What Matters When Repurposing Diffusion Models for General Dense Perception Tasks?

Authors: Guangkai Xu, Yongtao Ge, Mingyu Liu, Chengxiang Fan, Kangyang Xie, Zhiyue Zhao, Hao Chen, Chunhua Shen
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Transfer Learning, Diffusion Models, Visual Perception

Abstract:
Extensive pre-training with large data is indispensable for downstream geometry and semantic visual perception tasks. Thanks to large-scale text-to-image (T2I) pretraining, recent works show promising results by simply fine-tuning T2I diffusion models for a few dense perception tasks. However, several crucial design decisions in this process still lack comprehensive justification, encompassing the necessity of the multi-step diffusion mechanism, training strategy, inference ensemble strategy, and fine-tuning data quality. In this work, we conduct a thorough investigation into critical factors that affect transfer efficiency and performance when using diffusion priors. Our key findings are: 1) High-quality fine-tuning data is paramount for both semantic and geometry perception tasks. 2) As a special case of the diffusion scheduler by setting its hyper-parameters, the multi-step generation can be simplified to a one-step fine-tuning paradigm without any loss of performance, while significantly speeding up inference. 3) Apart from fine-tuning the diffusion model with only latent space supervision, task-specific supervision can be beneficial to enhance fine-grained details. These observations culminate in the development of GenPercept, an effective deterministic one-step fine-tuning paradigm tailored for dense visual perception tasks exploiting diffusion priors. Different from the previous multi-step methods, our paradigm offers a much faster inference speed, and can be seamlessly integrated with customized perception decoders and loss functions for task-specific supervision, which can be critical for improving the fine-grained details of predictions. Comprehensive experiments on a diverse set of dense visual perceptual tasks, including monocular depth estimation, surface normal estimation, image segmentation, and matting, are performed to demonstrate the remarkable adaptability and effectiveness of our proposed method. Code: https://github.com/aim-uofa/GenPercept

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1729829164528

thread_type:
official_review

<thread_object>
creator: reviewer_qgjw

<content>
summary:
This paper presents a novel approach to harnessing pretrained diffusion models for general dense perception tasks, including monocular depth estimation, surface normal estimation, dichotomous image segmentation, semantic segmentation, and human pose estimation. The central idea of this work is to utilize the diffusion model as a robust pretrained backbone, subsequently fine-tuning it for a variety of downstream applications.

strengths:
1. This paper provides systematical analysis of the design space of diffusion model for dense perception task, as shown in the 5 findings in paper. The core design of this paper is to employ a deterministic one step perception.
2. By using the proposed training protocol, the proposed GenPercept harnesses the power of the pre-trained UNet from diffusion models for dense visual perception tasks, including monocular depth estimation, surface normal estimation, dichotomous image segmentation and semantic segmentation. 
3. This paper presents a solid analysis accompanied by extensive experiments. The results are not only intuitive but also promising, demonstrating a strong potential for downstream application.

weaknesses:
1. Given that GenPercept is currently trained on a relatively small dataset, it tends to lag behind models that benefit from extensive data training. To enhance its performance, it would be beneficial to scale GenPercept's training with a larger volume of data in the future.
2. Given the robust prior established by training on the extensive LAION dataset, the question arises: what would be the outcome of employing alternative self-supervised methods, such as MAE or CLIP, using the same LAION dataset? A comparative analysis of these approaches against the diffusion pretrain would provide valuable insights into their relative efficacy and potential advantages.

questions:
1. Trained with synthetic data, why does GenPercept not suffer from the sim2real domain gap?
2. In the supplementary materials, line 22 should utilize the citation format 'citep'.
3. Can GenPercept be applied to general perception tasks, such as detection?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730474519368

thread_type:
official_review

<thread_object>
creator: reviewer_7lhd

<content>
summary:
This paper researches the process of fine-tuning pre-trained text-to-image diffusion models for dense perception tasks like monocular depth estimation and semantic segmentation, and analyzes several design choices in that process. In this analysis, it considers the model architecture, training procedure, model initialization, dataset selection, and fine-tuning protocol. Based on this analysis, five findings are made:

1.	Diffusion models can be fine-tuned accurately and efficiently for dense perception tasks with a one-step, deterministic approach.
2.	The most useful prior knowledge of the pre-trained diffusion model is contained in the U-Net denoiser. The VAE decoder can be replaced with other components without problems.
3.	With the one-step model, multi-timestep training is irrelevant, and additional text inputs do not significantly impact performance.
4.	Fine-tuning the U-Net denoiser leads to better results than freezing it or applying low-rank adaptation.
5.	Training data quality affects the prediction quality.

Based on these findings, the paper presents GenPercept, a paradigm for fine-tuning diffusion models for dense prediction tasks. Experiments show that GenPercept is an effective method to fine-tune Stable Diffusion for various dense prediction tasks. On several benchmarks, GenPercept achieves a competitive performance.

strengths:
1.	Most importantly, the analysis of the design choices that play a role when fine-tuning diffusion models for monocular depth estimation, described in Sec. 3, leads to various relevant and useful insights. 

    a)	The finding that fine-tuning can be done accurately with a one-step, deterministic approach (Tab. 1) is useful because it allows for significantly more efficient inference. 

    b)	Moreover, by formulating the task as one-step estimation, the model can now te trained with task-specific losses, such as the image angular loss for surface normal estimation. With experiments (Tab. 7), this is shown to improve the results.

    c)	The finding that most of the useful prior knowledge is contained in the U-Net of the pre-trained diffusion model, and not in the VAE decoder (Tab. 2), is useful because it means that the VAE decoder can be replaced with task-specific decoders. This is shown to be significantly boost the semantic segmentation performance in Tab. 9 (per L418-L423). 

    d)	The finding that the use of multiple timesteps during training is not necessary (Tab. 3) is useful because it allows the training procedure to be simplified to single-timestep training.

    e)	The finding that the U-Net denoiser should be fine-tuned fully, without low-rank adaptation (Tab. 4), is useful because it provides clear guidelines for future methods that aim to fine-tune diffusion models for depth estimation and other dense perception tasks.

2.	The presentation of the main findings of the paper in the text boxes (e.g., L283 and L296) is very helpful and useful. These text boxes clearly summarize the impact of the results that have been discussed thus far, and they help the reader to see the most interesting results of the paper at a glance.

3.	The presented GenPercept model is valuable because it takes advantage of the aforementioned findings of the design-choice analysis. The application and evaluation of this model on multiple tasks and datasets provides the reader with insights into the performance that can be obtained when combining the best individual design choices. Moreover, these results can serve as a baseline for future works that aim to fine-tune diffusion models for multiple downstream perception tasks.

weaknesses:
1.	The scores for DepthFM in Tab. 6 and GeoWizard in both Tab. 6 and Tab. 7 do not correspond to the scores originally reported in the respective papers. The numbers in these tables should be altered to reflect the originally reported numbers, or the text should explain why the numbers differ from these original numbers. Some examples:

    a)	GeoWizard: 9.7 AbsRel and 92.1 $\delta_{1}$ on KITTI in original paper [a], but 12.9 AbsRel and 85.1 $\delta_{1}$ in this submitted manuscript. 

    b)	DepthFM: 8.3 AbsRel and 93.4 $\delta_{1}$ on KITTI in original paper [b], but 17.4 AbsRel and 71.8 $\delta_{1}$ in this submitted manuscript.

2.	The results in Tab. 8 (and Tab. 2 of the supp) make it seem like GenPercept achieves state-of-the-art results in dichotomous image segmentation. However, this table does not contain the results of top-performing model MVANet [c]. This model achieves a max$F_{\beta}$ score of 0.916 on Overall DIS-TE (1-4), compared to 0.863 by GenPercept. Even with results that are inferior to MVANet, GenPercept has value, but it should be clear to the reader that GenPercept does not achieve state-of-the-art results, so these results should be added to the tables.
3.	For the image matting task on the P3M-500-NP dataset, GenPercept is only compared to the ResNet-34 variant of P3M [d], not the Swin-T or ViTAE-S variants which achieve much better results, e.g., 7.59 SAD for ViTAE-S compared to 11.23 SAD for ResNet-34. By only reporting the results for the ResNet-34 variant, it seems like GenPercept performs similarly to the state of the art, whereas this is not the case. The results of P3M for ViTAE-S and/or Swin-T should be added to the paper, or the text should clearly explain why such a comparison is not necessary.
4.	One of the main benefits of the one-step inference procedure of GenPercept is the improved efficiency compared to Marigold and other diffusion-based methods. The efficiency is also mentioned as a key characteristic of the method (L539). However, the paper does not explicitly evaluate the efficiency of the model. Therefore, it is not clear what the exact speedup is over existing diffusion-based models, or how the efficiency compares to the other methods reported in Tab. 6, 7, 8, and 10. The paper would be significantly stronger if the efficiency of GenPercept was shown quantitatively, by reporting the inference time of GenPercept and other methods.
5.	L187-L188 states that the results indicate that increasing the ‘training challenges’ leads to improved model performance. However, the paper does not provide any explanation for this. Why does the performance increase when the training task becomes more challenging? This seems counterintuitive. The value of the paper would improve if the authors provided an explanation (or hypothesis) for this phenomenon, as this could provide insights into the way these diffusion models learn dense perception tasks.
6.	The paper does not clearly define $\beta_{start}$ and $\beta_{end}$, even though first experiment and finding (Tab. 1 & L204) are fully focused around changing the values of these parameters. Fig. 2(b) illustrates that they impact the proportion of noise that is added to the latents in different time steps, but an exact definition is not provided. In L107, the paper refers to the supplementary material for more details, but the supplementary material only mentions $\beta_{s}$ and $\beta_{t}$. The clarity of the paper would improve if a clear definition of $\beta_{start}$ and $\beta_{end}$ was provided.
7.	The paper does not evaluate if the multi-class image segmentation performance of GenPercept is competitive with existing image segmentation methods. Currently, the paper only conducts an experiment where GenPercept is trained on HyperSim images with 40 semantic classes, and evaluated on ADE20K images with 40 classes. As this is a newly proposed setting, these results are not comparable with existing segmentation methods. To see if fine-tuning diffusion models is truly valuable for image segmentation, GenPercept should be fine-tuned and evaluated on standard segmentation benchmarks like ADE20K, and compared to existing segmentation models like Mask2Former [e]. 
8.	A minor weakness of the paper is that there is not a version of GenPercept that clearly works better than the other. As discussed in L357-L359 and shown in Tab. 6, the GenPercept model trained for depth estimation scores better on NYU, ScanNet and ETH3D, while the model trained for disparity estimation scores significantly better on KITTI and DIODE. As a result, two different models are necessary for different situations, and it is not always clear which of the models works best for an unknown application domain. 

Some minor weaknesses, which do not significantly impact my rating:

9.	L230: The acronym “DPT” is not defined, nor is there a reference to a work where it is presented. As a result, it’s not clear what a “DPT encoder” is.
10.	The meaning of the “fine-tune” column in Tab. 9 is not clear. From the text in L418-L423, it appears that it refers to the use of the UPerNet decoder. If so, the text “UPerNet” seems more appropriate than “fine-tune”. If it means something else than this, this should be better specified in the paper.
11.	The ordering of the labels in the legend of Fig. 2 (b) is confusing, as there appears to be no logical ordering. The legend, and the figure as whole, would be easier to interpret if the labels were ordered in descending or ascending manner, e.g. starting with (0.0002125, 0.003) and ending with (1.0, 1.0), or vice versa.

12.	There are a few errors in the text:

    a) L024: "tailed for" - Do the authors mean "tailored for"?

    b)	L187-L188: "increasing training challenges lead to" => "increasing training challenges leads to"

[a] Fu et al., “GeoWizard: Unleashing the Diffusion Priors for 3D Geometry Estimation from a Single Image,” ECCV 2024.

[b] Gui et al., “DepthFM: Fast Monocular Depth Estimation with Flow Matching,” arXiv:2403.13788, 2024.

[c] Yu et al., "Multi-view Aggregation Network for Dichotomous Image Segmentation," CVPR 2024.

[d] Ma et al., "Rethinking Portrait Matting with Privacy Preserving," IJCV 2023.

[e] Cheng et al., “Masked-attention Mask Transformer for Universal Image Segmentation,” CVPR 2022.

questions:
The main reason that I currently give a slightly low rating is because of the incorrect numbers of existing models, the missing quantitative comparisons to some other existing models, the missing efficiency experiments, and some missing explanations (see the ‘weaknesses’ section). I would like to ask the authors to carefully address my concerns, answer the questions posed in the ‘weaknesses’ section, and revise the manuscript accordingly.

Additionally, I have some other questions/suggestions:

1.	From the text in L411-L423, it appears like the classes for semantic segmentation are always encoded into 3-channel colormaps, even when UPerNet is used. Is this really the case? If so, why isn’t the ‘regular’ semantic segmentation format used, with one channel for each individual class? If not, please clarify this in the text.
2.	In the related work section, it seems appropriate to also mention DINOv2, because it has shown to be very suitable for downstream visual perception tasks like depth estimation (e.g., with Depth Anything [g]) and semantic segmentation.

[f] Oquab et al., “DINOv2: Learning Robust Visual Features without Supervision,” TMLR 2024.

[g] Yang et al., “Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data,” CVPR 2024.

---

**Update after author discussion.** After reading the different reviews, the authors' response, and the revised manuscript, and having a discussion with the authors, I have decided to upgrade my rating from 5 to 6. 

By answering my questions and revising the manuscript based on my review and follow-up questions, the authors have convincingly addressed the majority of my concerns. Importantly, the paper now compares the proposed method more fairly to existing methods, provides insightful inference speed results, better explains the operation of the method, and discusses the cause of the impact of different $\beta$ values in a more nuanced way. As such, the I believe the paper is stronger than the initial submission, and I upgrade my rating.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730628108760

thread_type:
official_review

<thread_object>
creator: reviewer_6whq

<content>
summary:
This paper reveals key elements of diffusion models for downstream dense perception tasks. The authors did a full range of validation from the perspectives of model design and training data, unveiled some IMPORTANT factors, and proposed a new model named GenPercept,
The authors have conducted extensive experiments on five dense perception tasks, including monocular depth estimation, surface normal estimation, image segmentation, and matting. This extensive experimentation serves as a testament to the effectiveness and universality of their method.

strengths:
- The authors conduct extensive ablation studies from different aspects and finally figure out the key factors affecting transfer efficiency and performance when using pre-trained diffusion models.
- Starting from the inconsistency of the results of two existing methods, the authors dig deeper into the influencing factors and then propose their GenPercept. This approach can provide some new ideas for doing research.
- These findings are inspiring and can provide constructive insights into model design when adapting pre-trained models for downstream tasks.

weaknesses:
- The biggest weakness is that all conclusions are based on experiments using large synthetic datasets trained on deep estimation tasks. I have two concerns. First, would these conclusions still hold if the datasets were small? Second, would these conclusions still hold if the dataset was completely real? 
- It is not appropriate to use the P3M 10K dataset to verify the robustness of GenPercept in image matting. This is because the human matting dataset is extremely similar to the dichotomous image segmentation task if only the boundary regions are considered and the deterministic foreground regions are ignored. Instead, if the authors wish to validate its robustness in the image matting task, more types of objects (e.g., semi-transparent objects) should be included and then its performance should be observed.

questions:
What would be the comparison of generalization for OOD data for models trained using synthetic and real data?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730702430505

thread_type:
official_review

<thread_object>
creator: reviewer_nfdp

<content>
summary:
This paper investigates key factors affecting the transfer performance of pretrained diffusion models repurposed for dense visual perception tasks, emphasizing the importance of fine-tuning data quality, training strategies, and task-specific supervision. It introduces GenPercept, a one-step fine-tuning paradigm that enhances inference speed and improves fine-grained details in predictions across various perception tasks.

strengths:
1. The exploration of key factors influencing the transferability of text-to-image (T2I) diffusion models to dense visual perception tasks is intriguing and relevant.
2. The experiments are thorough and comprehensive, providing strong support for the findings.
3. The proposed deterministic one-step perception approach effectively integrates these findings and demonstrates comparable performance with minimal fine-tuning.

weaknesses:
There are some confusing aspects in the experimental setup and comparisons for the downstream tasks. See questions.

questions:
1. What training set was used for Table 1? Are you following the training data of DMP or Marigold? If so, why is the first row of Table 5 identical to the baseline? If not, why did you begin directly with a synthetic dataset?
2. For monocular depth estimation, why is DMP not included as a baseline?
3. The authors mention that using a customized head and loss could accelerate inference time. Could you provide a comparison to demonstrate this improvement?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732106429468

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response and Summary of Updates to Manuscript

comment:
We would like to express our sincere gratitude to the reviewers and editors for their time, effort, and constructive feedback in evaluating our work. Your thoughtful comments and suggestions have been invaluable in enhancing the quality and rigor of our manuscript. We have carefully read all feedback and have provided detailed responses to each of the reviewers' comments. We trust that these responses adequately address the concerns and suggestions raised.

We are greatly encouraged by the reviewers’ recognition of our efforts in investigating key factors influencing transfer performance (**nfdP**, **6WHQ**, **7LHd**, **QgJw**), the effectiveness and efficiency of our novel one-step GenPercept approach (**nfdP**, **6WHQ**, **7LHd**, **QgJw**), and the robustness of our experiments (**nfdP**, **QgJw**). Below, we provide a high-level summary of the revisions made to the manuscript in response to the reviewers' feedback, followed by a restatement of the key contributions of our work.

---

Here is the summary of updates that we've made to the draft, and the relevant revisions have been highlighted in red in the latest version of the manuscript.
- Clarified some experimental settings and analysis. (**nfdP**, **7LHd**)
- Compared with more related approaches in Table 6, Table 8, Table 9, and Table 10 of the main paper, and Table 4 of the supplementary. (**nfdP**, **7LHd**)
- Added the quantitative efficiency improvement experiment in Table 1 of the supplementary. (**nfdP**, **7LHd**)
- Explored the effectiveness of data volume in Table 1 of the supplementary. (**6WHQ**)
- Added more qualitative analysis experiments in Figure 5 and Figure 6 of the main paper and Figure 1, Figure 2, Figure 3, Figure 7, Figure 10, and Figure 11 of the supplementary material, which shows the generalization ability of GenPercept. (**7LHd**)
- Added more types of objects in image matting in Figure 11 of the supplementary. (**6WHQ**)

To end this update, we would like to summarize the primary contributions of our work. GenPercept presents an efficient and effective approach to leveraging the prior knowledge embedded in diffusion models trained on large-scale data for dense perception tasks. Unlike approaches only focused on achieving state-of-the-art performance on a specific dataset, _GenPercept emphasizes robustness, detail, and generalizability_ in dense visual perception tasks while _maintaining a unified network architecture_. This work opens new possibilities for realizing more generalizable dense perception frameworks with the benefits of diffusion model priors.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces GenPercept, a novel approach leveraging diffusion models to enhance dense visual perception tasks. The work claims improvements in inference speed and detail of predictions, substantiated through a series of experiments across multiple tasks.

The paper stands out for its rigorous experimental validation and the clear demonstration of the versatility and efficiency of diffusion models in dense perception contexts. The design space analysis and the results presented are commendable and represent the paper's key strengths.

Concerns are raised about the heavy reliance on extensive synthetic datasets, leading to questions about the model's performance with smaller or real-world data. The initial comparison with existing methods was not entirely convincing, and the paper lacked a quantitative evaluation of model efficiency. Additionally, some theoretical claims, such as "ground truth leakage," were not sufficiently clear or substantiated.

The decision leans toward acceptance due to the paper's robust experimental work and the novelty of applying diffusion models in this manner. Despite this, the unresolved issues around the real-world applicability and some theoretical ambiguities slightly detract from the paper's impact.