Aligning Generative Denoising with Discriminative Objectives Unleashes Diffusion for Visual Perception

Authors: Ziqi Pang, Xin Xu, Yu-Xiong Wang
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Diffusion for Perception, Diffusion Models, Visual Perception

Abstract:
With success in image generation, generative diffusion models are increasingly adopted for discriminative scenarios because generating pixels is a unified and natural perception interface. Although directly re-purposing their generative denoising process has established promising progress in specialist (e.g., depth estimation) and generalist models, the inherent gaps between a generative process and discriminative objectives are rarely investigated. For instance, generative models can tolerate deviations at intermediate sampling steps as long as the final distribution is reasonable, while discriminative tasks with rigorous ground truth for evaluation are sensitive to such errors. Without mitigating such gaps, diffusion for perception still struggles on tasks represented by multi-modal understanding (e.g., referring image segmentation). Motivated by these challenges, we analyze and improve the alignment between the generative diffusion process and perception objectives centering around the key observation: how perception quality evolves with the denoising process. (1) Notably, earlier denoising steps contribute more than later steps, necessitating a tailored learning objective for training: loss functions should reflect varied contributions of timesteps for each perception task. (2) Perception quality drops unexpectedly at later denoising steps, revealing the sensitiveness of perception to training-denoising distribution shift. We introduce diffusion-tailored data augmentation to simulate such drift in the training data. (3) We suggest a novel perspective to the long-standing question: why should a generative process be useful for discriminative tasks - interactivity. The denoising process can be leveraged as a controllable user interface adapting to users' correctional prompts and conducting multi-round interaction in an agentic workflow. Collectively, our insights enhance multiple generative diffusion-based perception models without architectural changes: state-of-the-art diffusion-based depth estimator, previously underplayed referring image segmentation models, and perception generalists. Our code is available at https://github.com/ziqipang/ADDP.

Decision: accept (poster)
Reviewer scores: 6, 8, 6, 6

Reviews and discussion:
<thread>
creation date:
1730016444812

thread_type:
official_review

<thread_object>
creator: reviewer_ziyy

<content>
summary:
This paper aims to improve the performance of repurposing the diffusion models as the deterministic task models. First, the authors analyze the performance changes through the denoising process of diffusion models in an RIS task, and they find that later timesteps are more influential to the mIOU but there is a decreasing tendency in mIOU when approaching the smaller timesteps. They hypothesize the reason for this finding as 1) the unequalized contribution of denoising steps and 2) the training-denoising distribution shift. To deal with the first challenge, they propose loss scaling and timestep sampling methods, which reflect varied contributions of each timestep for deterministic tasks. For the second challenge, data augmentation strategies suitable for each task are utilized to reduce exposure biases. Finally, they argue that the generative process in diffusion models is beneficial in reflecting users' feedback on perception tasks and propose agentic workflows that conduct multi-round interactions. The various experiments are conducted to show the effectiveness of their method mainly focusing on improving Marigold, InstructPix2Pix, and InstructCV. Technical details and additional experimental results are mainly presented in the Appendix.

strengths:
- This paper deals with deep explorations of the denoising procedure of diffusion models in visual perception tasks.
- The main strength of this paper is the performance improvement. As shown in the results, the proposed methods offer substantial performance enhancement for various models.

weaknesses:
- **W1-1) (Minor)** This paper introduces numerous techniques, but some details are obscured or inaccurately described. For example, in Eq. (2), $c_t^2$ is presented as a weight for each loss term based on the timestep, yet it is also used in the timestep sampling distribution in Fig. 2. 
- **W1-2) (Major)** Furthermore, I cannot find the details of the analysis in Fig. 1. It is unclear how the authors calculated mIOU scores during the sampling process. If these scores are derived using only the estimated $x_0$​ values partway through the sampling, they don’t reflect the final generated results. Using these incomplete samples to support the authors’ conclusions seems insufficient. In other words, simply examining the denoising behavior of partial generations does not fully represent the actual generation process. A more accurate approach would be to inspect denoising characteristics across the complete generation process, as this would better validate the claims.
- **W2) (Major)** The technical contribution of this paper in the proposal of 1) a loss scaling and timestep sampling methods and 2) the dataset augmentation for each task can be trivial without comparing baselines. 
  - **W2-1)** The authors introduce loss scaling and timestep sampling methods based on their observations of uneven contributions to performance. In the context of loss-weighting and timestep-sampling strategies within diffusion models [1, 2, 3, 4, 5], several studies have shown that emphasizing the loss at later timesteps can enhance model performance. The core operation of the authors’ method appears similar to these existing techniques. Therefore, without direct comparisons to these established methods, it is difficult to be convinced of any distinct advantage offered by the proposed approach in deterministic task setups. I suggest adding the comparative results with these methods.
  - **W2-2)** Similar to W2-1), [6] also deals with timestep-dependent data augmentation and it is necessary to compare this to validate the advantages of the proposed methods. Also, incorporating the data augmentation strategy for reducing exposure biases in diffusion models is already mentioned in [7]. In this regard, I suggest adding the comparative results with data augmentation works in diffusion model contexts.

- **W3) (Minor)** The proposed methods require a manual setup for data augmentation and $c_t$.

### **Reference**

[1] Perception Prioritized Training of Diffusion Models, CVPR 2022

[2] Efficient Diffusion Training via Min-SNR weighting Strategy, ICCV 2023.

[3] Addressing Negative Transfer in Diffusion Models, Neurips 2023

[4] A Closer Look at Timesteps is Worthy of Triple Speed-Up for Diffusion Model Training, Arxiv 2024.

[5] Soft Truncation: A Universal Training Technique of Score-based Diffusion Model for High Precision Score Estimation, ICML 2022

[6] TADA: Timestep-Aware Data Augmentation For Diffusion Models, Neurips 2023 workshop on Diffusion Models.

[7] ELUCIDATING THE EXPOSURE BIAS IN DIFFUSION MODELS, ICLR 2024.

questions:
Q1: Why is linear regression needed in estimating $c_t$​ instead of determining $c_t$ directly?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730384282995

thread_type:
official_review

<thread_object>
creator: reviewer_2cz7

<content>
summary:
This work proposes a method to bridge the gap between generative diffusion models and discriminative perception tasks by aligning the generative denoising process with perception objectives. One of its contributions is designing a loss function to reflect the varied contributions of timesteps. At the same time, it introduces diffusion-tailored data augmentation to simulate training-denoising distribution shifts and leverages the denoising process as an interactive interface for user correctional prompts. These enhancements improve performance without requiring architectural changes.

strengths:
1. The paper presents a new way to make generative diffusion models work better for visual tasks by aligning them with discriminative tasks and gets SOTA performance.
2. It explains the differences between generative and discriminative processes and offers specific solutions like adjusting timestep sampling and using data augmentation designed for diffusion.
3. The method takes advantage of the interactive denoising process, allowing for correctional prompts and multiple rounds of interaction, which sets it apart from traditional models.

weaknesses:
1. The data augmentation techniques used in this paper are quite standard for perception tasks and don't offer much innovation.
2. Although the paper demonstrates improvements in certain tasks, the applicability of these methods to a wider range of perception tasks and datasets still needs thorough validation.

questions:
The proposed methods demonstrate strong performance on zero-shot benchmarks, but how robust and generalizable are these models in real-world applications? Are there additional experiments or analyses that validate the model’s stability across different environments and conditions?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730386342002

thread_type:
official_review

<thread_object>
creator: reviewer_zlxb

<content>
summary:
This paper investigates the performance gap between diffusion models and traditional encoder-decoder methods in perceptual tasks (e.g., Referring Segmentation) through validation experiments. It analyzes the varying contribution importance of different timesteps in perception through timestep-contribution analysis, leading to improvements in both training strategy and data aspects. Additionally, the paper demonstrates the potential of diffusion models for interactive referring perception tasks using prompt-based guidance agents.

strengths:
1. The methodology and motivation are presented with exceptional clarity, with comprehensive consideration of variants for each method, including:
   - Various augmentation approaches
   - Different definitions of $c_{t}^2$ in timestep-aware training

2. The experimental section is thoroughly comprehensive, including:
   - Validation experiments in the introduction
   - Performance evaluations across multiple tasks (RIS, Depth Estimate, Generalist Perception)
   - Detailed ablation studies

3. The innovations stem from meaningful problem discovery and are non-trivial, with adaptive training and data augmentation for different timesteps representing logical explorations of the original problem.

weaknesses:
1. The paper appears overly content-rich for a conference paper, resulting in some crucial discussions being relegated to supplementary materials:
   - The revised objective for the augmented ground truth in Sec. B.3.2
   - Intensity of augmentations across timesteps in Sec. B.3.1
   These aspects are fundamental to understanding the methodology but lack explanation in the main text.

2. The "User interface" section seems disconnected from the main methodological contributions and shows limited performance improvements (as shown in Table 3). The whole content of this paper might be better suited as two separate conference papers.

3. Concerns about the baseline comparison: It's unclear whether the InstructPix2Pix baseline in RIS underwent equivalent training. The paper doesn't specify if the baseline was trained on specific RIS datasets, raising questions about the fairness of comparing a general image editing model with a task-specific fine-tuned model.

questions:
Please refer to the weaknesses section, particularly regarding:
1. The justification for baseline comparison methodology
2. The rationale behind including the user interface section in this paper
</content>
</thread_object>

</thread>
<thread>
creation date:
1730729511688

thread_type:
official_review

<thread_object>
creator: reviewer_apx4

<content>
summary:
The paper "Aligning Generative Denoising with Discriminative Objectives Unleashes Diffusion for Visual Perception" introduces a novel framework, ADDP, designed to enhance the performance of generative diffusion models on discriminative visual perception tasks. These tasks, such as depth estimation, referring image segmentation, and generalist perception, demand precise alignment with ground truth data, which has been a challenge for generative models.

This technique prioritizes earlier denoising steps, which have a more significant impact on final perception quality. By adjusting the learning objectives, ADDP ensures that the model focuses on the most critical stages of the denoising process. To mitigate the "training-denoising distribution shift" issue, ADDP employs task-specific data augmentation strategies, simulating the deviations that occur during the denoising process to improve the model's robustness and maintaining perception quality across different timesteps.
ADDP leverages the iterative nature of diffusion models to enable user interaction. By incorporating classifier-free guidance, users can provide corrective feedback to refine the model's output, offering a more flexible and interactive approach compared to traditional discriminative models.

ADDP has demonstrated significant improvements in depth estimation, referring image segmentation, and generalist perception tasks. By effectively aligning generative and discriminative objectives, ADDP bridges the gap between generative diffusion models and discriminative baselines, without requiring architectural changes or additional data. This work represents a significant step forward in the field of generative diffusion models, highlighting their potential as powerful tools for visual perception tasks.

strengths:
The paper introduces a novel framework, ADDP, addressing a previously unexplored gap in applying generative diffusion models to perception tasks. The paper's key contributions include a Contribution-Aware Timestep Sampling to prioritize early denoising steps for improved perception accuracy, as well as a Diffusion-Tailored Data Augmentation technique that simulates distribution shifts during the denoising process. The authors provide a Interactive User Interface to enable human-in-the-loop correction through the denoising process.

The paper demonstrates high quality through evaluation on various tasks, including depth estimation, referring image segmentation, and generalist perception and provides significant improvements over standard diffusion-based models without architectural changes.The authors provide ablation studies to justify the design choices and the individual contributions of each technique.
The paper is well-written and easy to understand, with clear structure, clear explanations of technical concepts, aided by diagrams and clear communication of key ideas.

The paper has significant implications in expanded model versatility, using for both generative and perception tasks and in possibilities for human-in-the-loop correction and user-guided generation. Overall, the paper presents a highly original, well-supported, and clearly articulated framework, expanding the utility of diffusion models into discriminative and perception tasks.

weaknesses:
Not enough explanation about how the  contribution factors are derived and adapted to each perception task. It would be helpful if the authors could discuss the potential challenges or limitations of this augmentation strategy, particularly for tasks where shifts may not be well-simulated by data corruption.

The interactive correctional guidance using classifier-free guidance is a compelling feature, but  the authors could elaborate on how correctional prompts are formulated and whether they require manual input.The paper demonstrates improved results with diffusion-tailored data augmentation, but it is unclear whether specific types of augmentation (e.g., color, shape changes) have a greater impact than others without clarify which augmentations are most effective.

While ADDP is demonstrated with specific models, it’s unclear how generalizable these methods are across different diffusion model architectures, such as conditional or latent diffusion models with varying noise schedules. Although ADDP is tailored for perception, it would be interesting to know if the authors have considered its potential for other tasks, such as text-to-image generation or other generative tasks with precision requirements.

questions:
1. Could the authors provide additional insights into how the contribution factors $𝑐_𝑡^2$   for each timestep are estimated and why they differ for various perception tasks? Specifically, how does the estimation process differ between depth estimation and referring image segmentation (RIS)?
2. Can the authors clarify how well the proposed augmentation strategy generalizes to other types of distribution shifts in diffusion-based perception tasks?
3.  How effective is the system without human-generated prompts?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732430830591

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response to All the Reviewers

comment:
We appreciate the feedback and suggestions received from all the reviewers. We are glad that the reviewers found our exploration of diffusion-based perception previously **unexplored** (apx4), **new** (2CZ7), **non-trivial** (zLxB), and **deep** (ZiYy). In addition, the reviewers recognize our **thorough evaluation** (apx4, zLxB, ZiYy), which validates the **substantial improvement** of our approach for diffusion-based perception models (apx4, zLxB, ZiYy). The reviewers also encourage our exploration as a logical expansion of the original problem (zLxB), with innovations **set itself apart from traditional methods** (2CZ7), and potentials for **implications in expanded model versatility** (apx4).

We address the individual concerns of the reviewers below. We appreciate the reviewers for pointing out relevant studies in the context of generation-oriented diffusion models or conventional discriminative models, such as timestep-sampling strategies and data augmentation. We will cite all of these works in our revised manuscript. In the individual responses below, we elaborate on the specific differences and provide corresponding experimental comparisons. These detailed discussions and empirical results demonstrate the fundamental distinctions and strengths of our approach for diffusion-based perception tasks. In light of this, here we would like to clarify the key challenges faced by diffusion-based perception models and our major contributions:

* **Learning Objective**: We notice the **uneven contribution of timesteps across the denoising process** and adjusted the timestep sampling weights during training. Unlike previous generative studies, we propose **an automatic approach of estimating such sampling weights from perception statistics** to reflect the unique properties of each perception task.

* **Training Data**: We mitigate the **training-denoising distribution shift** by **corrupting the ground truth perception data**to simulate the distribution shifts during training. Notably, **our data augmentations differ from existing discriminative and generative methods**.

* **User Interface**: We demonstrate the unique advantage of diffusion models to **interact with users or foundation models via correctional prompts**, compared with conventional discriminative models. Such a proposal answers the long-standing question of “how can the stochastic generative process be useful for discriminative tasks.”

We hope the clarifications above summarize the novel contributions of our study and the contexts of our discussion.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732690772719

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Revised Manuscript

comment:
We greatly appreciate the comments and discussions with the reviewers and have incorporated the new analysis and results from the discussion into the updated manuscript:

## For Reviewer apx4

* Additional experiments in Sec. B.3 (Appendix) on individual effectiveness of our augmentations.
* An additional sentence in Sec. 6 discussing generative tasks as future works.

## For Reviewer zLxB
* An additional sentence in Sec. 3.2 mentioning the key implementation details on our data augmentations that are discussed in the Appendix.
* Revising the caption of Table 2 on the fair comparison of our study.

## For Reviewer 2CZ7
* Revising the caption of Table 1, clarifying the setting of Marigold, and avoiding the confusion of “zeros-shot benchmarks.”

## For Reviewer ZiYy
* Revising the caption of Table 1 clarifying how we got the results of intermediate denoising steps.
* Adding Sec. B.5 and Sec. B.6 (Appendix) for the suggested comparisons with weighting schemes and augmentation methods from generative studies.
* Adding Sec. B.7 (Appendix) clarifying the consistency of our observations regarding the number of denoising steps.

Please let us know if you have any additional comments. We would be happy to discuss them and incorporate your feedback into the revision.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper addresses the question of generative processes like diffusion actually perform on perception tasks where there is a singular ground truth. The paper demonstrates that unlike image generation, perception tasks often rely significantly more heavily on certain timesteps (e.g. for segmentation almost all the work is done in the first set of initial timesteps). Based on this finding, the authors propose to estimate the importance of each step for each perception task and reweight the training steps of the diffusion models based on these importances. The paper also finds that later steps can 'corrupt' or worsen the result due to the divergence of the prediction from the ground truth. The paper proposes a diffusion tailored augmentation to simulate this drift in training data. Additionally the authors propose that the diffusion process allows users to interact with the perception task in order to correct the process.

The paper is very well written and presents both interesting analysis which leads to interesting design improvements. The paper shows strong quantitative improvements for depth estimation, semantic segmentation, and object detection. While the evaluation could have been more thorough, I do not find significant weaknesses in the paper. Reviewers pointed out that the "per-perception task" loss is a potential weakness as well as potentially weak baselines.

I agree with the reviewers and advocate for acceptance.