Aligning Generative Denoising with Discriminative Objectives Unleashes Diffusion for Visual Perception
Authors: Ziqi Pang
Xin Xu
Yu-Xiong Wang
Venue: ICLR 2025 Poster (2025)
Keywords: Diffusion for Perception
Diffusion Models
Visual Perception
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: With success in image generation, generative diffusion models are increasingly adopted for discriminative scenarios because generating pixels is a unified and natural perception interface. Although directly re-purposing their generative denoising process has established promising progress in specialist (e.g., depth estimation) and generalist models, the inherent gaps between a generative process and discriminative objectives are rarely investigated. For instance, generative models can tolerate deviations at intermediate sampling steps as long as the final distribution is reasonable, while discriminative tasks with rigorous ground truth for evaluation are sensitive to such errors. Without mitigating such gaps, diffusion for perception still struggles on tasks represented by multi-modal understanding (e.g., referring image segmentation). Motivated by these challenges, we analyze and improve the alignment between the generative diffusion process and perception objectives centering around the key observation: how perception quality evolves with the denoising process. (1) Notably, earlier denoising steps contribute more than later steps, necessitating a tailored learning objective for training: loss functions should reflect varied contributions of timesteps for each perception task. (2) Perception quality drops unexpectedly at later denoising steps, revealing the sensitiveness of perception to training-denoising distribution shift. We introduce diffusion-tailored data augmentation to simulate such drift in the training data. (3) We suggest a novel perspective to the long-standing question: why should a generative process be useful for discriminative tasks - interactivity. The denoising process can be leveraged as a controllable user interface adapting to users' correctional prompts and conducting multi-round interaction in an agentic workflow. Collectively, our insights enhance multiple generative diffusion-based perception models without architectural changes: state-of-the-art diffusion-based depth estimator, previously underplayed referring image segmentation models, and perception generalists. Our code is available at https://github.com/ziqipang/ADDP.

Published as a conference paper at ICLR 2025
ALIGNING GENERATIVE DENOISING WITH DISCRIMINATIVE OBJECTIVES UNLEASHES DIFFUSION FOR VISUAL PERCEPTION
Ziqi Pang" Xin Xu" Yu-Xiong Wang University of Illinois Urbana-Champaanr Champaign (ziqip2, xinx8, yxx)eillinoke edu
ABSTRACT
With success in image generation, generative diffusion models are increasingly adopted for discriminative scenarios because generating pixels is a unified and natural perception interface. Although directly re-purposing their generative de- noising process has established promising progress in specialist (e.g., depth esti- mation) and generalist models, the inherent gaps between generative process and discriminative objectives are rarely investigated. For instance, generative models can tolerate deviations intermediate sampling steps long as the final distri- bution is reasonable, while discriminative tasks with rigorous ground truth for evaluation are sensitive such errors. Without mitigating such gaps, diffusion for perception still struggles on tasks represented by multi modiil understanding (e.g.. referring image segmentation). Motivated by these challenges, we analyze and improve the alignment between the generative diffusion process and perception objectives centering around the key observation: how perception quality evolves with the denoising process. (1) Notably, earlier denoising steps contribute more than later steps, necessitating a tailored learning objective for training: loss func- tions should reflect varied contributions of timesteps for each perception task. (2) Perception quality drops unexpectedly at later denoising steps, revealing the sensitiveness of perception to training- denoissng distribution shift. We introduce diffusion-taill terriiion-aaloredd data augmentation to simulate such drift in the training data. (3) We suggest novel perspective to the long-standing question: why should a generative process be useful for discriminative tasks - interactivity. The denoising process can be leveraged as controllable user interface adapting to users' cor- rectional prompts and conducting multi-round interaction in an agentic workflow. Collectively, our insights enhance multiple generative diffusion- baassd perception models without architectural changes: state-of-the-an art diffusion based depth esti- mator, previously underplayed referring image segmentation models, and pereep- tion generalists. Our code is available at https://iihub..om/iiggop
INTRODUCTION
The success of diffusion models (Ho et al., 2020; Ramesh et al., 2022; Rombach et al., 2023) has gone beyond pure image generation recently. They emerge as attractive candidates for perception models (Gan et al., 2024; Ke et al., 2024) with the prior knowledge stored in their pre-trained weights and the vision of generalist models via unifying various perception tasks under the same pixel gener- ation interface. Recent approaches finetune pre-trained diffusion models and have made promising progress in both state-of-the-art depth estimation (Ke et al., 2024) specialist and generalist mod- els (Gan et al., 2024) supporting tasks from geometric depth estimation to semantic segmentation and detection. However, they commonly focus on designing the generative format without inves- tigating the fundamental gaps between the generative diffusion process and discriminative tasks: the generative process aims at sampling reasonable distributions, while discriminative tasks require precise matches with rigorous ground truth. Without addressing such discrepancies, generative dif- fusion models noticeably under-perform when perception tasks involve intricate multi-modal rea- soning, e.g., referring image segmentation (RIS), which consequently constrains the exploration of generative perception (Geng et al., 2023). Therefore, the investigation of this paper aspires to bridge the gaps between the generative denoising process and discriminative perception.
"Equal contribution.

---

Published as a conference paper at ICLR 2025
Conditinne
Gree Touits
Step 11 Regular Prediction
Reberlap "Weer"
Truiting Ditriiationn
"Perssn chiee Nutooled sver
Step Internet with Ituman, /Foundationn Nodels fnswer
Pronet
Clskd Lap
whet might
segrier cution ot
SHEFERINNGE
Frompe
Step Prediction with Corfectional Geistanee
8 Timestops
MAN HILEE
< 300
- 620
- 400
(a) Uimeeess Curve for RIS
(b) Illustration of Denoising Perception
(c) Interactivity Advantage with Corrections
We demonstrate the gaps between a generative denoising process and perception tasks using referring image segmentation (RIS), where the diffusion model learns to color the referred object with red masks. (a)(b) The perception quality (Intersetionnover-Uniinn IoU) intermediate denoising steps, which come from the same denoising trajectory, reveals the uneven contribution of timesteps and praining-denoisiieg
Figure 1:
distribution shift, addressed by our enhanced learning objective and training data. (c) We discover that the generative denoising process is also unique user interface for discriminative perception, because of its capabilities to interact with the correctional guidance from users or foundation models.
In principle, the most profound difference between generative denoising process and a conven- tional discriminative model is the iterative sampling procedure, where diffusion model gradually approximates the final prediction by sampling from a score function (Song et al., 2021) 1) step by step. However, such intuition does not align with the reality of perception. As an intuitive illustra- tion, we choose the challenging RIS task and inspect how the perception quality evolves during the denoising process (Fig. 1). Following previous diffusion-based perception (Gan et al., 2024; Geng et al., 2023), our model adopts an image editing format and specifies RIS as editing the target re- gion, i.e., the objects referred by the language prompt, red masks. Ideally, the denoising timesteps should gradually refine these red masks to distinguish the target object, but their IoU (Fig. Ia) and appearances (Fig. 1b) unveil the opposite: (I) the contribution of timesteps is significantly uneven; and (2) perception quality drops surprisingly at later denoising steps. Centering around these ob- servations, we align the training of the denoising process with the reality of the sampling process in diffusion-based perception models, including the learning objective and training data.
The uneven contribution of denoising steps (Fig. la) motivates us to enhance the learning objective of diffusion models by reflecting the perception contribution of every timestep in the loss functions. In conventional diffusion training. c.g., DDPM (Ho et al., 2020), the timesteps are treated uniformly to learn single-step denoising. However, perception tasks need to minimize the distance between the ground truth and accumulation of multi-step denoising, which necessitates enhancing the training of more critical steps. Moreover, the surprising decrease of perception quality (Fig. 1b) arises from the training-denoise distribution shift, which is unique under the diffusion- perception context: deviated distribution from sampling steps can still produce reasonable images, but they wrong for discriminative tasks with rigorous ground truth. To train diffusion models that are robust to such distribution shift, we leverage data augmentations to simulate the erroneous intermediate denoising steps by purposefully corrupting the ground truth. Such improvement to training data addresses distribution shifts in the denoising process and maintains the perception quality until later steps.
Finally, we suggest a novel perspective to the long-standing question: how can the stochastic gen- erative process be useful for discriminative tasks? This becomes an increasingly important question when diffusion models are used as feature extractors (Zhao et al., 2023) without the denoising pro- cess. Instead, we propose that the generative process enables an interactive and interpretable user interface. Specifically, diffusion model can be guided by the correctional prompts from users to adjust their predictions progressively (as in Fig. Ic) with classifier free guidance (Ho & Salimans, 2022), which is rare ability for conventional single-step discriminative models. For example, our diffusion model enables using language as the multi-round reasoning interface for RIS in an agentic workflow (Ng, 2024) built from GPT4 (Achiam et al., 2023).
To conclude, we have made the following contributions to align the generative denoising process in diffusion models for perception:
2

---

Published as a conference paper at ICLR 2025
1. Learning objective. We illustrate the uneven contribution across denoising timesteps and reveal such importance by specifying the sampling weights of timesteps accordingly.
2. Training data. We demonstrate the training- deeriising distribution shift and introduce diffusion- tailored data augmentation to effectuate especially the later denoising steps.
3. User interface. We suggest the unique interactivity advantage of diffusion models for percep- tion: they can progressively modify the predictions via the correctional prompts from humans or foundation models, which is essential for an agentic workflow and human-involved applications.
Our insights are collectively named "ADDP" (Aligning Diffusion Denoising with Perception). Its enhancements generalize across diverse generative diffusion-basee perception models, includ- ing fate--of-the-art diffusion- based depth estimator Marigold (Ke et al., 2024) and generalist In- structCV (Gan al., 2024). Our ADDP also extends the usability of diffusion-based perception to multi-modal referring image segmentation, where we enable a diffusion model to catch up with some discriminative baselines for the first time. We hope ADDP overcomes the limitations of generative diffusion-based perception and unlocks new opportunities in this domain.
PRELIMINARIES
Diffusion Models. Diffusion models have been analyzed in multiple formulations (Ho al., 2020; Karras et al., 2022; Song et al., 1021), and here we adopt the DDPM (Ho et al., 2020) style since most diffusion-based perception models are implemented in DDPM's way. In DDPM, diffusson models learn the image distribution P(x) via a reverse Markov chain with length T. It gradually denoises a random variable xy, which commonly follows Gaussian distribution, into the target variable Xo- During training, the model learns a denoising objective & with a neural network (-).)
1~ Uniform (T),...,"))
N (0,1), X =V yāxx + V U,e, L (,.,))lll (x, (1)
To synthesize high-resolution images, recent latent diffusion models, e.g., Stable Diffusion (Rom- bach et al., 2023), encode images into latent space for denoising. By training at scale (Schuhmann
al., 2021), these models can integrate text conditions as Eg DJ, where D denotes a language description. Such prior knowledge is the basis of using latent diffusion models for perception.
Visual Perception Tasks. We investigate diverse perception tasks and diffusion models to under- stand the gaps between generative models and discriminative objectives. (1) Depth Estimation. We focus on the -ffte-of-thh-art diffusion-based Marigold (Ke et al., 2024). Its diffusion model operates as Eo (It), where x, is the image latent for depth maps and I is latent of the input image. (2) Referring Image Segmentation (RIS). RIS involves an input image and a referring description D for the target object. We treat RIS as an image editing task and adopt the common framework of InstructPix2Pix (Brooks et al., 2023). Concretely, the objective is to "edit the pixels of the tar- get object to red"' via the diffusion model of Eg I,D,), where x, is the latent of the image with red segments on the target object. Experimental analysis of our editing formats is in Sec. B. (3) Generalist Perception. We follow InstructCV (Gan et al., 2024) to build a generalist multi-task perception model for depth estimation, semantic segmentation, and object detection. This generalist model similarly uses InstructPix2Pix unify diverse tasks into image editing. Concretely, the model operates as Eg (x, 1, I)), where D is the description prompt of the task e.g., "Detect % #cCateoryy% and the output x, is image latent for depth maps, segmentation masks, or bounding boxes.
METHOD
Motivated by the gaps between the diffusion process and perception objectives (Fig. 1), we propose the corresponding alignments in Fig. 2, which are simple and plug-and-play for diffusion models. (1) Sec. 3.1: improving the learning objective by resembling the uneven contribution of timesteps.
Conventional Diffusion Training
Aligning Diffusion Deneising with Perception (Ours)
Classifier-tree free Guidanne for Generation
(Simpiited from DDPM atal,
(Simplified from Ho & Salimans, 2022)
1: repeat
Learning Objective: Timestop Sampling
1: Sample x,> ".
2: Ground truch X3
I Estimate Contribution Factor of Timestep: Q
2: for: T.... do
3: le Un.
2. Multinomial (cf, -. sf......)
b Horm Guided score
4:
e-2(0.D)
Training Data: Augmentation Augmentation(Ae_]
3:
(1 + D.., (.)
L x
1x,- S T-
4: (0..))
2. x (aat 111 2,5
6: Take gradient descent step on
Volie 2)1
User Interface; Correctional Guidance
5: Xx-1
Generate correctional prompts
7: until convergence
2. Guided score
score (1 + w)ea (1. D. - (e, D".
for
Figure 2: Method overview.
We align the generative diffusion models with perception tasks from learning objective, training data, and user interface. Notations follow DDPM (Ho et al., 2020).
3

---

Published as a conference paper at ICLR 2025
(2) Sec. 3.2: simulating the distribution shift by improving the training data with data augmenta- tion. (3) Sec. 3.3: re- re-purposing classifier free guidance to enable interactive user interfaces via the generative denoising process.
3.1 LEARNING OBJECTIVE: CONTRIBUTION-WWARR TIMESTEP SAMPLING
We observe the uneven contribution of denoising timesteps in the evolution of perception quality (Fig. 1), e.g. earlier steps closer to r 7= T have a more significant influence than later steps closer to 00 This aligns with image generation observations (Zhang et al., 2024b) where earlier steps conduct more influential "semantic planning." However, such properties are not reflected in the learning objective of conventional diffusion training (Eqn. 1): the timesteps are uniformly sampled, and the optimization targets are all g~>\ (0, I) with similar scales. So why and how should diffusion models tailor their training for distinct timesteps under perception scenarios?
Necessity of Distinguishing Timesteps in Diffusion Training. This design rooted in the differ- ent objectives of generative and discriminative tasks. Diffusion models can learn single-step score functions for generative tasks to sample reasonable images, but discriminative tasks require the fi- nal predictions, which are multi-ste step integral of score functions, to precisely match rigorous ground truths. To better explain its impact on the learning objective, we define contribution factors G, de- noting the relative contribution of timestep r for the final result xp. i.e., Xn x E. C,Ea (X), Then the distance between prediction xo and ground truth ig decomposed as below, where & is the ground truth noise at timestep r:
( 10 -01l oc Eii Br)
Σ Eelle Eg (1). 1=1
Before discussing the implication of Eqn. 2. we clarify the truncation terms for the right-hand Eo 12) As & is randomly sampled from M (0,I), the terms (& - Eg (x) are independent, making the whole truncation term zero; thus, this term can be ignored. (2) The intermediate x, in the denoising process may drift from the precise trajectory & for precisely sampling the ground truth Āg. Our approximation ignores the truncation errors caused by this drift since it is intractable over iterative sampling. With these analyses, proceed with the night-hand side of Eqn. 2 to improve the diffusion loss functions.
side. (1) (( En (1(E; (x(,
A natural implication from Eqn. 2 is that the contribution of needs to be reflected in the training loss of each timestep 2-= f|e, (x(xx This does not contradict the original diffusion objective since the model still learns to fit the score function on single timesteps. However, the errors from more influential timesteps are penalized more in our way, aligning better with the perception objective. As a side note, this principle is consistent with how rectified flow models re- reight the loss functions of timesteps to guide the optimization (Kingma & Gao, 2024). However, we additionally offer guidelines to utilize and estimate the values of ef for specific perception objectives. Concretely, one can: (1) scale the loss values by c?, or (2) scale the sampling probability of timesteps with a multinomial distribution using &i as sampling weights for t (Fig. 2). Both variants improve diffusion for perception, but the probability scaling method performs better (Sec. 4.3.1). The following parts discuss how to estimate the values of c,-
Deriving of from Diffusion Formulation. models since each step Ea (x) can be converted to xg following DDPM (Ho et al., 2020):
To start with, of is an inherent property of diffusion
VI oy E (x.
XO
04
Therefore, we can interpret --= as the relative importance of timestep 1, indicating how much of va Xn can be explained by Ea (x)..). Then we normalize them to acquire o² = (-- E I-
Deriving of from Perception Statistics. The above derivation is discussed in terms of the latents x, (Equ. 2). However, we find the contribution of timesteps sensitive to the perception tasks and diffusion models. For example, Marigold (Ke et al., 2024) for depth estimation exhibits a smoother precision (8)) curve during denoising (Fig. 3) compared with the IoU in RIS (Fig. la). Therefore, of becomes a unique property for each diffusion-based perception scenario and motivates us to estimate each task statistically with the same principle of c7 how much of the final prediction can
We use here accommodate varying noise schedulers and the normalization of estimating C:
4

---

Published as a conference paper at ICLR 2025
Original Image
Editing Target (rg)
Original Image
Beiting Target (sa)
Small Augmentation (46)
Augmentation (x))
Small Augmentation (x))
Large Augmentation (6))
[a] Depth Estt matin Augmentatiion
(b) RIS Augmestation (Referring front zebra)
Figure 4: Data augmentation of (a) Gaussian blurring for depth estimation, and (b) color shape / location for RIS. We use large small intensities of augmentations to simulate different scales of distribution shifts at the earlier later steps of denoising.
be explained by an intermediate denoising step? Concretely, can derive this via the coefficient of determination in re- gression analysis, denoted as R², which measures the good- ness of fit. This procedure involves three steps. (1) Data collection. We apply a diffusion- based perception baseline to validation samples and get 7 metric values of interme- diate denoising steps. Without loss of generality, we take loU from RIS as an example and acquire (ooU,itsTT.ss tial regression. We initialize the estimation with the first step of by running linear regression of :o Bo The R value of this regression, denoted as (R²),, is the pro- portion of the final IoU explained by the first denoising step, so we adopt it as c7. (3) Iterative estimation. We itera- tively add new timesteps into the regression model and set - - (R²) indicating the increase in explained IoU with the new timestamp. Please note that (R²), -RR)) non-negative since adding new variables can only improve the regression fit. More discussions and implementation details are in Sec. A.2.
Sare
(2) Ini-
Figure 3: Evolution of 8j (intu- itively the "accuracy" for depth es- timation) from Marigold (Ke al., 2024) shows smoother patterns than RIS. We copy the RIS curve from Fig. 1a here for easier comparison.
(R+):++ is
3.2 TRAINING DATA: DIFFUSION-
DATA AUGMENTATION
In RIS scenarios, observe the unexpected IoU drop at later denoising steps (Fig. la), where the masks gradually deviate from correct regions and show hallucinated shapes (Fig. 1b). This re- veals the training-denoisimg distribution shift: x, during training (Eqn. 1) comes from the ground truth, while it might deviate from the ideal sampling path during inference. Generative model stud- ies (Ning al., 2023; 2024) call this "exposure bias." However, is even more critical for discrim- inative scenarios: shifted x, might still produce reasonable images belonging to the distribution of ground truth but no longer the desired ground truth of that sample precisely.
Necessity of Simulating Distribution Shift. The ideal solution for distribution shift is to train the diffusion models with x; sampled from the actual denoising process. However, this is computation- ally infeasible due to the iterative nature of denoising. Therefore, we take step back and introduce the solution of simalating the distribution shift for training with augmentations to the ground truth.
niffusion-tailoreed Data Augmentation. We purposefully corrupt the ground truth x into x so that the x, for training (Eqn. 1) reflects distribution shift. Such corruption depends on the timesteps by using more intense augmentation for earlier timesteps: Intuitively, the perception results are coarser at the initial denoising and should be simulated with larger deviations from the real ground truth. When incorporated into the training pipeline of DDPM, the procedure becomes:
X Augment(xo.1), (O,T) X; = Vorlo + /1 O4E.
(4)
We design different augmentations to capture the typical distribution shift for each task in Fig. 4. For instance, the RIS format is a red mask, so our designed augmentation involves color (color changes), location ( transformations to masks), and shape (random erasing of mask parts); while depth estimation mimics coarse boundaries and adopts Gaussian blur. As critical implementation details, we discover that Xn-prediction of diffusion models are more suitable for such data augmen- tation than its mathematically equivalent & E-prediction, and the benefits of varying augmentation intensities w.r. timesteps. More details are discussed in Sec. A.3.
Discussion: Distinctions with Conventional Data Augmentation. Our data augmentation sig- nificantly improves the performance and effectuated later denoising steps (Sec. 4.3). Moreover, our
5

---

Published as a conference paper at ICLR 2025
insights are different from conventional data augmentation. Compared with perception studies, e-g-, RIS, cannot adopt augmentation since masks close to image borders disable "random cropping," and referring with "up/down/lefttighh" disable "random flipping." Compared with diffusion studies, extend the boundary of the common practice merely training on ground truth denoising trajecto- ries the diffusers Eg (.) can explicitly learn to correct problematic input into precise prediction.
3.3 USER INTERFACE: INTERACTIVITY
CORRECTIONAL GUIDANCE
We suggest novel perspective on the value of diffusion denoising process for discriminative tasks: with perception tasks intrinsically deterministic, why and how would the generative sampling diffu- sion models be helpful? This long-standing problem is increasingly important with emerging studies using pre-trained diffusion mod- els single-step generative models or feature extractors (Parmar et al., 2024; Xu al., 2024), without leveraging the multi-step gen- erative process. Besides the vision of unifying perception into pixel synthesis, we demonstrate that a generative model can serve as an interactive interface for perception, which is especially critical for human-involved applications and beyond the capabilities of con- ventional discriminative models.
Original Image (f)
Ground Truth Mask
Prediction from D
Prediction from D D
Interactivity via Denoising. Diffusion models can be expressed with score-matching formulation (Song et al., 2020), where Ea (x,.)) matches a score function Vx, p(x) This provides a nat- ural way to control the generation with explicit user guidance from the interaction with humans or foundation models. Multi-modal un- derstanding, iee., the previously overlooked RIS in diffusion-based perception, is an ideal application to demonstrate the unique inter- activity value. We show intuitive example in Fig. Ic.
Referring (D):
A plass Correctional Prompt (D"): White plate with yollow and red fruits
sitting
holder
Figure 5:
Interacting with correctional guidance D=
Correctional Guidance via Classifier-free Guidance. Our approach is based on classifier-free guidance (Ho & Salimans, 2022): given a condition C, adding (x, Eg (x,,, 0)) (ø denotes empty condition) to the original prediction can nudge the diffusion result to align better with C. In the case of RIS, we consider both conditions of referring description 1) and image /. If the model makes an error and the user can specify the error with language descriptions D=, the original prediction can be corrected by adding (Eg (,, D,I) Eg (x, ., D .)) (Fig. 5). Here, we refer to D correctional guidance, similar negative prompts (Podell et al., 2024) but grounded in perception and vision- language reasoning. Based on the compositionally of score functions (Liu et al., 2022), we utilize the correctional guidance D in addition to the image guidance term of wy:
Eg Eg = Eo (x,1, 4. 01) + wy (E0(x,,t, E 1.
(5)
+wo (o( Eo
"D and wn are scalers for the correctional guidance strength. By setting wp > wp. Eqn. 5 increases the margins between denoising from and D More discussion and details are Sec. A.4.
Integration with Agentic Workflows. To validate the value of such user interface at large scale, we construct an agentic workflow (Ng, 2024) with GPT4o (Achiam al., 2023) to automat- ically generate the correctional prompts D Concretely, we propose a two-round proof-of-concept workflow. (1) Use LLaVA (Liu et al., 2023c) to provide detailed caption of the original image (2) A language-only GPT40 guesses top-k confusing objects in the image based on the referring D and image caption. (3) Our diffusion model generates & new predictions from each correctional prompt using Eqn. 5, and applies majority vote to produce a final mask. More details are in Sec. A.4.
4 EXPERIMENTS
4.1 DATASETS AND IMPLEMENTATION DETAILS
Our insights are generalizable for diffusion- perception and cover diverse scenarios. (1) We im- prove the state-of- the-art art diffusion ffffee Marigold (Ke et al., 2024) for depth estimation, following the same zero-shot evaluation setups. (2) We investigate RIS, where previous diffusion-baan mod- els show large gaps to discriminative counterparts. We follow the standard practice by fine-tuning an

---

Published as a conference paper at ICLR 2025
Method
AbsRell &,1
ETHE3D
ScanNet AbsRell 6.1
AbsRel_ 8,1
NYUv2
AbsRell 8,1
Diade
AbsRell
KITTI
6,1
Average
Rank
DiverseDepth (Yin al,, 2020)
22.8
69.4
10.9
88.2
11.7
87.5
37.6
63.1
19.0
70.4
7.4
MiDaS (Ranftl et al. 2020)
18.4
75.2
12.1
84.6
11.1
88.5
33.2
71.5
23.6
63.0
7.1
LeReS (Yin et 2021)
17.1
77.7
9.1
91.7
9.0
91.6
27.1
76.6
14.9
78.4
5.1
Oumidata (Eftekhar et al., 2021)
16.6
77.8
7.5
93.6
7.4
94.5
33.9
74.2
14.9
83.5
4.7
HDN (Zhang al, 2023)
12.1
83.3
8.0
93.9
6.9
94.8
24.6
78.0
11.5
86.7
3.1
DPT (Ranfi) et al., 2021)
7.8
94.6
8.2
93.4
9.8
90.3
18.2
75.8
10.0
90.1
3.8
Marigold (Ke al., 2024)
7.1
95.1
6.9
94.5
95.9
31.0
77.2
10.5
90.4
2.4
+ADDP (Ours)
6.3
96.1
6.3
95.6
5.6
96.3
29.6
77.5
10.0
90.6
1.4
Table 1: Comparison with diffusion- based depth estimator Marigold (Ke et al., 2024) with identical
pre-training and zero-shot generalization to real-world benchmarks. Bold numbers are the best, underscored are the second best. Our method ADDP uses contribution timestep sampling ("Sampling") " and ceffiusion-taillrres data augmentation ("Aug") and consistently improves Marigold across these scenarios.
RefCOCO test-A
RefCOCO+ test-A
G-Ref
Method
val
Test-B
val
test-B
val
test
Discriminative Encoder- Decoder Based
MCN (Luo et ul., 2020)
62.44
64.20
59.71
50.62
$4.99
44.69
49.22
49.40
(Feng et al., 2021)
62.76
65.69
59.67
51.50
55.24
43.01
51,93
VLT (Ding 2022)
65.65
68.29
62.73
55.50
59.20
49.36
52.99
56.65
ReSTR (Kim et al., 2022b)
67.22
69.30
64.45
55.78
60.44
48.2
CRIS (Wang et ul., 2022)
70.47
73.18
66.10
62.27
53.60
59.87
60.36
LAVT (Yang et al., 2022)
72.73
75.82
68.79
62.14
68.38
55.10
61.24
62.09
VPD (Zhao et al., 2023)
73.25
62. 69
61,96
ReLA (Liu al., 2023a)
73.82
76.48
70.18
66.04
71.02
$7.65
65.00
65.97
PVD (Cheng et al. 2024)
74.82
77.11
69.52
63.38
68.60
56.92
63.13
63.62
UNINEXT (Yan et al., 2023)
77.90
79.68
75.77
66.20
71.22
59.01
70.04
70.52
Generative Image Synthesis Based
Unified-O0 (Lu al., 2022)
46.42
46.06
40.50
42.17
40.15
48.74
49.13
InstructDiffusion (Geng et al., 2025)
61.74
65,20
60.17
46.57
52.32
39.04
51.17
52.02
InstraciPix2Pix-SDI..
60.87
63.70
58.39
44.98
51.93
35.31
43.99
45.43
+ ADDP (Ours)
66.86
67.39
63.72
55.35
58.72
48.45
55.85
57.05
InstruciPh22PxxSS22.0
64.96
66.72
62.63
47.13
53.32
38,
50.28
50.58
ADDP (Ours)
69.14
70.27
67.46
57.58
61.65
51.67
59.05
59.60
Table 2: RIS Comparison. Our insights collectively mitigate the gaps between generative and dis- criminative ones by large progress. Although not achieving the state of the art, our improvements empower the common diffusion baseline, i.e., RIS finetuned InstructPix2Pix, to catch up with some representative discriminative baselines for the first time. We hope this improved baseline removes the constraints and encourages new opportunities for perception with generative diffusion models.
InstructPix2Pix (Brooks et al., 2023) model on RefCOCO (Yu et al., 2016), RefCOCO+ (Yu et al., 2016), and G-Ref (Nagaraja et al., 2016) separately for 60 epochs. (3) We follow InstructCV (Gan et al., 2024) and prove the effectiveness of our insights under a multi-task generalist setting, where a single model addresses depth estimation, semantic segmentation, and object detection. Due to space limits, the detailed training and evaluation setups for these experiments are in Sec. A.
4.2 MAIN RESULTS
4.2.1 DEPTH ESTIMATION
Diffusion-basee perception methods are already effective for depth estimation, represented by the recent Marigold et,,, 2024). Although only trained on synthetic depth, Marigold performs com- petitively in zero-shot way. We apply both of our improvements on learning objectives (Sec. 3.1) and training data (Sec. 3.2) to Marigold and show the quantitative comparison following Marigold's style in Table 1. Notably, our proposed techniques consistently improve Marigold across all the benchmarks. We conduct detailed ablations of the two techniques for depth estimation in Sec. B.4.
4.2.2 REFERRING IMAGE SEGMENTATION (RIS)
Improvement of Generative RIS We format RIS as an image editing problem and separately train on RefCOCO, ReiCOCO+, and G-Ref, for a fair comparison with previous studies. As in Ta- ble 2, we focus on the comparisons with generative methods and include discriminative approaches for context. We first emphasize the significant challenge of RIS for generative perception methods, despite their strong performance in other tasks like depth estimation. This limitation constrains the development of generative perception research. Note, our claim is not on achieving state-of the-art RIS performance. Rather, demonstrate that our ADDP, with its and-paay insights, substan-
7

---

Published as a conference paper at ICLR 2025
Original Ground Touth
Prediction
Prediction
D
Original Image
Ground
Prediction from
from D
Preductoo
Beferving, (D): The cnll Giraffi Correctional Prompt (0 Thegiruffe che forgguand
Referring (P): Mon anmpit
Correctional Prompt() Cluir
Figure 6: Interactive interface enables diffusion models adaptively correct their predictions via language models' guidance D Such capabilities of progressiveness are beyond conventional dis- criminative models and are an emerging advantage of the generative denoising process in perception.
NYUv2 (Depth Estimation) ADE20K (Semantic Segmentation) COCO (Object Detection) RMSE,
Method
mloUt
mAP@0.51
InstructCV
0.302
46.67
46.6
+ADDP (Ours)
0.288
48.40
48.1
Table Generalist Perception. We follow InstructCV (Gan et al., 2024) and build multi-task gen- eralist perception model using InstructPix2Pix without task-specific components. Our techniques show consistent improvement across these three fundamental perception tasks.
tially narrows the gap between generative and discriminative methods, enabling common diffusion framework InstructPix2Pix to catch up with some RIS baselines for the first time, without modifying the model introducing extra data. We hope our enhanced diffusion-based method inspires further exploration of generative perception in tackling multi-modal understanding challenges.
From Table 2, we have the following conclusions. (1) Compared with other generative methods, especially the baseline of InstructPix2Pix, we significantly and consistently improve all the RIS sub- sets through the integration of better learning objective (contribution-awe)
timestamp sampling) (2) Compared with the few existing meth- ods adapting pre-trained image generative models for perception, we outperform them by a large margin without tailoring the model architecture using more data for RIS, especially another InstructPix2Pix
and training data (diffusion-tailores data augmentation).
method- InstructDiffusion (Geng et al., 2023). These indicate that our dis- covered insights are critical for designing generative perception models and open new opportunities for diffusion based perception. (3) Our insights generalize across different stable diffusion mod- els (Rombach et al., 2023), enhancing them by a large margin. More ablations are in Sec. 4.3.
Effectiveness of Generative Denoising as Interactive User Interfaces. To validate the benefits of interactivity (Sec. 3.3) and support the value of the generative process for perception, evaluate our proof-of-concep agentic workflow with correctional guidance on the validation sets of RIS, Please note that our Table 2 does not use such in- teractive reasoning to avoid unfair comparison with other RIS methods. As in Table 3, workflow can improve the grounding and gain larger advan- lages on harder scenarios (G-Ref), especially the most challenging G-Ref. In Fig. 6, we further illus- trate qualitative examples of how our generated correctional prompts modify the grounding results via the reasoning conducted by language models. These results indicate that the interactive interface of diffusion models beneficial for perception tasks involving reasoning or user interaction.
Method
RefCOCo ReCCOCO+ G-Ref
InstrucPix2Pix
50.28
+Sampling+Ang (Our))
66.86
55.35
55.85
+Comectional Guidannce (Ours)
66.93
56.13
$6.98
Table Effectiveness of correctional guid- ance, especially on hard scenarios (G-Ref).
4.2.3 GENERALIST PERCEPTION
key motivation for using diffusion models for perception is the vision of building generalist per- ception models by unifying diverse tasks into image generation. We follow InstructCV (Gan et al., 2024) in this endeavor and solve three fundamental perception tasks simultancously: depth estima- tion, semantic segmentation, and object detection. They are formatted as image editing, addressed with the InstructPix2Pix (Brooks et al., 2023) framework. As shown in Table 4, our ADDP consis- tently improves the InstructCV using vanilla InstruetPixPix across all three tasks. This validates the effectiveness of our approach for broader diffusion- based perception models.
4.3 ABLATION STUDIES
We analyze the effectiveness of our insights through a series of ablation studies on the most chal- lenging RIS task. Without special mention, the experiments conducted on the RefCOCO bench- mark with 20 epochs of training as Sec. 4.1 and evaluation on the RefCOCO's validation set. More ablation studies provided in Sec. B.
8

---

Published as a conference paper at ICLR 2025
4.3.1 CONTRIBUTION-AWAR TIMESTAMP SAMPLING
In Table S, we analyze the strategies proposed in Sec. 3.1: enlarging the contribution of earlier
denoising steps in learning objectives. Specifically, original DDPM strategy, where the timesteps are uniformly sampled, and the losses are not scaled;
compare four strategies: (1) Uniform: the
(2) Loss Scaling (Diffusion): scaling the loss of timestep by ? estimated from the diffusion formulation. Prob Scaling (Diffusion): Sampling the timesteps by I~ Multinomial (,..,,,) where 2j is derived from the diffu- sion formulation. (4) Prob Scaling (Perception Stats): Sam- pling the timesteps with the c, estimated from the perception (loU) statistics. As shown in Table 5, reflecting the contribu- tion of timesteps in either sampling or loss weights enhances the uniform baseline. With of from diffusion weights, seal- ing the sampling probability is better than scaling the loss, trained with more iterations at earlier denoising steps under the probability scaling. Moreover, c7 estimated from the perception tasks performs the best, since this most closely aligned with the objective. These comparisons support our design in Sec. 3.1 to improve the learning objective of diffusion for perception.
(3)
Method
cf
oloU
Uniform
N/A
56,42
Loss Scaling
Diffusion
58.21
Proh Scaling
Diffusion
63.05
Prob Scaling
Perception Stats
64.00
Table 5: Strategies of using the con- tribution cî in diffusion training.
which is likely due to that (x is
4.3.2 CORRCIION-TALLORE DATA AUGMENTATION
Effectiveness of Data Augmentation. and after applying data augmentation. Specifically, we calculate the IoU at the 2nd, 20th, 40th, 60th, 80th, and 100th sampling out of 100 denoising steps in to- tal. Fig. 7 validates diflusion-tailored data augmentation from two aspects. (1) The quality of masks significantly improves. Such increase mostly comes from the earlier denoising steps, indicating the benefits of providing more challenging inputs to the diffusion model and enforcing the model to correct the er- rors. (2) The trend of IoU-Timestep curve shows that IoU keeps increasing slowly after 8000 contrasting the decrease of In- structPix2Pix and JinstrucPPxx2Pxx-Sampnngg between 808 and 1 200. Despite subsequent slight drop in metrics at the final stage, our data augmentation largely decreases the overall drops after the early steps. Therefore, our enhanced training data indeed aligns the denoising process with perception tasks by mit- distribution shifts.
Fig. 7, we compare the IoU-Timestep curves before
Vizzoolampa
Figure 7: IoU-Timestep curves. Our data augmentation de- creases the training-denoising distribution shifts.
igating the
Data Augmentation Intensity. In Fig. 8, we investigate the relationship between the intensity of data augmentation and fi- nal performance. As in Sec. 3.2 and Sec. A.3, the intensity of data augmentation specifies the corruption to the ground truth. For RIS, larger intensities indicate larger changes in ground truth masks' color, location, and shape. We regard the intensity used in Table 2 and Table 5 as the base level (1 x), which introduces visually reasonable corruptions. We also evaluate performance under conditions of no augmentation (0x), reduced intensity (0.5x), and increased intensity (2x). As demonstrated in Fig. 8, higher augmentation intensity leads to improved performance, indicating that more intense data augmentation enhances the dis- criminative capabilities of diffusion models. These findings val- idate the effectiveness of our data augmentation strategy. We utilize the median intensity data aug- exieiieeeeemee denoising phase. Further investigation of more intense augmentations will be our future work.
Os intensity
0.Sxintensity
olo 65.00
olol:: 66.05 2x intensity
Ex intensity
atall: 67.39 Figure 8: Augmentation Inten- sity.
aloU: 66.19
mentation in the main experiments, because it visually aligns with our observed data drift during the
RELATED WORK
Diffusion Models. Diffusion models (Ho et al., 2020; Karras et al., 2022; 2024; Sohl-Diekstein et al., 2015) are probabilistic models denoising from Gaussian noises, guided by a reverse Marko- vian process. They exhibit better training stability than generative adversarial networks (Esser et al., 2021; Goodfellow et al., 2014) or variational auto-encoders (Kingma & Welling, 2014; Van Den Qord et al., 2017). The recent advances in diffusion models have achieved outstanding text-to-

---

Published as a conference paper at ICLR 2025
image synthesis ability (Ramesh et al., 2022), especially with the latent diffusion models represented by the Stable Diffusion series (Esser et al., 2024; Podell et al., 2024; Rombach et al., 2023). The capabilities of generating realistic images with conditioning have motivated numerous applications represented by image editing (Brooks et al., 2023; Hertz et al., 2022; Sheynin et al., 2024) and con- trollable image generation (Zhang al., 2023). Besides training stability, diffusion models have the intuition of score-matching functions (Song et al., 2021) and support guidance at the denoising time (Dhariwal & Nichol, 2021; Ho & Salimans, 2022), which is crucial for improving the con- sistency with conditioning. The success of diffusion models is also progressing quickly in other modalities, such as 3D (Poole et al., 2022) and video generation (Ho et al., 2022). Our study mainly improves such diffusion models under a perception perspective (Gan et al., 2024; Geng et al., 2023; Ke et al., 2024; Xing et al., 2023). Concretely, our insights are plug-and-play alignment to train better diffusion-based perception models, effectuating the generative denoising process for percep- tion objectives. Furthermore, we illustrate how classifier-free guidance (Ho & Salimans, 2022) can be uniquely re-purposed for vision-language reasoning and imply the unique value of generative models for discriminative tasks.
Diffusion Models for Perception. Recent studies adopting pre-trained diffusion models for per- ception, egg., Stable Diffusion (Rombach et al., 2023), can be categorized into three groups. (1) Diffusion models can synthesize virtual training examples (Nguyen et al., 2024; Tian et al., 2023; 2024; Wu et al., 2023) for perception models. (2) The most profound trend is to leverage pre-trained backbones in diffusion models as feature extractors in perception tasks, supporting tasks like seg- mentation (Xu et al., 2023; Zhao et al., 2023), depth estimation (Xu et al., 2024; Zhao et al., 2023), 3D understanding (Man et al., 2024), and finding correspondence (Hedlin al., 2024; Luo al., 2024; Namekata et al., 2024; Tang et al., 2023; Zhang et al., 2024a). However, these methods do not fully leverage the generative capabilities of diffusion models. (3) Our focus is the last category, which unleashes the generation ability of diffusion models and envisions pixel synthesis as the pivot to developing generalist models (Gan al., 2024; Geng et al., 2023; Xing et al., 2023). Learning perception tasks also improve the precision of generation, e.g., EMU-Edit (Sheynin et al., 2024). Despite the success in depth estimation (Ke et al., 2024), we notice that generative perception re- mains challenging and inferior to discriminative methods on domains like multi-modal reasoning. Our studies enhance the training and inference of diffusion models by aligning the denoising pro- cess with discriminative tasks from the perspectives of learning objectives and training data. Such improvements bring consistent improvement under several scenarios and critically empower com- petitive diffusion-batt based baselines for multi-model understanding. In addition, we suggest the unique value of the generative process for visual perception as interactive user interfaces. We hope our dis- coveries open new opportunities and enable more studies in perception using diffusion models.
CONCLUSION
This study investigates the missing parts of diffusion models for perception tasks from the fundamen- tal distinction between generative and discriminative tasks: generation requires sampling diverse and reasonable contents, while discriminative perception needs a precise match with the rigorous ground truth. We unveil the gap between the conventional diffusion denoising process and perception tasks and propose plug-and-play enhancements in learning objective (contribution-and aware timestamp sam- pling) and training data Lifffusion-tailored data augmentation). In addition, we highlight the unique advantage of diffusion models as interactive and interpretable user interface for perception tasks, empowering multi- reasoning via agentic workflows. We hope our insights will foster further exploration and improvement of generative models for perception.
Discussion and Limitations. We investigate a wide range of diffusion-based perception models and unlock significantly improved baselines. However, we acknowledge that generative perception inherently challenging: the methods using the denoising process, instead of treating diffusion models as feature extractors, might still underperform on challenging tasks, such as the RIS in our paper. Moreover, the diffusion models are pre-trained for image generation purposes without align- ment with perception use cases. For example, Stable Diffusion (Podell et al., 2024; Esser et al., 2024) series employ data filtering to only train on highly aesthetic images, which potentially hurts the gen- eralization to perception data. Therefore, how to guide diffusion models for perception tasks during the pre-training stage will be meaningful for future work. Finally, our ADDP methodology could be relevant for generative tasks with relatively well-defined ground truth, such as super-resolution and 3D reconstruction. We hope ADDP can inspire further exploration in such directions.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported in part by NSF Grant 2106825, NIFA Award 2020-67021-32799 the Toy- ota Research Institute, the IBM-IIlinois Discovery Accelerator Institute, the Amazon-lliinois Center on AI for Interactive Conversational Experiences, Snap Inc., and the Jump ARCHES endowment through the Health Care Engineering Systems Center at Illinois and the OSF Foundation. This work used computational resources, including the NCSA Delta and DeltaAl supercomputers through al- locations CIS230012 and CIS240387 from the Advanced Cyberinfraatrructure Coordination Ecosys- tem: Services & Support (ACCESS) program, as well as the TACC Frontera supereomputer, Ama- zon Web Services (AWS), and OpenAl API through the National Artificial Intelligence Research Resource (NAIRR) Pilot.
ETHICS STATEMENT
The studies conducted in this paper do not have explicit ethics concerns. However, our method po- tentially shares the social biases of pre-trained diffusion models during data filtering, annotation, and training stages. Therefore, we aim to understand such generative models for perception scenarios and encourage cautious applications with human involvement.
REPRODUCIBILITY STATEMENT
We ensure the reproducibility all the results in the paper. The implementation details are enumer- ated in Sec. 4.1 and Sec. A. We have released the code http://gsitubb.oom//agponnng/D
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, llge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, el. GPT-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
Tim Brooks, Aleksander Holynski, and Alexei A Efros. InstructPix2Pix: Learning follow image editing instructions. In CVPR, 2023.
Yohann Cabon, Naila Murray, and Martin Humenberger. arXiv:2001.107.: 2020.
Virtual kitti 2. arXiv preprint
Zesen Cheng, Kehan Li, Peng Jin, Siheng Li, Xiangyang Ji, Li Yuan, Chang Liu, and Jie Chen. Parallel vertex diffusion for unified visual grounding. In AAAI, 2024.
Jooyoung Choi, Jungbeom Lee, Chaehun Shin, Sungwon Kim, Hyunwoo Kim, and Sungroh Yoon. Perception prioritized training of diffusion models. In CVPR, 2022.
Angela Dai, Angel X Chang. Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. ScanNet: Richly-annotated 3D reconstructions of indoor scenes. In CVPR, 2017.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. In NeurlPS, 2021.
Henghui Ding, Chang Liu, Suchen Wang, and Xudong Jiang. Vlt: Vision-language transformer and query generation for referring segmentation. TPAMI, 2022.
Ainaz Eftekhar, Alexander Sax, Jitendra Malik, and Amir Zamir. Omnidata: A scalable pipeline for making multi-task mid-level vision datasets from 3D scans. In ICCV, 2021.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In CVPR, 2021.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In ICML, 2024.
Guang Feng, Zhiwei Hu, Lihe Zhang, and Huchuan Lu. Encoder fusion network with co-attention embedding for referring image segmentation. In CVPR, 2021.
11

---

Published as a conference paper at ICLR 2025
Yulu Gan, Sungwoo Park, Alexander Schubert, Anthony Philippakis, and Ahmed M Alaa In- structCV: Instruction-uun -otaae diffusion models as vision generalists. In ICLR, 2024.
Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? The
KITTI vision benchmark suite. In CVPR, 2012.
Zigang Geng, Binxin Yang, Tiankai Hang, Chen Li, Shuyang Gu, Ting Zhang, Jianmin Bao, Zheng Zhang, Han Hu, Dong Chen, et al. InstructDiffusion: A generalist modeling interface for vision tasks. arXiv preprint arXiv:2309.0338 03895, 2023.
Hyojun Go, Yunsung Lee, Seunghyun Lee, Shinhyeok Oh, Hyeongdon Moon, and Seungtaek Choi. Addressing negative transfer in diffusion models. In NeurIPS, 2024.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. NeurIPS, 2014.
Tiankai Hang. Shuyang Gu, Chen Li, Jianmin Bao, Dong Chen, Han Hu, Xin Geng, and Baining Guo. Efficient diffusion training via min-snr weighting strategy. In ICCV, 2023.
Eric Hedlin, Gopal Sharma, Shweta Mahajan, Hossam Isack, Abhishek Kar, Andrea Tagliasacchi, and Kwang Moo Yi. Unsupervised semantic correspondence using stable diffusion. In NeurlPS, 2024.
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-prompl image editing with cross attention control. arXiv preprint 0rXiv:2208001666, 2022.
Jonathan and Tim Salimans.
Classifier-free diffusion guidance.
arXiv preprint
1rXi1:2207.12598, 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurlPS, 2020.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David Fleet. Video diffusion models. arXiv:2204.03458, 2022.
Aapo Hyvärinen and Peter Dayan, Estimation of normooomal statistical models by score match- ing. JMLR, 2005.
Tero Karras, Miia Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. NeurlPS, 2022.
Tero Karras, Miika Aittala, Jaakko Lehtinen, Janne Hellsten, Timo Aila, and Samuli Laine. Analyz- ing and improving the training dynamics of diffusion models. In CVPR, 2024.
Bingxin Ke, Anton Obukhov, Shengyu Huang, Nando Metzger, Rodrigo Caye Daudt, and Konrad Schindler. Repurposing diffusion-basst image generators for monocular depth estimation. In CVPR, 2024.
Dongjun Kim, Seungjae Shin, Kyungwoo Song, Wanmo Kang, and II-Chul Moon. Soft truncation: A universal training technique of score-based diffusion model for high precision score estimation. In ICML, 2022a.
Namyup Kim, Dongwon Kim, Cuiling Lan, Wenjun Zeng, and Suha Kwak. ResTR: Convolution- free referring image segmentation using transformers. In CVPR, 2022b.
Diederik Kingma and Ruiqi Gao. Understanding diffusion objectives as the elbo with simple data augmentation. NewrlPS, 2024.
Diederik Kingma. Adam: A method for stochastic optimization. ICLR, 2014.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. In ICLR, 2014.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: Common objects in context. In ECCV, 2014.
12

---

Published as a conference paper at ICLR 2025
Chang Liu, Henghui Ding, and Xudong Jiang. GRES: Generalized referring expression segmenta- tion. In CVPR, 2023a.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. arXiv:2310.33744, 2023b.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurlPS, 2023c.
Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B Tenenbaum. Compositional visual generation with composable diffusion models. In ECCV, 2022.
Loshchilov. Decoupled weight decay regularization. arXiv preprint arXiv: 2017.
lya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. arXiv
preprint 108,:386,8.03383,, 2016.
Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. Unified-IO: A unified model for vision, language, and multi-modal tasks. In ICLR, 2022.
Gen Luo, Yiyi Zhou, Xiaoshuai Sun, Liujuan Cao, Chenglin Wu, Cheng Deng, and Rongrong Ji. Multi-task collaborative network for joint referring expression comprehension and segmentation. In CVPR, 2020.
Grace Luo, Lisa Dunlap, Dong Huk Park, Aleksander Holynski, and Trevor Darrell. Diffusion hyperfeatures: Searching through time and space for semantic correspondence. In NeurIPS, 2024.
Yunze Man, Shuhong Zheng, Zhipeng Bao, Martial Hebert, Liang-Yan Gui, and Yu- Yu-Xiong Wang. Lexicon3D: Probing visual foundation models for complex 3D scene understanding. arXiv preprint ar arXiv:2409.03777. 2024.
Varun Nagaraja, Vlad I Morariu, and Larry Davis. Modeling context between objects for referring expression understanding. In ECCV,
Koichi Namekata, Amirmojtaba Sabour, Sanja Fidler, and Seung Wook Kim. EmerDiff: Emerging pixel-level semantic knowledge diffusion models. In ICLR, 2024.
Andrew Ng. The batch issue 242. https: :/www///wwww/// issue-24 242/, 2024.
Quang Nguyen, Truong Vu, Anh Tran, and Khoi Nguyen. Dataset diffusion: Diffusion-based syn- thetic data generation for pixel-level semantic segmentation. In NeurlPS, 2024.
Mang Ning, Enver Sangineto, Angelo Porrello, Simone Calderara, and Rita Cucchiara. Input per- turbation reduces exposure bias in diffusion models. In ICML, 2023.
Mang Ning, Mingxiao Li, Jianlin Su, Albert Ali Salah, and Itir Onal Ertugrul. Elucidating the exposure bias in diffusion models. In ICLR, 2024.
NaHyeon Park, Kunhee Kim, Song Park, Jung-Woo Ha, and Hyunjung Shim. TADA: Timestep- aware data augmentation for diffusion models. NeurIPS Workshop, 2023.
Gaurav Parmar, Taesung Park, Srinivasa Narasimhan, and Jun-Yan Zhu. One-step image translation with text-to-image models. arXiv preprint arXiv:2403.1203: 2024.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhor, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: Improving latent diffusion models for high-resolution image synthesis. In ICLR, 2024.
Ben Poole. Ajay Jain, Jonathan T Barron, and Ben Mildenhall. DreamFusion: Text-to-3D using 2D diffusion. ICLR, 2022.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.0mm 06155, 2022.
13

---

Published as a conference paper at ICLR 2025
René Ranftl, Katrin Lasinger. David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. TPAMI, 44(3): 1623-1637, 2020.
René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. In ICCV, 2021.
Mike Roberts, Jason Ramapuram, Anurag Ranjan, Atulit Kumar, Miguel Angel Bautista, Nathan Paczan, Russ Webb, and Joshua M Susskind. HyperSim: photorealistic synthetic dataset for holistic indoor scene understanding. In ICCVV,002)
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, 2023.
Olaf Ronneherger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional networks for biomed- ical image segmentation. In MICCAI, 2015.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In ICLR, 2022.
Thomas Schops, Johannes L Schonberger, Silvano Galliani, Torsten Sattler, Konrad Schindler, Marc Pollefeys, and Andreas Geiger. A multi-view stereo benchmark with high-resolution images and multi-camera videos. In CVPR, 2017.
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarezyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. LAION-400M: Open dataset of CLIP-filteed 400 million image-text pairs. arXiv preprint arXiv:2111.02114., 2021.
Shelly Sheynin, Adam Polyak, Uriel Singer, Yuval Kirstain, Amit Zohar, Oron Ashual, Devi Parikh, and Yaniv Taigman. Emu Edit: Precise image editing via recognition and generation tasks. In CVPR, 2024.
Aleksandar Shtedritski, Christian Rupprecht, and Andrea Vedaldi. What does clip know about a red circle? visual prompt engineering for vlms. In ICCV, 2023.
Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and sup- port inference from RGBD images. In ECCV, 2012.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics.. In (CML,0015
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR, 2020.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. ICLR, 2021.
Luming Tang, Menglin Jia, Qianqian Wang, Cheng Perng Phoo, and Bharath Hariharan. Emergent correspondence from image diffusion. In NeurIPS. 2023.
Yonglong Tian, Lijie Fan, Kaifeng Chen, Dina Katabi, Dilip Krishnan, and Phillip Isola. Learning vision from models rivals learning vision from data. arXiv preprint arXiv:2312.17742, 2023.
Yonglong Tian, Lijie Fan, Phillip Isola, Huiwen Chang, and Dilip Krishnan. StableRep: Synthetic images from text-to-image models make strong visual representation learners. NeurIPS, 2024.
Aaron Van Den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learn- ing. In NewrIPS, 2017.
Igor Vasiljevic, Nick Kolkin, Shanyi Zhang, Ruotian Luo. Haochen Wang, Falcon Z Dai, Andrea F Daniele, Mohammadreza Mostajabi, Steven Basart, Matthew R Walter, et al. Diode: A dense indoor and outdoor depth dataset. arXiv preprint arXiv: arXiv:1908.000
(et.: 004660 2019.
14

---

Published as a conference paper at ICLR 2025
Kai Wang, Mingjia Shi, Yukun Zhou, Zekai Li, Zhihang Yuan, Yuzhang Shang, Xiaojiang Peng, Hanwang Zhang, and Yang You. closer look at time steps worthy of triple speed-up for diffusion model training. arXiv preprint arXiv:2405.1740: 2024.
Zhaoqing Wang, Yu Lu, Qiang Li, Xunqiang Tao, Yandong Guo, Mingming Gong, and Tongliang Liu. CRIS: Clip-driven referring image segmentation. In CVPR, 2022.
Weijia Wu, Yuzhong Zhao, Mike Zheng Shou, Hong Zhou, and Chunhua Shen. Diffumask: Synthe- sizing images with pixel-level annotations for semantic segmentation using diffusion models. In ICCV, 2023.
Zhen Xing. Qi Dai, Zihao Zhang. Hui Zhang, Han Hu, Zuxuan Wu, and Yu-Gang Jiang. VID- iff: Translating videos via multi-modal instructions with diffusion models. arXiv preprint arXiv:2311.18837 2023.
Guangkai Xu, Yongtao Ge, Mingyu Liu, Chengxiang Fan, Kangyang Xie, Zhiyue Zhao, Hao Chen, and Chunhua Shen. Diffusion models trained with large data are transferable visual models. arXiv preprint arXiv:2403.0660 2024.
Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello. Open- vocabulary panoptic segmentation with text-to-image diffusion models. In CVPR, 2023.
Bin Yan, Yi Jiang, Jiannan Wu, Dong Wang, Ping Luo, Zehuan Yuan, and Huchuan Lu. Universal instance perception as object discovery and retrieval. In CVPR, 2023.
Zhao Yang, Jiaqi Wang, Yansong Tang, Kai Chen, Hengshuang Zhao, and Philip HS Torr. LAVT: Language-aware vision transformer for referring image segmentation. In CVPR, 2022.
Wei Yin, Xinlong Wang. Chunhua Shen, Yifan Liu, Zhi Tian, Songcen Xu, Changming Sun, and Dou Renyin. Diversedepth: Affine-invariant depth prediction using diverse data. arXiv preprint arXiv:2002.00569 2020.
Wei Yin, Jianming Zhang, Oliver Wang, Simon Niklaus, Long Mai, Simon Chen, and Chunhua Shen. Learning to recover 3D scene shape from single image. In CVPR, 2021.
Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara Berg. Modeling context in referring expressions. In ECCV, 2016.
Chi Zhang, Wei Yin, Billzb Wang, Gang Yu, Bin Fu, and Chunhua Shen. Hierarchical normalization for robust monocular depth estimation. In NeurIPS, 2022.
Junyi Zhang, Charles Herrmann, Junhwa Hur, Luisa Polania Cabrera, Varun Jampani, Deqing Sun, and Hsuan Yang. A tale of two features: Stable diffusion complements dino for zero-shot semantic correspondence. In NeurlPS, 2024a.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. ICCV, 2023.
Wentianding Zhang, Haozhe Liu, Jinheng Xie, Francesco Faccio, Mike Zheng Shou, and Jürgen Schmidhuber. Cross-attention makes inference cumbersome in text-to-image diffusson models. arXiv preprint 24XX::22004.02747, 2024b.
Wenliang Zhao, Yongming Rao, Zuyan Liu, Benlin Liu, Jie Zhou, and Jiwen Lu. Unleashing text- to-image diffusion models for visual perception. In ICCV, 2023.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In CVPR, 2017.
Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. IJCV, 2019.
15

---

depth eval- Published as a conference paper at ICLR 2025
A IMPLEMENTATION DETAILS
A.1 DATASETS AND IMPLEMENTATION DETAILS
A.1.1 DEPTH ESTIMATION
We strictly follow the setting in Marigold (Ke et al., 2024) for both training and evaluation. Con- cretely, the model is trained on the virtual depth maps in Hypersim (Roberts et al., 2021) and Virtual- KITTI (Cabon al., 2020) with initialization from stable diffusion (Rombach et al., 2023). The evaluation is conducted in a zero-shot style on multiple real-world datasets, including NYUv2 (Sil- berman et al., 2012 2). ScanNet (Dai et al., 2017), DIODE (Vasiljevic al., 2019), KITTI (Geiger et al., 2012), and ETH3D (Schops et al., 2017). The evaluation follows uation (Ranftl et al., 2020). We adopt the two metrics used in Marigold: Absolute Mean Relative Error (AbsRel) and 6n accuracy, which measure the overall errors and precision of depth estima- tion. For both training and evaluation, we keep Marigold's codebase and prper-parameters identical. Most importantly, our diffusion model is trained with 1000 steps of DDPM (Ho et al., 2020) and denoised with 50 steps of DDIM (Song et al., 2020). Our improvement in learning objectives and training data are described later in Sec. A.2 (contribution-aver timestep sampling) and Sec. A.3 duffusiiontailoree data augmentation).
A.1.2 REFERRING IMAGE SEGMENTATION
Datasets and Evaluation. We follow the standard practice of separately training models on Re- fCOCO (Yu al., 2016), RefCOCO+ (Yu et al., 2016), and G-Ref (Nagaraja et al., 2016) (UMD split), which are created from the MSCOCO dataset (Lin et al., 2014), then evaluate on their vali- dation and test sets. We adopt the standard metric of "overall intersection over union" (oIoU). This metric accumulates the intersection and unions across the whole dataset and emphasizes larger ob- jects. Before evaluating, we resize our prediction and ground truth the resolution of 512 x 512 following previous works (Wang et al., 2022).
Training and Inference. Our baseline is InstructPix2Pixx (Brooks et al., 2023), taking the referring prompts as the text conditioning and input image as the image conditioning. Our diffusion model trained with 1000 steps of DDPM (Ho et al., 2020) and denoised with 100 steps of DDIM (Song et al., 2020). During training, we will resize all the images to the resolution of 256 x 256, optimizing with the AdamW W optimizer (Kingma, 2014; Loshchilov, 2017), batch size of 128, learning rate of 10-4, and cosine annealing scheduler (Loshchilov & Hutter, 2016). The classifier-frs frreerr--eeeeee guidance (Ho & Salimans, 2022) weights are manually tuned on the RefCOCO validation set to guarantee optimal performance, which is 1.5 for image conditioning and 7.5 for text conditioning in the InstructPix2Pix model and 1.5 for image conditioning and 3.0 for text conditioning in our enhanced model. The models presented in Table 2 are trained with 60 epochs, where each epoch indicates enumerating each image once. The detailed configurations for our proposed insights (Sec. 3) are described in Sec. A.2 (contribution-aww) aware timestep sampling). Sec. A.3 dliffusion-tailores data augmentation), and Sec. A.4 (correctional guidance).
Evaluation processic... During the evaluation of diffusion for perception, we convert the edited images with red masks into binary masks for loU computation. Since the generated images might not have "perfect" red color with RGB values of (255,0,0). we apply color threshold d, to convert the image mask.
00 [V.,, (255.0, 0. 2
M, - 1, [V.,) (255, 0,5,,0,011 0,
(A)
J.j and M.j denote the RGB value of the image and mask value at pixel (I,j). This equation
intuitively means that recognize pixel as masked region if its color is close enough (within 6.) to the perfect red color. In our experiments on the validation set of RefCOCO (Yu et al., 2016), we find that ò, 50 is consistently reasonable threshold. Finally, we clarify that this protocol is only for the convenience of evaluation and is not our major concern.
A.1.3 GENERALIST INSTRUCTCV
For the generalist model, we primarily follow the experimental setup of InstructCV (Gan et al., 2024), which adopts InstructPix2Pix (Brooks et al., 2023) to conduct perception in the form of image editing. Although the original InstructCV also conducts image classification, its format of "turning the whole image red if the image matches a certain class" is less intuitive, and we find that including image classification causes large variances in the remaining three tasks. Therefore, we focus on depth estimation, semantic segmentation, and object detection. Specifically, we use NYUv2 (Silberman et al., 2012) for depth estimation, ADE20K (Zhou et al., 2017; 2019) for
16

---

Published as a conference paper at ICLR 2025
mantic segmentation, and COCO (Lin et al., 2014) for object detection. We mainly focus the contribution-aven aware timestep sampling for Table 4, with details Sec. A.2. We empirically re- weight the ratio of training samples from each dataset to 0.3, 0.3, and 0.4, respectively. The model trained for 20 epochs, with 100k iterations per epoch.
A.2 LEARNING OBJECTIVES: CONTRIBUTION-AWAN TIMESTEP SAMPLING
We mainly describe how estimate and use the e? defined in Sec. 3.1 to improve the training of diffusion models. The list of tasks are in Sec.
A.2.1 ESTIMATING of VALUES
Protocols. To avoid variance of timesteps, merge the 1000 steps from DDPM (Ho et al., 2020) into 10 groups, where the 100 timesteps using Marigold (Ke et al., 2024), we infer pre-trained Marigold model on NYUv2 (Silberman al., 2012) (654 samples) to get the results at intermediate denoising steps. The metric RMSE used. (2) For RIS task, estimate the results with IOU. The weights from SD1.5 are directly adopted in SD2.0 and SDXL. (3) For generalist perception with InstructCV (Gan et al., 2024), each from NYUv2 (Silberman et al., 2012), ADE20K (Zhou et al., 2017: 2019), and COCO (Lin al., 2014). We estimate of for each task separately and adopt the average of value as the shared sampling weight.
group share the same c/. (1) For the depth estimation
N 1,000 validation samples from RefCOCO (Yu et al., 2016) to
N validation samples
Estimated of Values. In the RIS experiments, discover that ==0 has extremely low prob- ability, while the earlier ones closer to 1=7 have large weights. However, we empirically round up their probabilities to 1% avoid any timesteps from insufficient training. A comparison of the probabilities between diffusion-ban -based formulation and empirical estimation from perception statis- ties (RIS and depth estimation) is in Fig. A. As clearly illustrated, (1) the weights estimated from IoU statistics are more unevenly distributed than diffusion weights, indicating the importance of the first few steps for multi-modal understanding. (2) The weights estimated for Marigold and gener- alist perception are smoother than RIS, which validates that the evolution of perception quality is unique property for each diffusion- based perception model.
100
Diffusion Formulation
Sampling Probability
RIS Stats
Marigold Stats
Generalist Stats
20
Uniform Sampling
999-900
899-800
799-700
699-600
599-500
1994400
399-300
299-200
199-100
99-0
400
TimeSteps
Figure A: Comparison of the sampling weights derived from normalized of, estimated by diffusion formulation or RIS/Marigold/Generiin perception statistics (Sec. 3.1). The dashed blue line de- notes the probability of uniform sampling from one of the ten timestamp groups.
A.2.2 ALGORITHMS
Algorithm A demonstrates the process for estimating the contribution factors of. and Algorithm B illustrates how the contribution factors are utilized during diffusion training via timestamp sampling-
A.3
DIFTUSION-TALLREED DATA AUGMENTATION
We introduce the detailed implementation of diffusion-tailored data augmentation (Sec. 3.2), which aims simulating the grining-denoising distribution shift during the training time.
17

---

Published as a conference paper at ICLR 2025
Algorithm of Estimation
Require: DDPM-trained Diffusion- Perception Model, Perception Quality Metric Q(-). Evaluate V samples with T intermediate steps, and obtain S =Qriisssss
2: Φ t- (UT++. -Qr+
Dummy initializing
3: Perform linear regression on Qo = + B +1+ and compute (R²)r+)
4: f=r/=7,7 ...... do
5: Perform linear regression on Qo. = + Σx pr
Qa the final result
6: Compute (R²),
b how well Qip is explained by (Qisheeer
7: -
(+-(2))+1 (R²),
8: end for
9: return (G)
Algorithm B Contribution-awat Timestep Sampling
Require:
Contribution factors (c
repeat
2:
3: I~-uulinomial((?....f
4: Take gradient descent step on VellE Be(v0,o vI 1
5:
until converge or reach the maximum iteration
A.3.1 DESIGNS AND IMPLEMENTATIONS
Depth Estimation. In the experiments of using Marigold (Ke et al., 2024) for depth estimation, we use Gaussian blur as data augmentation to simulate the rough prediction results at the early denoising steps. Concretely, our Gaussian blur has a kernel size of 31 pixels with the maximum intensity of 10 X/TT, where is the current timestep, and T 1000 is the maximum timestep in DDPM. An intuitive illustration Fig. 4a.
Considering the potential drift of predictions that can occur during the denoising steps, we apply augmentation on three different aspects during training: color, location, and shape. For color changes, we randomly adjust the color mask's ' brightness, contrast, and saturation with the maximum intensity of 0.2 x/ x1/T. For location changes, we randomly rotate, translate, or scale the mask region. The maximum rotation is 10 degrees, the maximum translation is 0.05 of the image scale, and the scale changes is confined between 0.95 and 1.05 times. For erasing, we randomly crop out parts of the mask with a scale between 0.01 and 0.05. The intuitive demonstration is in Fig. 4b.
RIS.
We acknowledge that our design of augmentation might not the optimal one. However, we mainly aim simulate the imperfect intermediate denoising results and illustrate the broad space of data augmentation for diffusion- based perception.
A.3.2 PREDICTION TARGET
During the training of diffusion models, it is common practice to adopt Gaussian noise as the objective. However, the integration of data augmentation introduces additional complexity, thereby altering the gradient direction of V vxx().. To account for the impact of data augmentation, we rede- fine the training objective as
0,50
Vaa
(Augment(zo.1) -nn)
(B)
C4
Despite this adjustment, empirical results indicate that predicting e' is still less effective than predict- ing xp. A potential explanation for this observation is that correcting the learning target introduces additional discrepancies between the training and denoising phases. In contrast, predicting xo helps mitigate this issue by directly predicting the final outputs, thereby enhancing model consistency across different phases. The ablation study presented Sec. B.3. In addition, we incorporate classifier free guidance (Ho & Salimans, 2022) when generating Xo. In accordance with the formu- lation in InstructPix2Pix (Brooks et al., 2023), we set wD 3.0 and w/ = 1.5.
18

---

Published as a conference paper at ICLR 2025
A.4 INTERACTIVITY WITH CORRECTIONAL PROMPTS
This section provides the detailed steps of how we convert the generative denoising process into interactive user interfaces. The details of our evaluation on RIS with an agentic workflow is also discussed.
A.4.1 FORMULATION OF CORRECTIONAL PROMPTS
We justify the formulation of our correctional prompts (See. 3.3) following the derivation in In- structPix2Pix (Brooks et al., 2023). We treat the correctional prompt D as an auxiliary condition alongside the image and referring 1), so the objective conditional probability for denoising is:
D,x, .I) P(D.D" [bx))((DD.Ix))((( P(D,D" ,1)
D(x|D,D .1)
(C)
By taking the logarithm and derivative of the conditional probability above, we have the score func- tion (Hyvärinen & Dayan, 2005) as below, corresponding to our correctional guidance in Eqn. 5.
(,log((x(DDD) .I) -V, V, l/
(D)
VxD /1x) V, P(D,D .xx).
A.4.2 PROMPTS AND WORKFLOW
When building the agentic workflow in Sec. 3.3, we leverage the following three steps as depicted in Fig. B, where our correctional guidance votes the masks for right predictions. Please note that the motivation of our workflow is to simulate the rough thinking process of how a human interacts and correct the predictions of referring segmentation.
(1) LL Captioning. We first prompt LLaVA (Liu et al., 2023b;c) to create detailed caption of each image, specifically the 1lava - v1.6 - vicuna - 13b, with the prompt showing in Fig. B. The special consideration of our prompt is to pay attention to both foreground and background objects, which both frequently appear in the referring segmentation dataset. After this step, we can input the image descriptions for foundation models to reason.
(2) Correctional Guidance Generation with GPT. Then we prompt GPT4 (Achiam et al., 2023) to analyze the captions and referring expressions to name the confusing objects for referring segmen- tation, which act as the correctional prompts. We acknowledge that this not an ideal agentic work- flow but is computationally feasible within our budget since each validation set of RefCOCO (Yu et al., 2016) has around 10,000 referring phrases. Specifically, we use gpt 40 2024 05 - 13 to conduct the prompts in the middle of Fig. 5. We explicitly use examples to guide GPT into reasoning the confounding objects for referring segmentation while providing the arguments. The output of the GPT will contain three confounding objects, namely the correctional prompts.
(3) Referring Mask Prediction. We independently predict the referring masks for each of the 3=3 correctional prompts, applying guidance weights of w/ = 1.5, wp = 2.0, and wD 330 in accordance with Eqn. 5. The final mask is derived from pixel-wise majority voting across the k= 3 masks, as depicted in Fig. B.
Step LarW Captizzigg
Steg Dennetioral Gulfarce Gecacation
Step Maak Gezeration nrighal Piwlttionn
apert cocortrr state slactinee entire Suethaatt,, you sfIt tii paldero and alouust The antere crter anggeir ancriv lnnitarcia cbject
aueic de ineerioe the acdrtoer
ther
Hhc dol orgge to staritane cauitle aros copatte sidde algcttten, Haaee ande acordin
Fasiglt start paioorrii
ane
contasinn cas ane talte it tangros then are taa
Sy Rt: Le1Sentitth
Lomediion Cher paapttakeeep
the alokn contuee dn rieee sentiee tee cets (ioos)) 5. (ouetiy cur war tall (16 (ntosoe
Ostgen senitceee
Maaco foo sale couid (000) insecttaniin the dary frost tall sell cne de cocluded are nencture cals cestoe festing rabien Crnnte onds hear sierite Westne
Crese Me sendete beft het 2iler of aetendimes
Whithheee Bag
Welod Mark
Clares, creairy Noat aeecuuzee
Figure B: Our workflow of generating correctional prompts shows the advantage of the interactivity of diffusion- based perception.
19

---

Published as a conference paper at ICLR 2025
B ADDITIONAL ABLATION STUDIES
B.I REFERRING IMAGE SEGMENTATION MASK FORMAT
As described in Sec. 2, we format referring image segmentation (RIS) as an image editing problem: painting the regions of target objects with solid red masks. This section analyzes why we chose red masks in our experiments.
Original Image
Hinary
Transparent Mask
Red Mask [ours]
Figure C: Comparison of mask encoding methods for RIS. Note that ground truth masks are used here for demonstration purposes.
Prediction
Post-processing
Transparent
Red (ours)
Figure D: Comparison of post-processing. The solid red mask simplifies the post-processing for evaluation, which also encourages to adopt it as the RIS format.
Solid or Transparent Masks. Previous studies that have adopted different formats: InstructDif- fusion (Geng al., 2023) utilizes transparent masks, InstructCV (Gan al., 2024) employs binary masks with RGB channels, while our strategy is a solid red mask, as visually compared in Fig. C. When trained with InstructPix2Pix (Brooks et al., 2023) for 20 epochs, the use of binary masks results in a sub-optimal result, achieving an oloU of 28. 18%, significantly lower than the 56.42% obtained by red masks. This performance gap likely attributable to the lack contextual infor- mation in the generated binary masks, which makes the grounding harder. As presented in Table 2, our red masks (InstructPix2Pix) achieve performance comparable to transparent masks (Instruct- Diffusion) with fewer training iterations, thereby underscoring the effectiveness of red masks. In addition, red masks facilitate simpler post-processing via straightforward pixel-wise thresholding, whereas applying thresholding to transparent masks produces excessively noisy results, shown in Fig. D. Although incorporating an additional U-Net (Ronneberger et al., 2015) for segmentation poot-processing might enhance the results (Geng et al., 2023), this introduces additional complexity to the evaluation process. Therefore, given the effectiveness and efficiency, we choose to use red masks our experiments.
Color of Masks. We additionally analyze the influence of colors on the RIS performance. These experiments train the baseline InstructPix2Pix and our enhancements with 40 epochs RefCOCO. As shown in Table A, our insights consistently improve RIS for all the scenarios. The red masks perform the best, so we chose as our default setting. We hypothesize that the influence color is similar to the discovery in visual prompt tuning (Shtedritski et al., 2023): distribution of images affect model's 's performance under different colors of visual prompts.
Red
Blue
Green
InstructPix2Pix
60.15
53.88
53.54
+Sampling+Auu
66.51
65.87
65.72
Table A: Comparison of mask colors for RIS.
20

---

Published as a conference paper at ICLR 2025
B.2 INTERMEDIATE DENOISING RESULTS
As discussed in Sec. 4.3, with contribution-aware timestep sampling and diffusion-tailoned data augmentation, we observe less pronounced decrease IoU. Since the final outputs of diffusion models are typically obtained at timestep 1= O,, we base main results on timestep 7 =0 for fair comparisons in Table 2. However, as indicated by Fig. 7, the IoU at timestep 200 is the highest among all timesteps. Hence, we also report results from this timestep in Table B, which show slightly better predictions than those at timestep 1=0. Further bridging this gap indicates better alignment between the denoising process and perception objective, and will be the future work.
RefCOCO
RefCOCO+
G-Ref
Timestamp
val
test-A
test-B
val
lest-A
test-B
val
test
1=0 0
66.86
67.39
63.72
55.35
58.72
48.45
55.85
57.05
202
67.34
68.36
64.44
56.12
59.87
49.06
57.41
57.90
Table B: Comparison on the oloU of RIS at intermediate denoising steps. The SD1 bassed RIS
models from Table 2 are used for this comparison.
B.3 TRAINING DATA: DATA AUGMENTATION FOR DIFFUSION MODELS
xo-prediction v.s. E-prediction. As mentioned in Sec. A.3.2, using corrupted input as data aug- mentation (Sec. 3.2) is sensitive to the prediction target of the scheduler in diffusion models. For example, the default e-prediction in DDPM (Ho al., 2020) might lead to problematic score func- tions when the input is not sampled from the ground truth trajectories, while xo to-prediction (Ramesh et al., 2022) or velocity prediction (Salimans & Ho, 2022). Here, the experiments are all conducted with the data augmentation proposed in Sec. 3.2. E-prediction refers to predicting the corrected noise e', as defined Eqn. B. Directly fitting Gaussian noise & does not produce reasonable results and not included in the table. The results indicate that prediction yields better performance. Further- more, the results from InstructPix2Pix demonstrate that switching to Xi prediiciiionn alone does not enhance performance, indicating that the improvements in InstructPix2Mask are from data augmen- tation rather than merely switching to xo-prediction.
Wimeestamp-depende Data Augmentation. In Table D, we investigate the impact of varying the intensity of data augmentation across timesteps. We compare the linearly increasing intensity with the constant intensity. The results indicate that the dynamic strategy outperforms the static one. This improvement likely due to the fact that tinestamp-depeneen augmentation aligns more closely with the original diffusion formulation by introducing less noise as approaches 0, since our goal is to simulate the distribution shift.
k E-prediction xo-prediction
Intensity oloU
InstructPix2Pix ADDP
56.42
53.39
constant
66.07
59.64
66.19
lincar 66.19
Table C: Analysis Training objectives.
Table D:
Augmentation Scales.
Analysis of Individual Augmentations We compare the three types of augmentation, color, shape, and location corruptions, to the ground truth masks proposed Sec. 3.2. As shown in Table E, we discovered that the augmentations have different effectiveness in enhancing diffusion-basee percep- tion models.
Color
Shape
Location
oloU
ADDP (w/o Aug)
64.0
Color Only
64.2
Shape Only
64.5
Location Only
65,9
ADDP (All Augs)
66.2
Table E: Comparison of augmentation types for RIS.
B.4 ADDITIONAL ANALYSIS ON MARIGOLD-BASEN DEPTH ESTIMATION
In addition to the analysis in Table 1, we analyze the separate effects of contribution-awee aware timestep sampling (Sec. 3.1) and tafiloed data augmentation (Sec. 3.2) for Marigold-based depth
21

---

Published as a conference paper at ICLR 2025
estimation. As shown in Table F, our proposed techniques from ADDP both improve Marigold step by step. Notably, our data augmentation enhances Marigold even though its perception qual- ity (Fig. 3) does not show significant drops RIS, indicating the vast existence of distribution shifts. This supports the effectiveness of our proposed techniques in addition to Sec. 4.3: both our contribution-aware timestep sampling (Sec. 3.1) and seffiusiin-taillored data augmentation (Sec. 3.2) are beneficial for diffusion- based perception.
Method
AbeRell
ETH3D
6.1
SeanNet AbsRell
6.7
AbsRelj
NYUv2
AbsRall
Diode
5.1
AbsRell
KITTI
6,1
AbsRell
Average
6.1
Marigold (Ke et al., 2024)
7.1
95.1
6.9
94.5
6.0
95.9
31.0
77.2
10.5
90.4
12.3
90.6
2.9
a -Sampling (Ours)
6.1
96.1
6.4
95.3
5.7
96.1
30.6
77.2
10.3
90.4
11,9
91.0
2.0
SSaplingg (Ouns)
6.3
96.1
6.3
95.6
56
96.3
29.6
77.5
10.0
90.6
11.6
91.2
Table F: Ablation analysis the effects of contribution timestep sampling ("Sampling") and cnffusion-tallore data augmentation ("Aug") for Marigold. Both of them contribute positively to diffusion- -ased perception.
B.5 COMPARISON WITH SAMPLING WEIGHTS FROM GENERATIVE STUDIES
Our ADDP leverages the e, estimated from the perception statistics to control the sampling of timesteps. Studies on diffusion models designed for generative tasks also notice the benefits of better loss scaling or probability scaling of timesteps. The comparison between our ADDP and their weights is in Table G. For fair comparison, we scale the probability of timestep sampling for all the methods and disable our data augmentations.
Method
oloU
Uniform (DDPM (Hlo al., 2020))
56.42
Estimated e, (Ours)
64.00
P2 (Choi et al., 2022)
57.90
Min-SNR (Hang al., 2023)
26.13
MLT (Go et al., 2024)
55.60
SpeedD (Wang et al., 2024)
57.89
Soft Truncation (Kim et al., 2022a)
57.49
Table G: Comparison with Sampling
Such an improvement primarily arises from better align- ment between our weights estimated from the perception statistics with the perception tasks, whereas the others are mainly considered from generative task perspective. This demonstrates the effectiveness and necessity of our investigation diffusion- based perception instead of fully relying the results from generative tasks as guidance.
Weights from Generative Models' Studies.
B.6 COMPARISON WITH AUGMENTATION STRATEGIES FROM GENERATIVE STUDIES
The studies of generative models also notice the distri- bution shift between training and denoising, termed as com-
Method
oloU
"exposure bias" (Ning et al., 2024). Therefore, pare the augmentation strategies with our method under liffusion-based perception setting. Since TADA (Park et al., 2023) primarily focuses on the intensity of data aug- mentation, we replace our linear intensity schedule with that from TADA while keeping the identical set of aug- mentations: color, shape, and location. As shown in Ta- ble H. our method outperforms these related approaches from generative task perspective, indicating the uniqueness of our diffusion- based perception in- vestigation.
Augmentation
64.00
w/ Augmentation (Ours)
66.19
TADA (Park et al., 2023)
65.76
Epsilon Scaling (Ning al., 2024)
64.13
Table H: Comparison with Augmenta- tion Strategies from Generative Mod- els' Studies.
B.7 INFLUENCE OF SAMPLING STEPS ON OUR OBSERVATIONS
When observing the behaviors of diffusion models for perception tasks (Fig. 1), we primarily fol- low the setting of InstructPix2Pix (Brooks al., 2023) and adopt 100 sampling steps. However, the number of sampling steps might influence the properties of a diffusion model. We analyze the number of sampling steps to check our observation. Specifically, we let an InstructPix2Pix base- line model trained on RefCOCO generate intermediate referring segmentation results at different sampling steps: 50, 100, and 200. The evaluation of IoU regarding timesteps is in Fig. E.
As shown in Fig. E, observations consistent across different sampling steps: (1) the contri- butions across timesteps are significantly uneven, with earlier steps contributing more than the later ones; (2) the distribution shift between training and denoising causes the performance drop later
22

---

Published as a conference paper at ICLR 2025
0.62
0.58
0.54
0.50
0.8
50 strge
100
980
900
800
600
400
200
TimeSteps
Figure E: Observations with different sampling steps.
denoising steps. When using fewer sampling steps, only the earlier steps suffer from insufficient diffusion denoising, but the performance quickly catches up after =00. These combined verified that our observed issues of diffusion-based perception consistent.
23


Reviews:
[{"review_id": "qXFZpWuVMU", "reviewer": "Reviewer_apx4", "summary": "The paper \"Aligning Generative Denoising with Discriminative Objectives Unleashes Diffusion for Visual Perception\" introduces a novel framework, ADDP, designed to enhance the performance of generative diffusion models on discriminative visual perception tasks. These tasks, such as depth estimation, referring image segmentation, and generalist perception, demand precise alignment with ground truth data, which has been a challenge for generative models.\n\nThis technique prioritizes earlier denoising steps, which have a more significant impact on final perception quality. By adjusting the learning objectives, ADDP ensures that the model focuses on the most critical stages of the denoising process. To mitigate the \"training-denoising distribution shift\" issue, ADDP employs task-specific data augmentation strategies, simulating the deviations that occur during the denoising process to improve the model's robustness and maintaining perception quality across different timesteps.\nADDP leverages the iterative nature of diffusion models to enable user interaction. By incorporating classifier-free guidance, users can provide corrective feedback to refine the model's output, offering a more flexible and interactive approach compared to traditional discriminative models.\n\nADDP has demonstrated significant improvements in depth estimation, referring image segmentation, and generalist perception tasks. By effectively aligning generative and discriminative objectives, ADDP bridges the gap between generative diffusion models and discriminative baselines, without requiring architectural changes or additional data. This work represents a significant step forward in the field of generative diffusion models, highlighting their potential as powerful tools for visual perception tasks.", "questions": "1. Could the authors provide additional insights into how the contribution factors $𝑐_𝑡^2$   for each timestep are estimated and why they differ for various perception tasks? Specifically, how does the estimation process differ between depth estimation and referring image segmentation (RIS)?\n2. Can the authors clarify how well the proposed augmentation strategy generalizes to other types of distribution shifts in diffusion-based perception tasks?\n3.  How effective is the system without human-generated prompts?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper introduces a novel framework, ADDP, addressing a previously unexplored gap in applying generative diffusion models to perception tasks. The paper's key contributions include a Contribution-Aware Timestep Sampling to prioritize early denoising steps for improved perception accuracy, as well as a Diffusion-Tailored Data Augmentation technique that simulates distribution shifts during the denoising process. The authors provide a Interactive User Interface to enable human-in-the-loop correction through the denoising process.\n\nThe paper demonstrates high quality through evaluation on various tasks, including depth estimation, referring image segmentation, and generalist perception and provides significant improvements over standard diffusion-based models without architectural changes.The authors provide ablation studies to justify the design choices and the individual contributions of each technique.\nThe paper is well-written and easy to understand, with clear structure, clear explanations of technical concepts, aided by diagrams and clear communication of key ideas.\n\nThe paper has significant implications in expanded model versatility, using for both generative and perception tasks and in possibilities for human-in-the-loop correction and user-guided generation. Overall, the paper presents a highly original, well-supported, and clearly articulated framework, expanding the utility of diffusion models into discriminative and perception tasks.", "weaknesses": "Not enough explanation about how the  contribution factors are derived and adapted to each perception task. It would be helpful if the authors could discuss the potential challenges or limitations of this augmentation strategy, particularly for tasks where shifts may not be well-simulated by data corruption.\n\nThe interactive correctional guidance using classifier-free guidance is a compelling feature, but  the authors could elaborate on how correctional prompts are formulated and whether they require manual input.The paper demonstrates improved results with diffusion-tailored data augmentation, but it is unclear whether specific types of augmentation (e.g., color, shape changes) have a greater impact than others without clarify which augmentations are most effective.\n\nWhile ADDP is demonstrated with specific models, it’s unclear how generalizable these methods are across different diffusion model architectures, such as conditional or latent diffusion models with varying noise schedules. Although ADDP is tailored for perception, it would be interesting to know if the authors have considered its potential for other tasks, such as text-to-image generation or other generative tasks with precision requirements.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zi3PqWYokS", "reviewer": "Reviewer_zLxB", "summary": "This paper investigates the performance gap between diffusion models and traditional encoder-decoder methods in perceptual tasks (e.g., Referring Segmentation) through validation experiments. It analyzes the varying contribution importance of different timesteps in perception through timestep-contribution analysis, leading to improvements in both training strategy and data aspects. Additionally, the paper demonstrates the potential of diffusion models for interactive referring perception tasks using prompt-based guidance agents.", "questions": "Please refer to the weaknesses section, particularly regarding:\n1. The justification for baseline comparison methodology\n2. The rationale behind including the user interface section in this paper", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. The methodology and motivation are presented with exceptional clarity, with comprehensive consideration of variants for each method, including:\n   - Various augmentation approaches\n   - Different definitions of $c_{t}^2$ in timestep-aware training\n\n2. The experimental section is thoroughly comprehensive, including:\n   - Validation experiments in the introduction\n   - Performance evaluations across multiple tasks (RIS, Depth Estimate, Generalist Perception)\n   - Detailed ablation studies\n\n3. The innovations stem from meaningful problem discovery and are non-trivial, with adaptive training and data augmentation for different timesteps representing logical explorations of the original problem.", "weaknesses": "1. The paper appears overly content-rich for a conference paper, resulting in some crucial discussions being relegated to supplementary materials:\n   - The revised objective for the augmented ground truth in Sec. B.3.2\n   - Intensity of augmentations across timesteps in Sec. B.3.1\n   These aspects are fundamental to understanding the methodology but lack explanation in the main text.\n\n2. The \"User interface\" section seems disconnected from the main methodological contributions and shows limited performance improvements (as shown in Table 3). The whole content of this paper might be better suited as two separate conference papers.\n\n3. Concerns about the baseline comparison: It's unclear whether the InstructPix2Pix baseline in RIS underwent equivalent training. The paper doesn't specify if the baseline was trained on specific RIS datasets, raising questions about the fairness of comparing a general image editing model with a task-specific fine-tuned model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AztDvpsRcq", "reviewer": "Reviewer_2CZ7", "summary": "This work proposes a method to bridge the gap between generative diffusion models and discriminative perception tasks by aligning the generative denoising process with perception objectives. One of its contributions is designing a loss function to reflect the varied contributions of timesteps. At the same time, it introduces diffusion-tailored data augmentation to simulate training-denoising distribution shifts and leverages the denoising process as an interactive interface for user correctional prompts. These enhancements improve performance without requiring architectural changes.", "questions": "The proposed methods demonstrate strong performance on zero-shot benchmarks, but how robust and generalizable are these models in real-world applications? Are there additional experiments or analyses that validate the model’s stability across different environments and conditions?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper presents a new way to make generative diffusion models work better for visual tasks by aligning them with discriminative tasks and gets SOTA performance.\n2. It explains the differences between generative and discriminative processes and offers specific solutions like adjusting timestep sampling and using data augmentation designed for diffusion.\n3. The method takes advantage of the interactive denoising process, allowing for correctional prompts and multiple rounds of interaction, which sets it apart from traditional models.", "weaknesses": "1. The data augmentation techniques used in this paper are quite standard for perception tasks and don't offer much innovation.\n2. Although the paper demonstrates improvements in certain tasks, the applicability of these methods to a wider range of perception tasks and datasets still needs thorough validation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Rk6Aubkbls", "reviewer": "Reviewer_ZiYy", "summary": "This paper aims to improve the performance of repurposing the diffusion models as the deterministic task models. First, the authors analyze the performance changes through the denoising process of diffusion models in an RIS task, and they find that later timesteps are more influential to the mIOU but there is a decreasing tendency in mIOU when approaching the smaller timesteps. They hypothesize the reason for this finding as 1) the unequalized contribution of denoising steps and 2) the training-denoising distribution shift. To deal with the first challenge, they propose loss scaling and timestep sampling methods, which reflect varied contributions of each timestep for deterministic tasks. For the second challenge, data augmentation strategies suitable for each task are utilized to reduce exposure biases. Finally, they argue that the generative process in diffusion models is beneficial in reflecting users' feedback on perception tasks and propose agentic workflows that conduct multi-round interactions. The various experiments are conducted to show the effectiveness of their method mainly focusing on improving Marigold, InstructPix2Pix, and InstructCV. Technical details and additional experimental results are mainly presented in the Appendix.", "questions": "Q1: Why is linear regression needed in estimating $c_t$​ instead of determining $c_t$ directly?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- This paper deals with deep explorations of the denoising procedure of diffusion models in visual perception tasks.\n- The main strength of this paper is the performance improvement. As shown in the results, the proposed methods offer substantial performance enhancement for various models.", "weaknesses": "- **W1-1) (Minor)** This paper introduces numerous techniques, but some details are obscured or inaccurately described. For example, in Eq. (2), $c_t^2$ is presented as a weight for each loss term based on the timestep, yet it is also used in the timestep sampling distribution in Fig. 2. \n- **W1-2) (Major)** Furthermore, I cannot find the details of the analysis in Fig. 1. It is unclear how the authors calculated mIOU scores during the sampling process. If these scores are derived using only the estimated $x_0$​ values partway through the sampling, they don’t reflect the final generated results. Using these incomplete samples to support the authors’ conclusions seems insufficient. In other words, simply examining the denoising behavior of partial generations does not fully represent the actual generation process. A more accurate approach would be to inspect denoising characteristics across the complete generation process, as this would better validate the claims.\n- **W2) (Major)** The technical contribution of this paper in the proposal of 1) a loss scaling and timestep sampling methods and 2) the dataset augmentation for each task can be trivial without comparing baselines. \n  - **W2-1)** The authors introduce loss scaling and timestep sampling methods based on their observations of uneven contributions to performance. In the context of loss-weighting and timestep-sampling strategies within diffusion models [1, 2, 3, 4, 5], several studies have shown that emphasizing the loss at later timesteps can enhance model performance. The core operation of the authors’ method appears similar to these existing techniques. Therefore, without direct comparisons to these established methods, it is difficult to be convinced of any distinct advantage offered by the proposed approach in deterministic task setups. I suggest adding the comparative results with these methods.\n  - **W2-2)** Similar to W2-1), [6] also deals with timestep-dependent data augmentation and it is necessary to compare this to validate the advantages of the proposed methods. Also, incorporating the data augmentation strategy for reducing exposure biases in diffusion models is already mentioned in [7]. In this regard, I suggest adding the comparative results with data augmentation works in diffusion model contexts.\n\n- **W3) (Minor)** The proposed methods require a manual setup for data augmentation and $c_t$.\n\n### **Reference**\n\n[1] Perception Prioritized Training of Diffusion Models, CVPR 2022\n\n[2] Efficient Diffusion Training via Min-SNR weighting Strategy, ICCV 2023.\n\n[3] Addressing Negative Transfer in Diffusion Models, Neurips 2023\n\n[4] A Closer Look at Timesteps is Worthy of Triple Speed-Up for Diffusion Model Training, Arxiv 2024.\n\n[5] Soft Truncation: A Universal Training Technique of Score-based Diffusion Model for High Precision Score Estimation, ICML 2022\n\n[6] TADA: Timestep-Aware Data Augmentation For Diffusion Models, Neurips 2023 workshop on Diffusion Models.\n\n[7] ELUCIDATING THE EXPOSURE BIAS IN DIFFUSION MODELS, ICLR 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)