Reconciling Visual Perception and Generation in Diffusion Models
Authors: Liulei Li
Yi Yang
Wenguan Wang
Venue: ICLR 2026 Poster (2026)
Keywords: Visual Perception
Image Classification
Object Detection
Semantic Segmentation
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We present GenRep, a unified image understanding and synthesis model that jointly conducts discriminative learning and generative modeling in one training session.

Abstract: We present \textsc{GenRep}, a unified image understanding and synthesis model that jointly conducts  discriminative learning and generative modeling in one training session. By leveraging  Monte Carlo approximation, \textsc{GenRep} distills distributional knowledge embedded in diffusion models to guide the discriminative learning for visual perception tasks. Simultaneously, a semantic-driven image generation process is established, where  high-level semantics learned from perception tasks can be used to inform image synthesis, creating a positive feedback loop for mutual boosts. Moreover, to reconcile the learning process for both tasks, a gradient alignment strategy is proposed to symmetrically modify the optimization directions of perception and generation losses. These designs empower \textsc{GenRep} to be a versatile and powerful model that achieves top-leading performance on both image understanding and generation benchmarks.

Published as a conference paper at ICLR 2026
RECONCILING VISUAL PERCEPTION AND GENERATION IN DIFFUSION MODELS
Liulei Li', Yi Yang', Wenguan Wang't
The State Key Lab of Frain-Machine Intelligence, Zhejiang University
ABSTRACT
We present GENREP, a unified image understanding and synthesis model that jointly conducts discriminative learning and generative modeling in training session. By leveraging Monte Carlo approximation, GENREP distills distributional knowledge embedded in diffusion models guide the discriminative learning for visual perception tasks. Simultaneously,, a semantic-driven image generation process established, where high-level semantics learned from perception tasks can be used to inform image synthesis, creating a positive feedback loop for mutual boosts. Moreover, to reconcile the learning process for both tasks, a gradient alignment strategy is proposed to symmetrically modify the optimization directions perception and generation losses. These designs empower GENREP to be a versatile and powerful model that achieves top-leading performance on both image understanding and generation benchmarks. Our code is available at GENREP
INTRODUCTION
Broadly, there are two fundamental goals in the field of computer vision: visual understanding which extracts meaningful cues from scenes, and image generation which aims to create new visual contents. The former is typically solved through visual representation learning, i.e. transforming raw pixel data into features or embeddings that can capture high-level semantics (Bengio al.. (0033) in a discriminative manner. This leads to strong performance in downstream tasks such as visual recognition and semantic segmentation. On the other hand, image generation relies on generative modeling and emphasizes the learning of underlying patterns and distributions within data (Croitoru) al. 2023). thereby enabling the synthesis of new samples that faithfully resemble the original one.
Since visual understanding and synthesis have long been addressed with different paradigms, most existing work excels in either synthesizing realistic outputs or interpreting input data, but seldom do both on unified basis. This brings several drawbacks: @ Representations learned in a discriminative manner for visual perception tasks often generalize poorly to unseen patterns (Pourpanah et al.) 2022) and overlook fine-grained details (Huynh & Elhamifar 2020). This stems from their narrow focus on decision boundary between classes (Jebara 2012), rather than capturing the underlying data distribution like generative models. 2 Modern generative models such as GANs (Goodfellllw lt al. 2014)) or diffusion models (Sohl-Dickstein et al.) Rombach et al. (022) exhibit a lower-level understanding of semantics due to the reliance on low-level reconstruction loss (Zhang al. 2023a). As a result, they tend to underperform discriminative approaches in scene understanding tasks. The divergence in technological protocols for image understanding and synthesis diffuses the research endeavors, and hinders innovations and insights achieved in one paradigm to enhance the other.
This stimulates us to rethink the perceived incompleteness in discriminate-basee based representation learning and generative modeling, and seek to bridge this gap by preserving both synthesis and understanding abilities within the same model. Our idea is motivated by the observations that: diffusion models facilitate downstream visual perception tasks (Zhao et al. 2023 Yang & Wang 2023;; ii) high-quality discriminative representations accelerate the generative learning of diffusion models (Yu et al. 2024). This reveals the potential commonality of representations learned via two paradigms, and forms the basis for devising a unified visual understanding and generation framework.
Corresponding Author: Wenguan Wang.

---

Published as a conference paper at ICLR 2026
Distributional Knowledge Transfer
Collic:
Step
Ruiking 0.32
Semantic Driven
Dachshund: 0:26
Step
Noise Free
Add Noise
Figure Unified image Understanding and synthesis within diffusion models: a) semantic driven image generation; b) distributional knowledge transfer from diffusion models for visual perception.
Building on this premise, we introduce GENREP, which reconciles the learning processes of down- stream visual perception tasks and image generation in diffusion models while enabling the mutual benefits, First, to enhance visual understanding, GENREP leverages the distributional knowledge captured by generative modeling (Fig. aa)). Assuming the irttusioonbbseed image generation can capture object distributions p(ælyy) with class labels as conditional inputs, we approximate it through Markov Chain Monte Carlo where intermediate outputs during reverse diffusion are utilized as repre- sentative samples. As such, the class-wise posterior probabilities p(y) x) can be retrieved via Bayes' theorem, which serves as supplementary guidance for the discriminative learning of downstream perception tasks. Second, to enable image generation informed by visual understanding, we propose semantic-driven generation learning strategy to guide image synthesis with high-level semantics derived from perception tasks the noise distribution on semantic embeddings delivered by the perception branch, encouraging the generated images to faithfully reflect the desired content. Finally, GENREP fosters the iteratively mutual enhancement between visual perception and image generation through joint optimization strategy, which aligns the gradient of the generation loss with the direction of the perception loss at each training step. This aims to harmonize the learned representations for both tasks, so as to deliver a single and cohesive model capable of effectively tackling both visual perception and generation.
It refines the reverse diffusion process by conditioning
By exploring the interplay between visual perception and generation, GENREP offers several com- pelling advantages over disjointed paradigms: First, unlike prior diffusion- basedd work (Zhu et al.) 2024a Xu et al. 2023 2024) that often compromises the generation ability for visual understanding. our approach holds superior performance for both tasks. Second, it moves from purely deterministic modeling to joint discriminative and generative learning, thereby demonstrating notably low expected calibration errors and benefits perception under vocabbacyy scenarios. Third, through joint optimization and gradient alignment, a feedback loop is established, where the unique strengths of two learning paradigms can be leveraged to enhance each other. Fourth, the construction of shared feature space for both perception and generation tasks facilitates the emergence of more robust and transferable representations, which improves the generalization across a variety of downstream tasks.
For thorough examination, we experiment GENREP on both visual perception and generation tasks. It consistently demonstrates remarkable performance across benchmarks, including 57.8 for out-of- the-distribution generalization on ObjectNet (Barbu et al.) 2019). 92.9 for fine-grained classification on CUB-200 (Wah al. 2011), 0.057 AbsRel for monocular depth estimation on NYUv2 (Silber- man et al.] 2012). 34.7/54.6 mloU for open/close-set semantic segmentation on ADE-20K (Zhou 2017) 7) and 56.5/36.0 AP for open-vocabulary object detection on MS COCO (Lin al. /LVIS v v1.0 et al., 2019). in leverage of advanced diffusion architectures such as CNN- based Latent Diffusion Models (LDM) (Rombach et al., 2022) and ViT-based Diffusion Transformers (DiT) (Peelees & Xie 2023). Furthermore, GENREP improves the image generation quality, achieving
al.
2014
top-leading performance on CelebA-H((Kanne et al. 2017), LSUN-Churcoe Yuu et al. 2015). and
ImageNet(Deng et al. 2009) under the class-conditioned setup.
2 RELATED WORK
Diffusion Models for Visual Perception. Work applying diffusion models for downstream perception tasks can be broadly classified into two categories. The first treats the prediction process as a denoising task, where noisy inputs refined to recover clean ground truth. This paradigm contains noise-to-
box for object/action detection (Chen et al., 20236] (Ho et al. 2023) Nag et al. 2023); noise-to-point
for object tracking (Xie et al. 2024b) and pose estimation (Shan al. 2023 Feng et al.) 2023):
2

---

Published as a conference paper at ICLR 2026
and noise-to- toise-to-map which directly synthesizes colorful masks for depth estimation (Ke et al. segmentation (Li et al. 2023c Ji et al., 2023), , and anomaly detection (Zhang et al., 2023b). On the
2024).
other hand, recent research highlights that diffusion models undergoing large-scale pre gra-training exhibit certain representation abilities, enabling them to extract meaningful features for downstream visual
perception tasks (Zhao et al. [023] Yang & Wang 2023 Kondapaneni et al. 2024; Chen et al. 2025).
On this basis, significant trend has emerged, where the diffiusion models are utilized as backbones for image classification (Clark & Jaini 2023), image segmentation (Zhu et al.) 2024a] Xu et al. 2023),
3D Object Detection (uu et al. 2024), numan-objec interaction detection (Li et al. (2024b). and referring video object segmentation (Zhu et al. (024b). This also facilitates correspondence matching by calculating cosine similarity between diffusion features Tang et al. 2023; Zhang et al. 2023a).
Though demonstrating promising performance, these work often sacrifices the image generation
capabilities of models. In contrast, our work seeks to enable both image generation and understanding within the same model, while the distribution knowledge is explicitly transferred from diffusion models to inform and guide the discriminative learning process.
Joint Discriminative and Generative Learning. Substantial research has emerged to combine the strengths of both discriminative and generative learning even before the deep learning era. To address the data- data-intensive and limited generalization inherent in purely discriminative methods, researchers incorporated generative techniques to manage noisy inputs (Jaakkola & Haussler 1999) and unlabeled samples (Bernardo et al.; 2007). Similarly, there are interests in the "discriminative training' of gener- ative models mitigate mismatches between real and model-specified data distributions (Tu, Holub & Perona 2005 Yakhnenko et al. 20055 Chen et al. 2024). More recently, complementary learning methods simultaneously learn data distributions leveraging advanced generative models, such as Generative Adversarial Networks (GANs)(Xu et al.] 2020), Variational Autoencoders (VAEs) (Chen
al. 2023a Kolesnikov et al., 2022), and Gaussian Mixture Models (Liang et al. (022).. resulting in generative classifiers for discriminative tasks. Additionally, generative models are trained to capture the distribution of known classes open-vocabulary recognition which facilitates the recognition of novel classes (Perera et al. 2020), and tuning diffusion models with discrimina- tive adapter has proven effective in improving the alignment between text prompts and generated images (Qu et al. 2024). However, most existing work merely focuses the one-direction enhance- ment, e.g., discriminative learning to improve image generation or generative learning to enhance visual perception. In contrast, GENREP builds feedback loop enable mutual boosts between generative and discriminative learning, while within a unified model.
Unified Image Understanding and Synthesis. In recent years, there has been a notable surge in integrating image comprehension and generation within the same model. The first research direction built upon LLMs, and distinguishes itself by implementing image generation in an auto-regressive DoaneerDDong et al., (2024), delivering a Tokenizer-Dettkenizzo framework that enables token-by- token generation of multimodal outputs for synthesis and understanding tasks (Zhu et al. 2023 Ge
et al. [024 Fang et al. 2024 Li et al. 2024a] Wu et al. (025). Another line of work utilizes diffusion models, which frames perception tasks as the generation of colorful maps (Qi et al. 2024] Wang el al., 2024 Yang al. 2025) or text embeddings (Huang et al.) 2023). Though retaining the generative capability, this kind of solution still falls in low-level reconstruction, lacking high-level modeling on semantics. A notable exception performs discriminative learning using features from diffusion models, and update the generative component in a mean teacher manner (Zheng et al.) 2024). However, the image generation capability in this approach is primarily optimized for augmenting perception tasks, leaving its potential for general-purpose image synthesis largely unexplored. To overcome these limitations, GENREP respects and harnesses the unique characteristics of both paradigms. Specifically, it enhances representation learning for perception tasks with generative modeling to consummate the decision boundary, and uses high-level semantics obtained from diseriminative learning instruct the sampling stage (i.e., reverse diffusion) of image synthesis.
3 METHODOLOGY
3.1 PRELIMINARY: DIFFUSION MODELS FOR VISUAL PERCEPTION
2024 have demonstrated that features processed by latent diffusion models contain certain visual cues, which can be used to tackle complex perception tasks. Specifically, given an input sample x and its corresponding textual class label y, x is first.
Empirical studies (Zhoo et al. 2023 Yu et al.
3

---

Published as a conference paper at ICLR 2026
encoded into the latent space using the encoder & of pre-trained generator (Le., VQGAN), yielding a = [ After a single noise-free forward pass through the denoising network Eg with the encoded label co(y) as the condition,
obtain ã co( O, Co (y)) which extracts features distinctive for the given class y. Following (Zhao et al. 2023 the extracted features are enhanced by aggregating intermediate outputs of four decoder blocks cg at different own- wwnnssapling factors with FL et al., 2017), so as to deliver the final input representations for task-specific decoders. GENREP follows this pipeline to enable downstream visual perception, and further seeks to bridge the historically parallel image generation and understanding tasks, yielding a single model capable of addressing both tasks.
3.2 GENREP: RECONCILE VISUAL PERCEPTION AND IMAGE GENERATION
In this section, we first detail how to distill knowledge of visual distributions from diffusion models to enhance discriminative visual perception, and then outline the perception-inspire image generation learning, emphasizing how gained insights from visual perception are utilized to improve generative capabilities. Finally, we address the reconciliation of these dual learning objectives, illustrating how GENREP yields balanced and unified model proficient in both tasks.
Generative Visual Perception Learning. Assuming the diffusion models can capture visual distribu- tions via generative modeling, the conditional distribution for sample æ (i.e., p( (x)ly) can be derived with class label y as the conditional input. Since the exact computation of p(æ) y) is intractable, we approximate it following the principle of Markov Chain Monte Carlo (MCMC) (Geyer 1992). Specifically, we observe that during the reverse diffusion process, a sequence of intermediate states
2T + TT-1 7 > To naturally constitutes a non-stationary Markov chain (Norris 1998). The transition kernel pe p=( (x)) at each step can be parameterized as:
po (0:z-11as (z.) = (2 (( (Z, t), ()))))o oo( t)),
(1)
where N is Gaussian distribution, and 140, dg are networks parameterized by 0 to predict the mean 14 and variance o, for N at time t. This structure is analogous to MCMC methods where samples are drawn from a sequence of transitions rather than independent draws from a static distribution.
However, leveraging the chain directly for approximation introduces two challenges: i) the initial states of the reverse chain correspond to nearly pure noise, which would degrade the approximation quality; and ii) samples drawn sequentially from a Markov chain are temporally correlated, which conflicts the independent assumption that strengthens Monte Carlo methods. To mitigate these issues, we adopt two techniques, known as burn-in and thinning, commonly used in MCMC (Link & Eaton 2012). For the burn-in period, we discard the first uninformative steps of the chain. For thinning. we reduce the correlation by selecting samples at a fixed interval of k-th step. Empirical evaluations show that k- 2 provide good trade-off between sample quality and quantity. Following the practice of MCMC, we then leverage the trajectory of a single reverse diffusion process to estimate p(æ) yy):
p(zly) R= ET=NN(ss Ht,y Ft,y),
(2)
where T represents the total number of reverse diffusion steps after burn-in and thinning. This allows for highly efficient estimation by avoiding the need to generate a large set of fully denoised samples 2o (i.e., massive full reverse diffusion runs) for each condition as required by standard Monte Carlo methods. The posterior distribution p(ylaz) (az) is then computed substitute into the Bayes' theorem:
p(y)p(æ)) y)
p(y (r) Evey p(azy")
(3)
where p(y) = 1/ is assumed to be uniformly distributed. This a standard choice (Kingma et al., 2014 Tran et al. 2019) which creates a non-informative prior that allows the posterior distribution to be shaped primarily by the learned likelihood p(aly), which contains the rich distributional knowledge we aim to distill. While (Li al.] 2023a) also uses diffusion models to estimate conditional distributions with Monte Carlo methods, it approximates log p(æ y) by averaging the noise prediction error derived from the forward diffusion process. In contrast, this work directly approximates p(æ> y)) by averaging Gaussian PDF values predicted during the reverse generative process (i.e., Eq. [). The motivation (i.e., correct conditioning enjoying accurate noise prediction vs patterns of samples generated with the same conditions being consist), computational basis (i.e., noise prediction error vs Gaussian probability densities), and diffusion process (i.e., forward noising-adding vy reverse
4

---

Published as a conference paper at ICLR 2026
Egen distil
p( at y
Gradient Aligument
Ma
base
Seml ar
buse fo
110 07
Figure 2: The overall pipeline of GenRep ($3.2). First, our proposed generative visual perception learning (i.e., region on transfers distribution knowledge from diffusion models. in leverage of intermediate denoised images samples to approximate the conditional distribution (i.e., p(=: ly)) Second, semantic-driven generation learning (i.e., region on utilizes the semantic embeddings (i.e., ( learned from visual perception tasks to guide the image generation process. Finally, gradients generated by these two type losses are aligned via [6 to deliver a unified model that excels both image generation and synthesis tasks.
generation) are all different. Our approach requires significantly fewer diffusion steps (i.e., 1000 vs 200), and thus excels in computational efficiency.
To inform the discriminative perception process with distributional knowledge, minimize the Bullback-Leibler (KL) divergence between p(y x)) computed by generative modeling and ((yl2)) obtained by applying the so f ftmax operation to the output logits of task-specific decoders:
Lgen distil = DxL ( = Eye P(yl log p(y
The final objective combines Egen distil with the conventional discriminative loss Laise for each percep- tion task (e.g., Cross Entropy loss for classification, Smooth & loss for bounding box regression):
distil
Cpercept
Laise + Lienndistil-
Eq 5 bridges the gap between generative and discriminative frameworks with Leeentiitill regularizer. Unlike standard discriminative loss that encourages overconfident predictions, Lienndistil leverages the generative likelihood create a soft posterior that faithfully reflects ambiguity for similar classes.
Cemantic-Driven Generation Learning. To enhance image generation, we propose semantic-aware noise adjustment strategy which leverages high-level semantics learned through visual perception. Assuming there is well-trained denoising network optimized for visual perception (i.e., Epereept in Eq the intermediate output representation which contains rich semantic cues is denoted as Dsem- During the reverse diffusion process, the noise parameters (i.e., mean (40 and variance ag in Eq [ are dynamically modulated according to Feem- Specifically, the prediction for pi is augmented as:
N t, Fsem) = H4
hase (æx, t) + fiem (Ie, Zsem),
(6)
where hhue (x t) is the baseline mean value predicted by the underlying diffusion model at time step t and frem is semantic correction function implemented as:
I (xt; (esem)
W!r. concat(a:,, Rsem);
with W. being learned weight matrix. Conceptually, Na determines the primary denoising direction, and adjusts it by steering the denoising trajectory towards the desired semantic target encoded in Tsem- On the other hand, the variance aee controls the uncertainty of reverse diffusion. Dynamically modulating dg allows the model to adaptively control the influence of semantic guidance. The variance prediction network is correspondingly augmented as:
del (a t, Øsem) = 0
buse (xx t) (1 + f (X
(
where of (2 t) is the baseline variance predicted following the pipeline of improved DDPM D & Dhariwal 2021), and (Ixsm) is a semantic scaling factor computed as:
f (Zeem) = MLP(Z
where MLP maps the semantic embedding to a scalar. Intuitively, a positive for (Fsem) increases variance, encouraging broader exploration towards the semantic target when the current sample is far.
5

---

Published as a conference paper at ICLR 2026
Conversely, a negative value reduces variance, promoting finer refinement near the target semantics. The overall training objective combines the standard reconstruction loss for latent diffusion models (I.e,, FLDM) and the representation alignment loss (i.e., Crep align):
Egenera LLDM + Crepaalign
(10)
where Liepaalinn minimizes the cosine similarity between æ and Zsem as in Yu et al. 2024). During inference, the explicit semantic representation z Eq ol Eq can be directly replaced with the current noisy sample as the enhanced denoising network has already learned to capture necessary semantic cues with the knowledge preserved in model weights.
Gradient Alignment for Weight Merge. To reconcile the optimization of visual perception loss (i.e., percepp 5 and image generation loss (i.e., Lgenera in Eq. T0p within a single model, a gradient alignment mechanism is introduced to address potential conflicts between the two training objectives by symmetrically modifying their respective gradients according to the severity of the conflict. Let V pereepe and V geuura denote the gradients derived from Epereepe and Egeneras respectively. We decompose each gradient into components parallel and orthogonal to the other gradient:
Lpprreppe Lggneraa V Epercopts
V Vlll genera
[epercplll 1
L L.ggara
V
V
Lgenera
VC Vegenera
genera>
(11)
V percopp
Cg V Egenera
V percopt
V percept
Epereept
V
pereeppe
(12)
V
Here the parallel components capture movements in the same opposite gradient directions of two tasks, while the orthogonal components are gradient directions that not affect the objective the
other task tast (Farajtabar et al. 2020). The aligned gradients for both tasks are then reconstructed as:
V aligned
L genera + a VV genera
(13)
gererr
V aligned percept
V
C percppp
C ppercept-
(14)
This approach selectively dampens gradient components parallel to the other, while fully preserving the orthogonal ones. Consequently, non-confficting information is retained, and interference smoothly reduced based on the conflict level. Here a is adaptive retention factor governing the damping and defined according to the cosine similarity between two original gradients:
Cerreept
Lggarra
cos_sim
(15)
|
Lgenera
We want when cos sim = 1 (no damping needed) and to decrease towards 0 as cos sim + -1 (maximum damping). A simple and effective formulation is the scaled and shifted power function:
= ((cos_sim + 1)/2)*. Here, h = 2 is a hyperparameter controlling the sharpness of the damping. The final gradients used for the model update are a weighted sum of aligned gradients:
symmetric = wp
V
aaiinndd pereept
Wi
V
aaligned
genera
(16)
where WP = 0.7 and wy = 0.3 scale task weights. As such, GENREP effectively manages gradient conflicts during joint learning, and encourages balanced optimization across two objectives.
3.3 IMPLEMENTATION DETAILS
Network Architecture. GENREP is built upon LDM-8 (RombaRommmhhheee et al. 2022 (Peblees &
Xie| (023) with 200/250 DDPM steps during inference. To ensure fair comparisons with existing work, the diffusion model is initialized with weights pretrained ImageNet (Deng al., 2009) and the LAION dataset (Schuhmann et al.) 2022) 2021), respectively. This facilitates comparison against conventional diseriminative-baved perception models pretrained on ImageNet-1K. 1K, and other diffiusion-batt based perception approaches pretrained on large-scale image-text pairs. Task-specific decoders are designed following representative work with details provided in Appendix.
Training Strategy. GENREP is first optimized with solely task-specific perception loss (Epercepe 5q
yielding denoising network eø which encodes high-level semantic cues into the intermediate output 2, (resulting in ). Subsequently, the images generation loss (Cgavera Eq steps in. A new denoising network _nniieed copied from sem is optimized where at each training step:

---

Published as a conference paper at ICLR 2026
Table 1: Quantitative results for fine- line-grained bird classi-
Table 2: Quantitative results for monocular depth esti-
fication on CUB-200 (Wah al. [201]) esst and OOD generalization on ObjectNet (Barbu et al. 2019) test.
mation on NYUV2(Silbenna) et al.
2012 val.
Model
Pre
Training 5g
AbbaRel
Model
Pro-Training CUB- 200 ObjectNet
BTS
et
2019
ImageNet
0.8620.996
0.108
ResNer-50
et
ImageNet
84.5
37.2
P3Depth
Pari ai
4122
ImageNet 0.8980.996
0.104
Swin.S
10 el
214
ImageNet
88.2
38,9
TransDepih
Zhap
ImageNet 0.9000.996
0.106
Con-NeXt-S
ImageNet
88.5
39.5
AdaaBins
Biaal cl
2021
ImageNet
0.9030.997
0.103
HorNes-S
K30 al
2024
ImageNet
89.1
39.3
DPT (annt 2011
ImageNet 0,9040,998
0.110
GENRET LM
ImageNet
90.5
51.1
RinsFomfer
40740
ImageNet 0.9250.997
0.094
Swin B
Liu el
202
ImageNet
90.6
40.3
ZoeDepth [Tail cl
2021
ImageNet 0.9510.999
0.077
ConxNeXi-B
ImageNet
90.9
40.9
GEEEEE mm
ImageNet 0.96600:999
'00700"
HorNos-B
K30 ct 21 2024
ImageNet
91.2
40.6
GENREPac nx
ImageNet 0.968
0.064
GENKKF 13
ImageNet
92.1
54.7
VPD (Zha al.
LAION-5B 0,9640.999
Clark al. Clark & Jaini 2023
LAION-SB
91.3
49.4
BCoDepth Farm et au
2024
LAION-SB 0.9780.997
0.059
LAION S8
91.8
52.5
DepthAnything Taing.
62M Depth 0.984 1.000
0.056
GEN KEN
LAION-SB
92.9
57.8
GF GENKEFIDA
LAION:5B 5B 0.9821.000
0.037
Gradients for both ppereepi and gerner are computed using the same input image;
Gradients are aligned according to Eq Eq
16 to update weights of attention blocks in e emiilied,
Parameters of sem are updated in a momentum manner: @sem + mnf=cm + (1 m Junifiodd with
0.999. This maintains stable semantic features Usem for image generation learning.
4 EXPERIMENT
Datasets. The experiments are conducted on nine datasets. Concretely, CUB-200 (Wah et al. 2011) for fine-graine bird classification, ObjectNet (Barbu et al.] 2019) for out-of-the--istribbe generation, NYUv2(Silhermaa et al., 2012) for depth estimation, ADE20K (Zhou et al.) 201 7) for
open/close set semantic segmentation, MS COCO(Lin et al., 2014) and LVIS Guptaa et al. 2019)
for open-vocabulary object detection, ImageNet (Deng et al. 2009). CelebA-HQ (aobb--QQKarrass
(Karras et al. 2017).
and LSUN-Churches Yu et al.) 2015) for image generation. Details are provided in Appendix.
Evaluation Metrics. For fine- fine-grained classification on CUB-200 and out-of-the-distributio general- ization ObjectNet, we report the top-1 accuracy. For depth estimation. following (Li et al. 2024c).
we report the accuracy under the threshold (ò, < 1.25', i = 1, 3) and mean absolute relative error (AbsRel). For close-set and open- vecebblaayul semantic segmentation, following (Xu et al., 2022a Cho et al. 2024), GENREP is trained on the training set of ADE20K and COCO Stuff, respectively. The evaluation is conducted on the validation set of ADE20K with the mloU score reported. For open-vocabulary object detection, consistent with prior work (Zang et al. [222 Wu et al. 2023a). we report the AP50 score for base, novel, and all classes, denoted AP So. AP AP50 on MS COCO, APr. APc, c. AP,, and AP for rare (novel), common, frequent, and all categories on LVIS. For image generation, following (Rombach et al. 2022). we report the FID, IS, precision, and recall scores.
Training. For visual classification, we use standard data augmentation techniques, including random cropping and horizontal flipping during training to enhance generalization. The Adam W optimizer with learning rate of le-3 and a weight decay of 0.05 is adopted. The batch size is set to 256 with 50 epochs training. For depth estimation, following (Li et al.) 2024c), we train the model for 40K steps with a batch size of 16, and use the Adam optimizer with learning rate of le and weight decay of 5e -2, For semantic segmentation, following (Cheng et al. 2022, Xie et al. 2024a Li et al. 2023b), the model is optimized with Adam W using learning rate of 2e and weight decay of le for 80K iterations on COCO Stuff for open- vocabulla,,,, and 160K iterations on ADE20K for close-set. Input images are cropped to the 768 x 768 pixels. For open-vocabulary object detection, following (Zhao et al., 22024] ZZhan et al. 2024). we train GENREP for 40K steps on MS COCO and 80K steps on LViS vI.0 with batch size of 16, and adopt the Adam optimizer with learning rate of 2e -3 and a weight decay of le Given the simultancous training of both perception and generation in GENREP, the training procedure for image synthesis is aligned with perception tasks.
4.1 COMPARISON WITH STAT
Visual Recognition. As shown in Table I benefited from the low-level modeling ability of diffusson models, GENREP yields remarkable performance on the bird classification task which prioritizes fine-graines cues. Furthermore, the knowledge transfer from diffusion models allows GENREP to achieve a t accuracy of 54.7% 14.7%/57.8%% % for out-of- distributtton generalization on ObjectNet, surpassing prior diffusion-based methods (Clark & Jaini 2023
Liet al. 2023a) by 8.4 3.4%
7

---

Published as a conference paper at ICLR 2026
Table 3: Quantitative results for closed-set semantic
Table 4: Quantitative results for voca-lacyy seman-
segmentation on ADE20K (hhou al. 2017) val.
tic segmentation on ADE20K (Zhou al.
2017 val.
Madel
Pre. Triiningg
Backhone
Intol[
Model
Pre- Traniin
Backbone
mioUT
3e+ Chen et al.
2018
ImugeNet
ResNett101
45.5
CroupViT [Xu et
20224
ImageNet
VITSS
10.6
OCRNet Yuan
2021
ImageNett
HRNet. W48
45.7
ZegFormer Ding ei
2048
ImageNet
VIT-
18.0
UperNet Xtad al
A018
ImugeNeei
Swin-S
47.7
SimBaseline of
212210
ImageNer
VIT-B
20.5
SegMenter
Struec ot
ImageNet
DefT-B
47.1
PACL Muknoti et
2023
ImageNet
VIT-B
31.4
K-Net
Zianx et
A121
ImageNet
Swin-S
49.7
OVS. (Liine 2)
ImageNer
VIT-B
24,8
SegFormer
Aie et mi
2021
ImageNer
MrT-RS
50.0
CAT Sey (C CCn
ImageNer
VIT-B
27.2
Musk2[oormer lemoosiion
InugeNet
Swin
51.3
SED www.ent
ImageNet
31.6
GENREP
ImageNat
LDM
"52.2"
G GENKET
ImageNaa
LEM
32.5
GENREP
ImageNet
52.8
GENREP
ImageNet
DIT
34.1
SDN TTn et al
2022
LAION-3B
LDM
51.1
OVDil (Krzzzia et al.
2024
LAION-SB
LDM
14.1
VPD Zi0
LAION-5B
LDM
53.7
ODISE XU
2023
LAION-5B
LDM
28.7
GENKET
LAION-5B
LDM
54.6
GENKET
LAION-5B
LDM
34.7
Table 5: Open-vocabulary detection on MS COCO Lin al.
2014
and LViS v1.0 Gupta et al.
2019) val.
Visual- Linguistic
MS COCO
LViS v1.0
Model
Models
AP
APt
APaa
AP,
AP, 1
AP,
AP
Guu al. 2022
CLIIP
27.6
59.9
51.2
16.1
20.0
28.3
22.5
OV-DETR Zang et al 2022
CLIP
29.4
61.0
52.7
17.4
25.0
32.5
26.6
OADP
al
CLIP
35.6
55.8
50.5
26.0
28.7
26.0
BARON Wü et al.
27450
CLIP
34.0
60.4
53.5
23.2
29.3
32.5
29.5
CORA Wi BI
20230
CLIP
35.1
35.5
35.4
28.1
BIND Zhang
2024
CLIP
36,3
$4.7
50.2
29.4
30.6
33.5
31.4
SAS Det
Zhao el
2014
CLIP
37.4
58.5
53.0
29.1
32.4
36.8
33,5
GENRRF
TDM
41.8
60L8
3ST
"30.S
33.3
"34.""
GENREP
DiT
43.4
61.5
56.5
31.6
33.7
37.3
36.0
Table 6: Quantitative results for class-conditional image
Table Quantitative results for image generation on
generation on (Deng et al.
2009) 256
256.
CelebA-HQ and LSUN-Churches 256x256.
Model
FIDD
IST
Precision
Recallf
Model
FIDI
Precision 1 Recalf
BigGAN (Buck et al
2018
6.95
171.4
0.87
0.28
CH
StyleGAN
2.30
265.1
0.78
0.53
PUGAN
Karras
ADM Dhariwal val & Nichor
2021
4.59
186.7
0.82
0.52
MeMt a 2022
7.16
158.7
TOM
1 KKomhach
W11
55.11
0..7
00399
Jabr et al
2025
3.42 182.0
+GENKET
3.84
0.78
0.54
VDM++ Kinema Gao
2125
2.12
267.7
LSUN-Churches
BDDM
Kunbach el
2012
7.77
201.6
0.84
0.35
PGGAN
2017
6.42
+GENKEP
6.92 213.7
0.89
0.44
StyleGAN Karras et a 2019
4.21
DIT Peebles & Xie
2023
2.27
278.2
0.83
0.57
LDM-8 Rombach
4.02
0.64
0.52
+GENKRP
2.09 283.8
0.88
0.58
+GENKET
3.12
0.69
0.58
Depth Estimation. For depth estimation, as shown in Table 2 GENREP achieves an impressive score of 0.064 in term of AbsRel. This verifies our core design conduct both generative and discriminative learning. Moreover, after initializing weights from Stable Diffusion pretraining on LAAON--B((cchil mann al. 2022), GENREP achieves comparable performance to DepthAnything (aai al. 2024) which is pretrained on 1.5M labeled and 62M unlabeled depth samples.
Semantic Segmentation. A detailed comparison of GENREP against top-leading approaches for semantic segmentation is provided in Tables 3 Built upon LDM (Rombach et al. 2022). GEN- REP achieves 52.2 %54.6 6.6% mloU for close-set semantic segmentation on ADE20K, beating all competitors. Moreover, for open- vocabulayyy semantic segmentation, our method delivers 6.0 % gain over ODISE (Xu et al. 2023). Leveraging DiT & Xie 2023) as the backbone observes similar trends, and builds new SOTA on two setups.
GENREP demonstrates remarkable accuracy over existing work for open- vocabulayyyl object detection on MS COCO (e.g., 41.8% % v.s. 37.4% in terms of AP and LViS v1.0 (e.g., 30.5% % v.s. 29.1 in terms of AP,). When using the Transformer-ban based diffusion models (i.e., DiT (Peebles & Xie 2023)). the performance boosts to 43.4% 4 AP and 31.6 % AP,.
Object Detection. As shown in Table 5
(Deng Image Generation. Image generation results on ImageNet (Deng et al. 2009), CelebA-HO (arras et al. 2017), and
As seen, GENREP boosts the performance to new SOTAs across metrics, proving the effectiveness of the overall design.
Yu et al. 2015) presented in Tables 6 7
4.2
QUALITATIVE RESULTS
Fig presents visualization results for visual perception ADE20K, NYUv2, MS COCO, and for image generation on ImageNet, CelebA-HQ, LSUN-Churches It can be observed that GENREP could effectively handle challenging scenarios, while synthesizing high-quality images.
8

---

Published as a conference paper at ICLR 2026
fork
dinning ta
pizza
laptop
gotted
mouse
Figure 3: Visualization results for image understanding on ADE20K (Zhou et al.
2017), NYUv2 Sil
berman et al. 2012), MS Coco et al. 2014), and for image generation on ImageNet (Deng et al.
2009) LSUN- Churchess Yu et al. 2015).
Table Analysis of essential components in GENREP.
Table 11: Analysis expected calibration error (ECE).
Generative Vis. Semantic-DD Gradient Align.
ObjectNet
CUB-200
ADE20K
MS COCO
mloUf FID
Perception Generation
0.237
0.095
0.484
0,382
45.4
27.8
13.27
0.208
0.076
0.425
0.343
47.8
30,9
12.96
44.1
25.6
7.45
Table 12: Strategies for semantic-driven generation.
49.4
31.5
7.23
51.1
32,5
6.92
Noise Adjust.
Lippulign
Top
minU/
FDD.
49.6
30.9
7.16
Table 9: Analysis of the thinning interval
50.3
31.5
7.38
32.5
Interval &
ObjectNet
CUB-200
ADE20K
MS COCO
1
50.3
89.2
30.7
53.5
Table 13: Analysis of directions for gradient alignment.
2
51.1
32.5
55.1
Gradient Align.
Top-11
mloLi
FIDJ
3
51.3
90.7
31.8
54.6
4
50.5
90.3
31.5
53.5
geten in Eg
48.7
Villiee
5
48.9
90.0
30.9
52.2
Vinnerss Eq
50.1
32.5
6.92
Table 10: Analysis of burn-in sample number m.
Table 14: Analysis of feature robustness on ObjectNet.
25
50
100
125
Model
(Clcan)
t=10
t=20
t=50
ObjectNet ADE20K
50.4
51.1
50.2
48.8
47.6
Swin- Trin-Trranffrmer
40.3
23.1
11.5
4.6
32.3
32.5
32.1
31.3
29.2
GENREP
51.1
48.7
44.5
37.2
4.3 DIAGNOSTIC EXPERIMENTS
For in-depth analysis, we conduct ablative studies with LDM (Rombach et al. 2022) as the denoising network. Unless otherwise specified, all experiments GENREPLDM pretrained ImageNet.
Key Component Analysis. We investigate the essential designs of GENREP, í.e., generative visual perception learning, serinnic-driven generation learning, and gradient alignment for weight merge in $3.2 in Table First, our generative visual perception learning strategy proves to be broadly effective visual perception tasks, yielding notable performance improvements. Second, with semantic-driven generation learning, GENREP delivers promising gains for the image generation task. Third, after combining them (i.e., row #3), both image generation and understanding tasks enjoy further boosts, which reveals positive feedback loop is established. Finally, with gradient alignment unify the optimization direction, GENREP achieves the best performance on all three datasets.
Thinning Interval. We analyze the impact of varying thinning intervals for MCMC approximation in Table 9 seen, setting i: 1, i.e., using all intermediate samples for approximation yields moderate improvement over the baseline (row #1 in 8). When & = 2, GENREP enjoys large performance gain. However, further increasing leads to decline in performance. This because larger à reduces the number of available samples and leads to a high-variance distributional estimate, indicating the trade-off between inference efficiency with fewer samples and approximation accuracy.
Burn-in Phase. We examines the impact of dsicarding first samples during reverse diffusion that are heavily noised (i.e., the burn-in strategy in standard MCMC) in Table [0] with the thinning interval A 2. Empirically, sufficiently noisy initial samples while retaining enough samples to support reliable estimation.
find that
50 provides favorable balance, which removes
Confidence Calibration. We evaluate the expected calibration enror (ECE) for predictions output by discriminative visual perception heads. ECE quantifies the alignment between predicted probabilities and the true likelihood of outcomes, serving crucial metric for assessing model reliability. As

---

Published as a conference paper at ICLR 2026
Table 15: Runtime comparison of closed-set semantic segmentation models on ADE20K val.
Trainable
Training Time
Inference Speed (FPS)
Method
Backhone
mloU
Params (M)
(GPU Hours)
V++pLabV3+ CChen et 2018
ResNet-101
83
14.2
45.5
SETR Zheng el al.
2021
ViT-L
308
623
9.7
46.2
UperNee(XXaao Xino et al.
2018
Swin-S
81
104
15.2
47.7
MaskFormer Cheng 2021
Swin-S
63
53
19.6
49.8
GENREP (perception only)
LDM
79
12.6
49.3
GENREP
LDM
54
87
12.6
52.2
shown in Table I1 the incorporation of generative distillation loss (i.e., Lgonndistil in Eq.4) leads to substantial reduction in ECE. This also indicates distribution knowledge in diffusion models can be effectively transferred with Lgen distil to improve the reliability of discriminative models.
Semantic-Driven Generation. We examine the impact of semantic-awan noise adjustment and representation alignment (i.e., frep aliga) in Table 12 The results demonstrate that both techniques independently contribute to improved generation quality. After combining them together, the FID score shows significant improvement, highlighting the complementary nature of these designs.
Gradient A Alignment. We probe different gradient alignment strategies in Table T3 As seen, while projecting perception loss in the direction of generation loss (i.e., Vplieeped obtains better image generation performance, there significant drop in perception performance. After balancing the trading off, we adopt symmetric strategy which treats both tasks equally during conflict resolution (i.e., V calioned and performs better in perception tasks while maintaining good generation quality.
Representation Robustness. To probe whether GENREP preserves good representation under noisy inputs, we provide with latents corrupted by = 10, 20, and capabilities forward diffusion steps. The results summarized in Table 14 offer empirical evidence for the robustness of learned representations, which stems directly from our model design. The perception module uses the denoising network the backbone, which is trained extract semantic structure from noisy input, and remains effective when operating on corrupted latents. Furthermore, the conditional distribution p(æly) for knowledge distillation aggregates noised states throughout the reverse diffusion. This encourages the model to learn noise-tolerant features that are predictive of the correct semantic labels.
4.4 RUNTIME ANALYSIS
We present detailed runtime analysis Table 15 It is important emphasize that GENREP is designed as truly unified model that simultancously masters visual perception and image generation within a single training process. The competitors, in contrast, are optimized exclusively for segmenta- tion. From this unified perspective, the efficiency of GENREP is remarkable. With a total training cost 87 GPU hours, it not only learns a strong image generator but also delivers a SOTA perception model that achieves 52.2% mloU, surpassing all listed specialist models. To further isolate the cost of our proposed generative distillation, we compare the full GENREP to a perception-one variant that removes the MCMC-based approximation. As seen, the full model incurs modest 10.1 1% increase in training time, yet elevates mloU by significant 2.9 points.
5 CONCLUSION
In this work, we reconcile visual perception and image generation within a unified model, termed GENREP. This leads to joint diseriminative and generative learning, where the unique properties of both paradigms are preserved and utilized to enhance each other. To achieve an optimal state for both image understanding and synthesis tasks, a gradient alignment strategy is proposed to pull close the weights optimized for two tasks. Empirical results suggest that GENREP achieves superior performance on six perception benchmarks, and greatly improves the image generation ability. Beyond the strong empirical results, our framework naturally inherits the flexible multimodal conditioning capabilities from LDM. This positions GENREP as a promising foundation to reconcile multimodal understanding and generation in unified model, key direction for future work.
10

---

Published as a conference paper at ICLR 2026
Acknowledgement. This work supported by National Science and Technology Major Project (No. 20232D0121300), National Natural Science Foundation of China (No. 62372405), Zhejiang Provincial Natural Science Foundation of China (No. LD25F020001), Beijing Natural Science Foundation (1.252036), Fundamental Research Funds for the Central Universities (226-2025-00057), Jiangsu Provincial Scientific Research Center of Applied Mathematics (No. BK20233002), and CCF-Tencent Open Fund.
This paper explores the reconciliation of visual perception and generation in diffusion models. It does not introduce new ethical concerns beyond those well established in the community. We do not identify any specific risks that warrant ethical review. For the potential misuse in deepfake generation, we encourage responsible deployment and support discussions on policy and regulatory frameworks to ensure the ethical application of generative models.
Ethics Statement.
Reproducibility. GENREP is implemented in PyTorch and trained on four Tesla A100 GPUs. Testing carried on the same machine. Our code is available at GENNEE
REFERENCES
Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Joshua Tenenbaum, and Boris Katz. Objectnet: large-scale bins-condd dataset for pushing the limits of object recognition models. In NeurIPS, 2019.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE TPAMI, 1798))179881825 2013.
JM Bernardo, MJ Bayarri, JO Berger, AP Dawid, D Heckerman, AFM Smith, and M West. Generative or discriminative? getting the best of both worlds. Bayesian statistics, 8(3):3-24 2007.
Shariq Farooq Bhat, Ibraheem Alhashim, and Peter Wonka. Adabins: Depth estimation using adaptive bins. In CVPR, 2021.
Shariq Farooq Bhat, Reiner Birkl, Diana Wofk, Peter Wonka, and Matthias Müller. Zoedepth: Zero-shot transfer by combining relative and metric depth. arXiv preprint arXiv:2302. /2288, 2023.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. In ICLR, 2018.
Guikun Chen, Xia Li, Yi Yang, and Wenguan Wang. Neural clustering based visual representation learning. CVPR, 2024.
Jiaqi Chen, Jiachen Lu, Xiatian Zhu, and Li Zhang. Generative semantic segmentation. In CVPR, 2023a.
Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder- decoder with atrous separable convolution for semantic image segmentation. In ECCV, 2018.
Mu Chen, Liulei Li, Wenguan Wang, and Yi Yang. Diffvsgg: Diffusion-driven online video scene graph generation. In CVPR, 2025.
Shoufa Chen, Peize Sun, Yibing Song, and Ping Luo. Diffusiondet: Diffusion model for object detection. In ICCV, 2023b.
Bowen Cheng, Alexander G Schwing, and Alexander Kirillov. Per-pixel classification is not all you need for semantic segmentation. In NewrIPS, 2021.
Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked- attention mask transformer for universal image segmentation. In CVPR, 2022.
Seokju Cho. Heeseong Shin, Sunghwan Hong, Anurag Arab, Paul Hongsuck Sco, and Seungryong Kim. Cat-seg: Cost aggregation for open- apen-vocabulary semantic segmentation. In CVPR, 2024.
Kevin Clark and Priyank Jaini. Text-to- toaee diffusion models are zero-shot classifiers. In NewrIPS, 2023.
11

---

Published as a conference paper at ICLR 2026
Florinel-Alin Croitoru, Vlad Hondru, Radu Tudor lonescu, and Mubarak Shah. Diffusion models in vision: A survey, IEEE TPAMI, 45(9):10800-1886) 2023.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale
hierarchical image database. In CVPR, 2009.
Prafulla Dhariwal and Alex Nichol. Diffusion models beat gans on image synthesis. In NeurIPS, 2021.
Jian Ding, Nan Xue, Gui-Song Xia, and Dengxin Dai. Decoupling zero-shot semantic segmentation. In CVPR, 2022.
Runpei Dong, Chunrui Han, Yuang Peng, Zekun Qi, Zheng Ge, Jinrong Yang, Liang Zhao, Jianjian Sun, Hongyu Zhou, Haoran Wei, et al. Dreamllm: Synergistic multimodal comprehension and creation. In ICLR, 2024.
Rongyao Fang. Chengqi Duan, Kun Wang, Hao Li, Hao Tian, Xingyu Zeng, Rui Zhao, Jifeng Dai, Hongsheng Li, and Xihui Liu. Puma: Empowering unified milm with multi-granule visual generation. arXiv preprint arXiv:2410. 13861, 2024.
Mehrdad Farajtabar, Navid Azizan, Alex Mott, and Ang Li. Orthogonal gradient descent for continual learning. In AISTATS, 2020.
Runyang Feng, Yixing Gao, Tze Ho Elden Tse, Xueqing Ma. and Hyung Jin Chang. Diffpose: Spatiotemporal diffusion model for video-based human pose estimation. In ICCV, 2023.
Yuying Ge, Sijie Zhao, Jinguo Zhu, Yixiao Ge, Kun Yi, Lin Song, Chen Li, Xiaohan Ding, and Ying Shan. Seed-x: Multimodal models with unified multi-granularity comprehension and generation. arXiv preprint arXiv:240..1336, 2024.
Charles J Geyer. Practical markov chain monte carlo. Statistical science, pp. 473 483, 1992.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NeurIPS, 2014.
Xiuye Gu, Tsung-Yr Yi Lin, Weicheng Kuo, and Yin Cui. Open-vocabulary object detection via vision and language knowledge distillation. ICLR, 2022.
Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: dataset for large vocabulary instance
segmentation. In CVPR, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CVPR, 2016.
Cheng-Ju Ho, Chen-Hsuan Tai, Yen- Yu Lin, Ming-Hsuan Yang, and Yi-Hsuan Tsai. Diffusion-ss3d: diffusion model for semi-supervised 3d object detection. In NewrIPS, 2023,
Jonathan Ho, Chitwan Saharia, William Chan, David Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. 23 2022.
Alex Holub and Pietro Perona. A discriminative framework for modelling object classes. In CVPR, 2005.
Runhui Huang, Jianhua Han, Guansong Lu, Xiaodan Liang, Yihan Zeng. Wei Zhang, and Hang Xu. Diffdis: Empowering generative diffusion model with cross-modal discrimination capability. In ICCV, 2023.
Dat Huynh and Ehsan Elhamifar. Compositional zero-shot learning via fine-grained dense feature composition. In NeurIPS, 2020.
Tommi S Jaakkola and David Haussler. Exploiting generative models in discriminative classifiers, In NeurIPS, 1999.
Allan Jabri, David J Fleet, and Ting Chen. Scalable adaptive computation for iterative generation. In ICMI,, 2023.
12

---

Published as a conference paper at ICLR 2026
Tony Jebara. Machine learning: discriminative and generative, volume 755. Springer Science & Business Media, 2012.
Yuanfeng Ji, Zhe Chen, Enze Xie, Lanqing Hong, Xihui Liu, Zhaoqiang Liu, Tong Lu, Zhenguo Li, and Ping Luo. Ddp: Diffusion model for dense visual prediction. In CVPR, 2023.
Laurynas Karazija, Iro Laina, Andrea Vedaldi, and Christian Rupprecht. Diffusion models for open-vocabulary segmentation. ECCV, 2024.
T Karras, S Laine, and T Aila. A style-based generator architecture for generative adversarial networks. IEEE TPAMI, 43(12):4217 42288221172, 2021.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv: 2017.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In CVPR, 2019.
Bingxin Ke, Anton Obukhov, Shengyu Huang, Nando Metzger, Rodrigo Caye Daudt, and Konrad Schindler. Repurposing diffusion-based image generators for monocular depth estimation. In CVPR, 2024.
Diederik P Kingma and Ruiqi Gao. Understanding diffusion objectives as the elbo with simple data augmentation. NeurIPS, 2023.
Diederik P Kingma, Danilo J Rezende, Shakir Mohamed, and Max Welling, Semi-supervised learning with deep generative models. NeurlPS, 2014.
Alexander Kolesnikov, André Susano Pinto, Lucas Beyer, Xiaohua Zhai, Jeremiah Harmsen, and Neil Houlsby. Uvim: unified modeling approach for vision with learned guiding codes. In NeurIPS, 2022.
Neehar Kondapaneni, Markus Marks, Manuel Knott, Rogério Guimaraes, and Pictro Perona. Text- image alignment for diffusion-based perception. In CVPR, 2024.
Jin Han Lee, Myung-Kyu Han, Dong Wook Ko, and II Hong Suh. From big small: Multi-scale local planar guidance for monocular depth estimation. arXiv preprint 1rXiv:1907.10326, 2019.
Alexander C Li, Mihir Prabhudesai, Shivam Duggal, Ellis Brown, and Deepak Pathak. Your diffusion model is secretly a zero-shot classifier. In ICCV, 2023a.
Hao Li, Changyao Tian, Jie Shao, Xizhou Zhu, Zhaokai Wang, Jinguo Zhu, Wenhan Dou, Xiaogang Wang, Hongsheng Li, Lewei Lu, et al. Synergen-vl: Towards synergistic image understanding and generation with vision experts and token folding. arXiv preprint arXiv:2412.09604, 2024a.
Liulei Li, Wenguan Wang, and Yi Yang. Logicseg: Parsing visual semantics with neural logic learning
and reasoning. In ICCV, 2023b.
Liulei Li, Wenguan Wang, and Yi Yang. Human-object interaction detection collaborated with large relation-driven diffusion models. In NeurlPS, 2024b.
Zhenyu Li, Xuyang Wang. Xianming Liu, and Junjun Jiang. Binsformer: Revisiting adaptive bins for monocular depth estimation. IEFE TIP, 2024c.
Ziyi Li, Qinye Zhou, Xiaoyun Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xie. Open-vocabulary object segmentation with diffusion models. In ICCV, 2023c.
Chen Liang, Wenguan Wang, Jiaxu Miao, and Yi Yang. Gmmseg: Gaussian mixture based generative semantic segmentation models. NeurlPS, 2022.
Feng Liang, Bichen Wu, Xiaoliang Dai, Kumpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu. Open-vocabulay semantic segmentation with musk-adapted clip. In CVPR, 2023.
13

---

Published as a conference paper at ICLR 2026
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr
Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
Tsung- Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. CVPR, 2017.
William A Link and Mitchell J Eaton. On thinning of chains in meme. Methods in ecology and evolution, 3(1):112-115, 2012.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie.
A convnet for the 2020s. In CVPR, 2022.
Chenlin Meng, Kristy Choi, Jiaming Song, and Stefano Ermon. Concrete score matching: Generalized score matching for discrete data. In NeurlPS, 2022.
Jishnu Mukhoti, Tsung-Yi Yu Lin, Omid Poursaeed, Rui Wang, Ashish Shah, Philip HS Torr, and Ser-Nam Lim. Open vocabulary semantic segmentation with patch aligned contrastive learning. In CVPR, 2023.
Sauradip Nag, Xiatian Zhu, Jiankang Deng, Yi-Zhe Song, and Tao Xiang. Difftad: Temporal action detection with proposal denoising diffusion. In ICCV, 2023.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In ICML, 2021.
James R Norris. Markov chains. Number 2. Cambridge university press, 1998.
Vaishakh Patil, Christos Sakaridis, Alexander Liniger, and Luc Van Gool. P3depth: Monocular depth estimation with a piecewise planarity prior. CVPR, 2022.
Suraj Patni, Aradhye Agarwal, and Chetan Arora. Ecodepth: Effective conditioning of diffusion models for monocular depth estimation. In CVPR, 2024.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In ICCV, 2023.
Pramuditha Perera, Vlad Morariu, Rajiv Jain, Varun Manjunatha, Curtis Wigington, Vicente Ordonez, and Vishal M Patel. recognition. In CVPR, 2020.
feature representations for open-set
Farhad Pourpanah, Moloud Abdar, Yuxuan Luo, Xinlei Zhou, Ran Wang, Chee Peng Lim, Xi-Zhao Wang, and QM Jonathan Wu. A review generalized zero-shot learning methods. IEEE TPAMI, 45(4):4051-4070, 2022.
Lu Qi, Lehan Yang, Weidong Guo, Yu Xu, Bo Du, Varun Jampani, and Ming-Hsuan Yang. Unigs: Unified representation for image generation and segmentation. In CVPR, 2024.
Leigang Qu, Wenjie Wang. Yongqi Li, Hanwang Zhang, Liqiang Nie, and Tat-Seng Chua. Discrimi- native probing and tuning for text-to-image generaattnntttooonnnnnoooooo In CVPR, 2024.
René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. In ICCV, 2021.
Yongming Rao, Wenliang Zhao, Yansong Tang, Jie Zhou, Ser Nam Lim, and Jiwen Lu. Hornet: Efficient high-order spatial interactions with recursive gated convolutions. NeurIPS, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, 2022.
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:21 1..02 2021.
14

---

Published as a conference paper at ICLR 2026
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. In NeurIPS, 2022.
Wenkang Shan, Zhenhua Liu, Xinfeng Zhang, Zhao Wang, Kai Han, Shanshe Wang. Siwei Ma, and Wen Gao. Diffusion-based 3d human pose estimation with multi-hypothesis aggregation. In ICCV, 2023.
Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and support inference from rgbd images. In ECCV, 2012.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In ICML.
Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid. Segmenter: Transformer for semantic segmentation. In ICCV, 2021.
Haoru Tan, Sitong Wu, and Jimin Pi. Semantic diffusion network for semantic segmentation. In NeurIPS, 2022.
Luming Tang, Menglin Jia, Qianqian Wang, Cheng Perng Phoo, and Bharath Hariharan. Emergent correspondence from image diffusion. In NeurlPS, 2023.
Toan Tran, Thanh-Tour Toan Do, Ian Reid, and Gustavo Carneiro Bayesian generative active deep learning. In ICML, 2019.
Zhuowen Tu. Learning generative models via discriminative approaches. In CVPR.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd birds-200-2011 dataset. 2011.
Chaoyang Wang, Xiangtai Li, Lu Qi, Henghui Ding, Yunhai Tong, and Ming-Hsuai Yang. Semflow: Binding semantic segmentation and image synthesis via rectified flow. In NeurIPS, 2024.
Luting Wang, Yi Liu, Penghui Du, Zihan Ding, Yue Liao, Qiaosong Qi, Biaolong Chen, and Si Liu. Object-aware distillation pyramid for open-vocabulary object detection. In CVPR, 2023.
Size Wu, Wenwei Zhang, Sheng Jin, Wentao Liu, and Chen Change Loy. Aligning bag of regions for open-vocabulary object detection. In CVPR, 2023a.
Xiaoshi Wu, Feng Zhu, Rui Zhao, and Hongsheng Li. Cora: Adapting clip for open-vocabulary detection with region prompting and anchor pre-matching. In CVPR, 2023b.
Yecheng Wu, Zhuoyang Zhang, Junyu Chen, Haotian Tang, Dacheng Li, Yunhao Fang, Ligeng Zhu, Enze Xie, Hongxu Yin, Li Yi, et al. Vila-u: unified foundation model integrating visual understanding and generation. In ICLR, 2025.
Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In ECCV, 2018.
Bin Xie, Jiale Cao, Jin Xie, Fahad Shahbaz Khan, and Yanwei Pang. Sed: A simple encoder-decoder for open- open-vocabulary semantic segmentation. In CVPR, 2024a.
Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: simple and efficient design for semantic segmentation with transformers. In NeurIPS, 2021.
Fei Xie, Zhongdao Wang, and Chao Ma. Diffusiontrack: Point set diffusion model for visual object tracking. CVPR, 2024b.
Chenfeng Xu, Huan Ling, Sanja Fidler, and Or Litany. 3difftection: 3d object detection with geometry-aware diffusion features. In CVPR, 2024.
Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong
Wang. Groupvit: Semantic segmentation emerges from text supervision. In CVPR, 2022a.
15

---

Published as a conference paper at ICLR 2026
Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello. Open- vocabulary panoptic segmentation with text-to-image diffusion models. In CVPR, 2023.
Mengde Xu, Zheng Zhang, Fangyun Wei, Yutong Lin, Yue Cao, Han Hu, and Xiang Bai. A simple baseline for open-vocabulary semantic segmentation with pre-trained rision-language model. In ECCV, 2022b.
Yanwu Xu, Mingming Gong, Junxiang Chen, Tongliang Liu, Kun Zhang, and Kayhan Batmanghelich. Generative Generative-diseriiinan discriminatiiee complementary learning. AAAI, 2020.
Oksana Yakhnenko, Adrian Silvescu, and Vasant Honavar. Discriminatively trained markov model for sequence classification. 2005.
Lehan Yang, Lu Qi, Xiangtai Li. Sheng Li, Varun Jampani, and Ming-Hsure Yang. Unified dense prediction of video diffusion. In CVPR, 2025.
Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. In CVPR, 2024.
Xingyi Yang and Xinchao Wang. Diffusion model representation leamner In ICCV, 2023.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv: 150.0355000.3655, 2015.
Sihyun Yu, Sangkyung Kwak, Huiwon Jang, Jongheon Jeong, Jonathan Huang, Jinwoo Shin, and Saining Xie. Representation alignment for generation: Training diffusion transformers is easier than you think. arXiv preprint arXiv:2410.06944 2024.
Yuhui Yuan, Xilin Chen, and Jingdong Wang. Object-contextual representations for semantic segmentation. In ECCV, 2020.
Yuhang Zang, Wei Li, Kaiyang Zhou, Chen Huang, and Chen Change Loy. Open-vocabulary detr with conditional matching. In ECCV, 2022.
Heng Zhang, Qiuyu Zhao, Linyu Zheng, Hao Zeng, Zhiwei Ge, Tianhao Li, and Sulong Xu. Exploring region-word alignment in built-in detector for open-vocabulary object detection. In CVPR, 2024.
Junyi Zhang, Charles Herrmann, Junhwa Hur. Luisa F Polanía, Varun Jampani, Deqing Sun, and Ming-Hsuan Yang. tale of two features: stable diffusion complements dino for zero-shot semantic correspondence. In NeurlPS, 2023a.
Wenwei Zhang. Jiangmiao Pang, Kai Chen, and Chen Change Loy. K-net: towards unified image segmentation. In NewrIPS, 2021.
Xinyi Zhang, Naiqi Li. Jiawei Li, Tao Dai, Yong Jiang, and Shu-Tar Xia. Unsupervised surface anomaly detection with diffusion probabilistic model. In ICCV, 2023b.
Jiawei Zhao, Ke Yan, Yifan Zhao, Xiaowei Guo, Feiyue Huang, and Jia Li. Transformer-base dual relation graph for multi-label image recognition. In ICCV, 2021.
Shiyu Zhao, Samuel Schulter, Long Zhao, Zhixing Zhang, Yumin Suh, Manmohan Chandraker, Dimitris N Metaxas, et al. Taming self-training for open-vocabulary object detection. In CVPR, 2024.
Wenliang Zhao, Yongming Rao, Zuyan Liu, Benlin Liu, Jie Zhou, and Jiwen Lu. Unleashing text-to-image diffusion models for visual perception. In ICCV, 2023.
Shuhong Zheng, Zhipeng Bao, Ruoyu Zhao, Martial Hebert, and Yu- Yu-Xiong Wang. Diff-2-in-l: Bridging generation and dense perception with diffusion models. In ICIR, 2024.
Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip HS Torr, et al. Rethinking semantic segmentation from requence-to-sequence perspective with transformers. In CVPR, 2021.
16

---

Published as a conference paper at ICLR 2026
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In CVPR, 2017.
Jinguo Zhu, Xiaohan Ding, Yixiao Ge, Yuying Ge, Sijie Zhao, Hengshuang Zhao, Xiaohua Wang, and Ying Shan. Vl-gpt: A generative pre-trained transformer for vision and language understanding and generation. arXiv preprint arXiv:2312.09251, 2023.
Xiaoyu Zhu, Hao Zhou, Pengfei Xing, Long Zhao, Hao Xu, Junwei Liang, Alexander Hauptmann, Ting Liu, and Andrew Gallagher. Open Open- apen-vocabulary 3d semantic segmentation with text-to-image diffusion models. In ECCV, 2024a.
Zixin Zhu, Xuelu Feng, Dongdong Chen, Junsong Yuan, Chunming Qiao, and Gang Hua. Exploring pre-trained text-to-video diffusion models for referring video object segmentation. In ECCV, 2024b.
17

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 DECLARATION OF LLM USAGE
The LLM was used solely for grammar checking and did not contribute to the core methodological design or the originality of the research.
A.2 LIMITATIONS
One potential limitation of GENREP is its computational cost, which introduces a trade-off between model performance and inference efficiency. Our reliance on a diffusion model backbone results in a lower inference speed compared to highly specialized perception architectures. As detailed in Table GENREP operates 12.6 FPS for semantic segmentation, whereas models like MaskFormer achieves 19.6 FPS and DeepLabV3+ achieves 14.2 FPS. This may constrain the usage of GENREP in latency-sensitive applications, such as real-time analysis or autonomous systems. This trade-off motivated by the substantial benefits unified approach provides, including 2.3% mloU improvement over MaskFormer and, crucially, stronger generalization to work aligns with the growing trend of using large-scale generative models unlock new capabilities in visual understanding, which often involves an initial focus on performance over efficiency. We consider the optimization of unified models a vital direction for future research. Promising avenues include knowledge distillation to yield lightweight architectures, developing more efficient diffusion sampling techniques tailored for perception, and model quantization. Bridging this efficiency gap will benefit the deployment of powerful unified perception and generation models in practical scenarios.
13
data. Our
Furthermore, in diffusion models, the mean of the data distribution far more dominant than the variance (Nichol & Dhariwal (021). Consequently, the learned variance can be less precise. Our method mitigates this by considering p(æ)) yy) as a regularizer for the discriminative task, rather than to obtain an exact posterior. Therefore, even the approximate dominated by an accurate mean, it can still offer a smoother and richer supervisory signal than relying solely on a one-hot label.
A.3 DATASET
CUB-200 Wah et al. 10011 is widely-used fine-graines dataset for bird species classification. It comprises 200 bird species with 5,994/5,744 749 samples for training/testing. ObjectNet Barbu et al. (2019) is challenging dataset designed to evaluate object recognition robustness real-world scenarios. contains 50,000 images of 313 classes for out-of-the- distribution evaluation.
ADE20K Zhou et al. (2117) is a densely annotated scene parsing dataset for the semantic seg- mentation task. It contains 20,210/2,000 images divided into 150 object and stuff categories for training/validation.
MS COCO [in et al. 2014) large-scale dataset contains 80 object categories with pixel- wise and bounding box annotations. It contains 118,287 and 5,000 images used for training and validation. For open-vocabulary detection, following Gu et al. (022)) the object categories is split
into 48 base and 17 novel.
LVIS v1.0 Gupta et al. (2019) is long-tail distribution benchmark containing 100 100,000/19 00111/980/2000000 for train/va l/test. Following prior work for open-vocabulary object detection Gu et al. (2022), the model is trained on 461 common and 405 frequent classes. The rest 337 rare classes
considered novel and used for testing.
NYUv2 Silberman et al. (2012) is popular benchmark for indoor scene understanding. It contains RGB-D images captured using Microsoft Kinect sensor in 464 indoor environments. Following existing work, 24,23 1/652 image-depth pairs are used for training/validation.
ImageNet Deng et al. (2009) is large-scale dataset commonly used for object recognition. It contains 1.2M images for training and 50,000 for validation, covering a wide range of 1,000 categories.
CelebA-HQ Karras et al. (2017) is high-quality version of the CelebA dataset, comprising 30,000
images resolution of (024x1024 pixels. It is widely used in computer vision research areas like image generation, super-resolution, and face synthesis.
LSUN-Church Yu et al. (2015) is subset of the Large-scale Scene Understanding (LSUN) dataset which focuses specifically outdoor church scenes. It contains over 126,000 high-resolution images, each resized such that the shorter side measures 256 pixels.
18

---

Published as a conference paper at ICLR 2026
A.4 IMPLEMENTATION DETAILS FOR TASK-SPECIFIC DECODERS
The task-specific decoders are designed following representative work. Specifically, the classification head for visual recognition is single-layer MLP. To enable generalization to oul-of-the-distribuuion classes, the model computes the similarity between pooled features and text embeddings of class labels. For semantic segmentation, GENREP leverages Musk2Formerihhnng et al. (2022). and calculates cosine similarities between class queries and label embeddings for open- vocabuocyy prediction. In open-vocabulary object detection, we follow Wu et al. (023a) to adopt a region proposal network, and map region features into pseudo words, which are then compared with class labels. The design of the object decoder follows Zhang et al. (2024) which utilizes stye- Transformer-decoder with 6 layers each containing 8 attention heads and a hidden dimension of 256. For depth estimation, we follow Li et al. (2024c) which employs a MaskFormer like architecture, and predicts the depth value linear combination of bin centers.
A.5 MONTE CARLO APPROXIMATION
We study the impact of varying sampling interval while keeping the total number of samples used for approximation constant in Table 16 As shown, when the number of samples held constant, performance consistently improves with a larger stride k. This validates that more independent samples (larger k) yield better distributional approximation. It also confirms that the performance decline in Table 9 is caused by the diminishing sample size, not inherent flaw in the thinning strategy.
Table 16: Analysis of the thinning interval A with fixed number of sampled intermediate states.
k
sample
T
ObjectNet (Top-1 Ace f)
ADE20K (mloU f)
2
75
75+2+50=200
51.1
32.5
3
75
75+3450=275
51.6
33.0
4
75
75+4+50=350
51.9
33.2
Since intermediate outputs of reverse diffusion are noisy or partially denoised versions of the data, it may cause mismatch the target distribution p(æ ly). We explore two strategies mitigate this: discarding the first m samples that noised heavily (i.e., burn-in); ii) importance re-weighting assign higher weights to later denoising steps in Equation 2 For importance re-weighting, we explore approaches, which are Linear Scaling (LS), Exponential Scaling (ES), and Power Scaling (PS):
t
(t-1))
t
LS: W
ES: W.
PS: w,
(17)
TIT+1)
Σ c(i-1)"
1P
The experimental results summarized below, with the thinning interval k 2, power factor 2. As observed, importance re-weighting leads to poor performance, possibly due to over emphasis on a small number of samples.
Table 17: Analysis of different important re-weight approaches for sample aggregation.
weightthting
N/A
LS
PS
ES
ObjectNet
51.1
49.2
49.8
48.7
ADE20K
32.5
29.3
30.0
29.1
A.6 ABLATION ON HYPERPARAMETER
The key hyperparameters of GENREP are the task weights (Wi, Wy in Eq. 17) and the alignment damping factor (a in Eq. 14-15). We ablate these hyperparameters below. As shown, the performance is relatively robust to minor variations. To obtain a balanced performance between perception and generation, we set wp 0.7, Wy = 0.3, and use the squared formulation for a.
A.7 PSEUDO CODE
For easier understanding, provide the pseudo code for generative visual perception learning with knowledge distillation in Algorithm
19

---

Published as a conference paper at ICLR 2026
Table 18: Analysis of task weights (wy, w, in Eq 16)
and the damping factor (a in Eq 13
W
Wg
ObjectNet (Top-I Acct)
ImageNett 256 (FIDL)
0.7
0.3
(*)²
51.1
6,92
0.6
0.4
(*)²
50.8
6.84
0.8
0.2
(*)²
51.5
7.12
0.7
0.3
(*)
50.7
6.98
0.7
0.3
(*))
51.3
7.04
Algorithm Generative Visual Perception Learning via Knowledge Distillation.
1: Hyperparameters:
2: T <- total diffusion steps
3: & < (Thinning interval]
4: m 50 (Burn-in steps}
5: Initialize models:
6: diffusion model <- PretraneeDDffuusioMood(((
7: _ek_decoder +- (hsskSpecfficDecodeer)
8: horapasams +- diffusion model.attention ]locks[[]]
9: Freeze all parameters except atfention blocks:
10: freeze_al laceeallllpaaaaa wawaaaaerrsaaiffuustttttt mmode!)
11: unfreeze (hro params)
12: for each (x., V/mue) training data do
13: Step 1: Reverse diffusion process
14: æT + sample samplc_noise(r))
noise(
reverse peerreesampleseeeee
16:
for = T 1,..., do
21 f diffusion model..everse.stepl... t. Vuwe)
18:
if < T - m and T mod k = then reverse samples.append(or)
20:
end if end for
21:
22:
Step 2: Estimate p(xly)
23:
Pis f {( (s.meen))
s reverse samples) sevelee_sample]
24:
Tist (s.vianne
25:
p(rly) +0
26:
for μr, a (Muis, Tlia) do
27:
p(x'y) () + p(xly) + ((),))
(Add Gaussian component)
end for
29:
p(rly) + p(zly)/(T//4
30:
Step 3: Compute generative posterior p(y(r))
31:
prior + 1/num_classes
logits gen < p( .og..o((((() log(prior)
33:
ply z) < softmax (lonmaxxxllitt...... gen)
34:
Step 4: Compute discriminative probability (v)2))
35:
logits
task task_decoder(2>
36:
q x) + softmax (logits
37:
Step 5: Loss computation
38:
lossaise +- crwss cross_entropyy(at | Mave)
lossgen_disu KLdivergenceef xv, x))
40:
total loss + losSaise + lossgen.disil
Step Backpropagation
42:
optimizer.zero_gnee -grad))
43:
44:
opiimizer.sepphhooparaaass taskdeecoder
45: end for
20


Reviews:
[{"review_id": "v5ucfrlZGl", "reviewer": "Reviewer_8E75", "summary": "This paper present GenRep, a unified image understanding and synthesis model that jointly conducts discriminative learning and generative modeling in one training session. This paper adopts a novel gradient alignment strategy to guide the joint optimization of perception and generation.", "questions": "- can the proposed methods be extended to more visual understanding tasks, such as OCR, captioning?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- the idea to distill distributional knowledge embedded in diffusion models to guide the discriminative learning for visual perception tasks is novel\n- the experimental results are comprehensive \n- the paper is well-written \n- the derivation is clear", "weaknesses": "- the gradient alignment may introduce extra computational overhead during training.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LOu4nUbccP", "reviewer": "Reviewer_fxcY", "summary": "This paper presents GENREP, a framework aiming to unify image understanding (perception) and image synthesis (generation). The method comprises three main components: \n1) Generative Visual Perception Learning: Using Monte Carlo approximation (a mature approach from Generative Classifier) on intermediate (noisy) samples from the reverse diffusion process to estimate the class posterior $p(y|x)$, which serves as an auxiliary distillation loss $\\mathcal{L}_{gen \\underline{ } distil}$ to guide the perception task. \n2) Semantic-Driven Generation: Using high-level semantic representations ($x_{sem}$) from the perception task to dynamically modulate the mean ($\\mu_\\theta$) and variance ($\\sigma_\\theta$) of each step in the generative process. \n3) Gradient Alignment: Using a gradient projection strategy from multi-task learning to reconcile conflicts between the perception and generation objectives. \n\nThe authors claim this unified model achieves SOTA performance on numerous perception and generation benchmarks.", "questions": "1.  Validity of MC Approximation: Can you provide stronger evidence that strided sampling ($k=2$) is sufficient to yield (near) independent samples from the reverse diffusion process? Furthermore, can you quantify the bias introduced by using noisy, intermediate samples to estimate the final, clean distribution $p(x|y)$?\n2.  Question on \"Positive Feedback Loop\": I am curious to see from Table 8 (rows #1, #3), it shows that  adding Semantic-Driven Generation (Percept -> Gen) *hurts* perception (Top-1/mIoU drops). Some explanations or investigations? This is helpful for the claims of \"positive feedback loop\".\n3.  Quantification of Training Cost: Compared to a standard perception baseline or a simpler unified framework (e.g., Diff-2-in-1), please quantify the extra training cost (e.g., in GPU-hours) introduced by the iterative Monte Carlo sampling required by your generative perception loss (Algorithm 1 in Appendix A.7)?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. Meaningful Goal: The attempt to unify discriminative learning (perception) and generative modeling (generation) within a single framework that allows for mutual benefits is a valuable research direction.\n2. Interesting Core Idea: The paper links generation and perception of a same Diffusion model via Bayes' theorem, which is a reasonable and natural synergy.\n3. Extensive Empirical Evaluation: The paper validates its approach across a wide range of benchmarks, including classification, segmentation, etc, achieving competitive results.", "weaknesses": "1.  The Necessity of Framework: The paper claims its novelty lies in adding \"high-level semantics\" to overcome the \"low-level reconstruction\" limitations of other unified models. However, the proposed mechanisms (MC estimation of $p(y|x)$ and explicit $\\mu_\\theta, \\sigma_\\theta$ modulation) are exceptionally complex (especially for computation), and I am not aware of how the so-called \"high-level semantics\" are contributing or whether they are necessary. In contrast, other unified frameworks (e.g., Diff-2-in-1 [1]) appear to achieve a similar unified goal via a much simpler self-training/mean-teacher loop, even without explicit \"high-level semantics\". This severely questions the necessity of GENREP's convoluted approach (too much cost for limited improvement, or meaningless innovation). Also, please conduct a more comprehensive literature review and a more detailed comparison of similar unified diffusion models, like [1].\n2.  Theoretically Flawed MC Approximation: The \"Generative Visual Perception Learning\" relies on an MC approximation of $p(x|y)$. This estimation is theoretically flawed in several major ways:\n    * Correlated Samples: It uses samples from a Markov chain (the reverse diffusion process). The claim that strided sampling ($k=2$) is sufficient to \"break\" this correlation is unsubstantiated and highly unlikely.\n    * Biased Samples: It uses *intermediate, partially-denoised* samples to approximate the *final, clean* data distribution $p(x|y)$. This is a fundamentally biased estimator.\n    * Computational Overhead: As shown in Algorithm 1, this approximation must be computed *inside every training iteration*, adding massive, unanalyzed computational overhead. Can this approximation be further accelerated?\n\n\n[1] Zheng, S., Bao, Z., Zhao, R., Hebert, M., & Wang, Y. X. Diff-2-in-1: Bridging Generation and Dense Perception with Diffusion Models. In The Thirteenth International Conference on Learning Representations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ovvmc7VozR", "reviewer": "Reviewer_aqJQ", "summary": "This paper proposes a unified model using GenRep pipeline that jointly learns discriminative and generative modeling with a single diffusion model. The paper aims for mutual enhancement and achieves this by jointly optimizing the two objectives. For example, for perception, the model distills distributional knowledge from the generative diffusion process using Monte Carlo approximation to better approximate conditional distributions. For generation, learned high-level representations are used to guide the reverse diffusion process. Then gradient alignment mechanism is used to resolve conflicts in the updates.", "questions": "- Training stability and sensitivity: While the paper reports stable convergence, the model jointly optimizes several interdependent losses (perception, generation, distillation, and alignment). How sensitive is GENREP to hyperparameter choices and architectural balance between these objectives? Did the authors observe instability or mode collapse when tuning these components across different datasets or backbones?\n- Sampling interval (k) trade-off: Table 9 suggests performance peaks at k = 2 but declines when k increases further. Could the authors elaborate on the underlying cause of this bias–variance trade-off? Intuitively, higher k should yield more independent samples — is the degradation due to sparse coverage of the diffusion trajectory or reduced effective sample size?\n- Could the GENREP architecture be extended to incorporate text-conditioned or open-vocabulary supervision (e.g., via CLIP embeddings)? How might the gradient alignment mechanism adapt in that scenario?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- High-performing framework: GENREP successfully combines discriminative and generative paradigms without sacrificing either performance, achieving state-of-the-art or highly competitive results on both perception and generation, across various datasets\n- Mutual learning synergy: The dual-branch setup intuitively builds a feedback loop — discriminative features improve the quality and faithfulness of generated images, while generative modeling provides richer distributional supervision for perception tasks. This is supported by the improved performance on downstream tasks \n- Generalization across domains: The model exhibits strong out-of-distribution performance (e.g., ObjectNet) and robustness across task types, indicating that the shared latent space supports better transferability and adaptability.\n- Clean optimization design: The proposed gradient alignment mechanism provides a clear way to balance conflicting gradients, promoting stable joint optimization.", "weaknesses": "- Computational overhead: While GENREP achieves impressive performance, integrating a diffusion model for perception tasks increases computational cost and reduces inference efficiency (e.g., 12.6 FPS vs. MaskFormer’s 19.6 FPS). This may limit its applicability in latency-sensitive or large-scale deployment scenarios.\n- Approximation sensitivity and bias–variance trade-off: The performance of the generative visual perception component depends on the strided sampling interval (k) used in the Monte Carlo approximation. While a small k (e.g., 1) leads to correlated samples, increasing k beyond 2 reduces correlation but introduces higher variance and less accurate distributional estimates. The authors observe that performance peaks at k = 2 and degrades for larger k values, which is unintuitive because larger strides should, in theory, produce more independent samples and better approximate the true distribution.\n- Multimodal Scalability: GENREP focuses on purely visual integration, but extending this framework to text-conditioned or cross-modal setups could further test its generality. The model currently does not explore alignment with language-conditioned diffusion models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1WKUZCpJJO", "reviewer": "Reviewer_poem", "summary": "The paper proposes a unified framework, **GENREP**, that jointly trains discriminative visual understanding and generative image synthesis in a single run:\n(1) It uses a *Monte Carlo approximation along reverse diffusion trajectories* to distill distributional knowledge from a diffusion model, treating it as Bayesian-posterior evidence to aid downstream perception;\n(2) On the generative side, it introduces *semantics-driven generation*, where semantic embeddings from the perception branch jointly modulate both the mean and the variance to steer the reverse process toward target semantics; \n(3) It applies *symmetric gradient alignment* between perception and generation losses to mitigate multi-objective conflicts, forming a mutually reinforcing loop. \nThe authors report strong results on classification, depth estimation, open-vocabulary segmentation/detection, and class-conditional generation.", "questions": "1. How does the generative distillation perform on samples where there are multiple (or ambiguous) most-likely labels `y`? Is the estimated distribution accurate in such cases?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The motivation to unify and mutually promote visual perception and visual generation tasks by leveraging their different modeling characteristics is excellent.\n2. In terms of implementation, the generative capability is effectively used to construct a sample distribution to guide perception, while the feature extraction capability from the understanding branch is used to assist denoising-based generation. This approach makes good use of the valuable information from both tasks.\n3. The experiments are comprehensive. The paper provides detailed comparisons for both perception and generation, along with multi-level ablation studies that successfully demonstrate the effectiveness of the proposed solution.", "weaknesses": "1. Using the generative prior `p(x|y)` to estimate the posterior `p(y|x)` is inaccurate. The diffusion loss primarily learns the distribution of `x` given a specific condition `y`, and the vast majority of the learning signal is dominated by the mean, not the variance. (As demonstrated in \"Improved Denoising Diffusion Probabilistic Models,\" the supervision signal for the variance must be small, otherwise, the generation quality degrades). This strongly suggests that the variance modeling is inaccurate. More intuitively, the mean is far more important than the variance in diffusion-based generative modeling. Furthermore, the assumption of a uniform distribution for `y` itself is questionable. The pseudocode in the appendix indicates that during training, `p(y'|x)` for all classes `y'` is treated as equally probable, rather than being reestimated from `p(x|y')`. This is clearly inaccurate in cases where there are multiple most-likely labels `y`, and thus it cannot serve as an effective distributional supervision signal.\n2. The perception task is trained only on clean samples. It is uncertain whether the model can maintain good representation capabilities when applied to noised latents.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)