A Simple Approach to Unifying Diffusion-based Conditional Generation
Authors: Xirui Li
Charles Herrmann
Kelvin C.K. Chan
Yinxiao Li
Deqing Sun
Chao Ma
Ming-Hsuan Yang
Venue: ICLR 2025 Poster (2025)
Keywords: image generation
controllability
estimation
Primary area: generative models
TL;DR: We propose a simple architecture and training scheme that produces a single diffusion model that can be used for flexible conditional generation, estimation, and joint diffusion.

Abstract: Recent progress in image generation has sparked research into controlling these models through condition signals, with various methods addressing specific challenges in conditional generation. Instead of proposing another specialized technique, we introduce a simple, unified framework to handle diverse conditional generation tasks involving a specific image-condition correlation. By learning a joint distribution over a correlated image pair (e.g. image and depth) with a diffusion model, our approach enables versatile capabilities via different inference-time sampling schemes, including controllable image generation (e.g. depth to image), estimation (e.g. image to depth), signal guidance, joint generation (image \& depth), and coarse control. Previous attempts at unification often introduce complexity through multi-stage training, architectural modification, or increased parameter counts. In contrast, our simplified formulation requires a single, computationally efficient training stage, maintains the standard model input, and adds minimal learned parameters (15% of the base model). Moreover, our model supports additional capabilities like non-spatially aligned and coarse conditioning. Extensive results show that our single model can produce comparable results with specialized methods and better results than prior unified methods. We also demonstrate that multiple models can be effectively combined for multi-signal conditional generation.

Published as a conference paper at ICLR 2025
A SIMPLE APPROACH TO UNIFYING DIFFUSION- BASED CONDITIONAL GENERATION
Xirui Li'
Charles Herrmann
Kelvin C.K. Chan Ming-Hsuan Yang²
Yinxiao Li
Deqing Sun
Chao Ma l+
Shanghai Jiao Tong University
Google DeepMind
Project webpage: https:/ http//xxxiii////ee/eellleeeeeeea rui142
.itthg io/ /nuuoonooniii offoonniiiffuiooo
Edge
Sketch
Text inpainting
ICLR
ia
Depth-to-Impg Image (As CommoiNet)
Operg-tt-Dethhhcom (As Marigoldj
Flexible Input
UniCa Edge
Rough: depch (Ax LouseCautrol)
Appcctaace
Deyth
Pilmaos Deptlt
UnK ce-itopph
Combining
Multiple Models
Depth Guidanee (As Readout Guidance)
One Model for All
idancdy
Posc
Figure 1:
The proposed UniCon supports diverse generation behavior in model for targeted type of image and condition. UniCon also offers flexible conditional generation ability with natural support for free-form input and seamless integration of multiple models.
ABSTRACT
Recent progress in image generation has sparked research into controlling these models through condition signals, with various methods addressing specific chal- lenges in conditional generation. Instead of proposing another specialized tech- nique, we introduce a simple, unified framework to handle diverse conditional generation tasks involving a specific image-condition correlation. By learning joint distribution over a correlated image pair (e.g. image and depth) with dif- fusion model, our approach enables versatile capabilities via different inference- time sampling schemes, including controllable image generation (e.g. depth to image), estimation (e.g. image depth), signal guidance, joint generation (im- age & depth), and coarse control. Previous attempts at unification often introduce significant complexity through multi-stage training, architectural modification, or increased parameter counts. In contrast, our simple formulation requires single, computationally efficient training stage, maintains the standard model input, and adds minimal learned parameters (15% of the base model). Moreover, our model supports additional capabilities like non-spatially aligned and coarse condition- ing. Extensive results show that our single model produce comparable results with specialized methods and better results than prior unified methods. We also demonstrate that multiple models be effectively combined for multi-signal conditional generation.
"Corresponding author.

---

Published as a conference paper at ICLR 2025
INTRODUCTION
Text-to-image diffusion models, such Dall-E (OpenAI, 2023) and Imagen (Ho al., 2022a), have revolutionized the field of image generation, leading to contemporary models (Midjourney, 2023: Baldridge et al., 2024) that can produce images almost indistinguishable from real ones. This progress in generative modeling, particularly with diffusion models, has spawned new research areas and reshaped existing fields within computer vision. With advancements image quality, the gen- erative community has expanded its focus to controllability, resulting in many different approaches, each promoting a distinct scheme for guiding the generative process. ControlNet (Zhang et al., 2023b) highlights the effectiveness of using modalities like depth and edges as conditional input. Meanwhile, other works, such as Loose Control (Bhat et al., 2024) and Readout Guidance (Luo et al., 2024), propose alternative conditioning types (e.g. coarse depth maps) and control mecha- nisms (e.g. guidance through a prediction head). Concurrently, the estimation community has seen diffusion models advance the state-of-the-art for predicting various modalities from RGB images, g. Marigold (Ke et al., 2024) repurposes a pretrained image generator to generate depth instead. In addition, other work Stan et al. (2023) has explored joint diffusion, generating paired image and depth simultaneously.
Although typically addressed as separate tasks within distinct communities. these problems share a common underlying structure: conditional generation between correlated images. Consider the relationship between an image and its depth map: controllable generation translates depth im- age, estimation maps image to depth, guidance uses depth predictions to guide image generation, and joint generation produces image-depth pairs. This observation motivates us to unify all these tasks under global distribution modeling problem. While a few works (Qi et al., 2024; Zhang et al., 2023a) have also explored unified models capable of handling these diverse tasks, they often introduce significant complexity through multi-stage training, increased parameter counts, archi- tectural modifications. This additional complexity makes creating and using these models difficult, hindering their adoption.
In this paper, we propose UniCon, a unified diffusion model that learns an image-condition joint distribution with flexible model architecture and simple but effective training strategy support diverse inference behaviors. We propose an architecture adaptation to the standard image generator diffusion model that is more flexible than ControlNet (allowing for non-pixel-aligned conditioning signals) and more efficient to train, decreasing both the number of learned parameters and required training samples. Inspired by Diffusion Forcing (Chen et al., 2024a), we use a training scheme that disentangles the noise sampling of the image and the condition, allowing flexible sampling strategies at inference time to achieve different conditional generation tasks without explicit mask input.
As shown in Fig. 1, with the same model but different sampling schedules, UniCon can do: 1) controllable image generation in the form of ControlNet, Readout Guidance, and Loose Control, 2) estimation, and 3) joint generation. We train our models based on large text-to-image model for several different modalities (depth, edges, human poses, identity) and show that the behavior of our single model is similar to or better than specialized methods using standard image quality and align- ment metrics. We demonstrate significant improvements over prior unified models conditional generation, training efficiency, and generation flexibility. We also show that UniCon can combine multiple models for multi-signal conditional generation or switch our model to other base model checkpoints. Our models are trained in about 13 hours 2-4 Nvidia A800 80G GPUs, adding 15% parameters to the base model.
The main contributions of this work are:
Proposing a framework that unifies controllable generation, estimation, and joint genera- tion, including model adaptation, training strategy, and sampling methods allowing flexible conditional generation at inference.
Demonstrating that our architecture and training can work on large-scale text-to-image diffusion model with small number of learned parameters and a relatively small training data scale.
Showing that our unified models can perform similar to specialized methods or better than current unified approaches on different modalities.
2

---

Published as a conference paper at ICLR 2025
2 RELATED WORK
Controllable Generation. Fine-tuning text-to-image diffusion models to conditional image genera- tion on signals beyond text has gained significant popularity (Huang et al., 2023; Zhang et al., 2023b; Ye et al., 2023; Mou et al., 2023; Sohn et al., 2023). ControlNet (Zhang et al., 2023b) trains a con- trol network attached to pre-trained diffusion models to incorporate condition signals, such as edge maps, segmentation masks, and pose estimation. Based on ControlNet, LooseControl (Bhat et al., 2024) generalizes depth conditioning positions. Readout-Guide Guidance (Luo et al., 2024) proposes a new control scheme by adding prediction heads to internal features and guiding the generation by the predicted condition signal. DAG (Kim et al.. 2022) guides a diffusion model to generate geometrically plausible images using depth prior. Conditional editing tasks, such as DiftEdit (Couairon al., 2023), have enhanced conditional im- age manipulation by applying diffusion- based models for inpainting and editing tasks. Instead of one specific control behavior, our proposed framework provides diverse controlling abilities through different sampling strategies.
loose depth maps that specify scene boundaries and object
Estimation. Extracting signals like depth, surface normals, or segmentation maps from RGB im- ages has been longstanding challenge in computer vision. Typically, each task has been addressed in isolation or limited combinations.e.g joint depth and segmentation, but distinct from image gen- eration. Starting works like DDVM (Saxena et al., 2023a), these tasks have started to be addressed directly with diffusion models including depth prediction (Saxena et al., 2023b;a), optical flow pre- diction (Saxena et al., 2023a), correspondence matching (Nam et al., 2024), etc.. Recently, there has been considerable interest in either using generative features inside estimators (Xu et al., 2023; Zhao al., 2023) or explicitly fine-tuning image generators as estimators, such as Marigold (Ke al., 2024) which adds a clean RGB conditioning and fine-tunes the entire model to diffuse a depth map. While Marigold shares some similarities with our depth estimation setting for our RGBD model, our approach differs significantly in both goals and techniques. Unlike Marigold, which focuses solely on depth and discards image generation capabilities, our method retains the ability to perform image generation, depth estimation, and other tasks, through lightweight LoRA fine-tuning.
Joint and Unified Generation. While less common than controllability or estimation, several works have attempted to unify multiple images and modalities within a single model. LDM3D (Stan et al., 2023) jointly generates image and depth data in an RGBD latent space. Following approaches com- monly include an inpainting mask in the input to extend joint generation to bidirectional conditional generation. For example, UniGD (Qi et al., 2024) unifies image synthesis and segmentation through diffusion model trained with image, segmentation, and inpainting mask as inputs. JeDi (Zeng et al., 2024) learns a joint distribution over images that share a common object, facilitating personalized image generation. Among the most relevant works, JointNet (Zhang al., 2023a) adopts a symmet- ric ControlNet-like structure for generating both image and depth, utilizing an inpainting scheme to support depth-to-image and image-to-depth generation. Our approach presents several improve- ments over these unified methods. First, our training strategy enables flexible conditional generation without requiring an explicit mask input. Thus we can avoid concatenating multiple inputs in the feature dimension or adding inpainting masks, which allows our model to act like adapters that can be plugged into the base model checkpoints. Furthermore, our structure supports both loosely corre- lated image pairs (as in JeDi) and densely correlated pairs (as in JointNet), providing more versatile capabilitics across different scenarios.
PRELIMINARIES
Diffusion Model. Diffusion models (Sohl-Dickstein al., 2015; Ho et al., 2020; Song et al., 2020b)
are generative models that model a data distribution p(x) through an iterative denoising process. Consider a forward process gradually adding Gaussian noise c to data Xo p(x) with timesteps t= 1,..., T and noise schedule {az),
((1 xo) -N(x / VopXa, (1 TT) =(x))==-)=-(xx Vaxx--, I.)
=
(1)
where o IT= as and or ~ NO, I) reaches pure noise. Diffusion models learn to denoise It at any timestep by estimating X (x(x,, t) R Xo. According to common e-parameterization, we can train the model to predict the noise eo (x,, instead using the following least squares objective,
min Exo.e,t llllc co (x),
3

---

Published as a conference paper at ICLR 2025
Scparate timestep sampling
8g
Xa
2x
Loss function
Noisy Image Ntx
Frozen pretrained layers
Joint attention module
90
Yo
Optional condition LoRA
Specific LoRA adapters
Noisy condition Vty
pin
Self Attention
Cross Attention
F
Wo
W W
Fout
Joint Cross Attention
PrujOui ini..)
Self
Cross
F
Attention
Attention
pin
Woy
WKy
W
Four
Figure 2: UniCon pipeline. Given pair of image-condition inputs, our UniCon model processes them concurrently in two parallel branches, with injected joint cross-attention modules where fea- tures from two branches attend to each other. We use LoRA weights adapt our model from pretrained diffusion model. During training, we separately sample timesteps for each input and compute loss over both branches.
where x, AddNoise(Xx: t) V + VT- a,E and .N(O, is random noise. With the trained denoiser, one can adopt any sampler (Ho et al., 2020; Song et al., 2020a; Karras et al., 2022) sample new data from noise, Recent latent diffusion models (Rombach et al., 2022; Ramesh al., 2022) map image data into the latent space to improve performance and efficiency. We base our experiments Stable Diffusion (Rombach al., 2022), large-scale text-to-image latent diffusion model.
METHOD
Our method, UniCon, aims to train a unified diffusion model for diverse conditional image genera- tion tasks, such conditional generation clean, coarse, partial control signals, estimation, and joint generation. The key idea is to learn joint distribution over a correlated image pair x, y. which allows flexible conditional sampling. The image pair can have strict spatial alignment (image-depth, image-edge) or loose semantic correspondence (frames from one video clip). In Sec. 4.1, we first introduce our motivation for learning the joint distribution. Then, we elaborate on our model struc- ture and training pipeline in Sec. 4.2 and the sampling strategies for flexible conditional generation Sec. 4.3.
4.1 MOTIVATION
Image diffusion models offer significant flexibility when sampling the learned image distribu- tion p(x). In addition generating new image x p(x), one can perform image inpainting by conditioning partial image x and image editing by conditioning noisy image x,, corresponding to sampling conditional distributions p(x[x"") and p(x{xx). Our motivation is to generalize these abilities from a single image to a correlated image pair (x,y) by learning joint distribution p(x,y) y).
We then use the modeled joint distribution enable flexible conditional generation. If the condi- tional signal is encoded as image condition y. The trained joint diffusion model supports various conditional generation tasks that can be unified as sampling in the following conditional distribution.
we can train a diffusion model to denoise both image and
(x,y) p(xxx y (xx Y
my
(3)
4

---

Published as a conference paper at ICLR 2025
where x indicates x partially masked by mask my under noise level (timestep) tz.
Sampling in the joint conditional distribution can be regarded as a direct generalization of image inpainting and editing. The conditional generation and estimation are equivalent to inpainting image x or condition y (p(xlyo).p() p(y xn)). We can also inpaint image and condition for partial con- t trol y .Yo ")). Adding noise to y enables the model to accept coarse condition input (ly)) like SDEdit (Meng et al., 2021) does for image. We can control the condition fidelity by adjusting the noise level. To sum up, combining spatial masking May my and noise masking l,, by provides substantial possibilities in free-form conditional generation.
Based on this motivation, our goal is train unified diffusion model for targeted image pair (x. and develop sampling strategies to support the conditional sampling described in Eq. 3.
4.2 UNICON
Figure 2 illustrates the model structure and training pipeline of the proposed UniCon. Instead of training a new model from scratch, we leverage existing large-scale diffusion models (Rombach al., 2022)
starting point. Since these models have learned a strong image prior, it is more efficient to adapt the prior for a single image p(x) to model the joint distribution of a correlated image pair, p(x, y). than learn this distribution from scratch.
Given a noisy image pair (t,)) we feed them as a batch into the denoising network. They are simultaneously processed in two parallel branches, denoted x-branch and y-branch. By default, x is the image, and y is the condition. When the conditional image differs from a natural image, we add LoRA (Hu al., 2021) the y-branch, which serves to adapt the image generator to produce, e.g. depth or edges. Additional joint cross modules are injected parallel to the self-attention modules join two branches. During training, we separately sample the timesteps fz by for x, y and optimize the model with the standard diffusion MSE loss from both branches. Next we provide details on our joint cross-attention modules, LoRA adaptation, and training strategy.
Joint cross attention. The joint cross-attention module is the key component that enables ours model to learn a joint distribution p(x, y) given the marginal distributions p(x).p(y) entangles x-branch and y-branch with cross branch attention.
The UNet (Ronneberger et al., 2015) among the most common diffusion model implementation and consists of residual blocks and transformer blocks. As shown in prior work (Tumanyan et al., 2023), the self-attention modules in the transformer blocks are crucial in determining the image structure and appearance. Therefore, self-attention modules. The module receives the features from both branches as input, with its outputs being added to the self-attention output of the branches. Specifically, given the input features of two branches pin Fin, the output features pour y are computed as,
inject the joint tross-attentie modules in parallel to the
pioint pjoint JointCrossAutn x Fi
Four SelfAttn(F + Fioint Fout SelfAttn(F")) + Fioine
In the joint cross-attention, two features Fin, Fin attend to each other instead of attending to them selves as in the self-attention. First, features are projected into queries Qy. keys K,, Ky, and values V., Vg. g. We use different matrices to project x features and y features. For instance, Q. = F Q Qg = F - Then we perform cross-attention between x and y in bidirection,
Q.K Vd
O₂K?? Vd
2 Softmax
Vy:
O Softmax
V
(5)
where d the feature dimension and the feed-forward projection after the attention operation is omit- ted. Essentially, x aggregates y values V, according to the query-key similarity matrix Q, and vice versa. As common practice (Zhang et al., 2023b; Guo et al., 2024), we add zero-initian initiaiiit linear projection ProjOut at the end to ensure the training starts without disrupting the pretrained feature distribution. Fioint Fjoint ProjOut(/O... Oyl). O O, are concatenated in channel dimen- to enhance feature fusion. Otherwise, they are
sion if image x and condition y are fed forward separately.
5

---

Published as a conference paper at ICLR 2025
Compared to alternatives including feature residual (Zhang et al., 2023b), input concatenation (Stan et al., 2023), and backbone sharing (Liu et al., 2023), joint cross attention is compatible with im- age pairs without strict spatial correlation. We initialize all joint cross aoss-attention weights from the pretrained self-attention modules and train LoRA adapters for x, projection matrices.
LoRA adaption for condition branch and joint cross attention. Training all the parameters in our model for our conditional signal at least doubles the parameter number in pretrained image layers. Therefore, we instead adopt the Low-Rank Adaptation technique (Hu et al., 2021) (LoRA) to fine-tune the pretrained weights by adding low-rank trainable weight matrices. In addition to reducing trainable parameter numbers, using LoRA adapters allows to apply our model other checkpoints sharing the same structure as the training base model by plugging joint cross-attention and trained LoRA weights.
We freeze all pretrained layers in the x-branch to retain the natural image prior p(x). When the condition y is encoded as a pseudo-image falling out of natural image distribution, we add a LoRA adapter to the y-branch to adapt for the condition image distribution p(y), denoted as y-LoRA. y-LoRA applies to all projection matrices the self-attention and cross-attention modules.
For joint cross-attention, we initialize all weights from the pretrained self-attention modules. Then we add two sets of LoRA adapters to the pretrained projection matrices. xy-I.oRA includes LQ. L,_,, LV,, and yx-LoRA includes L Lv,, For instance, the adapted y,y query pro- jection matrices are We, = We + LQ_, We. = We + LQ, where We is the frozen query projection matrix from pretrained self-attention module.
Training with disentangled noise levels. One training objective adopted by previous methods (Stan et al., 2023; Liu et al., 2023) is ming E(xo,vo).e..lle cx(X,, Vr, Vr,t]] t)l where x, shares the same noise level. The model learns how to denoise the noisy (x, jointly and can generate new sam- ples (x,y) ~ p(x,y) y). However, models trained this way do not explicitly support conditional sampling. Some work (Zhang et al., 2023a) solves the problem by augmenting the input with con- dition mask and masked latents and finetuning the model with an inpainting target, yet this requires heavy training involving all model parameters.
Recently, Diffusion Forcing (Chen et al., 2024a) proposed a new training paradigm where the model is trained to denoise inputs with independent noise levels. Inspired by the idea, we separately sample the diffusion timesteps for x and y when training, leading to the following training objective,
min E Ex |c co (x Mty, tuz 11
where (Czz, Cy) and xt, AddNoise(x tz)) timessep-diisentaglled objective can directly perform conditional generation by denoising x,, while keeping by The noise added the input can be regarded as an implicit noise mask. Unlike explicit input masks, noise masking can interpolate between no mask (L = 0) and full mask (= T, enabling image generation with coarse conditions.
AddNoise ty). Models trained with the X1, x1,,
4.3 INFERENCE
Our timestep-disee disttep-eiset training allows UniCon models to process paired inputs with different noise levels each denoising step. Suppose we have a denosing sequence (xx,,))) X,, are sampled under independent noise schedules (t,s ..., (2) and (1....., (9) where t' S ++1
Sampling with independent noise schedules. The independent y noise schedule enables diverse sampling behaviors. First, we can jointly generate x, y by denoising them together with identical noise schedules, f 1 Vi. For conditional generation, we can sample x from noise with a clean condition input y, i.e. y; y, by Vi. We can similarly sample y conditioned the b-bachhh clean input and t Vi. Furthermore, our models allow sampling x with coarse control signal conditioning condition fidelity by adjusting the noise level E from 0 (no noise) to T (pure noise). Since the control signal is corrupted by noise, the condition image itself does not need to be precise. Therefore, we use artificially created edited condition images to loosely control image generation.
x by giving
a noisy condition image ys = AddNoise(y, (.). We can control the
Sampling with guidance. Since our model has an output for both branches, we can apply guidance to each of them for image inpainting or partial condition. Latent replacement is typical approach

---

Published as a conference paper at ICLR 2025
Deyth Iraege
Inege Depth
Reugh Dxysh Condition
Joint Deyth flange Geecralker
Partell Depth Canditie
Deyik Guidanee
satl
a cssliction
"Seuo sughls
guten on
Marionl
anthaad
Not supportl
Figure 3: Qualitative comparison of diverse mnage-Depthh generation tasks. We compare our single UniCon- UniCon-Depth model with other specialized methods and previous unified method Joint- Net (Zhang et al., 2023a) on diverse generation tasks.
Depth
SoftEdge
Pose
Figure 4: Conditional generation samples. We show our sample conditional generation results. For each model, the left column the input condition and the right column is the output image.
for inpainting where the noisy latents are partially replaced by exact samples from the forward process (Eq. 1) each step, 2, = (1 m)) AddNoise(z, 1) + m 2, where z, m are the given condition sample and mask. The method is an approximation to exact conditional sampling. Fol- lowing Ho et al. (2022b), we can add a guidance term to correct the sampling process and improve the condition adherence,
at V 2. z = Wy
20 (z:: 0)12
(7)
2
where z is guided noisy latents, 2n (z:: 0) [N VT- -GED (z:: 1))/ / is the predicted original sample and w,, is weighting factor. 2d indicates part of z where mask m = 0 and rn is the inverted mask. The guidance term leads the noisy latents toward reconstructing the masked condition area z For UniCon, above variables include both inputs,
(t,, 2 y), m (m,, my))
Sampling with multiple conditional signals. combine multiple UniCon models and extend the joint cross attention to include all image-condition pairs. In specific, the image feature is paired with each condition feature in the joint cross-attentios modules and processed by weights from corresponding models. Then the image branch aggregates all output features with weight factors to balance the strength of each condition.
sample with multiple conditional images, we
RESULTS
We base our experiments on Stable Diffusion (Rombach et al., 2022) (SD), a large-scale text-to- image diffusion model. We train 4 UniCon models, Depth, SoftEdgee, Human-Pose (Pose), Human- Identity (ID) on SDv1-55 The first three pair an image with a spatially aligned condition image. Following Luo et al. (2024), we train Depth, SoftEdge models on 16k images from PascalVOC (Ev- eringham et al., 2012) and Pose model on a subset with 9k human images. Depth, soft edge, and pose images are estimated by Septh-Anythingee
(Yang et al., 2024b), HED (Xie & Tu, 2015) and OpenPose (Cao et al., 2019). We train the ID model on 30k human face images from CelebA (Liu al., 2015) and use images with the same identity training image pairs. In addition, train

---

Published as a conference paper at ICLR 2025
Depth
SoffFdge
Pose
TD
0.50
0.56
0.51
0.67
Profornce
0.27
0.22
0.03
0.03
0.00
RG
Contolleet
UhiCin
RG
ContaalNe
CaeCan
RG
ContolNet
UniCon
IP-Adeptar
UnCCo
Figure 5: User study of conditional generation performance. We compare our UniCon against Readout-Guidann (Luo et al., 2024), ControlNet (Zhang al., 2023b) for spatially-aligna condi- tions (depth, softedge, pose) and against IP- P-Aaapter-Face (Ye et al., 2023) for ID condition.
Table I: Quantitative comparison of conditional generation for spatially We compare Readout-Guidance (Luo et al., 2024), ControlNet (Zhang et al., 2023b), and UniCon (Ours) on spatially-alignes conditions depth, softedge, and pose. For fair comparison, all methods are trained on Pascal VOC with the same annotator.
conditions.
Depth
Softtidge
Method
FID-6K
AlmRel (%)
FID-6K
EMSE (Ie-2) J.
FID-6K
PCK @ 0.1
Readout-Guidanee
Readout- Guidance
18.72
23.19
18.43
4.84
21.07
24.96
ControlNet
13.68
9.85
13.46
2.30
18.61
57.54
UniCon
13.21
9.26
12,80
2.28
17.51
61.97
an auxiliary Appearance model cooperate with other models, shown in Fig 8. is trained on random frame pairs from 6k videos in Panda70M (Chen et al., 2024b), aiming at generating images with similar visual appearance.
5.1
MAIN RESULTS
Qualitative results. In Fig. 3, we show sample results generated by our Depth model on different tasks and compare them with specific method for each task. Note our results generated by the same Depth model with different sampling strategy. First, our model can accept a clean depth or im- age perform Depth-to-Imagg
Image-to-Deplle generation. ControlNet (Zhang al., 2023b) and Marigold (Ke et al., 2024) respectively work for the two tasks with different structures. Compared ControlNet, UniCon supports generation with a rough partial condition. LooseControl (Bhat et al., 2024) finetunes ControlNet for generalized depth condition. Our model works for such created rough depth images without foreknowledge by conditioning noisy depth images. Similar to Readout-Cuidant (Luo et al., 2024), we can apply the guidance the depth output for conditional generation. addition image, such as using the border of the depth map to specify the overall scene structure. Finally, our model can jointly generate an image with its depth, which the goal of LDM3D (Stan et al., 2023).
complete depth image,
also possible to guide with part of depth
Fig shows sample results from all UniCon models. Apart from common spatial aligned control signals, our ID model works for loosely correlated human images. With the ID model, we can generate images of the same person in the condition image and utilize the input prompt to specify appearance and style.
Quantitative comparison. We compare UniCon with other methods on conditional generation and depth es- generate 6K 512 x 512 images conditioned on depth, soft edge, pose of random images from OpenImages (Krasin et al., 2017). We use Frechet Inception Distance (FID) (Heusel et al., 2017) to measure the distribution distance between generated images and real images cor- responding to the same input conditions. We also eval- uate the condition fidelity by Absolute Mean Relative Error (AbsRel) (Ke et al., 2024) for depth; Edge Mean Squared Error (EMSE) for soft edge; and Percentage of Correct Keypoints (PCK) (Yang & Ramanan, 2012) for pose. All metrics computed between the modalities estimated from real images and generated images. For
Table 2: Quantitative depth estimation comparison. We compare MiDaS (Ran- fil et al., 2020). DPT (Ranftl et al., 2021), Marigold (Ke et al., 2024), and our Depth- Metric model on zero-shot depth estima- tion benchmarks. We show results with- out test-time ensembling.
timation. For the conditional generation,
NYUv2
ScanNer
Mothod
AbsRel
81
AbsRel 1
81
MiDaS
11.1
88.5
12.1
84.6
DPT
9.8
90.3
8.2
93.4
Marigold
6.9
JointNet UniCon
13.7
81.9
14.7
79.5
7.9
93.9
9.2
91.9
depth estimation, evaluate on NYUv2 (Silberman al., 2012) and ScanNet (Dai al., 2017)
8

---

Published as a conference paper at ICLR 2025
"A loon a field flowers"
Imput
"A cat"
ICLR
Sampling Schedule
Output
ITP
ICLR
Figure 6: Flexible conditional generation via different sampling schedules. We annotate each image with its sampling schedule. Schedule bars represent the noise level from to Arrows indicate the noise sampling schedule. We apply the guidance in generation (orange arrow) for partial condition samples and use red borderlines to split areas to keep and areas to generate.
Training Step
Ablation
Data Scale Ablation
10.5
80
PD--K1
CestolNel
RDD-K 1
ConastNai
Alrikel
10.0
28
AbaRd
12.5
57
10.0
Trining
Training Supps
Figure 7: Ablation of training steps and data scale. We compare the depth conditional generation performance of UniCon and ControlNet on different training settings. We use the same training steps for data scale ablation and vice versa. UniCon-ND indicates trained without depth loss.
with AbsRel and 51 (Ranftl et al., 2021) as metric, following the protocol of affine invariant depth evaluation (Ranftl al., 2020).
We conduct a user study to compare UniCon models against their corresponding methods to evaluate shown in Fig. 5, UniCon demonstrates comparable
the conditional generation performance. performance to specialized methods in human preference (Fig. 5).
In addition, we compare UniCon for spatially- conditions (depth, softedge, pose) against Guidance (Luo et al., 2024) in terms of FID and condi-
ControlNet (Zhang et al., 2023b), tion fidelity metrics. As shown in Tab. 1, our UniCon achieves similar or better performance than Readout-Guidance (Luo et al., 2024) and ControlNet (Zhang et al., 2023b) FID and condition fidelity over all modalities.
We finetune our Depth model for 5K steps with Anyth-Any V2-Metric (Yang et al., 2024b) as the annotator for metric depth evaluation. As shown in Tab. 2. our Depth-Metric model performs similarly or better than MiDaS (Ranftl et al., 2020) and DPT (Ranftl et al., 2021). There is margin between our model and Marigold (Ke et al., 2024), which fine-tunes the whole diffusion model and solely focuses on depth estimation. In comparison, our model trains fewer parameters and targets on unified conditional generation.
Flexible conditional generation. We show diverse conditional generation samples in Fig. 6. Start- ing from a noisy condition image, our models can interpret the exact condition image to other mean- ings (dog to lion in Column 1) take rough condition as control (cat sketch in Column 2). Using guidance for partial conditioning enables us to condition on both input signals (Column 3) or re- paint an image with coarse condition signal (Column 4). We generate the "ICLR" edge image by replacing raw edges with text.
In addition, we can combine multiple UniCon models to enhance the control ability (Fig. 8). One in- teresting application is to combine loose conditions (ID, Appearance) with dense conditions (Depth, Pose). the bottom row, we use the same image for both and Appearance conditions to enhance both ID alignment and overall image appearance consistency. Similar to ControlNet, our models can apply to other customized checkpoints fine-tuned from our base model (Fig. 9).
Comparison with JointNet. We compare our method to the most relevant baseline, JointNet (Zhang et al., 2023a), which also supports multiple conditional generation tasks for image and depth. As demonstrated in Fig, 3, method shows better results in image-to-deplee partial condition, and depth guidance, with extra abilities of rough depth condition. Quantitatively, UniCon achieves su- perior depth estimation results compared to JointNet (Tab. 2).

---

Published as a conference paper at ICLR 2025
Depth
Inpui
SDvIl-5
Reallitio--4
Anything-95
Appeoratce
Deptt
Ouipar
ID Depperronne
Fose
Oxiput
Multi-signal conditional gener- ation samples. We can combine multiple models for different condition signals.
Figure 8:
Figure 9: Apply to other checkpoints. Our mod- els can apply to other checkpoints fine-tuned from the base model (SDv1.5).
5.2 ABLATION STUDY
Training steps and data scale. We ablate training steps and data scale for depth conditional
generation (Fig. 7). As the training step grows, our model improves steadily, showing a higher performance upper bound than ControlNet which starts overfit on the dataset. We observe sudden drop in condition fidelity when downscaling our training dataset. We attribute it to our joint cross-attention requiring certain data scale capture the image-condition correlation. On the other hand, UniCon achieves better condition fidelity with enough data.
Training setting and model alternatives. In Tab. 3, we test our Depth model with training setting and structure alterna- tives. First, we drop out the depth loss training, leading model that has similar depth-to-image generation perfor- mance but cannot denoise depth image. We also investigate the depth loss influence under different data scales (Fig. 7 Right), showing that joint modeling depth has a positive ef- feet on conditional generation. Further dropping the noise added to the depth image in training results in a ControlNet- like depth-control model. It improves the condition fidelity
Table 3: Ablation of training set- ting and model alternatives. We evaluate the conditional generation performance our Depth model un- der different settings.
FID-6K J
AbsRell
UniCon-Depth
13.21
9.26
Depth loss
13.18
9.23
Depth loss, noise
13.66
8.57
Eneoder
but harms generation quality. We also remove joint cross- attention modules in the UNet encoder against structure changes. Despite the slight performance drop. our method works with half of the attention modules. Finally, our model consistently improves when we scale up the training data to 200K images from Openlmages (Krasin et al., 2017).
Data (200K)
13.10
8.66
the robustness
6 DISCUSSION
We propose simple framework for unifying diffusion-based conditional generation. We consider all conditional generation tasks involving à specific image-condition correlation as sampling in global distribution and train diffusion model lear Our flexible model architecture adapts pretrained diffusion model to handle multi-input processing, alongside effective training and sam- pling strategies designed to support diverse generation tasks. The inherent flexibility of our approach opens up the possibility for wide range of applications, potentially encouraging further exploration into novel image-condition mappings. Additionally, our work demonstrates that large-scale diffu- sion models can be successfully adapted to accommodate non-aligned noise levels in input signals, suggesting a path toward enhancing existing multi nigti- diffusion models. As for limitations, some models dealing with loosely correlated image pairs, e.g. ID model, exhibit instability. We at- tribute this issue to the need for more training data and refined techniques to achieve satisfactory performance in such cases.
10

---

Published as a conference paper at ICLR 2025
Acknowledgements. This work supported in part by the National Natural Science Foundation of China (62322113, 62376156).
REFERENCES
Jason Baldridge, Jakob Bauer, Mukul Bhutani, Nicole Brichtova, Andrew Bunner, Kelvin Chan, Yichang Chen, Sander Dieleman, Yuqing Du, Zach Faton-Rosen, al. Imagen arXiv preprint Stti:2400.0000000 2024. 2
Shariq Farooq Bhat. Reiner Birkl, Diana Wofk, Peter Wonka, and Matthias Müller. Zoedepth: Zero- shot transfer by combining relative and metric depth. arXiv preprint arXiv:2302. 2023. 16, 18
Shariq Farooq Bhat, Niloy Mitra, and Peter Wonka. Loosecontrol: Lifting controlnet for generalized depth conditioning. In SIGGRAPH, 2024. 3.
Z. Cao, G. Hidalgo Martinez, T. Simon, S. Wei, and Y. A. Sheikh. Openpose: Realtime multi-person 2d pose estimation using part affinity fields. IEEE TPAMI, 2019. 7,
Boyuan Chen, Diego Marti Monso, Yilun Du, Max Simchowitz, Russ Tedrake, and Vincent Sitz- mann. Diffusion forcing: Next-token prediction meets full-sequence diffusion. arXiv preprint arXiv:2407.01392, 2024a. 2.6
Tsai-Shien Chen, Aliaksandr Siarohin, Willi Menapace, Ekaterina Deyneka, Hsiang-wei Chao, Byung Fun Jeon, Yuwei Fang, Hsin-Ying Ying Lee, Jian Ren, Ming-Hsuan Yang, et Panda-70m: Captioning 70m videos with multiple cross-modality teachers. In CVPR, 2024b. 8, 15
Guillaume Couairon. Mohamed Elgharib, Kiran Varanasi, Alexis Joly, Patrick Pérez, and Cordelia Schmid. Diffedit: Diffusion-based semantic image editing with mask guidance. arXiv preprint arXiv:2303.09367, 2023. 3
Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated reconstructions of indoor scenes. In CVPR, 2017. 8
M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The PASCAL Visual Object Classes Challenge 2012 (VOC2012) Results. htt::/www.ppaccal
2012. 7, 14
Xiao Fu, Wei Yin. Mu Hu, Kaixuan Wang, Yuexin Ma, Ping Tan, Shaojie Shen, Dahua Lin, and Xiaoxiao Long. Geowizard: Unleashing the diffusion priors for 3d geometry estimation from a single image. In ECCV, 2025. 16
Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang. Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffu- sion models without specific tuning. ICLR, 2024. 5
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Adv. Neural Inform. Process. Syst., 30, 2017. 8, 16
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NewrIPS 202/ Workshop on Deep Generative Models and Downstream Applications, 2021. 15
Jonathan Ho and Tim Salimans. arXiv:2207.12594 2022. 14
Classifier-free diffusion guidance.
arXiv preprint
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. NeurIPS. 2020. 3,
Jonathan Ho. William Chan, Chitwan Saharia, Jay Whang, Ruigi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint 20.0v22200.02300. 2022a. 2
11

---

Published as a conference paper at ICLR 2025
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J. Fleet. Video diffusion models. arXiv preprint arXiv:2204.03558, 2022b.
Edward Hu, Phillip Wallis, Zeyuan Aluen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In ICLR, 2021. 5, 6
Lianghua Huang, Di Chen, Yu Liu, Yujun Shen, Deli Zhao, and Jingren Zhou. Composer: Creative and controllable image synthesis with composable conditions. arXiv preprint arXiv:2302.09778, 2023. 3
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. NeurIPS, 2022. 4, 17
Bingxin Ke, Anton Obukhov, Shengyu Huang, Nando Metzger, Rodrigo Caye Daudt, and Konrad Schindler. Repurposing diffusion- bassed image generators for monocular depth estimation. In CVPR, 2024. 2, 3, 8, 9, 16, 17, 18, 20
Gyeongnyeon Kim, Wooseok Jang, Gyuseong Lee, Susung Hong. Junyoung Seo, and Seungryong Kim. Dag: Depth-aware guidance with denoising diffusion probabilistic models. arXiv preprint arXiv:2212.08880 2022. 3
Ivan Krasin, Tom Duerig, Neil Alldrin, Vittorio Ferrari, Sami Abu-El-Haija, Alina Kuznetsova, Hassan Rom, Jasper Uijlings, Stefan Popov, Andreas Veit, Serge Belongie, Victor Gomes, Abhi- nav Gupta, Chen Sun, Gal Chechik, David Cai, Zheyun Feng, Dhyanesh Narayanan, and Kevin Murphy. Openimages: A public dataset for large-scale multi-label and multi-class image classifi- cation. Dataset available from https://github.com com/openimages, 2017. 8, 10
http::/gtihubb.cmm/eennnnggg
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre- training for unified vision-language understanding and generation. In ICML, 2022. 14
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In ICMI, 2023. 14
Xian Liu, Jian Ren, Aliaksandr Siarohin, Ivan Skorokhodov, Yanyu Li, Dahua Lin, Xihui Liu, Ziwei Liu, and Sergey Tulyakov. Hyperhuman: Hyper-realistic human generation with latent structural diffusion. arXiv preprint arXiv:2310.08579, 2023.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild.
In ICCV, 2015. 7, 14
I Loshchilov. Decoupled weight decay regularization. arXiv preprint arXiv: 2017. 14
Grace Luo, Trevor Darrell, Oliver Wang. Dan B Goldman, and Aleksander Holynski. Readout guidance: Learning control from diffusion features. In CVPR, 2024. 2. 7, 9, 14, 15
Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. In ICLR, 2021.
Midjourney, https//www.midjoumey.coolll 2023.
Chong Mou, Xintao Wang, Lianghin Xie, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. arXiv preprint arXiv:2302.08453, 2023.
Jisu Nam, Gyuseong Lee, Sunwoo Kim, Hyconsu Kim, Hyoungwon Cho, Seyeon Kim, and Seun- gryong Kim. Diffusion model for dense matching. In ICLR, 2024. 3
OpenAl. Dall-e-2, https://opemii..om/ppoouuutuauat
2023. 2
Lu Qi, Lehan Yang. Weidong Guo, Yu Xu, Bo Du, Varun Jampani, and Ming- Yang. Unigs: Unified representation for image generation and segmentation. In CVPR, 2024. 2,
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.066 06/25, 2022. 4
12

---

Published as a conference paper at ICLR 2025
René Ranftl, Katrin Lasinger. David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE TPAMI, 44(3):1623-1637, 2020. 8,
René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction.
In ICCV, 2021. 8, 9
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, 2022. 5,
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. Medical Image Computing and Computer Intervention, 2015.
Saurabh Saxena, Charles Herrmann, Junhwa Hur, Abhishek Kar, Mohammad Norouzi, Deqing Sun, and David J. Fleet. The surprising effectiveness of diffusion models for optical flow and monoc- ular depth estimation. In NeurlPS, 2023a.
Saurabh Saxena, Junhwa Hur, Charles Hermaann Deqing Sun, and David J Fleet. Zero-shot metric depth with a field-of-view conditioned diffusion model. arXiv preprint 1rxi::2312.13252.. 2023b.
Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and sup- port inference from rgbd images. ECCV, 2012.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In ICML, 2015.
Kihyuk Sohn, Nataniel Ruiz, Kimin Lee, Daniel Castro Chin, Irina Blok, Huiwen Chang, Jarred Barber, Lu Jiang, Glenn Entis, Yuanzhen Li, al. Styledrop: Text-to-image generation any style. arXiv preprint wrXiv:2306.0098, 2023. 3
Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR, 2020a. 4, 16
Yang Song, Jascha Sohl-Dickstein, Diederík Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In ICML, 2020b.
Gabriela Ben Melech Stan, Diana Wofk, Scottie Fox, Alex Redden, Will Saxton, Jean Yu, Estelle Aflalo, Shao-Yen Tseng, Fabio Nonato, Matthias Muller, et al. Ldm3d: Latent diffusion model for 3d. arXiv preprint arXiv: 10853, 2023. 3, 6,
Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusion features for text-driven image-to-ing to-image translation. CVPR, 2023.
Saining Xie and Zhuowen Tu. Holistically-nested edge detection. In ICCV, 2015. 7, 14
Jiarui Xu. Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello. Open-
vocabulary panoptic segmentation with text-to-image diffusion models. In CVPR, 2023. 3
Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. In CVPR, 2024a. 16
Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiaogang Xu, Jiashi Feng, and Hengshuang
Zhao. Depth anything v2. arXiv preprint arXiv:2406.0999 09414, 2024b. 7, 14, 16. 17, 18, 20
Yi Yang and Deva Ramanan. Articulated human detection with flexible mixtures of parts. JEEE TPAMI, 35(12):2878-255 2012. 8, 16
Hu Ye, Jun Zhang, Sibo Liu. Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv preprint wrXiv:2308.06722 2023. 3,
13

---

Published as a conference paper at ICLR 2025
Yu Zeng. Vishal M Patel, Haochen Wang. Xun Huang, Ting- Chun Wang, Ming- Liu, and Yo- gesh Balaji. Jedi: Joint-image diffusion models for fintuning-free personalized text-to-image generation. In CVPR, 2024.
Jingyang Zhang, Shiwei Li, Yuanxun Lu, Tian Fang, David McKinnon, Yanghai Tsin, Long Quan, and Yao Yao. Jointnet: Extending eext-to-image diffusion for dense distribution modeling, arXiv preprint ar arXiv:2310.0634// 2023a. 2, 3, 6, 9, 17
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control text-to-image diffusion models. In ICCV, 2023b. 2, 3,5, 6, 8, 9, 14, 17, 18, 21
Shihao Zhao, Dongdong Chen, Yen-Chun Chen, Jianmin Bao, Shaozhe Hao, Lu Yuan, and Kwan- Yee K Wong. Uni-controlnet: All-in-one control to text-to-image diffusion models. NeurIPS, 2024.
Wenliang Zhao, Yongming Rao, Zuyan Liu, Benlin Liu, Jie Zhou, and Jiwen Lu. Unleashing text- to-image diffusion models for visual perception. In ICCV, 2023.
A IMPLEMENTATION DETAILS
Our model process input x, y in two parallel branches, shown in Fig. 2. In practice, we do not prepare two network branches to process two inputs. Instead, all inputs are concatenated in batch dimension and fed into the denoising UNet, batch of image inputs. They are simultaneously processed, with the condition LoRA selectively applying to the inputs. For joint cross-attention, we split x and y features and perform the cross-attention operation.
For the models we present in the paper, we use LoRA rank 64 for all adapters, including the condi- tion LoRA and the joint cross-attention LoRA. We only add the condition LoRA when the condition image falls out of natural image distribution, i.e. our Depth, SoftEdge, and Pose models. We addi- tionally incorporate a trigger word to the text prompts for these conditions, such as "depth.map".
B TRAINING
We train UniCon models, Depth, SoftEdge, Human-Pon (Pose). Human IdenantI (ID), and Ap- pearance Stable Diffusion v1.5 (Rombach et al., 2022). Stable Diffusion uses a variational au- toencoder (VAE) to define latent space for image generation. We adopt the VAE of the base model to encode and decode all type of images to and from the latent space, including annotated images like depth, edge and pose. For all models, we use AdamW (Loshchilov, 2017) optimizer with learn- ing rate le-4. The training images are resized to 512 resolution with random flipping and random cropping as data augmentation. The text prompts are generated by BLIP (Li et al., 2023; 2022) for datasets without captions. We drop out the text prompt input with rate of 0.1 to maintain the classifier-free guidance (Ho & Salimans, 2022) ability.
Depth, SoftEdge, Pose. 2024) to train on PaS (Everingham et al., 2012). Depth and SoftEdge model is trained on 16K images and Pose model is trained on 9K images of humans. To obtain the condition input, we first annotate training images with existing estimation methods. Depth, soft edge, and pose images are estimated by Depth-Anything-v2 (Yang et al., 2024b). HED (Xie & Tu, 2015) and OpenPose (Cao et al.. 2019). We encode all estimated modalities as images, following the annotators in Control- Net (Zhang et al., 2023b). For SoftEdge, we follow ControlNet (Zhang et al., 2023b) to quantize the edge maps into several levels to remove possible hidden patterns. We train Depth, SoftEdge models for 20K steps with batch size 32 and Pose model for 10K steps. Training 20K steps costs about 13 hours on two NVIDIA A800 80G GPUs. To adapt our Depth model for metric depth estimation, The
For spatially aligned conditions, follow Gradout-Guidance (Luo al.,
further fine-tune tune the model for 5K steps on images annotated with Depth Aachh-Ayytht produced Depth-Metric model is used for the depth estimation evaluation in Tab. 2.
ID, Appearance. We test our method on two cases of loosely correlated image pairs, the identity- preserving model and the eecerrance-preeevving model. The Identity model is trained on human images with the same identity. We collect 30K human images from CelebA (Liu et al., 2015), including about 5K identities with more than image. We randomly pair images with the same ID
14

---

Published as a conference paper at ICLR 2025
to generate 200K training image pairs (50 for each identity). We train the Appearance model on Panda-70M (Chen et al., 2024b). Due to issues in downloading YouTube videos in Panda- 70M, we only minimal subset with 6K videos for training. When training, a length of 16 and a resampled frame rate of 7 and randomly select two frames from loaded clips input image pairs. The IDD and Appearance models trained for 20k steps with batch size 64 distributed on 4 NVIDIA A800 80G GPUs.
load video clips with
C INFERENCE
Table 4: Example sampling schedules for different conditional generation targets given UniCon model of x.
Target
Example Sampling Schedule (50 steps)
p(x,y y)
(X50, Yo0), (X49: Y44), (x1, Y1), (Xo: Yo)
ply (x))
(Xo, Y50
(Xo, Y49), (xo. Y1 (Xo: No)
p yo)
(X5o. Yo), (X19. Yu). No), (Xo.
p(x
(x50: Y25 (X49. V25). (X18. Y24). .... (x1, Vi): (xo,
p(x(yd")
(X50. Ysu), (x49
(xr
((06" 1(06") )) (x0 yo ((66))
y
(16")
Sampling schedules. In Tab 4, we list example sampling schedules to show how our sampling strategies (discussed in Sec. 4.3) support the conditional sampling described in Eq. 3. For the partial conditioning case p(xly"" ), g(v"") means replacing latents and applying guidance according to give masked condition Yo (Sec. 4.3, Sampling with guidance). Note that the listed sampling schedule one possible schedule to achieve the target. We can alter or combine them to perform customized conditional generation. For guidance, we adopt an optimizer (e.g. Ad to compute the gradient and determine the weighting factor w, in Eq. instead of manually setting fixed weight factor, as suggested by Readout-Guidance (Luo et al., 2024).
Combining multiple models. As discussed in Sec. 4.3, to achieve multi mugti- control. We expand the details here. Suppose we have input image x and two input conditions y, z. We denote the model parameters for y and Oy Oz. Then the joint feature outputs Fjoint, Fjoint, Fjoint in Eq. 4 are computed as:
combine multiple UniCon models
F Ezy yz = JointCross (i F.j, By), F 22. F.z JointCrossAtmm Fig pjoint Way F by + waz az, Fjoint Was F Fya; pjoint W2s zmr3
(8)
where all w are weighting factors to balance the strength of each condition. To explain, we perform joint cross-attention between any maage-condition pairs yx and (x, 2) with corresponding model weights. Then the image branch will aggregate all output features as the final output.
Condition Guidance. Our ID and Appearance models that target loose correlation sometimes perform badly in the conditional generation, generating low-quality images or images not aligned with the condition. We attribute the problem the fact that loose condition has weaker influence on image generation as it allows more freedom and diversity in generated images than dense conditions. Similar problems have also been observed in text-to-image generation, with an effective solution called classifier-fre free guidance (Ho & Salimans, 2021). Therefore, we optionally utilize similar guidance scheme emphasize a certain condition signal for better condition alignment.
In specific. we alter the model output as Eg = Exep + k(Ejoint Esep) where Ejoint Exep are the model output with or without joint cross-attention and le is the guidance scale. Intuitively, the output towards the direction defined by Cjoint Exep. which means the condition signal from joint cross- attention enhanced. Furthermore, we can enable more fine-grained guidance over a specific signal when there are multiple condition signals by replacing Ejoint, Esep with <(W.on), <WWnne). Here Weon, Wune indicate the two sets of weighting factors used in Eq. Therefore e e(Wcon) e(Wune) the model output with different condition weighing factors. Take Eq 8 as example, if we want to emphasize y but not z, we can set Ways Wyz to in Woon and to in w Wonc while keeping other weights the same.
15

---

Published as a conference paper at ICLR 2025
Table 5: Relative depth estimation performance DA-2K evaluation benchmark. We com- pare UniCon-Depth model against Marigold (Ke et al., 2024), Geowizard" (Fu et al., 2025), Depth Anything VI (Yang al., 2024a) (DA V1), ZoeDepth (Bhat et al., 2023), Depth Anything V2 (Yang et al., 2024b) (DA V2). *; results from Depth Anything V2.
Method
Marigold
Geowizard*
DAT VI*
ZoeDopth
UniCon-Dephh (Ours)
DAT V2 (MIT-L)
Accuracy (%)
86.8
88.1
88.5
89.1
90.5
97.1
Table 6: Comparison between synchronous and asynchronous sampling scheduling. Syn- chronous: denoising image and depth together while replacing depth input with noisy GT depth each step. Asynchronous (default): denoising image with clean depth input. We test UniCon- Depth for depth-to-image generation.
Method
FID
CLIP Score
AbsRel
81
Sync
14.78
32.45
17.43
77.00
Asyne
13.21
32.11
9.26
91.02
D EVALUATION
D.1 EVALUATION METRICS
For conditional generation (Tab. 1), we compute Frechet Inception Distance (FID) (Heusel et al., 2017) between 6K generated images and corresponding real images. We measure condition fidelity using an estimation-na mattatiio-mmtching strategy. In specific, we estimate the condition modalities of the gen- erated images and real images. Then, we compute alignment metries over the attributes estimated on the generated images and on the real images, to measure the alignment between reference and generated images on the condition modality.
Depth alignment. We compute the Absolute Mean Relative Error (Ke et al., 2024) (AbsRel) on depth values estimated by Depth-Anything-V2 V2. We adopt the same affine-invariant evaluation pro- tocol as our depth estimation evaluation.
Edge alignment. For edge alignment, we simply compute the mean squared error on non-zero areas (i.e. edge areas) in the estimated edge maps, thus denoting it as Edge Mean Squared Error (EMSE). Because the edge map is nearly binary value map, EMSE can directly reflect the alignment of edges.
Pose alignment. For pose alignment, we compute the standard pose estimation metric Percentage of Correct Keypoints (Yang & Ramanan, 2012) (PCK) with adaptation to fit scenario. PCK measures the alignment between paired ground truth and predicted keypoints. However, the real and generated images in evaluation may include multiple humans, and we do not have matching between them. Therefore, we perform a greedy matching between real image keypoints and gen- erated image keypoints. In specific, compute pair-wise PCK across all sets of keypoints in two images. Then we match the keypoints set greedily in PCK descending order and obtain a matching between two groups of keypoints. Finally, we compute PCK over the matched keypoints.
For depth estimation evaluation, we follow the same effine-invariant evaluation setting as Marigold (Ke al., 2024), i.e. aligning the prediction with ground truth with the least squares fitting. Suppose a is the predicted depth and d is the GT depth. We compute AbsRel as My of a, d,l/d; where M is the total number of pixels. Another metric ó, defined as the proportion of pixels satisfying max(a;/ /d,, d,/a;) < 1.25.
D.2 SAMPLING SETTING
For conditional generation comparison in Tab. 1, we use the DDIM (Song et al., 2020a) scheduler with 1.a=1.0. We sample for 50 steps and use classifier-fres guidance scale of 7.5. We use identical
16

---

Published as a conference paper at ICLR 2025
lepining
lopabiig
bdge baiing
Ieuge Ocecriive
Tmags S3xa
IC'LR
Bap Eallig
Sepliiingg
Figure 10: Inpainting and editing samples. We show diverse inpainting and editing results using UniCon-Depth Depth and UniCon-Edge:
Table Inference Speed Comparison for conditional generation. We compare UniCon against ControlNet (Zhang et al., 2023b) on denoising iterations per second.
Method
ControlNet
UniCon (Remove Joint Modules)
UniCon
iteration/s 10.82
7.65
5.02
sampling settings for all comparison methods. For depth estimation in Tab. 2, we use the Euler Ancestral scheduler (Karras et al., 2022) sample 20 steps. Additionally, we find adding minor noise (10% of max noise timestep) to the input image helps improve the estimation quality.
For the qualitative comparison in Fig. 3, JointNet (Zhang et al., 2023a) does not support guidance their official implementation. Therefore, adopt the same guidance scheme their model generate the depth guidance sample. We tune our sampling setting to generate each sample, such as the noise level added to the input depth for the rough depth condition task.
the default sampling setting for specialized methods.
E ADDITIONAL RESULTS
User Study. We conduct pose, and identity conditions. For each model, users are asked to select their preferred image that aligns with the given image and text condition. We show the result Fig. where UniCon produces comparable results against other specialized methods.
study conditional generation performance for depth, softedge,
For the Appearance model, evaluating it independently is challenging because (1) Applying it in isolation often produces outputs that closely resemble the input image. We must adjust the condition weight or text prompt for specific use cases. Its effectiveness most apparent when combined with other models. Please check Fig. 1,4,8,11 for its use cases. (2) A suitable baseline for comparison is currently unavailable.
Synchronous and asynchronous sampling schedules. Our disentangled timestep sampling scheme during training enables flexible asynchronous sampling schedules at inference time, which is how UniCon handles diverse generation tasks. In Tab. 6, we compare the depth-to-image generation performance under synchronous and asynchronous sampling schedules. The results demonstrate that asynchronous noise level scheduling offers clear advantages in depth conditional generation.
For the synchronous setting, we denoise image and depth together while replacing depth input with noisy GT depth at each step. For the asynchronous setting, denoise the image with clean depth input, which is only possible due to our training strategy.
Quantitative depth estimation. Though our UniCon model still lags behind of-te--f-thh-arr (SOTA) Ieluusonnbasce depth estimators (e.g., Marigold (Ke et al., 2024)) on NYUv2 and ScanNet, existing benchmarks often lack scene diversity, noted in Depth Anything V2 (Yang al., 2024b). To ad-
17

---

Published as a conference paper at ICLR 2025
Table Latency Comparison for Depth Estimation. We compare UniCon against Marigold (Ke et al., 2024), ZoeDepth (Bhat et al., 2023), and Depth Anything V2 (Yang et al., 2024b) on denoising iterations per second, where Marigold and UniCon are diffusion- based. The latency tested on one A800 GPU.
Method
Marigold
Marigold(LCCM)
ZoeDepth
Depth Anything V2
UniCon
latency (A800)
2.3s
372ms
289ms
91ms
4.0s
dress this limitation, we further evaluate UniCon-Depti on the relative depth estimation benchmark DA-2K, introduced in Depth Anything V2, which comprises diverse high-resolution test images.
On DA-2K (Tab. 5), Our UniCon-Depth model is superior to other models except for Depth Any- thing V2 (which is used as our annotator), demonstrating that our model better generalizes diverse scenes.
Computation efficiency analysis. We provide an analysis of our inference cost. Tab. compares UniCon with ControlNet for controllable image generation. UniCon incurs additional inference overhead due the separate computation LoRA layers (condition LoRA and joint cross-attention LoRA) instead of fusing them into the pretrained weights. Therefore, separate LoRA layer computa- tion incurs heavy overhead. This overhead can potentially be mitigated by saving multiple attention weights (with and without LoRA) directly in the model.
Tab. compares UniCon with other depth estimation methods. Diffusion- -aase methods (e-g. Uni- Con and Marigold) generally exhibit higher latency than traditional approaches. However, UniCon can leverage techniques like Latent Consistency Models (LCM), as demonstrated by Marigold, to significantly improve inference speed.
In summary, while UniCon currently has higher inference costs, its computational efficiency can be enhanced through optimizations like LoRA weight fusion and LCM integration. We plan to explore these directions in future work.
Inpainting and editing using UniCon. UniCon supports diverse editing and inpainting behavior on both image input and condition input, as shown in Fig. 10. In addition to inpainting or editing one input conditioning on another, can simultancously edit and inpaint both inputs.
Empirically, we observe that our model is robust for editing tasks. However, its inpainting perfor- mance is less stable, which is limitation of the base model (Stable Diffusion). straightforward solution is to apply our method to an inpainting speciffc diffusion model, such as SD-Inpainting
Interpolating between noise schedules. In Fig. 11 show that we can interpolate the initial noise levels for input x, y to gradually change our sampling behavior in y-to-x, rough y-to-x, joint x, y generation, rough x-to-y, and x-to-y.
Qualitative results on depth estimation. We provide more qualitative results on depth estimation for our UniCon- -eptth model (Fig. 12). Our method performs better than Marigold (Ke et al., 2024) on most test cases, especially for images with ambiguous structure (Fig. 12 Line 2,5) or nen-realistic style (Fig. 12 Line 1,7,8). By learning the bidirectional correlation between image and depth, our model better preserves the natural image knowledge of the base model, while fine-tuning the whole model a conditional sampling manner (as Marigold) might destroy the pretrained image prior.
Qualitative results on conditional image generation. We provide more qualitative results of our Depth, Edge, and Pose model (Fig. 13). ControlNet (Zhang et al., 2023b) sometimes generates overrsaturated images (SoftEdge 2,4 in Fig. 13) or images not aligned with the condition (Pose 4 in Fig. 13), while it rarely happens to our method. We attribute it to that joint modeling stimulates our model to fully capture the bidirectional correlation between image and condition, while conditional modeling might learn to rely on minor clues in the condition image.
18

---

Published as a conference paper at ICLR 2025
Depth
SoftEdge
Appearance
X
X
Input
y x
T
Joint (x,y)
T-
Generation
x to
T
Initial
Noise level
Figure 11: Interpolating noise schedules. We can move from x-to-y generation to y-to-x gen- eration by interpolating the level of initial noise added to x, inputs. After adding the noise, we denoise y together to clean outputs.
19

---

Published as a conference paper at ICLR 2025
Input Image
Depth Anything V2
Marigold
UniCon (Ours)
Figure 12: Qualitative results on depth estimation. We compare UniCon-Depth against Depth Anything V2 (Yang et al., 2024b), Marigold (Ke et al., 2024). Test images are from Depth Anything V2 (Yang et al., 2024b).
20

---

Published as a conference paper at ICLR 2025
ContreINet
Uni- Eot
UniCon (Ours)
"a collection old muicouses sitting on top each other"
"wildebeent in the wildebeest migration
SuftEdge
"a woman is pacting drinks cooler"
"a pile of blondies for a crowd of mistionaries
"a reee house the with states"
and
the giving presentation
Depth
"a plane
"a dog with bow tie sitting floue"
sitting
stooo!
stuffed of wearing armay""
Pose
men
coats working macchive"
mun strting at slerk computers and
Figure 13: Qualitative results conditional image generation. We compare conditional image generation results against ControlNet (Zhang et al., 2023b), Uni-ControlNet (Zhao et al., 2024). Uni-ControlNet accepts different inputs for depth and softedge, generated by its default preprocessors (MiDaS, HED).
21


Reviews:
[{"review_id": "OSHfHmb9Fc", "reviewer": "Reviewer_zDcz", "summary": "The paper presents a diffusion model-based conditional image generation method as an add-on to current latent diffusion model (e.g., SD1.5). The model consists of joint cross attention layers and LoRA for conditional image branch, jointly denoising a concatenation of generated and conditional images. The training of the model is performed with disentangled noise levels from the two branches, inspired from Diffusion Forcing, allowing flexible conditional generation from different sampling schedules. The final model is compared to various problem-specific conditional model as well as with ControlNet, which solves similar problem with the proposed method with larger add-on network.", "questions": "These questions are from pure curiosity and not counted in my final scores.\n\n- Does the presented method applicable to applicable/robust to inpainting/editing tasks? Supporting this feature will make the proposed method far more persuasive.\n- This is a very minor concern, but do the authors want to extend this model beyond SD1.5, which seems to be deprecated in the generative AI community?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "The main strengths are in their lightweight configuration, good reported performance, and novelty in using independent timestep scheduling.\n\n- The writing of the paper is clear with comprehensive evaluations supporting the superiority of the proposed method.\n- The model is overall lightweight in terms of the size and the training time, compared to previous image conditional add-ons, e.g., ControlNets. This makes the method application-friendly.\n- Using disentangled noise level scheduling from Diffusion Forcing in paired generation for controllable image synthesis is quite novel. The demonstration in Figure 5 justifies this extension.", "weaknesses": "Although I believe the current version of the manuscript is above acceptance threshold, there are some limitations that prevents me recommending for higher honors (e.g., Highlight/Oral).\n\n- Although there are five image conditional models trained using the proposed framework, it seems that only three (Depth, SoftEdge, Pose) types are compared quantitatively. This asymmetry in the quantitative/qualitative demonstration makes the manuscript incomplete.\n- Moreover, there are also other metrics that compares the conditional generation. For example, for depth-conditional generation, there is $\\delta_1$ metric used throughout the methods (please see Table 1 of Marigold and Table 2 of Depth Anything) that compares the method in at least five datasets. For pose-conditional generation, there is multiple metrics such as CLIP Identity Observance and Pose Control Accuracy (as in Table 1 of FineControlNet). We can also use user preference study for the tasks that does not have quantitative measures (such as in identity preservation and appearance preservation). Also there is a CLIP score to report the quality of the generation as well as text-fidelity of the generation. I do not want to recommend showing all those metrics that I have mentioned (these are for examples), but I believe it is required to have a complete table of quantitative metrics at least for models presented.\n- It would be better to have a quantitative comparison in generated images from synchronous and asynchronous noise level scheduling in the inference time. The demonstration in the manuscript shows that the method works, but it does not show that this asynchronous generation leads to diversity and quality comparable to synchronous one.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "149jjD1ACn", "reviewer": "Reviewer_xQEt", "summary": "This paper introduces UniCon, a unified framework for handling diverse conditional generation tasks by learning a joint distribution over correlated image pairs using diffusion models. UniCon employs a simple architecture that adds minimal learned parameters (15% of base model) and uses a single efficient training stage while maintaining the standard model input. Through extensive experiments, the authors demonstrate that their single model can produce comparable or better results than specialized methods and prior unified approaches, while also showing that multiple models can be effectively combined for multi-signal conditional generation.", "questions": "1. What is the inference cost of depth prediction compared to other traditional and recent approaches?\n\n2. How does your method compare with ZoeDepth, Depth Anything, and Depth Anything v2?\n\n3. What are the specific advantages of joint distribution modeling over other approaches discussed in the paper, particularly in terms of inference, accuracy, and other metrics?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Unified framework for handling diverse conditional generation tasks through a joint distribution approach\n- Lightweight adaptation of existing diffusion models with minimal parameter overhead\n- Clear empirical demonstration of the proposed framework", "weaknesses": "1. **Missing Comparisons/References**\n\n* The paper lacks comparisons with several important recent methods in depth estimation, and this limits our understanding of where the method stands in relation to the current state-of-the-art in depth estimation.\n   - ZoeDepth\n   - Depth Anything\n   - Depth Anything v2 (which is only used as an annotator in this work)\n\n* In addition, it will be helpful to discuss \"DAG: Depth-Aware Guidance with Denoising Diffusion Probabilistic Models\" [Kim et al.], which appears to be a relevant line of work for conditional generation with depth information.\n\n2. **Technical Ambiguities**\n* Section 4.2 should explicitly state the model architecture, e.g., whether the approach uses two pre-trained diffusion models and requires multiple feed-forward passes.\n* This paper does not provide sufficient analysis of the inference cost. Based on current understanding, while it uses LoRA, it requires twice as many feed-forward passes, making the inference step complex and resource-intensive.\n\n3. **Joint Distribution Modeling**\n\n* While the paper presents joint distribution modeling as a key contribution, it should better justify why this particular formulation is advantageous over alternatives. The theoretical benefits of learning the joint distribution p(x,y) versus other approaches could be more thoroughly explained and experimented. For instance, their joint modeling might mutually enhance depth-aware image generation through improved representation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KkvtbkbOSw", "reviewer": "Reviewer_r1wX", "summary": "This paper proposes a framework for unifying different conditional generation tasks involving image-condition correlation in diffusion models. The proposed method learns the join probability distribution of the images and conditions during training and used for conditional generation during inference. The method mainly consist of parallel denoising paths for each condition, interacting with the image denoising path via cross-attention blocks. To maintain paramer efficiency, the base model for image generation is kept frozen, and Low Rank Adaptation (LoRA) is used to adapt the base model to each condition. The proposed method also allows for generation based on multople conditions. The authors evaluate their method on different conditional generation task involving conditions such as depth, edges, images, human poses, and identities.", "questions": "- It would be great if authors could provide an analysis on the computational cost of inference using their method.\n\n- In Section 4.2, line 233, authors mention \"When the conditional image differs from a natural image, we add a LoRA to the y-branch.\" How is the branch trained for conditioning on natural images, considering that the base image generation U-Net is fozen and no LoRA is used?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written and easy to follow.\n\n- The proposed method provides a parameter-efficient way to model the joint image-condition distribution, which is more versatile for different conditioning tasks compared to specialized conditional methods.\n\n- The authors provide sufficient experiments and comparisons for their method.\n\n- Based on the provided results, the proposed method seems effective in modeling the joint image-condition distributions, and performing conditional generation", "weaknesses": "- Conditional generation using the proposed method requires performing multiple denoising paths, which makes the inference compationally intensive compared to direct conditioning, especially for multiple conditions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "owiM5cV6kY", "reviewer": "Reviewer_TGFy", "summary": "The paper introduces UniCon, an efficient approach to unify various conditional generation adaptation paradigms by modeling a single joint distribution across multiple modalities. This is done by having two U-Net branches that interact via a joint cross-attention. By also having different noise levels per branch, as previously done in Diffusion Forcing, both joint generation and conditional generation in both directions are possible during inference. To improve parameter efficiency, non-RGB branches are initialized to the RGB branch's weights and adapted using LoRAs instead of training them from scratch.", "questions": "Regarding the weaknesses, I'd recommend the authors incorporate more related work into their quantitative comparisons and related work. Especially Uni-ControlNet seems important to compare against, as it is very closely related (except for the joint generation part).\nFor additional qualitative comparisons, I recommend following the example set by IP-Adapter, which provides a great overview of the qualitative performance of a wide range of methods.\n\nThe ablation study only focuses on RGB generation in a single setting. Have you also ablated effects in other sampling settings?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed method enables more flexible conditional generation than many previous methods (Cond -> RGB, RGB -> Cond, RGB & Cond joint generation) by posing conditional generation as a special case of a more general joint generation setup and combining it with different timesteps per sample as pioneered by Diffusion Forcing. Compared to the approaches prevalent in conditional diffusion adaptation methods, this enables more flexible generation scenarios. The quantitative and qualitative evaluations show competitive performance of the proposed method compared to more specialized approaches. The paper is easily understandable and presents the method clearly.", "weaknesses": "The method, as presented, is limited to only accepting conditioning in the form of 2D images/feature maps. As it has now become common to support those combined with feature vector conditioning, as done in, e.g., Uni-ControlNet [Zhao et al., NeurIPS 2023], Composer [Huang et al., ICML 2023], or CTRLorALTer [Stracke et al., ECCV 2024].\n\nThe paper would benefit from additional qualitative examples in the appendix, as is commonplace in similar works.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kPfcaSVs0x", "reviewer": "Reviewer_Rn1C", "summary": "This paper allows to generate diverse output modalities (image, depth maps, edge maps, poses) from diverse input modalities using a single generative model. The proposed UniCon model processes image and condition inputs “concurrently in two parallel branches, with injected joint cross-attention modules where features from two branches attend to each other.” LoRA weights are used to adapt the model from the pre-trained stable diffusion model. Experiments show that the approach performs on par with single generative approaches (controlNet, readout guidance, etc) in terms of FID, and depth, pose or edge estimation metrics, and outperform jointnet, a related multi-task generative approach in terms of results and speed.", "questions": "1/ Do you plan to release the source code to ease reproducibility of the approach? \n\n2/ Could you do a human study? \n\n3/ In figure 3, third column, the fact that both the loose control and Unicon approaches lead o a snowman on the beach would mean their latent space is really similar as both approaches build on Stable diffusion weights. Also the controlNet result (first column) is also very similar to the Unicon one. It makes question the generalization ability of the approach.\n\n4/ Why is there no quantitative comparison with the Loose control results?\n\n5/ How is the weight w_r set up in (7)", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The paper is well written and illustrated.\n* The approach is well motivated, unifying abilities of multiple generative models into a single one.\n* The approach is relatively less computationally intensive than many other diffusion based approaches. \n* The results are convincing.", "weaknesses": "* No open sourcing of the code / models is mentioned.\n* There is no human study comparing the effective quality of the approach compared to the others. There are nice qualitative examples in Fig 3, but a human study would be more convincing to demonstrate these examples are representative.  \n* One of the most concrete applications of generative image models would be serving downstream tasks, such as classification or depth estimation as demonstrated in this work. However, the estimation performance is far from SOTA performance. \n* There is a confusion in the architecture of Unets, at least a sentence to rephrase line 242: The Unet [2015] [...] does not contain transformer blocks as it is written. Unets employed in diffusion models yes, but the sentence with this reference is confusing for now.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)