More Than Generation: Unifying Generation and Depth Estimation via Text-to-Image Diffusion Models
Authors: Hongkai Lin
Dingkang Liang
Mingyang Du
Xin Zhou
Xiang Bai
Venue: NeurIPS 2025 poster ()
Keywords: unify model; image generation; depth estimation
Primary area: applications
TL;DR: A unified model for image generation and depth estimation, starting from a fixed-parameters pretrained text-to-image model.

Abstract: Generative depth estimation methods leverage the rich visual priors stored in pretrained text-to-image diffusion models, demonstrating astonishing zero-shot capability. However, parameter updates during training lead to catastrophic degradation in the image generation capability of the pretrained model. We introduce MERGE, a unified model for image generation and depth estimation, starting from a fixed-parameters pretrained text-to-image model. MERGE demonstrates that the pretrained text-to-image model can do more than image generation but also expand to depth estimation effortlessly. Specifically, MERGE introduces a plug-and-play framework that enables seamless switching between image generation and depth estimation modes through simple and pluggable converters. Meanwhile, we propose a Group Reuse Mechanism to encourage parameter reuse and improve the utilization of the additional learnable parameter. MERGE unleashes the powerful depth estimation capability of the pretrained text-to-image model while preserving its original image generation ability. Compared to other unified models for image generation and depth estimation, MERGE achieves state-of-the-art performance across multiple depth estimation benchmarks. The code and model will be made available.

More Than Generation: Unifying Generation and Depth Estimation via Text-to-Image Diffusion Models
Hongkai Lin
Dingkang Liang
Mingyang Du Xin Zhou
Xiang Bai
Huazhong University of Science and Technology (hklin, dkliang, xbai) edu. en
Project code: https: //git/uttttt com/ H- EmbodVis/MERGE
"Text to Image"
"Image to Depth"
MERGE
Figure 1: We present MERGE, a simple unified diffusion model for image generation and depth estimation. Its core lies in leveraging streamlined converters rich visual prior stored in generative image models. Our model, derived from fixed image generation models and fine-tuned pluggable converters with synthetic data, expands powerful zero-shot depth estimation capability.
Abstract
Generative depth estimation methods leverage the rich visual priors stored in pre-trained text-to-image diffusion models, demonstrating astonishing zero-shot capability. However, parameter updates during training lead to catastrophic degra- dation in the image generation capability the pre-trained model. We introduce MERGE, unified model for image generation and depth estimation, starting from fixed pre-trained sox-tto-mmaee model. MERGE demonstrates that the pre-trained text-to- to-image model can do more than image generation, but also expand to depth estimation effortlessly. Specifically, MERGE introduces play-and-ple framework that enables seamless switching between image generation and depth estimation
x Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

modes through simple and pluggable converters. Meanwhile, we propose a Group Reuse Mechanism encourage parameter reuse and improve the utilization of the additional learnable parameters. MERGE unleashes the powerful depth estimation capability of the pre-trained text-to-image model while preserving its original im- age generation ability. Compared to other unified models for image generation and depth estimation, MERGE achieves state-of-the-an performance across multiple depth estimation benchmarks.
(text) or Image
(text) Image
(text)
Image
(text) Image +
Pre-trained TeI Model
Pre-trained TuI Model
&
Diffusion Transformer
Pre-trained T2l Model
converters &
Pre-trained Tzl Model
Image or Depth
Large- scale
Image or Depth
Image or Depth
Image or Depth
(a) Marigold
(b) Parallel Model
(c) Data-driven
(d) Ours Method
Text-Image
Image- Depth i Training data
O Interaction
Text-to-Image
Image-to-Depth
Frozen
Learnable
Figure 2: The comparison between existing methods and ours shows that, unlike previous works, our method requires only a few additional parameters unleash its powerful depth estimation capability without compromising its inherent T21 generation ability.
Introduction
Diffusion models [12], especially text-to-image (T2I) models [1, 19, 38], have made astonishing progress in the quality of generated images. Meanwhile, emerging generative depth estimation research [10, 11, 16] reveals the latent potential for depth estimation tasks of advanced T2I models. Exploring a framework bridging image generation and depth estimation is an essential step toward building Unified Model used for generation and visual perception.
Marigold [16]. pioneer of generative depth estimation, shows powerful zero-shot depth estimation method leveraging pre-trained T2I diffusion models, as shown in Fig. 2(a). Even though the irre- versible degradation of its original generation capability due to parameter updates during training, the paradigm still makes it possible to unify image generation and perception within diffusion models. In contrast, JointNet [52] and UniCon [21] present an alternative strategy by utilizing a parallel dual-model interaction architecture to unify generation and depth estimation tasks, as shown in Fig. 2(b). The latest work, OneDiffusion [20], introduces a data-driven solution, as shown in Fig. 2(c), to train unified generation and perception model from scratch with massive multitask data (100M).
Despite remarkable progress, such approaches may be considered inefficient or resource-intensive solutions, potentially suboptimal choices. Considering rich visual prior is stored in advanced T21 models, natural question arises: Can T21 models effortlessly expand depth estimation capability without degrading their image generation capability? This paper explores a unified diffusion model without bells and whistles for image generation and depth estimation tasks, starting from a fixed T2I model. Our method, referred to as MERGE (as shown in Fig. 2(d)), demonstrates that T21 models can do More than RenERate imaGEs. With a few simple converters, they can effortlessly unleash the powerful depth estimation capability.
To enable depth estimation ability in fixed T2I diffusion model, we present a play-and-plug framework without bells and whistles. Before each transformer layer of the pre-trained T2I diffusion transformer (DiT), hereafter referred to as the T2I block, we introduce an identical, learnable T2I block as a converter. This converter transforms the latent features originally tailored for the T2I task into features suitable for the depth estimation task. This straightforward enables seamless switching between the original image generation and depth estimation models by skipping or running these converters. Meanwhile, considering the similarity of output features between pre-trained T2I blocks, we propose Group REuse (GRE) mechanism to improve the utilization of the converter. Specifically, pre-trained T2I blocks are divided into several groups, and shared aithin-group converter is used to effortlessly transform the T2I model into generative
say-and--luu design

---

depth estimation model. During inference, if the text-to-image generation capability is needed, these additional converters can be easily skipped, restoring the depth estimation model to the T2I model. In addition, some empirical studies further simplify the converters with virtually no impact on MERGE'S depth estimation performance. Combining these simple, flexible, and pluggable converters with the group reuse mechanism enables MERGE to unify image generation and depth estimation with no difficulty. Remarkably, MERGE presents a unified model starting from a fixed T2I model, requiring only few additional parameters and fine-tuning on depth estimation datasets.
We conduct comprehensive evaluations of MERGE on established depth estimation benchmarks (NYUv2 [40], ScanNet [6], and DIODE [43]). The results show that MERGE achieves state-of the- art performance across multiple metrics and benchmarks while introducing merely 12% additional trainable parameters. Especially on the NYUv2 benchmark, MERGE achieves 5.9 A.Rel and 95.4% 81, outperforming OneDiffusion, which is trained from scratch with massive data.
MERGE demonstrates a simple and effective approach to enabling pre-trained T21 diffusion models with depth estimation capability while preserving their inherent T2I generative ability. We hope our work provides valuable insights into the construction of unified models for generation and perception, while offering a cost-effective solution (~ 12% additional parameters) for extending the capabilities of pre-trained image generation models. The main contributions of this work are as follows: 1) We explore how to unleash the depth estimation capability from fixed pre-trained T2I models while preserving their original generation ability and propose simple method without bells and whistles called MERGE. The key lies in a flexible play-and-pluy framework that allows seamless switching between image generation and depth estimation modes. 2) We present a streamlined converter that effortlessly transforms features suited for image generation into depth estimation. Considering the feature similarity between different layers in pre-trained T2I models, we propose Group Reuse Mechanism, which encourages converter reuse to improve parameter efficiency.
2 Related Work
Text-to-Image Diffusion Model. Recently, text-to-image (T2I) generation models [1, 30, 39] based on the diffusion principle [12] make significant progress in image quality, diversity, and efficiency. Stable Diffusion [38], as the pioneering latent diffusion model, compresses the T2I generation process into the latent space using VAE [17], greatly improving computational efficiency and the quality of generated images. SDXL [32] introduces dual text encoders [15, 33] and fully upgrades Stable Diffusion, further enhancing the quality and resolution of generated images. VQ-Diffusion [9] proposes vector quantized diffusion model by replacing VAE with VAE [41], addressing the unidirectional bias problem. Unlike these denoising diffusion models based on UNet architectures, PixArt [4, 5] introduces novel text-to-image paradigm built upon Diffusion Transformers (DiT) [31]. By integrating vision-language models [27] to construct high-quality text-image pairs data, Pix achieves performance comparable advanced works at significantly reduced computational cost. MMDiT, an innovative Multimodal Diffusion Transformer architecture, is introduced by Stable Diffusion 3 [7] and demonstrates excellent image generation capability. The otate-of-the-arr method, FLUX [19]. builds a larger T2I diffusion model by mixing MMDiT with the Single-DiT, presenting stunning image generation capability. Distinguishing these excellent T2I works, this paper focuses on exploring how to unleash the depth estimation capability of pre-trained T2I models (specifically those transformer-basee architectures) while preserving their inherent T2I generation ability.
Diffusion-based Depth Estimation Model. Unlike discriminative perception methods [23, 24, 36, 45, 46, 48] and these works [22, 44, 47, 54] that treat T21 models as feature extractors, emerging research [14, 16, 25] reveals the compelling potential of pre-trained T21 models as generative depth estimation models. GeoWizard [8] leverages geometry switcher inspired by Wonder3D [29] to extend a single stable diffusion model to produce depth and normal results. Mcanwhile, a simple yet effective scene distribution decoupler strategy proposed in this work boosts the capture of 3D geometry. DepthFM [10] is the pioneer of combining Flow Matching [26] to explore generative monocular depth estimation, significantly improving inference speed while maintaining accuracy. BetterDepth [53] is a straightforward and powerful depth estimation framework that refines the results of generative depth estimation by using the output of foundation depth estimation model [48, 49] as a prior. Lotus [11] introduces tuning strategy called detail preserver that achieves more accurate and fine-grained predictions, especially on reflective objects. JointNet [52] and UniCon [21] propose unified architecture for image generation and depth estimation, leveraging parallel dual diffusion
3

---

Noise Sampler
Text Encoder
(text) or
Copy
Copy
Text
VAE
VAE Encoders
Encoder
Encoder
ImagezDepti
Texta Destelmage
convertees
Pretrained Tal Model
VAE Encodeng
Share
Group 1
Share
Group
VAE Decoder
Add noise Concat
Tel block
Converter
Initialization
* Frozen
Learnalle
Image
Depth
(a) Training phase of our MERGE
(b) Inference phase
Figure 3: The pipeline of MERGE. Starting from the fixed DiT-based text-to-image (T2I) model, where transformer layers (hereafter referred to T2I blocks) are divided into different groups. shared and learnable converter is inserted before each T2I block within group, transforming it into depth estimation model. can be reverted to the original T2I model by skipping these converters.
models with feature interaction operations between them. The latest work, OneDiffusion [20], presents massive data-driven unified model for image generation and depth estimation. In contrast to existing full-parameter fine-tuning methods, which catastrophically degrade the image generation inturcceintensive approaches, explore method that
ability of pre-trained T2I models, preserves the inherent generation capability of the T2I model while seamlessly transforming into a generative depth estimation model.
Preliminaries
In this section, we review latent text-to-image diffusion models, with Stable Diffusion [38] serving as representative of the latent diffusion paradigm.
Diffusion Models (DMs) [12] establish advanced text-to-image generation frameworks renowned for synthesizing photorealistic images through iterative denoising. DMs learn to reconstruct complex data distributions by approximating the reverse of predefined diffusion process. Denoting & as the random variable at the t-th timestep, the diffusion process is modeled as Markov Chain:
2t N WV (1 a.))
(I)
where a is fixed coefficient predefined in the noise schedule, and / refers to the identity matrix. A prominent variant, the Latent Diffusion Model (LDM) [38], innovatively shifts the diffusion process of standard DMs into latent space. This transition notably decreases computational costs while preserving the generative quality and flexibility of the original model. The resulting efficiency gain primarily arises from the reduced dimensionality of the latent space, which allows for lower training costs without compromising the model's generative capability.
Stable Diffusion, exemplary implementation of LDM, comprises AutoEncoder [41] and latent denoising model. The AutoEncoder e is designed to learn a latent space that is perceptually equivalent to the image space. Mcanwhile, the LDM is parameterized as a denoising model with the multimodal feature interaction module and trained on a large-scale dataset of text-image pairs via:
LLDM = Be c0(2,, l, To (1))
where e is the target noise, and is an RGB image. тo and are the pre pre-trained text encoder (e.g.,
CLIP [33], T5 [34]) and text prompts, respectively. This equation represents the equared error (MSE) between the target noise c and the noise predicted by the model, encapsulating the
core learning mechanism of the latent diffusion model. Some of the latest text-to-image works (e.g., Stable Diffusion 3.5 [7], FLUX [19]) attempt to introduce more advanced flow matching [26] as an optimization objective, as it can more directly establish a mapping between the noise and data distributions, further improving image generation quality and efficiency.

---

Our MERGE
To unleash the potential depth estimation capability of pre-traue nee-trained text-to-image (T2I) models while retaining their inherent image generation capability. We present MERGE, as shown in Fig. 3(a), which requires only simple modifications to the pre-trained T2I model that can effortlessly unify image generation and depth estimation. The core of MERGE lies play-and-plug framework and the Group Reuse Mechanism without bells and whistles, designed for unified image generation and depth estimation. In addition, we present important empirical investigations, which significantly reduce the number learnable parameters while keeping depth estimation capability virtually unaffected.
4.1 Play-and-Plug Framework
The T2I diffusion model learns the training data distri- bution to generate infinite data. Among these infinite pixel-level combinations that form images, there may exist underlying information highly correlated with the depth estimation modality. However, due to weak prompts and the sparse representation of depth information within the overall data distribution, extracting this latent feature of depth estimation modality from fixed pre-trained T21 model is challenging.
28
26
1.0
0.5
24
0.9
0.4
Layer Index
0.8
8
0.9
In contrast to previous works [10, 11, 16] on fine-tuning the denoising models with full parameters, which catas- trophically disrupt the inherent generation ability, we intro- duce play-and-plug learnable converters before each T2I block in the pre-trained T2I model to guide and unleash its potential depth estimation capability, This play-and-plug converter design more naturally leverages the visual pri- ors stored in the pre-trained T2I model, presenting highly comparable even better depth estimation capability than full- parammeer fine-tume tie-ttnnng with fewer training parameters. This straightforward strategy, without bells and whistles, demonstrates remarkable depth estimation capability and preserves the inherent image generation capability the pre-trained T2I model, which aspect that previous works do not explore.
4
0.9
0.8
0.1
0.1
0.1
2
1.0
0.9
0.8
0.7
0.1
0.1
2
6
8
24
26
28
Layer Index
Features similarity between TzT blocks
Figure 4: The cosine similarity be- tween the output features different T2I blocks within the PixArt [5] model.
4.2 Group Reuse Mechanism
The solution copying a learnable block before each pre-trained T2I block as converter is simple yet effective. However, this method, which can be seen as directly coupling two models, does efficiently leverage the powerful visual knowledge of the pre-trained T2I model to seamlessly guide its latent depth estimation capability, making an inefficient ap- proach. Interestingly, we observe that the closer two T2I blocks in the pre-trained T2I model are more similar in their output features most cases, shown in Fig. 4. Considering this observation, Group Reuse Mechanism (GRE) to encourage converter reuse and prove parameter utilization. Specifically, the T2I block in the pre-trained denoising model is divided into different groups, as shown Fig. 3(a). The T21 block in group shares converter, which transforms the latent features suitable for the image generation task into the depth estimation task. Since GRE considers the feature similarity between different T2I blocks, it significantly reduces the additional learnable parameter number minor performance cost.
Remove
Cross-Attention
Self-Attention
propose
Converter
Simplify
Self-Artention
Converter
Converter simplified
The collaboration between the play-and-plug framework and the Group Reuse Mechanism forms MERGE, which efficiently and flexibly unleashes the powerful zero-shot depth estimation potential of the pre-trained T2I model while retaining its inherent image generation capability.
Figure 5:
The pro- cess of converter simpli- fication, exemplified by PixArt [5].
5

---

4.3 Empirical Investigation
To smoothly transform the pre-trained T2I model into generative depth estimation model through MERGE, the converter in MERGE is copied from the first T21 block of the respective group in the pre-trained T21 model, as illustrated by Converter in Fig. 5. These converters are designed for T21 tasks and generally include components for multimodal feature interaction, such as Cross-Attention However, existing diffusion-based generative depth estimation methods typically set the text prompt to empty. Intuitively, we suspect that the multimodal feature interaction design in the converter might be redundant for MERGE, as an empty text prompt contains no useful information. The experiments in Sec. 5.3 confirm this hypothesis, demonstrating that directly removing this multimodal interaction design, as illustrated by Converter B in Fig. 5, has almost no effect on the depth estimation performance of MERGE. This simplified converter reduces the 25% learnable parameter number in MERGE used to unleash depth estimation capability.
Similarly, in some of the latest MMDiT-based methods, like FLUX [19], a dual-stream multimodal interaction block can be simplified into a single-stream block to serve as a converter for MERGE. Using these simplified converters, MERGE can naturally leverage the rich visual priors of the pre- trained T2I model to demonstrate powerful zero-shot depth estimation capability while preserving its inherent image generation ability, which full-parameter fine-tuning methods cannot achieve.
5 Experiments
Dataset and Evaluation metrics. Following prior work [16, 20, 21], we train our model on the Hypersim [37] and Virtual KITTI [3] datasets, which correspond to synthetic indoor and outdoor datasets, respectively, comprising total of 74k training samples. Subsequently, we evaluate its depth estimation performance on three real-world datasets: NYUv2 [40], ScanNet [6], and DIODE [43]. The quantitative evaluation metrics include absolute relative error (A.Rel) and the percentage of inlier pixels (61) with thresholds of 1.25.
Implementation Details. We implement MERGE using PyTorch and employ PixArt-Xl-2-512x4
and FI.UX.1-der our pre-trained text-to-image (T2I) models. The training is solely conducted on depth estimation data to build a unified image generation and depth estimation model. Following previous work [16], we double the input channel of the patchify layer, enabling to handle image conditions for generative depth estimation. During the depth estimation inference process, this patchify layer can seamlessly replace the patchify layer used in the T2I process. Regarding the group reuse mechanism, we adopt a default strategy of evenly dividing groups to avoid model-specific designs. Training our MERGE takes 30K iterations using a batch size of 32. We use the Adam optimizer with learning rates of le-4 and 3e-4 for PixArt and FLUX, respectively. Additionally, we follow the defaults provided in [16] for depth data preprocessing settings. For hyperparameters not mentioned in the MERGE training process, we follow the fine-tuning settings provided by the official pre-trained T2I model. All of our experiments are implemented on 8 NVIDIA H20 GPUs.
5.1 Main Results
We present two versions of MERGE with different parameter scales using the pre-trained PixArt [5] and FLUUX [19] models, referred to as MERGF-B and MERGE-L,, respectively. Unless otherwise specified, the number of groups for MERGE-B and MERGF-L. is set to 14 and 10, respectively.
Compared with the state-of-the-art. As shown in Tab. 1, MERGE-B achieves superior performance over JointNet [52] and UniCon [21] with only 1 110M additional learnable parameters. Distinguishing these methods that require running dual diffusion models in parallel, the play-and-plug framework of MERGE is more computationally efficient, as it only injects fewer converters in sequential manner when needed. Notably, due to the feature interaction design between the parallel diffusion models, registers new knowledge into the pre-trained T2I model, which affects its original image generation capability, whereas MERGE does not.
Compared to the latest work, OneDiffusion [20], which is driven by 100M data, our MERGE-L achieves superior performance, surpassing OneDiffusion by 0.9 A.Rel on NYUv2 and 1.1% 81 on DIODE, while requiring only approximately 12% additional parameters of the pre-trained T2I model. Remarkably, the learnable parameter of MERGE needs only about half the learnable parameters of OneDiffusion By leveraging the powerful visual knowledge stored in the fixed pre-trained T2I

---

Table I: Quantitative comparisons on zero-shot depth estimation. We compare our MERGE with works capable of both image generation and depth estimation zero-shot depth estimation benchmarks. "-B" and "-L." refer to MERGE based on the pre-trained Pix [5] and FLUX [19] models, respectively. "#Param." refers to the learnable parameter number, and "(-%)" in "#Param." represents the proportion of trainable parameters relative the size of the original model. Bold numbers are the best.
NYUv2
ScanNet
DIODE
Method
Reference
Training Data
#Param.
A.Rel 61
A.Rel 81 A.Rel + 61 f
These discriminative methods have depth estimation capability only.
DPT [35]
ICCV 21
1.2M
123M(100%)
9.8
90.3
8.2
93.4
18.2
75.8
HDN [51]
NeurIPS 22
300K
123M(100%)
6.9
94.8
8.0
93.9
24.6
78.0
DepthAnything [48]
CVPR 24
63.5M
335M(100%)
4.3
98.1
4.2
98.0
27.7
75.9
DepthAnythingy2 [49]
NeurlPS 24
62.5M
1.3((100%)
4.4
97.9
6.5
95.4
These generative methods have depth extimation capability only.
Marigold [16]
CVPR 24
74K
889M(100%)
5.5
96.4
6.4
95.1
30.8
77.3
GeoWizard [8]
ECCV 24
280K
889M(100%)
5.2
96.6
6.1
95.3
29.7
79.2
DepthFM [10]
AAAT
63K
889M(100%)
6.5
95.6
22.5
80.0
Lotus [11]
ICLR
59K
889M(100%)
5.4
96.8
5.9
95.7
22.9
72.9
These generative methods support buth image generation and depth estimation.
JointNet [52]
ICLR
65M
889M(100%)
13.7
81.9
14.7
79.5
UniCon [21]
ICLR
16K
125M(15%)
7.9
93.9
9.2
91.9
OneDiffusion [20]
CVPR 25
100M
2.8B(100%)
6.8
95.2
29.4
75.2
MERGE-B (ours)
74K
110M(18%)
7.5
94.2
9.9
89.8
32.5
74.9
MERGE-L (ours)
74K
1.4B(12%)
5.9
95.4
7.1
94.0
31.4
76.3
model, MERGE unifies image generation and depth estimation with less than one-thousandth of OneDiffusion' 's training data scale. Moreover, MERGE demonstrates overall superior depth estimation capability compared to OneDiffusion.
In addition, in the qualitative comparison, as shown in Fig. 6, MERGE demonstrates superior visual results, particularly in hollow and reflective regions, highlighted by the black boxed areas.
Compared with efficient low-rank fine-tuning methods. We also present the comparison results between the efficient low-rank fine-tuning method and MERGE based on the same pre-trained T2I model. To ensure a fair comparison, we adjust the rank to match the number of learnable parameters in MERGE-B. The results shown in Tab. 2, compared to LoRA [13] and DoRA [28], MERGE-B performs better on all datasets. Unlike LoRA and DoRA, which fine-tune parameters at a finer-grad greined layer level, MERGE shows structured converter at higher level enable depth estimation ability and proposes group reuse mechanism to improve parameter utilization.
Compared with full-parameter fine-tuning. To further demonstrate the excellent zero-shot depth estimation capability of MERGE, Tab. 3 presents comparison with the Marigold [16] paradigm, generative depth estimation approach that full-parameter fine-tunes the pre-trained T2I diffusion model. We use the same pre-trained T2I model to present the Pix version of Marigold (Marigold-P) As shown in Tab. MERGE-B achieves results highly comparable the full-parameter fine-tuning used by Marigold while requiring only 18% additional learnable parameters. Notably, if the converter injected before each pre- traitre T2I block, referred to as MERGE-BAL -B-28. it performs better on the NYUv2 dataset with only 37% learnable parameters of Marigold-P. More importantly, MERGE retains the image generation capability of the pre-trained T21 model thanks to the play-and-plug framework, which the Marigold paradigm can not achieve due to full-parameter fine-tuning disrupting the original feature distribution.
Compared with traditional discrininative methods. In terms of quantitative performance, state- of-the-art traditional discriminative methods (e.g., DepthAnything [49]) still lead in monocular depth estimation, as shown in Tab. 1, particularly on challenging benchmarks like DIODE [43] that
7

---

UniCon (ICLR 25)
OneDiffusion (CVPR 25)
Input RGB Image
Ours
Ground Truth
Figure 6: Qualitative comparison between MERGE and other unified image generation and depth estimation methods on the NYUv2 benchmark. MERGE shows better depth estimation results, especially in detailed areas (such as free-form voids and reflect regions).
Table 2: Compared with low-rank fine-tuning methods based on the same text-to-image model [5].
NYUv2
ScanNet
DIODE
Method
Rank
#Param.
A.Rel +
81 f
A.Rel 4
51
A.Rel
81 f
LoRA [13]
128
110M
8.7
92.3
10.8
88.0
32,9
73.9
DoRA [28]
128
110M
8.6
92.4
10.6
88.4
32.8
74.3
MERGE-B
110M
7.5
94.2
9.9
89.8
32.5
74.9
Table 3: Compared with full-parameter fine-tuning. For a fair comparison, we adopt the same pre-trained T21 model present the PixArt variant of Marigold [16], referred to Marigold-P. "-28" indicates the number groups is 28.
Support Tasks
NYUv2
ScanNet
DIODE
Method
#Param.
Generation
Depth
A.Rel
81
A.Rel
81
A.Rel
81
Marigold-P
x
596M
7.4
94.2
9.5
90.1
31.5
76.0
MERGE-B
110M
7.5
94.2
9.9
89.8
32.5
74.9
MERGE-B-28
224M
7.0
94.7
9.2
91.1
31.8
75.5
include diverse outdoor scenes. However, it is important to note that the performance gap narrows significantly on indoor datasets, for example, our MERGE achieves highly competitive performance to DepthAnything v2 on the NYUv2 [40] benchmark (A.Rel: 4.4 us. 5.9, 81: 97.9% us. 95.4%).
5.2 Generalization Normal Estimation Task
To further present the generality of our MERGE, we extend the zero-shot surface normal estimation. Following prior work [11], we employ the Hypersim [37] and Virtual KITTI [3] datasets as training data. Specifically, we filter out samples in Hypersim with invalid annotations and combine them with 20K Virtual KITTI training data, resulting in 59K training samples. All training hyperparameters are consistent with those used for the depth estimation task. We quantitatively evaluate the normal estimation performance of MERGE on NYUv2 [40], ScanNet [6], iBims-1 [18]. and Sintel [2], reporting mean angular error (m.) as well as the percentage of pixels with angular error below 11.25

---

Table 4:
Quantitative comparisons on normal estimation. "-B" and "-L" refer MERGE based on the pre-trained PixArt [5] and FLUX [19] models, respectively. Bold numbers are the best.
Support Tasks
NYUv2
ScanNet
iBims-1
Sintel
Method
Reference
Gen. Normal m. 11.25
m. 4 11.25' t tm. 11.225 1m. 11.25
Marigold [16]
CVPR 24
X
20.9
50.5
21.3
45.6
18.5
64.7
GeoWizard [8]
ECCV 24
X
V
18.9
50.7
17.4
53.8
19.3
63.0
40.3
12.3
StableNormal [50] SIGGRAPH 24
v
18.6
53.5
17.1
57.4
18.2
65.0
36.7
14.1
Lotus [11]
ICLR
59.4
15.1
17.2
66.2
33.6
21.0
MERGE-B (ours)
V
V
21.6
52.1
21.8
50.1
19.7
63.5
41.4
15.2
MERGE-L (ours)
v
20.1
54.3
18.3
60.2
18.2
66.2
36.9
21.3
Table 5: Ablation of the composition of converters. Here, GRE is disabled (i.e., unique converter is inserted before each T21 block). SA/CA means the Self-Attention/Cros-ttteee
Setting
SA
CA
FFN
FFN Scale
#Param.
A.Rel
51 t
A
RRGD
V
V
X4
596M
6.9
94.8
X
x4
447M
94.6
X
X
V
x4
298M
93.6
V
X
X
149M
7.4
94.2
B
V
X
V
x1
224M
7.0
94.7
As shown in Tab. even compared with existing task-specific generative methods based full- parameter fine-tuning, our MERGE-L achieves highly comparable or superior results across multiple benchmarks. More importantly, MERGE preserves the original image generation capability of the pre-trained text-to-image model, which others can not achieve due to full-parameter fine-tuning disrupting the original feature distribution.
5.3
Ablation Studies
Unless otherwise specified, conduct ablation studies PixArt [5] using the NYUv2 dataset [40], and the training hyperparameters remain consistent with those mentioned earlier.
Ablation the composition of the converter. We first validate the hypothesis in Sec. 4.3 under the setting of not enabling GRE, where the multimodal interaction design, Cross-Attention is redundant for MERGE'S converters, it fails to extract effective information from empty text prompt. The results, shown in setting A and setting B Tab. 5, demonstrate almost no performance difference between with and without parameter number. Meanwhile, we further explore the impact of other components in the converter, such Self-Attention and the Feedforward Network (FFN). As demonstrated by the results of setting and setting D, removing Self-Attention or FFN reduces the learnable parameters of the converter, and it significantly degrades the depth estimation performance.
while the latter reduces approximately 25% the learnable
may be ascribed to these operations' sole focus on latent image features, and removing these components would significantly weaken the model's representational capacity. Additionally, an interesting finding is that reducing the expansion rate of the FFN from 4 to in converters, as illustrated in setting E, causes almost no impact performance. This significantly reduces the learable parameter number of the converter by approximately 36%.
Ablation on the Group Reuse Mechanism (GRE). To demonstrate the effectiveness of the Group Reuse Mechanism, we divide two adjacent T2I blocks into group, resulting in a to- tal of 14 groups with no overlap between them. When the converter is fixedly inserted before the first T2I block of each group, i.c., without GRE, it causes a significant decrease depth estimation performance (A.Rel from 7.0 + 15.6. 81 from 94.7 + 78.8), as shown in the first two rows of Tab. 6. However, when sharing converter within group, the same parameter number yields significantly improved result (A.Rel from 15.6 - 7.5. 51 from 78.8 94.2).

---

Compared to inserting converter before each pre-trained T21 block, GRE reduces the learnable parameter number by about half, with only a slight performance cost. Tab. 6 also presents the impact of different group divisions on depth estimation performance. The results show that as the number of divided groups increases, the depth esti- mation performance of MERGE gradually improves. It is reasonable since features between consecutive layers share similarities but still exhibit differences. With more groups, these differences are more effectively balanced. Considering the trade-off between computational cost and performance, we ultimately adopt the setting of 14 groups as the default configuration for MERGE-B.
Table 6: Ablation of Group Reuse Mech- anism (GRE).
Groups GRE #Param.
A.Rel
81 f
28
x
224M
7.0
94.7
14
x
110M
15.6
78.8
14
V
110M
7.5
94.2
7
V
56M
7.8
93.5
4
<
32M
9.3
91.0
Ablation on the number of converters. We also demon- strate the impact of stacking varying numbers of converters on depth estimation performance. The results in Tab. 7 show that stacking multiple converters to create a larger one is ineffective and even has negative effects. We sus- pect this is due to the significant increase in model depth, making optimization extremely challenging.
Table 7: Ablation of converter number.
Number
#Param.
A.Rel +
81 f
1
110M
7.5
94.2
2
224M
7.6
93.8
3
335M
8.2
92.9
Ablation on the initialization methods of the converter. We further investigate the impact of different initializa- tion methods for the converter on the depth estimation performance. The results in Tab. 8 show that initializing the converter with the pre-trained T2I block (the first T2I block of each group) consistently outperforms random ini- tialization. This initialization strategy enables the converter to process the input features seamlessly.
Table 8: Ablation of initialization type.
Init. Type
A. 4
81
Random Pre-trained
7.9 7.5
93.5
94.2
Ablation on different text prompts. We finally investi- gate the impact of different types of text prompts on the depth estimation performance of MERGE. Specifically, we conduct experiments using three types of prompts: the default empty prompt, a fixed "depth map" prompt, and dense captions generated by LeVV---igtnnigg-Mi -PPT
Table 9: Ablation of the text prompt.
Prompt Type
A.Rel &
81 f
Empty prompt "Depth map"
7.5
94.2
7.4
94.3
Dense caption
7.3
94.4
7B [27]. As shown in Tab. 9, our results demonstrate that more specific captions provide perfor- mance boost for our method. Considering the trade-off between the additional cost of obtaining dense captions and the resulting benefits, we utilize an empty prompt by default.
6 Conclusion
This paper presents MERGE, method that starts from fixed pre-trained text-to-image (T2I) model and unleashes its depth estimation capability while preserving its inherent image generation ability. Specifically, MERGE introduces a unified model without bells and whistles, which can seamlessly switch between original image generation and depth estimation modes through pluggable converters. Meanwhile, through empirical research, MERGE presents a simple and effective converter. Moreover, considering the similarity of output features across pre-trained T2I blocks, MERGE proposes a Group Reuse Mechanism to encourage parameter reuse, significantly reducing the additional parameter number. Extensive experiments demonstrate the effectiveness of MERGE. MERGE presents new perspective for unifying multiple generative tasks besides the existing approaches that rely on massive data. Mcanwhile, this play-and-pll strategy offers cost-effective solution for extending the functionality of existing generative models.
Limitation. Despite the promising results achieved, our method still has limitations. For example, performing semantic segmentation using MERGE is highly challenging. One possible solution is to map semantic IDs to the RGB space [20]. However, the randomness of the denoising model makes it difficult to achieve stable and accurate segmentation result by one-shot ID mapping from generated RGB results. In addition, integrating bit encoding methods, e.g.. LDMSeg [42], is incompatible with the VAE of existing pre-trained text-to-image diffusion models. We consider this an important direction for future exploration.
10

---

Acknowledgement.
This work was supported by the NSFC (62225603, 62441615, and 623B2038).
References
[1] Jason Baldridge, Jakob Bauer, Mukul Bhutani, Nicole Brichtova, Andrew Bunner, Kelvin Chan, Yichang Chen, Sander Dieleman, Yuqing Du, Zach Eaton-Rosen, al. Imagen 3. arXiv preprint arXiv:2408.07009 2024.
[2] Daniel Butler, Jonas Wulff, Garrett B Stanley, and Michael J Black. A naturalistic open souree movic for optical flow evaluation, In Proc. of European Conference on Computer Vision, 2012.
[3] Yohann Cabon, Naila Murray, and Martin Humenberger. Virtual kitti 2. arXiv preprint arXiv:2001. 2020.
[4] Junsong Chen, Chongjian Ge, Enze Xie. Yue Wu, Lewei Yao, Xiaozhe Ren, Zhongdao Wang, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-o: Weak-to-strong training of diffusion transformer for 4k text-to-imase -mage generation. In Proc. of European Conference on Computer Vision, 2024.
[5] Junsong Chen, Jincheng Yu, Chongjian Ge. Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, et al. Pixart-alpha: Fast training of diffusion transformer for photorealistic text-to-image synthesis. In Pruc. of Intl. Conf. Learning Reprexentations, 2024.
[6] Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: dichly-annotated 3d reconstructions of indoor scenes. In Proc. of IEEE Intl. Conf. on Computer Vision and Pattern Recognition, 2017.
[7] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, al. Scaling rectified flow transformers for high-resolution image synthesis. In Proc. of Intl. Conf. on Machine Learning, 2024.
[8] Xiao Fu, Wei Yin, Mu Hu, Kaixuan Wang, Yuexin Ma, Ping Tan, Shaojie Shen, Dahua I Lin, and Xiaoxiao Long, Geowizard: Unlcashing the diffusion priors for 3d geometry estimation from single image. In Proc. of European Conference on Computer Vision, 2024.
[9] Shuyang Gu. Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for text-to-image synthesis. In Proc. of IEEF Intl, Conf. Computer Vision and Pattern Recognition, 2022.
[10] Ming Gui, Johannes Schusterbauer, Ulrich Prestel, Pingchuan Ma, Dmytro Kotovenko, Olga Grebenkova, Stefan Andreas Baumann, Vincent Tao Hu, and Bjöm Ommer. Depthfm: Fast monocular depth estimation with flow matching. In Proc. of the AAAI Conf. on Artificial Intelligence, 2025.
[11] Jing He, Haodong Li. Wei Yin, Yixun Liang, Leheng Li, Kaiqiang Zhou, Hongbo Zhang, Bingbing Liu, and King-Cong Chen. Lotus: diflusion-based visual foundation model for high-quality dense prediction. In Proc. of Inll. Conf. on Learning Representations, 2025.
[12] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Proc. of Advances in Neural Information Processing Systems, 2020.
[13] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In Proc. of Intl. Conf. on Learning Representations, 2022.
[14] Wenho Hu, Xiangjun Gao, Xiaoyu Li, Sijie 7hao, Xiaodong Cun, Yong Zhang, Long Quan, and Ying Shan. Depthcrafter: Generating consistent long depth sequences for open-world videos. In Proc. of IEFF Intl. Conf. on Computer Vision and Puttern Recognition, 2025.
[15] Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openelip, July 2021.
[16] Bingxin Ke, Anton Ohukhov, Shengyu Huang, Nando Metzger, Rodrigo Caye Daudt, and Konrad Schindler. Repurposing diffusion--aase image generators for monocular depth estimation. In Proc. of IEEE Intl, Conf. on Computer Vision and Pattern Recognition, 2024.
[17] Diederik P Kingma. Auto-encoding variational bayes. In Proc. of Intl. Conf. on Learning Representations. 2014.

---

[18] Tobias Koch, Lukas Liebel. Friedrich Fraundorfer, and Marco Korner. Evaluation of can-based single-image depth estimation methods. In Proc. of European Conference on Computer Vision, 2018.
[19] Black Forest Labs. Flux. https:/giitub..oombbaaaffsssstti
2024.
[20] Duong H. Le, Tuan Pham, Sangho Lee. Christopher Clark, Aniruddha Kembhavi, Stephan Mandt, Ranjay Krishna, and Jiasen Lu. One diffusion to generate them all. Proc. of IEEF Intl. Conf. on Computer Vision and Pattern Recognition, 2025.
[21] Xirui Li, Charles Herrmann, Kelvin CK Chan, Yinxiao Li, Deqing Sun, Chao Ma, and Ming- Hing-Hsuan Yang. A simple approach unifying diffusion-basse conditional generation. In Proc. of Intl. Conf. Learning Representations, 2025.
[22] Ziyi Li, Qinye Zhou, Xiaoyun Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xie. Spea-voca object segmentation with diffusion models. In Proc. of IEEE Intl. Conf. on Computer Vision, 2023.
[23] Dingkang Liang, Tianrui Feng. Xin Zhou, Yumeng Zhang, Zhikang Zou, and Xiang Bai. Parameter- efficient fine-tuning in spectral domain for point cloud learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025.
[24] Dingkang Liang, Wei Hua, Chunsheng Shi, Zhikang Zou, Xiaoqing Ye, and Xiang Bai. Sood++: Leveraging unlabeled data to boost oriented object detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025.
[25] Hongkai Lin, Dingkang Liang. Zhenghao Qi, and Xiang Bai. A unified image-dense annotation generation model for underwater scenes. In Proc. of IEEE Intl. Conf. on Computer Vision and Pattern Recognition, 2025.
[26] Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. Proc. of Intl. Conf. on Learning Representations, 2023.
[27] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jac Lec. Visual instruction tuning. In Proc. of Advances in Neural Information Processing Systems, 2024.
[28] Shih Yang Liu, Chien-Ye Yi Wang. Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang- Kwang-Ting Cheng, and Hung Chen. Dora: Weight-decompossed low-rank adaptation. In Proc. of Intl. Conf. on Machine Learning, 2024.
[29] Xiaoxiao Long, Yuan-Chen Guo, Cheng Lin, Yuan Liu, Zhiyang Dou, Lingjie Liu, Yuexin Ma. Song-Iaai Zhang, Marc Habermann, Christian Thcobalt, ct al. Wonder3d: Single image to 3d using cross rross-domain diffusion. In Proc. of IEEE Intl. Conf. on Computer Vision and Pattern Recognition, 2024.
[30] Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Megrew. 1lya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. In Proc. of Insl. Conf. on Machine Learning, 2022.
[31] William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proc. of IEEE Intl. Conf. on Computer Vision, 2023.
[32] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. In Proc. of Intl. Conf. Learning Representations, 2024.
[33] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In Proc. of Intl. Conf. on Learning Representations, 2021.
[34] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research, 2020.
[35] René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. In Proc. of IEEE Inl. Conf. on Computer Vision, 2021.
[36] René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross- transfer. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020.
12

---

[37] Mike Roberts, Jason Ramapuram, Anurag Ranjan, Atulit Kumar, Miguel Angel Bautista. Nathan Paczan, Russ Webb, and Joshua M Susskind. Hypersim: photorealistic synthetic dataset for holistic indoor scene understanding. In Proc. of IEEE Intl. Conf. on Computer Vision, 2021.
[38] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In Proc. of IEEE Intl. Conf. on Computer Vision and Puttern Recognition, 2022.
[39] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan. Tim Salimans, et al. Photorealistic text-to image diffusion models with deep language understanding. In Proc. of Advances in Neural Information Processing Systems, 2022.
[40] Nathan Silberman, Derek Hoiem, Pushmect Kohli, and Rob Fergus. Indoor segmentation and support inference from rgbd images. In Proc. of European Conference on Computer Vision, 2012.
[41] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. In Proc. of Advances in Neural Information Processing Systems, 2017.
[42] Wouter Van Gansbeke and Bert De Brabandere. A simple latent diffusion approach for panoptic segmenta- tion and mask inpainting. In Proc. of European Conference on Computer Vision, 2024.
[43] Igor Vasiljevic, Nick Kolkin, Shanyi Zhang, Ruotian Luo. Haochen Wang, Falcon Z Dai, Andrea F Daniele. Mohammadreza Mostajabi, Steven Basart, Matthew R Walter, et al. Diode: A dense indoor and outdoor depth dataset. arXiv preprint arxiv: 1908. 00463, 2019.
[44] Weijia Wu, Yuzhong Zhao, Hao Chen, Yuchao Gu, Rui Zhao, Yefei He, Hong Zhou, Mike Zheng Shou, and Chunhua Shen. Datasetdm: Syuthesizing data with perception annotations using diffusion models. In Proc. of Advances in Neural Information Processing Systems, 2023.
[45] Gangwei Xu, Xianqi Wang. Zhaoxing Zhang, Junda Cheng, Chunyuan Liao, and Xin Yang. Igev++: Iterative multi-range geometry encoding volumes for stereo matching. IEEE Transactions on Pateenn Analysis and Machine Intelligence, 2025.
[46] Gangwei Xu, Yun Wang, Junda Cheng, Jinhui Tang, and Xin Yang. Accurate and efficient stereo matching via attention concatenation volume. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
[47] Jiarui Xu, Sifci Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello. Open- vocabulary panoptic segmentation with text-to-image diffusion models. In Proc. of IEFE Intl. Conf. on Computer Vision and Pattern Recognition, 2023.
[48] Lihe Yang, Bingyi Kang. Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hongshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. In Proc. of IEEE Inil. Conf. on Computer Vision and Pattern Recognition, 2024.
[49] Lihe Yang. Bingyi Kang, Zilong Huang, Zhen Zhao, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao Depth anything v2. Proc. of Advances in Neural Information Processing Systems, 2024.
[50] Chongjie Ye, Lingteng Qiu, Xiandong Gu, Qi 7ao, Yushuang Wu, Zilong Dong, Liefeng Bo, Yuliang Xiu, and Xianguang Han. Stablenormal: Reducing diffusion variance for stable and sharp normal. In ACM Transactions ON Graphics, 2024.
[51] Chi Zhang, Wei Yin, Billzb Wang, Gang Yu, Bin Fu, and Chunhua Shen. Hierarchical normalization for robust monocular depth estimation. In Proc. of Advances in Neural Information Processing Systems, 2022.
[52] Jingyang Zhang, Shiwei Li, Yuanxun Lu, Tian Fang, David McKinnon, Yanghai Tsin, Long Quan, and Yao Yao. Jointnet: Extending text-to-image diffusion for dense distribution modeling. In Proc. of Inil. Conf. on Learning Representations, 2024.
[53] Xiang Zhang, Bingxin Ke, Hayko Riemenschneider, Nando Metzger, Anton Ohukhov, Markus Gross, Konrad Schindler, and Christopher Schrvers. Betterdepth: Plug-and-play diffusion refiner for zero-shot. monocular depth estimation. In Proc. of Advances in Neural Information Processing Systems, 2024.
[54] Wenliang Zhao, Yongming Rao, Zuyan Liu, Benlin Liu, Jie Zhou, and Jiwen Lu. Unleashing text-to-image diffusion models for visual perception. In Proc. of IEEE Intl. Conf. on Computer Vision, 2023.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of the work in the section of conclusion.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't It acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
14

---

Justification:
This paper does not involve theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide detailed information about experiments and the source code that reproduce the reported results.
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [Yes]
Justification:
We will release of code and data.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nipp cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /Codeeuomiisseeeeeeee
/nips. wwtpccccpppcccaecccccccccccccclt / ssssssooiccyyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
We specify all the training and test details in the main text.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We follow the convention in prior works and report the performance number on the standard benchmarks.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
16

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
See experiments part.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /ttprr//////.../eeeeeeeeee cc. htte/public/eeeie cttpppllltttthhhhccueeeeee...eee ctt.idellllllleecccccccccc ines?
Answer: [Yes]
Justification:
We follow the NeurlPS Code of Ethic.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: no societal impacts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: No such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We properly credited the creators or original owners of assets (e.g., code, data, models), used in the paper and conformed the license and terms.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
18

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We communicated the details of the code/model part of their submission.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/cooe/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
19

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy (https / 220 for what should or should not be described.
20


Reviews:
[{"review_id": "033VbhyN3K", "reviewer": "Reviewer_RUJQ", "summary": "Authors present a unified framework for image generation and depth estimation with a fixed T2I diffusion model. Specifically, they introduce an identical, learnable T2I block as a converter before each transformer layer of the pre-trained T2I diffusion transformer. The play-and-plug design enables seamless switching between the original T2I model and depth estimation model by skipping or running these converters. Moreover,considering the similarity of output features across pre-trained T2I blocks, MERGE proposes a Group Reuse Mechanism (GRE) to encourage parameter reuse that significantly reduces the number of parameters. Extensive experiments demonstrate the effectiveness of MERGE for the task of depth estimation compared to existing approaches that rely on  massive data.", "questions": "- I would suggest to extend the work to at least one more generation task other than style transfer like tasks.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 1, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- Plug-and-play design offers flexibility of the network capability and such design can theoretically extend to other image modality.\n- Thorough ablation studies of the the converter to show the effectiveness of the proposed design details.\n\nWeakness:\n- The research topic is worth discussing. As diffusion models by default are good at image style transfer, it is not surprising to levarage the generation power to do the depth estimation as depth maps can be treated as another style of the image with same structures. As the motivation of doing depth estimation with diffusion network does not bring much novelty, the next question is whether the proposal brings a better depth estimation solution. However, from the Table 1 comparison we can see the depth estimation expert models (Marigold, GeoWizard and etc.) outperforms the proposed method with less training parameters and close quantity of training data. So for real world applications, users can simply load two models to have both functionalities other than switching on and off the converter. Even though the study in Table 2 shows that the proposed method is better the Lora/Dora fine-tuning methods, indicating an superior design of GRE, the value of the proposal in real world senarios is not clear unless the work shows the possibilities to extend the work to other image generation tasks, like OneDiffusion. \n- Minor typo. Plug-and-play and play-and-plug are both used in the text.", "was_revised": true, "final_justification": "Comment:\nAuthors added another normal estimation analysis in the rebuttal and demonstrated better performance than the compared methods, which somewhat mitigates my concerns of the generalization ability to other image generation tasks. However, in my initial review, I suggest to see some experiment results other than style transfer like tasks. As the normal estimation is very similar to depth information, using stuctural information from original image to convert the pixel values to another format, I am not completely convinced at this point. The proposal and rebuttal are 100% clear delivering the message that the framework is capable of maintaining good image generation abilities across similar tasks, but the rebuttal seems avoid answering the core question raised in the review. One thing I may overlooked in the inital review is the evaluation of out-of-distribution (OOD). I think this part the proposal did a good job and should add values to the final score. So I lift the rating to boarderline accept.", "rebuttal": "1. **To weakness 1: “… not surprising to levarage the generation power to do the depth estimation … comparison we can see the depth estimation expert models (Marigold, GeoWizard and etc.) … . So for real world applications …”**\n    \n    **Reply**: Good question! In general, image generation and geometric estimation (e.g., depth estimation) are considered two totally different tasks. The emergence of recent works in the generative field, such as Show-o[1], UniCon[2], and OneDiffusion[3], indicates that the unified of previously separate tasks, including understanding, generation, and perception, is a promising research direction. So, our method aims to explore **a new paradigm for constructing a unified model for image generation and geometric perception in a simple framework, instead of simply pushing the performance border of a single task.**  \n    \n    Although methods like Marigold and GeoWizard achieve strong performance through full fine-tuning, their image generation capability is inevitably compromised. Thus, a relatively fair comparison would be with UniCon[2] (ICLR 25) and OneDiffusion[3] (CVPR 25), which also possess image generation and geometric perception capabilities. As shown in the table, even when compared to Marigold, our A.Rel metric shows a difference of less than 1 across three depth estimation benchmarks. Moreover, in the evaluation of normal estimation, our MERGE achieves an overall better performance. More importantly, our MERGE retains the capability for image generation, which Marigold does not.\n    \n    Considering the early exploration of merging geometric perception capabilities while preserving the original image generation capabilities of pre-trained models, we believe our work is valuable to researchers in this field.\n    \n    | Method\\Depth | NYUv2-A.Rel ↓ | NYUv2-δ1 ↑ | Scannet-A.Rel ↓ | Scannet-δ1 ↑ | DIODE-A.Rel ↓ | DIODE-δ1 ↑ |\n    | --- | --- | --- | --- | --- | --- | --- |\n    | Marigold | 5.5  | 96.4  | 6.4  | 95.1  | 30.8  | 77.3 |\n    | GeoWizard  | 5.2  | 96.6  | 6.1  | 95.3 | 29.7 | 79.2 |\n    | MERGE (Ours) | 5.9 | 95.4 | 7.1 | 94.0 | 31.4 | 76.3 |\n    \n    | Method\\Normal | NYUv2-m.↓ | NYUv2-11.25◦↑ | Scannet-m.↓ | Scannet-11.25◦↑ |  iBims-1-m.↓ |  iBims-1-11.25◦↑ |\n    | --- | --- | --- | --- | --- | --- | --- |\n    | Marigold | 20.9  | 50.5  | 21.3  | 45.6  | 18.5  | 64.7 |\n    | GeoWizard  | **18.9** | 50.7 | **17.4** | 53.8 | 19.3  | 63.0 |\n    | MERGE (Ours) | 20.1 | **54.3**  | 18.3  | **60.2**  | **18.2**  | **66.2** |\n2. **To weakness 1: “… the value of the proposal in real world senarios is not clear unless the work shows the possibilities to extend the work to other image generation tasks … .”**\n    \n    **Reply**: Good suggestion! We would like to clarify that our current experiments align with the evaluation of out-of-distribution (OOD), which involves training on synthetic datasets and testing on real-world datasets. The experiment results represent impressive performance, demonstrating the potential of our method in the real world. Additionally, following your suggestion, we extend our method to the normal generation task. Please refer to Appendix A.2 for specific details. Our method achieves highly competitive or even better performance than current state-of-the-art full fine-tuning models. Critically, these models lose their image generation capabilities after fine-tuning, whereas our MERGE method preserves the original text-to-image generation ability. \n    \n    In addition, image generation tasks like depth-to-image controllable generation are similar to generative depth estimation, as both belong to per-pixel-level conditional generation tasks. Therefore, we also explore extensions to these types of image generation tasks. However, due to the constraints of the rebuttal format, we cannot show the visualization results (these tasks typically lack quantitative evaluation). We commit to presenting these visualization in the final version.\n    \n3. **To weakness 2: “Minor typo… .”**\n    \n    **Reply**: typos: Plug-and-play -> play-and-plug\n    \n\n[1] Show-o: One Single Transformer to Unify Multimodal Understanding and Generation, ICLR 2025.\n\n[2] A Simple Approach to Unifying Diffusion-based Conditional Generation, ICLR 2025.\n\n[3] One Diffusion to Generate Them All, CVPR 2025."}, {"review_id": "ghposrlIyj", "reviewer": "Reviewer_Xt7v", "summary": "This paper introduces MERGE, an unified diffusion model designed for both image generation and depth estimation. MERGE leverages a fixed, pre-trained text-to-image diffusion model and enhances it with plug-and-play converters. Group Reuse Mechanism encourages parameter reuse and improves efficiency. The model demonstrates superior depth estimation performance while uniquely preserving the original image generation capabilities of the pre-trained T2I model.", "questions": "- Please refer to the weaknesses section.\n- Would it be feasible to extend this framework to enable co-generation of an image and its corresponding depth map simultaneously from a text prompt?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n- Enabling powerful depth estimation without irreversibly degrading the inherent image generation capability of the pre-trained DiT is impressive.\n- The proposed Group Reuse Mechanism is a well-reasoned design choice that improves parameter utilization and efficiency, which outperforms while maintaining similar parameter counts.\n- It demonstrates superior performance on multiple depth estimation benchmarks, especially when considering its significantly lower training data and additional parameter requirements compared to other unified models.\n- The paper is written clearly and is easy to follow.\n\nWeaknesses\n- The paper would benefit from a more explicit and in-depth discussion on the primary motivation for preserving the pre-trained T2I model's image generation capability. While the paper mentions catastrophic degradation, the advantages of retaining the original generative ability when only depth estimation is needed are not fully highlighted. Specifically, the lower performance compared to Marigold, which is a full fine-tuning method without generation preservation, in Table 1 raises questions about the trade-off. Clarifying why a model capable of both image generation and depth estimation is important would strengthen this argument.\n\n- Following on from the point above, if a key benefit of preserving the T2I prior is enhanced generalization capability, demonstrating its robustness on OOD data would further validate the importance of maintaining the generative prior.\n\n- The comparison in Table 3 is relevant to the discussion on motivating T2I prior preservation. Providing more details on this would strengthen the paper's rationale.", "was_revised": true, "final_justification": "The authors’ rebuttal has alleviated some of my concerns. The arguments regarding OOD performance help to strengthen the paper, and I am convinced that the proposed method demonstrates strong performance under the same problem setup.\n\nI am not fully convinced that the problem setup—“depth inference while preserving image generation priors”—is a necessary direction for the field. As Reviewer RUJQ raised, the authors have not provided a convincing reason why it is essential to preserve image generation priors for depth generation. If specialized methods that do not retain the generative prior can achieve better performance, it would be more natural to use those methods. The actual advantage of performing both image generation and depth inference within a single model is not clearly presented in the paper.\n\n(Update) Thanks to the authors’ additional comment, I have come to agree with the necessity of this problem setup to some extent. Considering the scalability of image generation models, I also agree that this could be a step toward more promising research directions in the future. While I would not raise my score above borderline accept, I am leaning toward acceptance and will maintain my current rating.", "rebuttal": "1. **To weakness 1: “… motivation for preserving the pre-trained T2I model's image generation capability. … the lower performance compared to Marigold, … raises questions about the trade-off. Clarifying why a model capable of both image generation and depth estimation … .”**\n    \n    **Reply:** Thanks. Recent works, such as Show-o[1], UniCon[2], and OneDiffusion[3], reveal a promising research direction that merges image generation capabilities with understanding or perception capabilities in a single model, which are originally independent. Thus, our method aims to explore a new paradigm that merges the previously independent image generation and geometric perception (e.g., depth estimation) capabilities into a unified model, rather than pushing the performance border of a single task. So, we would like to clarify that a fair and meaningful comparison should be made with unified-based methods like UniCon (ICLR 25) and OneDiffusion (CVPR 25) rather than with specialized models (e.g., Marigold). Notably, even compared to Marigold, our method shows only a minor performance gap (e.g., only 0.4 A.Rel gap on the NYUv2 benchmark). More importantly, our method preserves the original image generation capability of the pre-trained model, which Marigold does not. We believe that trading a minor difference in single-task performance to preserve the model's original image generation capability is a valuable trade-off for developing unified models for image generation and perception in the community.\n    \n2. **To weakness 2: “Following on from the point above, if a key benefit of preserving the T2I prior is enhanced generalization capability, demonstrating its robustness on OOD data would further validate the importance of maintaining the generative prior.”**\n    \n    **Reply:** Good suggestion! We apologize for not clearly explaining the experiment setting. Actually, our current training on synthetic datasets and evaluation on three real-world datasets (NYUv2, ScanNet, and DIODE) aligns with the out-of-distribution (OOD) test setting. Our MERGE outperforms other unified models, such as JointNet (ICLR 24), UniCon (ICLR 25), and OneDiffusion (CVPR 25), on these real-world benchmarks, proving that preserving and leveraging the robust visual priors from Text-to-Image models is crucial for enhancing a model's zero-shot generalization capabilities. Thanks for your suggestion. We will explicitly clarify our experimental setup in the final version.\n    \n3. **To weakness 3: “The comparison in Table 3 is relevant to the discussion on motivating T2I prior preservation. Providing more details on this would strengthen the paper's rationale.**”\n    \n    **Reply:** Thanks. Previous work [4] suggests that the catastrophic forgetting during the fine-tuning of Text-to-Image (T2I) models often leads to vague predictions in highly detailed areas. Thus, a naive idea is that preserving the T2I prior might alleviate this issue. So, we provide a direct and fair comparison between our method and Marigold under an identical T2I model. The results show that our MERGE not only achieves highly competitive and even better performance compared to the Marigold-P with fewer learnable parameters but also preserves the original generation capability, an advantage not supported by the full fine-tuning approach.\n    \n4. **To question 2: “Would it be feasible to extend this framework to enable co-generation of an image and its corresponding depth map simultaneously from a text prompt?”**\n    \n    **Reply:** This is a very insightful question, and the answer is yes! In Fig. 1 of the main text, we present examples of generating an image and its corresponding depth estimation annotation using a two-stage method (text-to-image -> image-to-depth). A possible one-stage solution to simultaneously generate the image and its corresponding depth map involves branching the features for image generation and depth estimation. After passing through a shared Text-to-Image block, one branch converts the features for the depth estimation task via our converters, while the other branch skips the converters to complete the original image denoising process. This allows for the simultaneous output of both the image and its corresponding depth map. Follow your suggestion, we will refine this solution in our future work.\n    \n\n[1] Show-o: One Single Transformer to Unify Multimodal Understanding and Generation, ICLR 2025.\n\n[2] A Simple Approach to Unifying Diffusion-based Conditional Generation, ICLR 2025.\n\n[3] One Diffusion to Generate Them All, CVPR 2025.\n\n[4] Lotus: Diffusion-based Visual Foundation Model for High-quality Dense Prediction, ICLR 2025."}, {"review_id": "ygcF9JENbK", "reviewer": "Reviewer_FL3u", "summary": "This paper proposes a unified model that can generate images and estimate depth. Starting from a pretrained text-to-image model, this work introduces a play-and-plug framework that can switch between different tasks. It also proposes a group reuse mechanism for better utilization of parameters. Experimental results show the effectiveness of proposed methods in both tasks.", "questions": "Please refer to the weaknesses.", "limitations": "This paper discusses the limitations.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n1. The proposed MERGE method adopts a few converters to switch between tasks, which is simple and efficient.\n2. This work conducts extensive experiments to show the effectiveness of the proposed method in both image generation and depth estimation tasks. \n3. This paper is well-organized and easy to follow. \n\n**Weaknesses:**\n1.   This paper highlights that the proposed method uses less training data compared to prior studies such as OneDiffusion. However, it is not discussed whether the training data is a subset of that used in previous studies or not. For example, OneDiffusion uses 40K images from Hypersim dataset,  without using Virtual KITTI used in this work. Therefore, it is not clear whether the performance improvement comes from different training sets or the proposed method. \n\n2.  This work shows the effectiveness of MERGE over prior studies in Table 1 and Figure 6. However, further analysis is preferred. For example, what kinds of scenarios is the proposed method particularly effective in? \n\n3. The paper compares depth estimation with diffusion-based methods like Lotus (Table 1), but lacks the comparison to traditional methods like Depth Anything[1] and Depth Anything V2 [2]. This may mislead the readers on the results of the proposed methods. \n\n[1] Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data, CVPR 2024.\n[2] Depth Anything V2. A More Capable Foundation Model for Monocular Depth Estimation, NeurIPS 2024.", "was_revised": true, "final_justification": "I have carefully written my reviews and actively discussed with the authors in multiple rounds. I have also read the comments from other reviewers.\n\nI gave a score of *borderline accept* because the paper proposes an interesting approach for depth estimation by introducing converters into diffusion models. This paper is well written.\n\nI did not give a higher score because I believe this work is better categorized as a single-task method, similar to Lotus, rather than a unified model as claimed. When used for image generation, the proposed method degrades to the original diffusion model without any improvement.  With converters, the proposed method can only support depth estimation.  In contrast, unified models such as OneDiffusion support a broader range of tasks, including segmentation, pose estimation, ID customization, and so on.  Compared to unified models like OneDiffusion, this work supports much fewer tasks.  Compared to single-task methods like Lotus, the  performance on depth estimation is lower. Considering these points, I will maintain my score and not raise it to a 5.", "rebuttal": "1. **To weakness 1: “… OneDiffusion uses 40K images from Hypersim dataset, without using Virtual KITTI used in this work. … the performance improvement comes from different training sets or the proposed method.”**\n    \n    **Reply:** Good question! To address the concern that the performance improvement might stem from different training sets, we train a MERGE-L model using only Hypersim as training data, without Virtual KITTI. The experimental results are shown in the table below. Compared to OneDiffusion, the MERGE-L model trained solely on Hypersim still achieves comparable or even better results on the depth estimation benchmarks. This result demonstrates that the performance gain is primarily attributable to our proposed MERGE method rather than a specific data combination. In addition, we would like to mention that the depth map training data for OneDiffusion includes the Hypersim dataset and approximately 500K high-quality data, while we use 74K depth map training data.\n    \n    | Method | Depth Map Training Data | NYUv2-A.Rel ↓ | NYUv2-δ1 ↑ |  DIODE-A.Rel ↓ |  DIODE-δ1 ↑ |\n    | --- | --- | --- | --- | --- | --- |\n    | OneDiffusion | Hypersim+500K depth data | 6.8  | 95.2 | 29.4 | 75.2 |\n    | MERGE-L (Ours) | Hypersim | 6.2 | 95.2 | 31.0 | 76.5 |\n2. **To weakness 2: “… what kinds of scenarios is the proposed method particularly effective in?”**\n    \n    **Reply:** Good question! Our current experimental setup is an out-of-distribution evaluation, where training is performed on synthetic data and evaluation is conducted on real-world datasets. Experimental results indicate that, compared to other unified models, our method demonstrates superior performance in zero-shot scenarios, especially in indoor scenarios. Additionally, Appendix B provides qualitative comparison results under challenging objects (Reflective Objects, Free-form Voids, and Filiform Objects). These results show that MERGE exhibits unique advantages in these challenging objects. We will highlight these findings in the main text of the final manuscript.\n    \n3. **To weakness 3: “… lacks the comparison to traditional methods like Depth Anything[1] and Depth Anything V2 [2]. This may mislead the readers on the results of the proposed methods.”**\n    \n    **Reply:** Good suggestion! To provide readers with a more complete overview of the current state of development in the depth estimation field, we commit to including the results of traditional methods like the Depth Anything series[1][2] in Table 1 in the final version.\n    \n\n[1] Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data, CVPR 2024. \n\n[2] Depth Anything V2. A More Capable Foundation Model for Monocular Depth Estimation, NeurIPS 2024."}, {"review_id": "upuO1sD819", "reviewer": "Reviewer_VXik", "summary": "This paper introduces MERGE, a plug-and-play framework that transforms a fixed, pre-trained text-to-image (T2I) diffusion model into a unified model capable of both image generation and depth estimation. The authors achieve this unification by injecting lightweight, learnable converters before each transformer block (T2I block) in the diffusion model, allowing seamless switching between tasks without degrading the original generative quality. This paper also proposes the Group Reuse Mechanism (GRE), which reuses converters across similar T2I blocks to reduce the number of additional trainable parameters, improving parameter efficiency. MERGE demonstrates state-of-the-art performance on multiple real-world depth estimation benchmarks (e.g., NYUv2, ScanNet).", "questions": "1. The connection between the feature similarity shown in Figure 4 and the motivation for reusing converters is not clearly established, while the key motivation of this paper: how to unify depth estimation and image generation.\nBesides, if the outputs of different T2I blocks are indeed highly similar, does this suggest redundancy in the DiT model’s internal representations? A deeper analysis would strengthen the justification for the Group Reuse Mechanism. \n\n2. In addition, the proposed augmentation strategy appears to lack generalizability across architectures beyond PixArt, such as Stable Diffusion or FLUX, which raises concerns about the robustness of the approach.", "limitations": "yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "I thank the authors for their efforts in this work. Below are some comments.    \n\n**Strengths**\n\n1. This manuscript is well-written and easy to follow. \n2. The authors perform extensive ablation experiments (Section 5.2), systematically evaluating key design choices including converter composition, Group Reuse Mechanism (GRE), initialization strategies, and prompt settings. This validates the effectiveness of the proposed method.\n\n**Weakness**\n\n1. The comparison with baselines is not entirely fair, as the proposed method is based on PixArt, whereas several of the compared baselines, such as JointNet or UniCon, are built on different models like Stable Diffusion.\n2. While the proposed converter introduces an incremental innovation by combining reused T2I model blocks with LoRA-like principles, the justification in Figure 4 appears model-specific and does not clearly generalize to architectures such as Stable Diffusion or FLUX. More discussion is included in # Question parts.\n3. The idea of the proposed converter is incremental; it is essentially a combination of reusing T2I model blocks and LoRA. \n4. While the paper claims to preserve the original image generation ability of the T2I model compared to the related work, it lacks quantitative or visual evaluations of image quality for the image generation tasks. This undermines a central claim of the work.\n5. The ablation on the number of converters appears to violate scaling laws, yet no in-depth analysis is provided to explain this phenomenon. The paper also lacks a clear discussion on the trade-off between depth estimation and image generation capabilities. Furthermore, there is no ablation study exploring configurations with fewer than 110M learnable parameters.", "was_revised": true, "final_justification": "I thank the authors for their efforts on the manuscript and rebuttal. Before the rebuttal, I raised concerns about generalization, unfair comparisons with baselines using Stable Diffusion, the motivation for reusing converters, and \\etc. The authors have addressed these concerns well in their response. Therefore, I recommend this work as borderline accept.", "rebuttal": "1. **To weakness 1: “The comparison with baselines is not entirely fair, … such as JointNet or UniCon, are built on different models like Stable Diffusion.”**\n    \n    **Reply:** Thanks. In the table below, we provide experiments on the Stable Diffusion version of MERGE (MERGE-SD). The experimental results demonstrate that MERGE-SD consistently outperforms JointNet and UniCon across multiple depth estimation benchmarks. We fully validate the advantages of our MERGE on multiple pre-trained Text-to-Image models, including Stable Diffusion, PixArt, and FLUX.\n    \n    | Method | NYUv2-A.Rel ↓ | NYUv2-δ1 ↑ | Scannet-A.Rel ↓ | Scannet-δ1 ↑ |\n    | --- | --- | --- | --- | --- |\n    | JointNet | 13.7  | 81.9  | 14.7  | 79.5 |\n    | UniCon | 7.9  | 93.9 | 9.2 | 91.9 |\n    | MERGE-SD (Ours) | 6.1 | 95.6 | 7.6 | 93.2 |\n2. **To weakness 2&3: “… converter introduces an incremental innovation by combining reused T2I model blocks with LoRA-like principles, the justification in Figure 4 appears model-specific and does not clearly generalize to architectures such as Stable Diffusion or FLUX. …”**\n    \n    **Reply:** Thanks. Regarding your opinion that our converter design represents \"incremental innovation,\" we would like to clarify that it fundamentally differs from methods like LoRA in motivation and architecture. **1) The core motivation is fundamentally different.** The motivation of LoRA stems from prior work [1], which revealed that pre-trained models possess an extremely low-dimensional parameter space. Fine-tuning within this space achieves the same effect across the entire parameter space. Essentially, LoRA provides an efficient implementation of full fine-tuning. Our intuition behind designing the converter is based on the fact that diffusion-based image generation and generative geometry estimation both belong to the latent diffusion process. The transition from image generation to geometry estimation may only lack a \"task-oriented\" converter, which guides the latent diffusion of image generation towards geometry estimation. **2) The architectural implementation has significant distinctions.** LoRA introduces low-rank matrices into existing layers to achieve efficient fine-tuning. In contrast, our approach injects a highly structured converter similar to Text-to-Image (T2I) blocks. The most direct advantage of our highly structured converter is that it enables converters to be initialized with the weights of pre-trained T2I models for parameter initialization, which LoRA does not support. The experimental results in the table below demonstrate that this initialization consistently outperforms random parameter initialization.\n    \n    | Init.Type | NYUv2-A.Rel ↓ | NYUv2-δ1 ↑ |\n    | --- | --- | --- |\n    | Random | 7.9  | 93.5 |\n    | Pre-trained | 7.5  | 94.2 |\n    \n    The motivation for our Group Reuse Mechanism (GRE) is inspired by observing the common characteristics of large Transformer models, such as Fig. 1 in [2], Fig. 3 in [3], and Fig. 4 in [4]. Specifically, adjacent layers often have a high degree of feature similarity. Fig. 4 in our main text uses PixArt as a visual example of this common phenomenon, rather than as comprehensive evidence.\n    \n3. **To weakness 4: “… lacks quantitative or visual evaluations of image quality for the image generation tasks. …”**\n    \n    **Reply:** Good suggestion! The design of our method is to ensure the complete preservation of image generation capabilities. Specifically, all additionally introduced converters are skipped when performing the \"text-to-image\" task. It means that the inference process of image generation remains fully consistent with the original, unmodified, pretrained T2I model. Following your suggestion, we provide the quantitative evaluation results for image generation below. In addition, we present some images generated by our MERGE model in Fig. 1 of the main text. We sincerely thank your suggestion and commit to explicitly including the quantitative evaluation results of image generation in the final version of the paper to strengthen our core claims.\n    \n    | Method | COCO FID-30K ↓ | GenEval ↑ |\n    | --- | --- | --- |\n    | SDXL | 6.63 | 0.55 |\n    | SD3-medium | 11.92 | 0.62 |\n    | JointNet | 14.15 | - |\n    | OneDiffusion | - |  0.65 |\n    | PixArt-α | 7.32 |  0.48 |\n    | MERGE-PixArt (Ours) | 7.32 |  0.48 |\n    | FLUX.1 dev | 10.15 |  0.67 |\n    | MERGE-FLUX (Ours) | 10.15 |  0.67 |\n4. **To weakness 5: “… the number of converters appears to violate scaling laws, yet no in-depth analysis is provided to explain this phenomenon. … discussion on the trade-off between depth estimation and image generation capabilities. … no ablation study exploring configurations with fewer than 110M learnable parameters.”**\n    \n    **Reply:** Thanks. Since converters are inserted into the pre-trained T2I model in a sequential manner, increasing the number of converters sharply increases the network depth. For example, when the number of converters is 3, the network depth expands 4x, which can lead to optimization challenges[5][6] and ultimately result in a decline in performance. \n    \n    It is an excellent question regarding the \"trade-off.\" Thanks to our plug-and-play design, our image generation capability remains constant and does not vary with changes in depth estimation performance.\n    \n    We present the ablation study with fewer than 110M parameters in Table 5, as shown in the table below. In this table, we explore settings with 56M (7 groups) and 32M (4 groups) learnable parameters by adjusting the number of groups in the group reuse mechanism (GRE). The experimental results show that our method remains effective under a lower parameter budget.\n    \n    | Groups  | GRE  | #Param. | NYUv2-A.Rel ↓ | NYUv2-δ1 ↑ |\n    | --- | --- | --- | --- | --- |\n    | 28  | × | 224M | 7.0  | 94.7 |\n    | 14 | × | 110M | 15.6  | 78.8 |\n    | 14 | ✔ | 110M | 7.5  | 94.2 |\n    | 7 | ✔ | 56M | 7.8  | 93.5 |\n    | 4 | ✔ | 32M |  9.3  | 91.0 |\n5. **To question 1: “… feature similarity shown in Figure 4 and the motivation for reusing converters is not clearly established. … redundancy in the DiT model’s internal representations? …”**\n    \n    **Reply:** Thanks. Fig. 4 of the main text reveals the high similarity between the output features of adjacent layers in the DiT model. Our idea is that if T2I block i and i+1 outputs are similar, the feature transformation for handling depth estimation tasks should also be similar. Therefore, sharing a converter may largely capture this common transformation.\n    \n    Regarding the question of “redundancy”, we argue that it might highlight the widespread phenomenon of redundancy in the DiT model’s internal representations. We also note that a recent study[7] demonstrates that a significant portion of computation in global self-attention mechanisms is wasted on calculating near-zero attention scores between distant tokens, resulting in only a limited number of effectively updated features. Moreover, as Transformer models become deeper, feature representations in the deeper layers become overly similar[6], leading to a loss of diversity and expressive power, making additional layers computationally redundant. These findings indicate that the DiT model indeed contains redundancy.\n    \n6. **To question 2: “…lack generalizability across architectures beyond PixArt… Stable Diffusion or FLUX, …”**\n    \n    **Reply:** Good suggestion! We would like to clarify that we discuss MERGE based on PixArt and FLUX in the paper. Additionally, we present the performance of MERGE based on Stable Diffusion in the table below. Our three variants achieve outstanding performance on multiple depth estimation benchmarks. These results directly demonstrate the strong generalizability of our method, which is not limited to the PixArt architecture alone.\n    \n    | Method | NYUv2-A.Rel ↓ | NYUv2-δ1 ↑ | Scannet-A.Rel ↓ | Scannet-δ1 ↑ | DIODE-A.Rel ↓ | DIODE-δ1 ↑ |\n    | --- | --- | --- | --- | --- | --- | --- |\n    | JointNet | 13.7  | 81.9  | 14.7  | 79.5 | - | - |\n    | UniCon | 7.9  | 93.9 | 9.2 | 91.9 | - | - |\n    | OneDiffusion | 6.8  | 95.2 | - | - | 29.4  | 75.2 |\n    | MERGE-PixArt (Ours) | 7.5 | 94.2 | 9.9 | 89.8 | 32.5 | 74.9 |\n    | MERGE-Stable-Diffusion (Ours) | 6.1 | 95.6 | 7.6 | 93.2 | 30.8 | 93.4 |\n    | MERGE-Flux (Ours) | 5.9  | 95.4  | 7.1 | 94.0 | 31.4 | 76.3 |\n\n[1] Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning, ACL 2021.\n\n[2] Do Vision Transformers See Like Convolutional Neural Networks?, NeurIPS 2021. \n\n[3] Skip-Attention: Improving Vision Transformers by Paying Less Attention, ICLR 2024.\n\n[4] Layerlink: Bridging remote sensing object detection and large vision models with efficient fine-tuning, PR 2025.\n\n[5] Going Deeper with Image Transformers, ICCV 2021.\n\n[6] Swin Transformer V2: Scaling Up Capacity and Resolution, CVPR 2022.\n\n[7] Swin DiT: Diffusion Transformer using Pseudo Shifted Windows, Arxiv 2025."}]

Author rebuttal:

Meta-review: This paper proposes a framework for unifying image generation and depth estimation using pretrained text-to-image diffusion models. Instead of fine-tuning all parameters, the paper introduces lightweight plug-and-play converters alongside a Group Reuse mechanism to enable switching between modes while mitigating catastrophic forgetting. Experiments demonstrate strong depth estimation results across benchmarks while preserving image generation quality.

All four reviewers rated the paper as borderline accept. Reviewers RUJQ and Xt7v question the necessity of preserving image generation priors when depth-specific methods could be stronger. Generalization to tasks beyond depth remains unconvincing. Reviewer VXik was initially concerned about fairness and motivation but found the rebuttal satisfactory. Reviewer FL3u argues the paper is closer to a single-task method (depth estimation only) than a true unified model. Unlike methods such as OneDiffusion, MERGE does not extend naturally to a broader range of tasks. Moreover, compared to single-task baselines like Lotus, its depth results are not competitive.

After rebuttal, the reviewers converged on borderline accept, highlighting that the paper is technically sound. The rebuttal helped mitigate concerns regarding comparisons and generalization, though questions remain about whether MERGE should be considered a “unified” model given its limited scope relative to other frameworks. Given the positive consensus of the reviewers,  the AC does not oppose the acceptance. The authors are strongly requested to improve the paper according to the reviews. It is hoped that the work will stimulate further discussion in this direction.
Decision: Accept (poster)