Minimal Impact ControlNet: Advancing Multi-ControlNet Integration
Authors: Shikun Sun
Min Zhou
Zixuan Wang
Xubin Li
Tiezheng Ge
Zijie Ye
Xiaoyu Qin
Junliang Xing
Bo Zheng
Jia Jia
Venue: ICLR 2025 Poster (2025)
Keywords: generative models
ControlNet
Primary area: generative models

Abstract: With the advancement of diffusion models, there is a growing demand for high-quality, controllable image generation, particularly through methods that utilize one or multiple control signals based on ControlNet. However, in current ControlNet training, each control is designed to influence all areas of an image, which can lead to conflicts when different control signals are expected to manage different parts of the image in practical applications. This issue is especially pronounced with edge-type control conditions, where regions lacking boundary information often represent low-frequency signals, referred to as silent control signals. When combining multiple ControlNets, these silent control signals can suppress the generation of textures in related areas, resulting in suboptimal outcomes. To address this problem, we propose Minimal Impact ControlNet. Our approach mitigates conflicts through three key strategies: constructing a balanced dataset, combining and injecting feature signals in a balanced manner, and addressing the asymmetry in the score function’s Jacobian matrix induced by ControlNet. These improvements enhance the compatibility of control signals, allowing for freer and more harmonious generation in areas with silent control signals.

Published as a conference paper at ICLR 2025
MINIMAL IMPACT CONTROLNET:
ADVANCING MULTI-CONTROLNET INTEGRATION
Shikun Sun ; Min Zhou?, Zixuan , Xubin Li?, Tiezheng Ge², Zijie Ye',
Xiaoyu Qin 11, Junliang Xing', Bo Zheng", Jía Jia 1,34f
1 "Department of Computer Science and Technology, Tsinghua University
Taobao & Tmall Group of Alibaba
$ BNRist Tsinghua University
*Key Laboratory of Pervasive Computing, Ministry of Education
(ssk21, , wangzi xu21 1 } @ma 1s. tssiiii niguaa.nua.. edu en, jzjjswweggma com [yunqi. zm, 1xb2047222 22, t ciezheng- gt bozheng) (xyqin, jlxing, jjia] @t sinhuu. edu. on
@ a baba-aa inc. com
ABSTRACT
With the advancement diffusion models, there is a growing demand for high- quality, controllable image generation, particularly through methods that utilize one or multiple control signals based on ControlNet. However, in current Control- Net training, each control is designed influence all areas of an image, which can lead to conflicts when different control signals are expected to manage different parts of the image in practical applications. This issue especially pronounced with edge-type control conditions, where regions lacking boundary information often represent wwwffreuency signals, referred as silent control signals. When combining multiple ControlNets, these silent control signals can suppress the gen- eration of textures in related areas, resulting in suboptimal outcomes. To address this problem, propose Minimal Impact ControlNet. Our approach mitigates confficts through three key strategies: constructing a balanced dataset, combining and injecting feature signals in a balanced manner, and addressing the asymmetry in the score function's 's Jacobian matrix induced by ControlNet. These improve- ments enhance the compatibility of control signals, allowing for freer and more harmonious generation in areas with silent control signals.
INTRODUCTION
Recent advancements in diffusion models (Sohl-Dickstein et al., 2015; Ho et al., 2020; Rombach et al., 2022; Podell et al., 2023) have significantly bolstered the field of image generation. These innovations particularly notable for the incorporation of controlled generation techniques, such as ControlNet (Zhang et al., 2023) and IP-Adapter (Ye et al., 2023), which allow precise manipula- tions using one more control signals. Despite these advancements, challenges remain, particularly when integrating multiple control signals.
The primary difficulty arises from the fact that during the training of ControlNet, each control is designed to influence all areas of an image. This can lead to conflicts when different control signals are expected manage different parts of the image in practical applications. This issue is especially pronounced with edge-type control conditions, where regions lacking boundary information often represent ow-frequency signals, referred as silent control signals by us.
As shown in Figure 1, our observations suggest that when combining multiple ControlNets, these silent control signals can suppress the generation of textures in areas where other control signals aim to generate details, resulting in suboptimal outcomes. This not only compromises the fidelity of the generated images but also restricts the flexibility and effectiveness of the control mechanisms within the model.
Work done when Shikun Sun was an intern at Taobao Tmall Group of Alibaba.
1 "ooresponning author

---

Published as a conference paper at ICLR 2025
To tackle these challenges, adhering the principle of "less is more", introduce the Minimal Impact ControlNet (MiControlNet), a novel framework designed to refine the integration of mul- tiple control signals within diffusion models. Our approach includes strategic modifications to the training data to reduce biases and utilizes a multi-objective optimization strategy during the feature combination phase, well addressing the asymmetry in the score function's Jacobian matrix induced by ControlNet. These methods aim to minimize conflicts between different control sig- nals and between control signals and the inherent features of the dataset, thereby ensuring better compatibility and fidelity in the generated images.
Canny
ControlNet
ControlNet
MIContrroNNe((Issagee
MICoontroNNe((-sttaee)
Openpose
Co
ControlNet1.3
Uni-ControlNet
Figure 1: The silent control signal from OpenPose ControlNet (outside the blue box) suppresses the high-freet frequenryyen control signal from Canny ControlNet (inside the red box). The black regions of the control signals represent the silent control signals.
To summarize, our main contributions are three-fold as follows:
First introduce silent control signals that should remain in- active when other control signals are engaged, improving the compactness of the generation.
Introduce silent control signals:
Feature injection and combination: Employ strategies based on multi-objective optimization principles improve model performance.
Theoretical contribution: Develop and integrate a conservativity loss function within a large modular network architecture to ensure more stable learning dynamics.
By addressing the fundamental issues in training data preparation and signal integration, MICon- trolNet enhances the model's 's ability to follow the correct control signals in areas previously affected by control signal conflicts. Additionally, it improves controllability in high-frequency regions.
PRELIMINARIES
2.1 DIFFUSION MODELS FOR TEXT TO IMAGE GENERATION
Diffusion Models SSohl-Dickstein et al., 2015; Ho et al., 2020; Rombach et al., 2022; Podell et al., 2023; Song et al., 2021) has gain great success generative models, especially in the text image generation task (Rombach et al., 2022; Podell et al., 2023).
Suppose the image data distribution is q(x) go(x)), where x X C RCHW We define forward process through a sequence of distributions, qe (X4) = Naa (1 - a?) with {a+} decreasing for t [0, T' Z. Here, au and ayr 0. In the generation process, initiate from Xy ~ (0 I) and iteratively generate sample of the previous timestep using a denoising network (x,,, l), trained by minimizing the prediction of added noise follows:
E
+ o.e, -<113-
2

---

Published as a conference paper at ICLR 2025
where wt) balances the losses across different timesteps, t is uniformly selected from 0 to T, and oe VI-at. This loss also serves as the learning objective for numerous control methods, such as ControlNet as described by Zhang et al. (2023).
Researchers Song al. (2021); Vincent (2011) have developed the theory of score-based diffusion models. They established a connection between the score of qt and (x t) as:
(x(,),
S (x,, 1) = Vx, log qt (x+,
2.2 CONTROLNET
ControlNet (Zhang et al., 2023) marks a substantial breakthrough in controlled generation for diffu- sion models, utilizing low-level features such as edges, poses, and depth maps to refine the generative process. The original ControlNet Zhang et al. (2023) paper and Figure 2 provide more intuitive explanation through graph; here, we opt for a formulaic approach introduce symbols that facilitate the proofs in subsequent sections.
Consider U-Net architecture where the encoder g and decoder D consist of layers (E)) J==1 and
[1, respectiell,, with indicating the layer index and representing the model parameters. parallels that of 8" but also integrates a control The architecture of ControlNet, C = {c image an input.
Ti=1, (c
Ignoring the input and output layers, we start with fj = x and fj = x + conv(c). The input of & D; and C; are f fo and (førre add (fgrres, f)r the output of & D; and C, are (f ff41 f ) f and {ff f respectively, where f", fd, f the direct outputs and peres and fares are the residual outputs.
141
For the convenience of calculating the Jacobian matrix of the score function, we clearly define the components of the whole encoder E, ControlNet C, and decoder 'D as follows:
E(x) = (feres feres
(3)
C(x, c) -(f peres
peres
(4)
and
D(L eg "e,..,, f4.1) =
(5)
where fo add (feres fores) and represents the score function.
2.3 SCORE FUNCTION AND ITS CONSERVATIVITY
During the parameterization of the score function s(x,, t)) with a neural network, there are no inher- constraints the conservativity of score functions (Salimans & Ho, 2021). The conservativity of vector field indicates that the field can be represented as the gradient of scalar-valued function. For instance, the score function s(x,, t) can be modeled as the gradient of log q: (x,, t). A straight- forward method to verify if a vector field is conservative involves computing the Jacobian matrix of the vector field and checking this matrix is symmetric. This symmetry consequence of the commutativity of the partial derivatives of the scalar function.
However, directly calculating the Jacobian matrix of the score function poses significant challenges. As an alternative, we utilize stochastic estimators and leverage the capabilities of modern neural network frameworks, such as the vector-Jacobian computation features in PyTorch, obtain an unbiased estimation of the trace of the Jacobian matrix and related values.
To enforce the conservativity the score function directly, suppose the Jacobian matrix of s, with respect to x, is denoted as J J We propose using the following loss function:
Loc
2 Et,x, JJ.. - JT JI 11::
(6)
3

---

Published as a conference paper at ICLR 2025
where F represents the Frobenius norm. That formula can be equivalently expressed as (Chao al., 2022):
Lgc x,, [tr(Js.,x, Jg xx xx..)))
where the trace the product of Jacobian matrices can be efficiently estimated using Hutchinson's trace estimator (Hutchinson, 1989). However, even such an estimator can be computationally ex- pensive, especially when dealing with large-scale neural networks.
2.4 MULTI-OBJECTIVE OPTIMIZATION
The goal of multi-objective optimization to find the Pareto optimal solution, a state where no sin- gle objective be improved without degrading others. Similar single-objective optimization, local Pareto optimality can also be achieved using gradient descent techniques. The Multiple Gra- dient Descent Algorithm (MGDA) (Désidéri, 2012), is one such method for attaining local Pareto optimal solutions. The central concept of MGDA is to balance all gradients towards a direction that forms acute angles with each gradient, thereby ensuring that no objective worsens as result of an optimization step. It also has applications in image and 3D generation tasks (Sun et al., 2023; Huang et al., 2024). We think the idea of forming an acute angle with each gradient a good way to balance the gradients, and we will use this idea in the feature combination and feature injection.
PROBLEMS IN MULTI-CONTROLNER COMBINATION
Our problem setting aligns with the current standard in the community, wherein each ControlNet is trained individually. At the sampling stage, these networks are combined according to different control signals, functioning as plug-ins. This setup ensures flexibility and modularity, allowing for the seamless integration of various control signals to enhance the model's generative capabilities. However, this approach has several limitations, particularly when combining multiple ControlNets.
Data Bias in Areas with "Silent" Control Signals. In the training of ControlNet models, a sig- nificant issue arises from the presence of "silent" control signals, particularly with edge condition signals. These "silent" control signals are characterized by empty conditions where the correspond- ing paired image areas are often blurred or lack high-frequency information. This leads to data bias during training, causing the model to suppress high-ffrquency information in the generated im- ages. While this suppression can be advantageous for strict generations in single-control scenarios, it poses a challenge in multi-ControlNet combination scenarios. When two control signals coexist in area one with high-freque fre-frreyency information and the other being "silent" control signal conflict arises. The model, influenced by the "silent" control signal, may undesirably suppress high- frequency information in the generated images. This conflict is problematic in multi-ControlNet combination scenarios, where the preservation of high-frequency details is crucial.
Optimal Ratios for Multi-ControlNet Combination. Another challenge in combining multiple ControlNets is determining the optimal ratios for merging various control signals. There is currently no clear guideline for the combination of different control signals, making this process difficult. The current practice involves combining control signals as plug-ins at the sampling stage, relying on user experimentation. This approach may lead to suboptimal results, as the model may not effectively balance the different control signals.
Conservativity of Conditional Score Function. Although the conservativity of diffusion models well researched, the situation differs for ControlNet, where another network tuned to control the diffusion model with much less data compared to the original diffusion model. Therefore, is essential to consider the conservativity of the enhanced score function when combining multiple ControlNets to ensure stability or seek improved performance.
4 MINIMAL IMPACT CONTROLNET
To address the aforementioned issues, we introduce the MIControlNet. The key idea MIControl- Net is minimize the impact of the ControlNet the original U-Net by reducing the conflicts of
4

---

Published as a conference paper at ICLR 2025
+prompt +timestep
+Noise
Encoder
Decoder
S
Original Image
Image
+Mask
Calculating J.
ControlNet
Caleulating
Prozen Trainable
Canny Edge
Canny Edge
Overview of our data flow. Masking specific areas of the conditions allows the silent control signals to generate more diverse patterns, while exhibiting reduced controllability when interacting with high-frequency control signals.
Figure 2:
each NontrolNet, thereby achieving the best combination of multiple ControlNets. We will intro- duce the details of MIControlNet in the following sections following the order of problem setting, training strategies and sampling strategics.
4.1 REBALANCE THE DISTRIBUTION
To solve the first problem, we apply simple but effective data augmentation techniques to rebalance the distribution of areas lacking control signals. Specifically, from images on the control signals, enhancing the diversity of the image areas corresponding to the silent control signals, which the This process will help the model learn to generate freeffreyency information in areas with silent control signals, thereby reducing data bias during training. Further details will be explained in the Appendix F.I.
will apply segmentation masks
as inpainting the image area with the silent control signals.
4.2 MINIMAL. IMPACT ON FRATURE INJECTION AND COMBINATION
To minimize the impact of the ControlNet signal of each layer feres on the original U-Net en- coder feature feres we draw inspiration from the MGDA algorithm Désidéri (2012), which involves forming acute angles with each vector. We employ restricted MGDA-based balancing algorithm to regulate the injection of control signals into each layer, which will keep the coefficient of original feature fere peres In detail, we will apply the following new dynamic add function to the feature injection process.
Feusible Domain
Combined Feature
Feasible Domain
Feuture U-Net
fz
Figure 3: The left image shows the feature injection process in MIControlNet, while the right image illustrates the feature combination process. The Feasible Domain is where the combined optimiza- tion direction aligns with the U-Net feature both control signal features fi and fz.

---

Published as a conference paper at ICLR 2025
Firstly, we calculate the coefficient of injected control signal X, for each layer i follows:
(V2 v1) V2
AV(1 v2) min
L, max
0
(8)
V2 V1 2
and then
v2)
X(V1, V2) 1- A5(VI, V2)
(9)
The new add function is defined as follows:
add (n (f.rres . -f =f peree + Ae( , ..eres) (eeen)-- f.res
(10)
The key constraint impose is to maintain the coefficient of feres the original U-Net data flow architecture. Practically, the range of X; is limited to [0, 20] to mitigate the risk of overpowering control signals that could suppress the original features. Further details the connection to MGDA are elaborated in the Appendix E.
ensuring the preservation of
To balance different control signals during the sampling stage, also utilize the concept of forming an acute angle with each gradient, which initially balances the various control signals before the injection.
For the feature maps associated with different control signals, denoted f;- peres:) and peres? employ combination strategy defined by the following equation:
feree add (1 (feree i
peresi2),
(11)
com where the adm function is explicitly defined as:
add (ram((pres)) (peres,1 p.res.2) = 10--,,60 ( p + M x 2
(12)
Then follow the same process as in the feature injection stage to calculate the coefficient of the combined feature map A; for each layer i.
4.3 MINIMAL IMPACT ON CONSERVATIVITY
To estimate the Loc in Eqn. we need apply the Hutchinson's estimator to the Jacobian matrix of the model. However, such estimation still needs to construct second order derivatives, which is computationally expensive.
Our insight is that the parameters of ControlNet primarily manage the additional conservativity introduces. By decomposing the Jacobian matrix of the model into two components, we can isolate and only calculate the conservativity loss specific to ControlNet, which follows the red line in Figure 2, making the process simpler and more efficient. Additionally, applying this conservativity loss ensures control over all the extra unconservativity introduced by ControlNet.
Suppose v fits a distribution whose expectation O and variance is I, by the Hutchinson's 's estimator, have unbiased estimation of Loc, which is
Lest Ey,,,x, Je. JJ 8.
VJ JS.x,
(13)
We propose the following proposition to decompose the Jacobian matrix of the model into the orig- inal Jacobian matrix from the U-Net and the additional Jacobian matrix introduced by ControlNet.
Proposition 4.1 (Decomposition Jacobian Matrix) a U-Net model augmented with Control- Net, the overall Jacobian matrix Js,x, be decomposed into the original Jacobian matrix Je from the U-Net and an additional Jacobian matrix J introduced by ControlNet:
J Js,x; Je J.. + J -
(14)

---

Published as a conference paper at ICLR 2025
And due to the large training data gap between traing the U-Net and ControlNet, propose the following assumption to depart them in parameters level.
Assumption 4.1 (Responsibility for Conservativity) In the U-Net model equipped with Control- Net, the conservativity of the original Jacobian matrix governed by the parameters of the U-Net. Meanwhile, the parameters of ControlNet are principally tasked with managing the additional con- servativity introduced by ControlNet, which can described as
V Bexx, = O,
(15)
where & is the parameters of ControlNet.
Then, we can ignore the parts does not containing J s, in Eqn. 13 and got new loss for optimiza- tion of the ControlNet, which is
fàc = Ev,,,x, v [2J JcT 2J 81,x, 8e + J JeT Jr
Be.xe St
Be, Be.x. v.
(16)
We have
Proposition 4.2 Under Assumption 4.1, the gradient of the conservativity loss of ControlNet is equal the gradient of the estimated conservativity loss, which is given by
voLac- VVC
(17)
However, due computation limitations, still fully remove the Js J term. We have the following simplified loss for the ControlNet optimization, which is
simple QC
Ex,,,x, V [ x
JeT J 8.
J x.
(18)
And we have the following proposition for the relationship between the simplified loss and the original loss.
Theorem 4.1 Suppose the Frobenius norm of B.x, uniformly bounded by M, we have
Coc S 2 v
Loe + +LQC
psimple
Csimple
(19)
which indicates that if the simplied loss is zero, the original loss also zero.
In practice, we just apply the simplified loss to optimize the Control
5 EXPERIMENTS
5.1 EXPERIMENT SETUP
Dataset. For training. we primarily use the MultiGen-200 dataset (Qin et al., 2023), subset of LIoON-Aeshetie (Schuhmann al., 2022), which provides conditions such as Canny (Canny, 1986), Hed (Xie & Tu, 2015), and OpenPose (Cao et al., 2017). This dataset also includes segmenta- tions, which facilitate balancing the ground truth of areas with silent control signals. For evaluation, randomly sample images from LAION-Aesthete For single control signal, we use the original prompts from the dataset. For multi control signals, we directly concate the corresponding prompts the prompts.
and use them and their extracted conditions.
Implementation. We initially train our model using balanced data and feature injection for the Canny, HED, Depth, and OpenPose conditions. Once the model converges, we label this phase as 1-stage. We then continue training the model with additional conservativity loss, labeling this phase as 2-stage. Further details are provided in the Appendix. For sampling, we apply balanced feature injection and combination in our models. More results are in the Appendix H.

---

Published as a conference paper at ICLR 2025
Baseline. We select the newest ControlNet v1.1 and Uni Uni-ControlNet (Zhao al., 2024) baseline. For the ControlNet baseline, we provide both the original feature combination and a bal- anced version, labeled ControlNet*. We also introduce fixed scaling factors for the first control signals, labeled ControlNet and ControlNer- maintaining a total scaling factor of 2.0 to match the original ControlNet and feature combination method. Additionally, include Control- Net** which is trained with the same data augmentation as our method.
Some other models, such as ControlNet++ (Li et al., 2024), are optimized for precise control, making them less suitable as baselines in our experimental settings.
5.2 SINGLE CONTROL SIGNAL
In this subsection, we primarily examine the improvements our method brings when using a single control signal. The main improvement lies in the inpainting ability of the silent control signal.
5.2.1
QUALITATIVE COMPARISON
We mainly conduct two qualitative comparisons:
Total Variance under Silent Control Signals. For the calculation of the total variance, we sam- ple 500 images from the LesoN-Assthetics dataset and calculate the total variance in the regions controlled by the silent control signals. We then compare the results with ControlNet. As shown in Figure 4a, the results indicate the ability of our method to generate more diverse texture patterns in these situations.
Asymmetry in the Jacobian Matrix. We analyze the asymmetry of the extra part of the Jaco- bian matrix introduced by ControlNet, discussed in Asym metric defined by Chao et al. (2022). This indicates the asymmetry of the Jacobian matrix. As shown in Figure 4b, our method reduces the asymmetry, leading to the MultiGen-20M dataset, using a batch size of 64 for 100 steps. We observed that after the second-stage training, the asymmetry introduced by ControlNet significantly diminishes, indicat- ing smaller impact on the original U-Net, There is also an interesting phenomenon where the decreases in Asym are similar on a logarithmic scale.
stable and consistent control. The Asym metric is estimated
We also compare the FID and convergence speed as described in Appendix G. Our model achieves similar image quality with faster convergence compared the ControlNet baseline.
3.6
10
ControlNet
(istage)
Ous stags)
Ours (2 stage)
(2-stagr)
Jacahion Asymmery (log sodee)
Toal Variance (1x10f)
3,0
2.8
10'
22
2.0
Conny
Hel
10
Cammy
Hed
Openpose
(a) Total variance under silent control signals.
(b) Asymmetry the Jacobian Matrix.
Figure 4: Two qualitative comparisons for single control signal.
5.2.2 VISUAL COMPARISON
We compare the visual results for single control signals of the ControlNet and our MIControINet in Figure 5. More results are in the Appendix H. Our method demonstrates the ability to generate more texture patterns in areas corresponding to silent control signals, which aligns with the quantitative results shown in Figure 4a.
8

---

Published as a conference paper at ICLR 2025
Canny/ Hed
ControlNet
MIControlNet (I-stage)
MIControlNet (2-stage)
Figure 5: Comparision of ControlNet and MIControlNet for single condition generation.
5.3 MULTI-CONTROL SIGNALS
this subsection, we examine the improvements introduced by our method when using multiple control signals. We randomly selected 2,000 images from the LAION-Aesthetics dataset and ex- tracted the central portion of two conditions in equal measure for sampling. These conditions were randomly resized and placed on either the left right side, with the remaining area filled by silent signals. We then used these modified control signals to generate images. To save space, we present both conditions in single image in Figure?
Table 1: The FID of the multi-condition scenario. Each condition is associated with its FID. the FID scores are presented with the best result highlighted in bold and the second best underlined.
Methods
Crenpose-Camny
Openpose-Hed
Canny-Hed
Hed-Depth
ControlNet
80.37 / 111.30
76.98 / 84.20
123.59 / 86.43
91.98 / 86.25 -/-
ControlNet 0.5
105,86 123.13
145.88 / 107.52
143.67 106.40
ControlNet 1.5
74.37 99.44
74.52 86.57
120.84 88.38
-/-
ControlNet*
77.43 89.57
76.69 / 78.31
122.10 / 85.45
78, 90.65
ControlNet*
92.98 84.02
87.33 78.49
77.02 75.46
74.28 81.16
Uni-ControlNet
96.50 / 74.55
139.87 76.06
88.77 75.47
73.68 / 89.94
Ours (I-stage)
76.13 77.22
70.32 68.42
74.19 70.26
71.16 71.93
Ours (2-stage)
75.77 72.25
73.45 71.74
71.34 / 69.35
69.68 71.18
5.3.1 QUALITATIVE COMPARISON
FIDs for each control signals. We calculate the FIDs for two conditions in multi-condition scenario. For each condition, we extract the relevant part the generated image and compute the FID against the original 1,000 images. As shown in Table 1, our 2-stage MIControlNet achieves the best FIDs most cases, indicating that our method outperforms the baselines and highlights its scenarios. Our feature injection and combination technique achieves
effectiveness in an average improvement of 9.79 over the vanilla ControlNet with silent control signal targeted data augmentation. The data augmentation alone achieves average improvement of 11.66.
Cycle consistency for each control signal. Table Appendix G.2 shows the L1 distance be- tween the extracted condition from the generated images and the original condition. Our MICon- trolNet achieves the lowest values in most cases, indicating better preservation of control signals (excluding silent control signals) in the generated images.

---

Published as a conference paper at ICLR 2025
5.3.2 VISUAL COMPARISON
Visual comparisons are shown in Figure In the first case, ControlNet fails apply the openpose condition, while ControlNet* succeeds. In the second case, ControlNet fails to meet the canny condition. In the final case, ControlNet fails both control signals. Our method effectively silences silent control signals when other control signals are active, allowing the useful control signals to dominate. Additional visual results are provided in Appendix H.
CONCLUSION
In this paper, we introduced MIControlNet, designed to minimize the impact of ControlNet for im- proved multi-control signal integration. Our approach involves rebalancing the data distribution in areas controlled by silent control signals, introducing a multi-objective perspective to feature com- bination, and reducing the asymmetry in the Jacobian matrix of the score function. These strategies enhance the balance and compatibility of multiple ControlNets without necessitating joint training, enabling more free and harmonious generation using multiple control signals.
ACKNOWLEDGEMENT
This work is funded by the National Key R&D Program of China under Grant No. 2024QY1400 the National Natural Science Foundation China No. 62425604. This work is supported by Tsinghua University Initiative Scientific Research Program. This work is also supported by Alibaba Group through Alibaba Innovative Research Program. We sincerely thank Qixin Wang for her assistance in creating the pretty figures for this paper.
REFERENCES
John Canny. A computational approach edge detection. IEEE Transactions on pattern analysis and machine intelligence, 1986.
Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. Realtime multi-perss piiliperson 2d pose estimation using part affinity fields. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2017.
Chen-Hao Chao, Wei-Fang Sun, Bo-Wun Cheng, and Chun-Yi Lee. On investigating the conserva- tive property of score-based generative models. arXiv preprint arXiv:2209.127533 2022.
Jean-Antoine Désidéri. Multiple-gradient descent algorithm (MGDA) for multiobjective optimiza- tion. Comptes Rendus Mathemalique, 350(5-6):313333) 2012.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, pp. 8840-6851, 2020.
Christian Horvat and Jean-Pascal Plister. On gauge freedom, conservativity and intrinsic dimension- ality estimation in diffusion models. arXiv preprint arXiv:2402.03845, 2024.
Shuo Huang, Shikun Sun, Zixuan Wang, Xiaoyu Qin, Yanmin Xiong, Yuan Zhang, Pengfei Wan, Di Zhang, and Jia Jia. Placiddreamer: Advancing harmony in text-to-3d generation. Proceed- ings of the 32nd ACM International Conference Multimedia, pp. 6880-6889, 2024.
Michael F Hutchinson. A stochastic estimator of the trace of the influence matrix for laplacian smoothing splines. Communications in
and Computation, 1989.
Chieh-Hsin Lai. Yuhta Takida, Naoki Murata, Toshimitsu Uesaka, Yuki Mitsufuji, and Stefano Er- mon. Fp-diffusion: Improving score-based diffusion models by enforcing the underlying score fokker-planck equation. In International Conference on Machine Learning, 2023.
Ming Li, Taojiannan Yang, Huafeng Kuang, Jie Wu, Zhaoning Wang, Xuefeng Xiao, and Chen Chen. Controlnet++: Improving conditional controls with efficient consistency feedback. arXiv preprint arXiv:2404.079888 2024.
10

---

Published as a conference paper at ICLR 2025
Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. arXiv preprint arXhv:2108.01073, 2021.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023.
Can Qin, Shu Zhang. Ning Yu, Yihao Feng. Xinyi Yang. Yingbo Zhou, Huan Wang. Juan Car- los Niebles, Caiming Xiong, Silvio Savarese, et al. Unicontrol: A unified diffusion model for controllable visual generation in the wild. arXiv preprint arXiv:2305. 147, 2023.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, 2022.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
Tim Salimans and Jonathan Ho. Should ebms model the energy or the score? In Energy Based Models Workshop-ICLR 2021, 2021.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, al. Laion-5b: An open large-scale dataset for training next generation image-text models, Advances in Neural Information Processing Systems, 2022.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In Proceedings of International Conference on Machine Learning. 2015.
Yang Song, Jascha Sohk-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In Interna- tional Conference on Learning Representations, 2021.
Shikun Sun, Longhui Wei, Junliang Xing, Jia Jia, and Qi Tian. Sddm: Score- -eore-decomposed diffusion models on manifolds for unpaired image-to-image translation. Proceedings of International Conference on Machine Learning, 2023.
Pascal Vincent. A connection between score matching and denoising autoencoders. Neural Compu- tation, 2011.
Saining Xie and Zhuowen Tu. Holistically-nestedd edge detection. In Proceedings of the IEEE international conference on computer vision, 2015.
Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv preprint 2tti::2300.0672211 2023.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. IEEE International Conference on Computer Vision (ICCV), 2023.
Shihao Zhao, Dongdong Chen, Yen-Chun Chen, Jianmin Bao, Shaozhe Hao, Lu Yuan, and Kwan- Yee K Wong. Uni-controlnet: All-in-one control text-to-image diffusion models. Advances in Newral Information Processing Systems, 2024.
11

---

Published as a conference paper at ICLR 2025
APPENDIX
A RELATED WORK
A.I IMAGE-BASED CONTROL METHODS FOR DIFFUSION MODELS
Image-based control methods are crucial for image generation. Following the success of diffusion models, numerous algorithms for controlled image generation have been developed, leading the creation of techniques such as SDEdit (Meng et al., 2021), ControlNet (Zhang et al., 2023), and DreamBooth (Ruiz et al., 2023).
A.2 CONSERVATIVITY DIFFUSION MODELS
With the significant success of score matching training algorithms in the unconstrained score ap- proach, this method has become a focal point in research. The score functions learned this manner no longer conservative, meaning they may not strictly adhere to the constraints of the original data distribution. This lack conservativity could impact model performance, and numerous stud- ies have explored this phenomenon (Salimans & Ho, 2021; Chao et al., 2022; Horvat & Pfister, 2024: Lai et al., 2023). Researchers have attempted to adjust for this by incorporating either soft or hard conservativity constraints, producing some interesting theoretical results in the process.
However, while conservativity has been extensively studied in foundational models, there a rel- ative lack of research conservativity in diffusion models that enhance control over generative capabilities through the addition of modules. Given the unique generation process of diffusion mod- els, implementing effective conservativity controls is particularly critical, potentially offering new perspectives on improving model stability and generation quality.
B DISCUSSION AND LIMITATIONS
Compared with mainstream methods developed from ControlNet, which exert control influence across the entire image. our approach have distinct cases. While mainstream ControlNet meth- ods offer broad control capabilities, MIControINet focuses on precise control in targeted areas, ad- dressing conflicts arising from multiple control signals.
Our primary focus is on improving controllability. However, our method has not yet fully explored the potential of prompt engineering and related techniques, such using negative prompts and sampling algorithms. There is significant room for improvement in these areas, which could further enhance the effectiveness and flexibility of controlled image generation.
The necessity of incorporating a conservativity loss is another crucial aspect of our approach. Due resource constraints, we could not fully implement the conservativity loss in large-scale models. We hope future work will address this limitation, potentially leading to more robust implementa- tions. Additionally, with the theoretical advancements in conservativity constraints, similar to the development of score matching, we anticipate the emergence of unbiased estimation algorithms for the trace Jacobian matrix that does not require second-order gradient backpropagation.
C BROADER IMPACT AND SAFEGUARDS
Generative AI has the potential produce harmful information. To mitigate these risks, it is crucial to implement comprehensive safeguards. Accordingly, we will integrate a safety checker into our released code.
D PROOFS
D.1 PROOF OF PROPOSITION 4.1
Proposition D.1 (Decompusition of Jacobian Matrix) In a U-Net model augmented with Control- Net, the overall Jacobian matrix Js.xx be decomposed into the original Jacobian matrix Je
12

---

Published as a conference paper at ICLR 2025
from the U-Net and an additional Jacobian matrix x, introduced by ControlNet:
Ja.x. Je. + J -,xx1
(20)
J x,
J s pf
1+1
J. J peeeeeee + Jee prree xi
J s pa Jpd peren xe + Jo padee pores porendparee
(21)
J J.. gores J gores + Ja Ja
J Jorees X
1+1
1+1
Σ J s. fe J peres +
Σ
Ja. deren peres
J
J x
Sx,x,
D.2 PROOF OF PROPOSITION 4.2
Proposition D.2 Under Assumption 4.1, the gradient of the conservativity loss of ControlNet is equal to the gradient of the estimated conservativity loss, which is given by
VoLoc = V Loc.
(22)
C Ev, TJ Je JT
s1 V Js, JSc
Exitin T [2 SJ Bexxs - 2J Jr
JcT
s.
+ Jr.
JcT
Se.X:
Bcix.
Jr. Jo.xel
(23)
+ E V [Je [J 81,0 x. 81
JeT Je J
= Cso + Ev. v [ee x, 8.
J
x
Je
85,x, V.
Because V [Je.
8 81.x
JcT Je Jc
x. x
= O, therefore, we have
Lo
(24)
D.3 PROOF OF THEOREM 4.1
Theorem D.1 Suppose the Frobenius norm of Je Ja,x is uniformly bounded by M, we have
Lig S 2V2M
/
csimple Csimple
(25)
which indicates that if the simplied loss is zero, the original loss is also zero.
13

---

Published as a conference paper at ICLR 2025
f.c F. V [2
JeT
2J
J
+ +J. JcT
J Bc.x, Sc
Beix,
Bc.x.
St.Xe
BeiXe Bri
+ Ev,t,x, v [J Br.x: 81.x.
JeT
J
s.
J
81,x, v
Eviti V [J J
J.T
sx J J V
L. v (J + I Seixa)
(J + Je T
(J se,20 + + (J + S J
Ev...x. V [J (Ssxx J.T - J Sx I J
-Ex, (ir s. + + xx (Jo. (Je.x. + J xx ') br (JSs,xx +J xx.x) (Jer + J IG
Ex tr (J JoT Jotxe) - tr (J st, Js. xx,
Ei,xt [ Sl:x1 )x
(J + J x
T
E Er,x |_
-Jcl 51,x1 1|2
F7
Bi
[] -Jot (x(x))) + (Ja
JCT B
1
E4
-JeT 11
2
< Ei,x
(IJ
JoT || + J
JJT
x.
1
Br,xs
|| |
Sc,xx 11?
2
E1,x1
| Se.xc
Se Il J
JcT
JeTxelly +
Ex
| 80,xx - -Jo l1?
sr,x,
(26)
Because that ||
S M, we have:
| JST J.txxll 11r S ||Je xx.. 11 | [ 2M.
(27)
Then, we have
foc S 2MMx ||Je Jot 11
Be,xa [ Ja -. 11
(28)
By Cauchy-Schwarz Inequality, we have
2 Exxx [|J..... sc Js.xxill ||p]
I Exxx [ -JoT JSTxx
(29)
Therefore, we have
Exxx [e JET -sixxilp 1| S V
Brxx [| JeT
(30)
Then,
Bexx [[ fōo 22VMM
JoT 1|2 Br,x x.
[|
JeT
(31)
80,Xe
which indicades
Cgc 2222M v Cae Cog
csimple
Csample
(32)
E MGDA FOR FEATURE INJECTION AND COMBINATION
The score function sy (x;) is defined as the gradient of the scalar value log p(x)). ). we can interpret addition operation in the score domain the combination of gradients from different optimization objectives. Thus, MGDA is applicable for optimizing this blend of diverse gradients. While the feature domain of U-Net may not present straightforward optimization objective, adapt the
14

---

Published as a conference paper at ICLR 2025
principle of forming acute angles between each feature map to mitigate conflicts among various features.
The distinction between feature injection and combination lies in the architecture of ControlNet. Feature injection involves adding the control signal to the original U-Net feature map, which sug- gests that the original U-Net feature map should ideally remain unchanged. Therefore, after balanc- ing the coefficients with MGDA, additional scaling is required to ensure this. In contrast, for feature combinations, can directly apply MGDA balance the feature maps without such constraints.
F IMPLEMENTATION DETAILS ABOUT MICONTROLNET
F.1 DATA REBALANCE DETAILS
Firstly, we segment the images into distinct regions. Subsequently, we randomly select portions of these segmentations, utilizing both the edge-like features within these selected areas and the original images to construct our training dataset. This approach ensures that edge-like features not included in the segmentations are converted into silent control signals. Consequently, the corresponding image regions retain figh-frequency information, crucial for detailed image generation in silent control signals.
F.2 TRAINING DETAILS
Our training process comprises two stages, all of which are conducted on the MultiGen-20M dataset (Qin et al., 2023) using our balanced control signals. In the first stage, we train the model using the add 'nj operation for epochs. For the OpenPose Model, which has less training data, the duration extends to epochs. In the subsequent stage, we integrate the LQC Csimple loss into the original
diffusion predicting noise loss with a coefficient of 0.01, and continue training for 2000 steps with an equivalent batch size of 128. All experiments are executed on eight NVIDIA A800 GPUs, each with 80GB memory. The first stage requires approximately 2 days, while the second stage is completed in about 7 hours.
F.3 SAMPLING DETAILS
For sampling with multi MIControlNets, we first apply the add operation for the feature com- bination of different MIControlNets. Then we apply the add operation to add the feature maps of MIControlNets that of the original U-Net.
G MORE EXPERIMENTS
G.1 THE SUDDEN CONVERGENCE OF MICONTROLNET AND FID
We evaluate the rapid convergence behavior of MIControlNet compared to the original Control illustrated in Figure 6. Notably, ControlNet often experiences sudden shifts performance particular training steps. To investigate further. we focused on these critical training milestones for both MIControlNet and ControlNet. Our results demonstrate that MIControlNet achieves earlier convergence while maintaining similar or improved generation quality compared to ControlNet.
G.2 MORE QUALITATIVE METRICS FOR MULTI-CONDITION EVALUATION
Table shows the I.1 distance between the extracted condition from the generated images and the original condition. Our MIControlNet achieves the lowest values in most cases, indicating better preservation control signals (excluding silent control signals) the generated images.
Table 3 presents the FID scores of various models using new conditioning approach, where the ground truth image is split into left and right sections, and conditions are extracted for each part. In Table 4, the ground truth image is divided into the central object and the surrounding areas, with conditions extracted accordingly. Our MIControlNet consistently suppresses baseline models across nearly all conditions, demonstrating its superior performance in multi-condition image generation.
15

---

Published as a conference paper at ICLR 2025
Hed
Model
100
ConzzllNei- - OpenPioe
MiCasteaNie OpenPtoee
OpenPose
OpenPose
96
110
92
Canny
Canny
58 Hed
5000
7000
30/0
9000
10000
11000 12000
Steps
Figure 6: FlD-convergence steps. circle represents ControlNet and square represents MIControlNet.
Table 2: The distance between the condition extracted from the generated image and the ground truth in the conflict area. The distances are L1 norm expressed in units of x 10a.
Methods
Openpose-Canny
Openpose-Hed
Canny-Hed
ControlNet
1.3903
1.7851
2.8626
1.3223
1.8310
2.8881
ControlNet
1.3848
1.9009
2.8123
ControlNet*
1.3833
1.9066
2,9381
Ours (I-stage)
0.9638
1.5080
1.9634
Ours (2-stage)
1.0729
1.6600
2.1954
Uni-ControINet
1.0808
1.7232
2.0951
G.3 QUALITATIVE METRICS UNDER DIFFERENT PROMPT CONDITIONS
Table presents the FID and Total Variance (TV, units of 1 x 104) for Canny and OpenPose conditions under three scenarios: no prompts, brief prompts, and detailed prompts.
We have the following findings:
MIControlNet. with or without the conservativity loss, demonstrates similar FID perfor- mance. However, with conservativity loss, MIControlNet exhibits improved pattern gener- ation ability under silent control signals, as highlighted in Table 5.
MIControlNet achieves comparable FID performance to the baseline but demonstrates sig- nificantly stronger performance in terms of total variance.
When comparing prompts, brief prompts, and detailed prompts, providing detailed prompts generally leads to better FID performance and smaller total variance.
Interestingly, for detailed prompts, the total variance tends to slightly increase. We hypoth- esize that this is due to the more detailed prompts offering finer control under silent control signals, thereby generating more diverse patterns.
16

---

Published as a conference paper at ICLR 2025
Table 3: The FIDs for the left-right split condition. ControlNet denotes ControlNet*s with our balanced feature combination sampling. ControlNet and ControlNet**L. represent Control- Ne sampling where the first control signal scaled by 0.5 and 1.5, respectively.
Methods
Openpose-Canny
Openpose-Hed
Canny-Hed
HED-Depth
ControlNet
63.8590
75.2076
58,8519
69.3344
Net
68.7007
69.4012
61.9949
62.1554
ee
67.5028
65.9667
66,9362
62.9284
Nen
80.1214
84.4420
68.6729
61.8351
Ne
66.1629
65,2359
65.5698
71.7198
ControlNet*
68.8815
68.0828
56.7660
71.9574
ControlNet
65,9972
108.2335
81.5698
75.4173
ControlNet 1.5
69.2258
66.4017
58.4945
93,3087
Ours(I -tagee)
64.7937
64.0063
55.9595
56.3233
Ours(2-stage stage)
62.5830
66.4729
54.3970
57.9421
Uni-ControlNet
71,2586
89.4048
56.8694
65.9861
Table 4: The FIDs for the central-outside split condition.
Methods
Openpose-Canny
Openpose-Hed
Canny-Hed
Hed-Depth
ControlNet
65.2961
75.3146
60.8962
73.4688
Cet
68.1888
63.8602
58.9864
59.8399
Cen
66.6215
66.4260
62.1626
62.1199
ee
70.8668
71.1370
63.1586
61.1704
ControlNet
67.3933
65.0094
61.3308
62.3237
ControlNet
72.1651
72.1254
67.7424
78.5825
ControlNe
72.5424
104.6354
82.8161
81.8447
ControlNer
67.6942
67.9931
62.5344
89.5242
Ours(I-stagee Issage))
62.9185
59.0101
54.9762
56,8017
Ours(2-stage)
61.6007
61.2391
56.3576
57.4142
CeitControlNet
60.5932
68.3847
57.8282
61.7094
G.4 ASYMMETRY ANALYSIS FOR REGULAR CONTROLNET
We calculate the asymmetry (Asym) for Regular ControlNet and compare it with our MIControlNet (I-stage) and MIControlNet (2-stage). The results shown in Table 6:
We have the following findings:
MIControINet (1-stage) performs slightly better than ControlNet in terms of asymmetry (Asym).
MIControlNet (2-stage) significantly outperforms both ControlNet and MIControlNet (1- stage) on Asym.
For each condition, MIControlNet (2-stage) demonstrates consistent improvements logarithmic scale.
G.5 A MORE CLEAR ABLATION
The FID scores for a thorough ablation study are shown in Table We observe that:
Our silent control signal-targeted data augmentation, feature injection & combination, and conservativity loss all lead to improvements FID scores.
The conservativity loss, particularly for Canny combined with other conditions, achieves a consistent improvement of approximately 3 points in FID.
17

---

Published as a conference paper at ICLR 2025
Table 5: FID and Total Variance (TV) for Canny and OpenPose conditions under different prompt scenarios.
(FID, TV)
ControlNet
MIControlNet (1-stage)
MIControlNet (2-stage)
Canny No Prompts
(109.6, 2.62)
(114,433544)
(123.9, 3.79)
Canny Brief Prompts
(89.34, 2.48)
(89.77, 3.24)
(90.18, 3.15)
Canny Detailed Prompts
(88.55, 2.47)
(90.21,
(89.37, 3.36)
Open No Prompts
(132.5, 3.34)
(131.9 3.39)
(133.0, 3.67)
OpenPose Brief Prompts
(97.08, 2.52)
(98.32, 2.71)
(98.14, 2.74)
OpenPose Detailed Prompts
(99.09, 2.70)
(95.34, 2.92)
(94.16, 2.91)
Table 6: Asymmetry (Asym) comparison across different conditions.
Condition
Canny
Hed
Openpose
ControlNet
56.75
22.41
6.454
MIControlNet (I-stage)
29.87
38.28
3.980
MIControlNet (2-stage)
0.1174
0.1894
0.0274
The improvements achieved through the conservativity loss are consistent, and we have further strengthened its theoretical foundation, particularly in the context of modular neural networks designed to optimize GPU memory usage and computational efficiency.
18

---

Published as a conference paper at ICLR 2025
Table 7: FID scores for different methods under various conditions. Lower scores indicate better performance.
Method
Openpose-Canny
Canny-Hed
Hed-Depth
Vanilla ControlNet
80.37 / 111.30
123.59 / 86.43
91.98 / 86.25
Data Augmentation
92.98 84.02
77.02 75.46
74.28 81.16
Our Feature Injection & Combination
76.13 / 77.22
74.19 / 70.26
71.16 / 71.93
Conservativity Loss
75.77 / 72.25
71.34 69.35
69.68 / 71.18
H
SAMPLES
Culti-Condition
ControlNet
ControlNet
MIControlNet (1-stage)
MIControlNet (2-slage)
Figure 7: Comparison of ControlNet and MIControlNet for multi-conditions generation.
19

---

Published as a conference paper at ICLR 2025
Canny
ControlNet
MIControlNet (I-stage)
MIControlNet (2-stage)
Figure 8: More visual results for single control signal.
20

---

Published as a conference paper at ICLR 2025
Hed
ControlNet
MIControINet (1-stage)
MIControINet (2-stage)
Figure 9: More visual results for single control signal.
21

---

Published as a conference paper at ICLR 2025
Openpose-Campe
ControlNet
ControlNet
MIControlNett (I-stage)
MIControlNet (2-stage)
Figure 10: More visual results for multi-control signals.
Z
Openpose-Hed
ControlNet
ControlNet
MIControlNet (1-stage)
MIControlNet (2-stuge)
Figure 11: More visual results for multi-control signals.
22

---

Published as a conference paper at ICLR 2025
Canny-Hed
ControlNet
ControlNet**
MIControlNet (I-stage)
MIControlNet (2-stage)
Figure 12: More visual results for multi-control signals.
23


Reviews:
[{"review_id": "yeroLXxPTu", "reviewer": "Reviewer_ESQy", "summary": "This paper presents a well-designed approach to multiple CtrlNet integration by addressing the challenge of control signal conflicts through a \"minimal impact\" strategy. \nSpecifically, the authors address several issues they found in multi-ControlNet integration.\n1. To handle data bias in silent control regions, they use data augmentation and segmentation masks. This adds texture to low-signal areas. \n2. To avoid conflicts between control signals, they use an MGDA-inspired feature injection method. This balances gradients so that no single control dominates. \n3. They introduce a conservativity loss function to ensure stable training. This function keeps CtrlNet's influence to original U-Net minimal and improves consistency in multi-control outputs.", "questions": "Please refer to the *weakness* section.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The solution proposed is clear and systematic. Experiments are comprehensive and convincing, covering various control signal combinations and providing robust quantitative and qualitative results. Limitation is discussed in the appendix.", "weaknesses": "Based on the methods presented, it appears that the proposed framework effectively supports scenarios involving more than two CtrlNets. Did the authors conduct experiments to evaluate performance in such cases? Since the experiments primarily focused on pairs of CtrlNets, I am curious about how well the approach scales to accommodate multiple CtrlNets.\n\nAdditionally, the authors mention the \"Conservativity of Conditional Score Function\", which seems to be a general principle that influences ControlNet performance, regardless of whether there is one or multiple CtrlNets in play. I wonder if the solutions proposed in this context could also enhance performance in standard single-CtrlNet applications. It might be beneficial for the authors to discuss this potential in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PR9FTI8Gfg", "reviewer": "Reviewer_ZUxW", "summary": "- The paper introduces Minimal Impact ControlNet (MIControlNet), a novel framework designed to refine the integration of multiple control signals within diffusion models for image generation.\n\n- The innovative approach of MIControlNet includes rebalancing data distribution, multi-objective feature combination, and reducing asymmetry in the Jacobian matrix to minimize conflicts between control signals.\n\n- Experimental results demonstrate that MIControlNet outperforms baselines, achieving better compatibility and fidelity in generated images with multiple control signals.", "questions": "See Weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The method improves the compatibility of multiple control signals, which is crucial for applications requiring simultaneous control from different sources.\n\n - By addressing conflicts between control signals, the approach leads to higher quality image generation, particularly in areas with silent control signals.", "weaknesses": "- In Figure 1, why is it not the canny signal that suppresses the openpose signal? Is there some analysis?\n\n- The authors should supplement corresponding **prompts**. Detailed prompts often lead to the generation of complex textures. Comparing the total variance and visual results under detailed/brief/null text conditions would enhance the persuasiveness of the proposed method in this paper.\n\nA good response to the Weaknesses will improve my initial rating.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0a5fzAmFtB", "reviewer": "Reviewer_DjNC", "summary": "In current ControlNet training, each control is designed to influence all areas of an image so that conflicts occur where lacking boundary information(low-frequency signals) especially in the black areas. The authors proposed the Minimal Impact ControlNet(MIControlNet) that can handle conflicts when different control signals are expected to manage different parts of the image. The introduced silent control signals which utilizes segmentation masks can rebalance the distribution of high-frequency information in the blank area. The authors reorganized the formula of ControlNet giving theoretical enhancement and ensuring more stable learning dynamics. The proposed multi-objective optimization strategies which utilizes Multiple Gradient Descent Algorithm (MGDA) improved the model performance.", "questions": "In line 121, $f^e_c$ seems weird. Doesn't the value under $f$ mean layer number?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The problem settings and overall paper structure are well organized.\n\nThe qualitative results for both single control signal and multi control signals utilizing segmentation masking are well aligned with handling the distribution of high-frequency information in the blank area. The blur effect shows in ControlNet even with single condition generation in silent area, however the MIControlNet shows more high-frequency informations in silent area.\n\nThe proof that gradient of the conservativity loss of ControlNet is equal to the gradient of the estimated conservativity loss and simplified loss for the ControlNet optimization are well explained.\n\nThe proposed model achieves the best FIDs in most cases within the multi-control signals.", "weaknesses": "In 2.2, the notations and symbols are confused and hard to read.\n\nThe proposed estimated conservativity loss function needs to construct second order derivatives, which is computationally expensive.\n\nThe authors proposed three main contributions with rebalancing the data distribution, feature injection and combination, and conservativity loss, however there are no ablation studies at each component in the paper. The only comparison is that both the original feature combination and a balanced version together between ControlNet and proposed model. For example, there are no comparison about Jacobian Asymmetry between the other baselines in Figure 4.(b). To claim the main contributions of each method, it would be beneficial to compare the performance improvements of each method against ControlNet.\n\nThe problem settings and the flow of the solutions are well-structured, however, the segmentation masking and the feature injection process shown in Figure 3 seems to be a little simple. Although the two main contributions(silent control signals, feature injection method) enhanced the model performance in the targeted problems, the method to apply the techniques are not quite novel. \n\nMinor:\n\nIn Figure 6, the legend seems to have lack of information.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "z5A3lEwHH1", "reviewer": "Reviewer_TnUC", "summary": "This article addresses the coordination of diverse control signals in diffusion models for precise manipulation, specifically focusing on the conflicts caused by low-frequency signals, termed \"silent control signals.\" Such signals often interfere with those that generate finer details, resulting in suboptimal outcomes. To tackle this, the authors propose the Minimal Impact ControlNet (MIControlNet) framework, which minimizes ControlNet’s influence on the original U-Net and mitigates conflicts through modified training data, multi-objective optimization, and adjustments to the Jacobian matrix asymmetry in the score function. A two-stage training process is used: first, balanced data and feature injection for various conditions, followed by a secondary phase incorporating a conservativity loss to enhance signal compatibility and image fidelity.", "questions": "* In Equations (8) and (9), what do v1 and v2 represent? Additionally, since λi varies across layers, should v1 and v2 also include subscripts to indicate properties at different layers within the network? \n\n* The concept of \"conservativity\" in ControlNet doesn’t seem to be a commonly used term. I find it somewhat confusing—could the authors provide further clarification? \n* In Figure 4(b), why is there no comparison with ControlNet? From the results, it appears that Stage-1 has a more significant impact than Stage-2. How do the authors view the balance between these two stages? This may relate to the previous question, as the authors mention that Stage-2 primarily enhances the \"conservativity\" of control signals. \n\n* Could this approach potentially be extended by first separating foreground and background for guided generation and then synthesizing the layers? This might allow for even greater control over specific elements.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "1. Analyzed challenges in training diffusion models with multiple control signals for ControlNet: \n\n* Data Bias: Handling biases in regions influenced by \"silent\" control signals.\n\n* Optimal Combination Ratios: Determining ideal ratios for integrating multiple ControlNet layers.\n\n* Conditional Score Function Stability: Ensuring balanced, stable updates across control signals (referred to as the “conservativity” of the score function).\n\n2. Introduced a conservativity loss function within a modular network structure, supported by theoretical analysis.\n3. Results demonstrate that the new loss function improves texture details specifically in areas affected by silent control signals, achieving finer and more accurate generation.", "weaknesses": "1. ControlNet is not yet a widely recognized concept. It is recommended that the authors briefly introduce its definition when first mentioned; otherwise, readers may feel confused starting from the abstract.\n\n2. Some phrases lack clarity. For example, in lines 18-19 of the abstract, \"..., the silent control signals can suppress the generation of textures in related areas,...\" leaves \"related areas\" ambiguous. Does this refer to areas that are texturally similar, spatially close, or something else?\n\n3. There is a lack of necessary citations, such as in lines 164-165, where it states \"to find the Pareto optimal solution, a state where no single objective can be improved...\". Adding citations here would provide more credibility to the explanation.\n\n4. The organization of the paper could be improved, as it is currently somewhat challenging to follow. Additionally, the main text does not sufficiently emphasize results showing the balancing of coefficients among multiple control signals. \n\n5. The three main issues highlighted in the problem analysis section lack clear support in the experimental results.\n\n6. Ablation studies are missing, which would help illustrate the individual contributions of each component to the overall outcome.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)