FlexControl: Computation-Aware Conditional Control with Differentiable Router for Text-to-Image Generation
Authors: Zheng Fang
Lichuan Xiang
Xu Cai
Kaicheng Zhou
Hongkai Wen
Venue: ICML 2025 poster (2025)
Keywords: Diffusion model
controllable image generation
dynamic route
data-driven
efficient inference
Primary area: deep_learning->generative_models_and_autoencoders

Abstract: Spatial conditioning control offers a powerful way to guide diffusion‐based generative models. Yet, most implementations (e.g., ControlNet) rely on ad-hoc heuristics to choose which network blocks to control — an approach that varies unpredictably with different tasks. To address this gap, we propose FlexControl, a novel framework that equips all diffusion blocks with control signals during training and employs a trainable gating mechanism to dynamically select which control signal to activate at each denoising step. By introducing a computation-aware loss, we can encourage the control signal to activate only when it benefits the generation quality. By eliminating manual control unit selection, FlexControl enhances adaptability across diverse tasks and streamlines the design pipeline with computation-aware training loss in an end-to-end training manner. Through comprehensive experiments on both UNet and DiT architectures on different control methods, we show that our method can upgrade existing controllable generative models in certain key aspects of interest. As evidenced by both quantitative and qualitative evaluations, FlexControl preserves or enhances image fidelity while also reducing computational overhead by selectively activating the most relevant blocks to control. These results underscore the potential of a flexible, data‐driven approach for controlled diffusion and open new avenues for efficient generative model design. The code will soon be available at https://github.com/Daryu-Fan/FlexControl.

FlexControl: Computation-Aware Conditional Control with Differentiable
Router for Text-to-lmage Generation
Zheng Fang"
Lichuan Xiang" 12
Xu Cai? Kaicheng Zhou? Hongkai Wen
30
20
SD1.5 5-based FlexContrd
Adtested contrc
FeecCnntoo 230 GFLOPS
(-50% spansty)
10
1.98 FID]
Routers
Acivadd canbollacts
FlexContral
RexControl 3.11 TFLOPs
Z
Z
(g(2,,c,1
7o
(-50% sparsty)
Co(2, c)
Inferance steps
(a) ControlNet
(b) FlexControl
(c) Statistics activated blocks
Figure 1. Dynamically inject conditional controls for image generation based timestep and task-specific sample. (a) The architecture of conventional controllable generative model ControlNet. (b) The architecture the proposed FlexControl. (c) Statistics on the number of activated control blocks of the FlexControl each denoising step. Here, "~50% spasity" indicates that the number of floating-poin operations (FLOOPs) of activated blocks is limited to 50% of trainable branch.
Abstract
with computation-awar training loss in end- to-end training manner. Through comprehensive experiments on both UNet and DiT architectures on different control methods, we show that our method can upgrade existing controllable gener- ative models in certain key aspects of interest. As evidenced by both quantitative and qualitative evaluations, FlexControl preserves enhances image fidelity while also reducing computational overhead by selectively activating the most rele- vant blocks to control. These results underscore the potential of a flexible, data-driven approach for controlled diffusion and open new avenues for efficient generative model design. The code will soon be available https: ////tttinn githuu..... com/ Daryu-Fan/PlexConte
Spatial conditioning control offers powerful way to guide diffusson-based generative mod- els. Yet, most implementations (e.g., Control- Net) rely on ad-hoe heuristics to choose which network blocks to control - an approach that varies unpredictably with different tasks. To ad- dress this gap, we propose FlexControl, novel framework that equips all diffusion blocks with control signals during training and employs trainable gating mechanism to dynamically select which control signal to activate at each denois- ing step. By introducing computation-awee -wwrre loss, we can encourage the control signal to ac- tivate only when benefits the generation qual- ity. By eliminating manual control unit selec- tion, FlexControl enhances adaptability across diverse tasks and streamlines the design pipeline
1. Introduction
Equal contribution 'Deaarment of Computer Science, Univer- sity of Warwick, Coventry, UK CCollov Labs. Correspondence to: Hongkai Wen <hongkai. wen @ war.wcck.c..kk..
Diffusion-baat based image generative models have recently gained widespread acceptance in the art and design commu- nity. Not only for their high-quality, photo-realistic image generation, but also due to the transformative capabilities of controllable unit like ControlNet (Zhang et al., 2023b), 121-Adapter (Mou al., 2024), OminiControl(Tann al.,
Proceedings of the nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

FlexControl: Computation-Aware Conditional Control with Differentiable Router for Text-to-lmage Generation
2024), elc., enable users to create images under diverse spa- tial conditions (e.g., layout, pose, shape, and form), allow generated images that satisfy various real-world demand.
Computation-aware controllable generation: Our approach significantly enhances controllability and im- age quality while maintaining a similar computational cost to the original controllable generative model by introducing a novel computation-aware training loss. Furthermore, this strategic allocation of computation to control units outperforms brute-force doubling, es- tablishing new Pareto frontiers in the control -uaalit v.s. compute trade-off.
Despite its growing popularity, controllable generative meth- ods typically rely on multiple design hyperparameters such as choosing which block to control for improved fi- delity and adherence to imput conditions, without a sys- tematic investigation of their effects. For example, the ControlNet variant based on SD1.5 (Stability, 2022) repli- cates encoder blocks and injects control information into the decoder, whereas T2I-Adapter applies control in the encoder. It remains unclear which block-level configuration is most effective, especially since optimal designs can vary by task. Complicating matters further, the controllable diffu- sion model is often trained on significantly smaller datasets than those used for the diffusion model's pre- pre-training. imply- ing that adding too much control could disrupt the pretrained representations and degrade image quality, while insuffi- cient control may fail to deliver the desired guidance. As evidence, a recent study (Ju et al., 2024) highlights that the number and placement of control blocks might significantly affect image quality in tasks such as inpainting. Moreover, the diffusion pipeline relies on heuristic strategy to decide which timesteps should receive control signals at inference, yet evidence is scarce regarding which approach consistently yields the best results. Collectively, these gaps emphasise the need for a more principled, comprehensive analysis of network design and inference strategy.
Universal plug-and-play integration: Our method seamlessly integrates with any dual-stream and LoRA- based control-model, introducing minimal additional parameters and zero architectural modifications to host models. It enables flexible switching between full con- trol and efficiency-optimized modes, depending on computational requirements.
2. Related Work
Text-to-image diffusion models. The diffusion proba- bilistic model was originally introduced by Sohl-Dickstein et al. (Sohl-Dickstein et al., 2015), which has been success- fully applied in the field of image synthesis and achieved impressive results (Dhariwal & Nichol, 2021; Kingma et al., 2021: Huang et al., 2023a;b; Jiang et al., 2023; Ren et al., 2022). The Latent Diffusion Models (LDMs) (Rombach et al., 2022), reduce computational demands by transfer- ring the diffusion process from the pixel space to the la- tent feature space. Such diffusion models (Stability, 2022; Nichol et al., 2022; Podell et al., 2023; Rombach et al., 2022; Saharia et al., 2022) typically encode text prompts as potential vectors through pre-trained language models (Radford et al., 2021; Raffel al., 2020), combined with UNet (Ronneberger et al., 2015) to predict noise to remove at each timestep. Recent studies explore Transformer-based architectures, which have yielded state-of-the th-he-rr- results for large-scale text-to-image generation tasks (Bao et al., 2023a;b; Peebles & Xie, 2023; Tu et al., 2022; Esser et al., 2024b), These frameworks leverage Transformers' capacity for modeling long-range dependencies and scaling mas- sive multimodal datasets, enabling breakthroughs in compo- sitional reasoning, dynamic resolution adaptation, and high- fidelity synthesis. However, their reliance on purely textual input-despite advances in cross-modal alignment-still poses challenges for precise spatial or stylistic control.
To dynamically adjust blocks being controlled based on timestep and conditional information while maintaining (or even improving) generation quality, we propose Flex- Control, a data-driven dynamic control method. Similar to existing controllable generative methods, as shown in Figure I(a), we adopted mostly the same control method set- ting as the original control while our FlexControl equipped with a router unit within the control signal in each block (see Figure 1(b)) to plan forward routes, activating control signal only when necessary based on the current latent vari- able. In contrast to other controllable generative models, FlexControl customizes the inference path for each input, minimizing potential redundant computations. In summary, our main contributions are as follows:
Data-driven dynamic control configuration: We in- troduce an automated router unit that dynamically se- lects blocks to control each timestep, eliminating the need for exhaustive architecture searches and retrain- ing. Our approach enables: (1) task-adaptive control configurations through end-to-end training, (2) tem- porally adaptive inference via per-timestep activation decisions, and (3) faster configuration design compared to manual search baselines by removing the need for configuration search and repeated training.
Controllable diffusion models. While state-of-the-ar text-to-image models achieve remarkable photorealism, their reliance on inherently bow-bandiitth, abstract textual input limits their ability to meet the nuanced and complex demands of real-world artistic and design applications. This underscores the growing need for frameworks like Con- trolNet (Zhang et al., 2023b) and T21-Adapter (Mou et al., 2024), which augment text prompts with spatial or structural
2

---

FlexControl: Computation-Aware Conditional Control with Differentiable Router for Text-to-Image Generation
constraints (e.g., sketches, depth maps, or poses), enabling finer-grained control over generation to bridge the gap be- tween creative intent and algorithmic output. Recent ad- vancements in controllable text-to-imase -atttto-mmag generation have di- versified across methodological approaches. Instance-basse methods, such as those by (Wang et al., 2024; Zhou et al., 2024) enable zero-shot generation of stylized images from single reference imput, prioritizing speed and flexibility. Meanwhile, an improvement in cross-attention constraint, proposed by (Chen et al., 2024a), guides generation along desired trajectories by refining latent space interactions. Prompt engineering has also emerged as a lightweight strat- egy for enhancing controllability, with works like (Ju et al., 2023; Zhang al., 2023e; Yang et al., 2023; Li et al., 2023) optimizing textual or hybrid prompts for fine-grained guid- ance. Additionally, multi-condition frameworks (Hu al., 2023; Qin et al., 2023; Zhao et al., 2024a; Li et al., 2025) integrate auxiliary inputs-such as segmentation maps or depth cues-to complement text prompts, improving align- ment with complex user intent. However, while these meth- ods expand generative versatility, many overlook the compu- tational overhead introduced by auxiliary networks, limiting their scalability for real-time applications.
Dynamic Neural Network. Unlike static models, dy- namic neural networks can adjust the inference trajectory of the neural network based on inputs, thus achieving superior trade-off between performance and efficiency by sacrificing cheap storage. (Bolukbasi et al., 2017; Han et al., 2022; Fan et al., 2024a; Elhoushi et al., 2024) adopt the method of early exit, allowing samples to exit the infer- ence process in the middle layers of the neural architecture based on the feature information they contain through dis- criminators or other judgment conditions to achieve depth adaptation. However, the effective layers are not necessarily continuous and the ineffective layers located at the front of the network cannot be omitted. (Herrmann et al., 2020; Yang et al., 2020; Li et al., 2021; Han et al., 2024) real- ize dynamic neural networks through adjustable network widths, but such methods are not applicable to the current mainstream Transformer architectures. To compensate for the above deficiencies, (Rao et al., 2023; Zeng et al., 2023; Elhoushi et al., 2024; Yang et al., 2025) alleviate the running latency caused by network redundancy through adaptively activating part of the network components based on samples. Recently, dynamic neural network technology has also been introduced into the diffusion model (Pu et al., 2024; Fan et al., 2024b; Zhao et al., 2024b) to accelerate the diffusion iteration process. However, in the field of controllable con- ditioning generation, the potential of dynamic architecture remains unexplored.
Improving diffusion efficiency. Efforts to improve dif- fusion model's efficiency have focused on architectural re- design, training optimization, and inference acceleration. NottrolNeXt (Peng et al., 2024) replaces ControlNet's 's bulky auxiliary branches with a streamlined architecture and sub- stitutes zero convolutions with Cross Normalization, slash- ing learnable parameters by 90% while maintaining stable training convergence. Beyond this, multi-expert diffusion frameworks (Lee et al., 2024; Zhang et al., 2023a) tailor denoising operations specific timesteps, though their com- putational demands hinder practicality. To reduce inference costs, pruning techniques (Fang et al., 2023; Kim et al., 2023; Ganjdanesh et al., 2024) trim redundant parameters from pre-trained denoising models, while distillation meth- ods (Hsiao et al., 2024) train lightweight guide models to minimize denoising steps. Inspired by RepVGG (Ding et al., 2021), RepControlNet (Deng et al., 2024) introduces novel reparameterization strategy: modal-specific adapters mod- ulate features during training, and their weights are later merged with the backbone, eliminating auxiliary compu- tations at inference. Unlike prior methods that rely on fixed heuristics, post-hoc pruning, or static architectural modifications, our FlexControl introduces dynamic, end- to-end trainable framework where block activation is both task-aware and compuutaiionawarce By integrating gat- ing mechanism with a computational efficiency object, our approach uniquely balances precision and resource usage, enabling adaptive control across diverse architectures with- out manual intervention - a paradigm shift from rigid, task-specific designs to flexible, generalizable control.
3. Methodology
3.1. Preliminaries
Denoising diffusion probabilistic model (DDPM) (Ho et al., 2020) aims to approximate the real data distribution q (xx) with the learned model distribution p (Xo) (Ho et al., 2020). It contains a forward diffusion process that progressively adds noise to the image and a reverse generation process that synthesizes the image by progressively eliminating noise. Formulaly, the forward process is T Tsepp Markov chain:
q (xir)/x0) = II (x
1=/
(1)
q (xx(xx= == No X -11
where { are the noise schedule, and {xt)__ are latent variables. Let a2 = 1- - Bt, the distribution of x for a given xe can be expressed as:
q (xx|xx) = N V1
(2)
Here, āj = Ii 0g is a differentiable function of timestep t, which is determined by the denoising sampler. Therefore,
3

---

FlexControl: Computation-Aware Conditional Control with Differentiable Router for Text-to- Generation
Rouler
Router
F Frazen
th-0.5
Gumbel No
Gr. 6_
Sigmoid
Block
Sigmoid
Block
y
W
Trainable
Block
Trainable Block
Batch h
1-M
Training Procedure
Inference Procedure
Forward
Element-wise
Poini-wise
Frozen
Trainable
Block of FlexControl
Propagation
Propagation
Multiplication
Addition
Multipiiatioo
SD Model
Figure 2. Overview of dynamic routing guided by the router unit. (a) In the training stage, Gumbel noise added to the discrete mask to make it continuous to assist the gradient backpropagation. (b) In the inference stage, the router unit controls whether to activate the control block and whether inject conditional control into the frozen block of the backhone according the input latent variahle. Once output the instruction of inactive, the corresponding control block and zero module will be skipped, and the current latent feature will automatically input to the next router unit.
the diffusion training loss can be formulated as:
3.2. Structure
Cg (xx,)
(Ae) liēa (xe,t) <122
Taking ControlNet as an example for explanation. Follow- ing its core design philosophy, we first freeze the powerful diffusion model backbone, fine-tune a trainable copy with zero modules learn spatial conditioning controls, and then inject the acquired latent control feature into the frozen backbone:
(3)
where e denotes a noise vector drawn from Gaussian dis- tribution, and éø refers to the predicted noise at timestep t by denoising model with parameters 0. (Ax) is pre- defined weighted function that takes into the signal-to-noise ratio A The reverse process first sample a Gaussian noise (xT) = = (xT;0 O, I). and then proceeding with the tran- sition probability density step by step:
y= F (x: θ) (F 2 9o); 0,2), (7)
where θ and θ are the weight parameters of the original model and the trainable copy respectively, Z represents zero modules and c the control element. Instead of just cloning the encoder and adding conditional controls only in the decoder blocks (Zhang al., 2023b), shown in Fig- ure I(a), copy all blocks of the original diffusion model to generate conditional controls and inject them into the corresponding blocks of the backbone in turn, as shown in Figure 1(b), which is similar to the strategy used BrushNet (Ju et al., 2024), and we call this structure ControlNet-Large Although more control signals may improve the genera- tion quality and controllability, it leads to huge redundant computation and multiplies the inference delay.
Po (x,) (x,- |x,,, Xo)
(4)
= (x Hig (X,, Xo) If
where P4 Xo)
X,
ي-اا vI-ai Eg (x,, and
Vae
of = I-a. 1- 1 Br are the mean and variance of posterior Gaussian distribution po (Xo).
In order to improve the efficiency of diffusion model, flow- based optimization strategy (Lipman al., 2023; Liu al., 2022; 2023) is introduced, which defines the forward pro- cess straight path between the real data distribution and the standard normal distribution:
To reduce computational redundancy and enhance image generation quality, we propose FlexControl, which intro- duces a lightweight router unit before each control block. The router generates binary mask M (0, 1) from the latent feature input to determining whether the underlin- ing control block needs be activated. Specifically, "0" indicates inactive, "I" indicates activate, and N represents the number of control blocks. Note that the router unit can be trained along with the control block.
x axX0 + b.e.
(5)
With Equation (5), vector field u, is constructed gen- erate path P. between the noise distribution and the data distribution. Meanwhile, the velocity is parameterized by the parameter 0 of a neural network to approximate ut- After variable recombination, the flow matching object can also be formulated as Equation (3) (Esser et al., 2024a). In the reverse stage, the ODE solver is used for fast sampling:
(t) = x (0) / vo (xxx) (7), dT.
The mask generation process of the router is data-driven, enabling independent path planning and adaptive decision-
(6)

---

FlexControl: Computation-Aware Conditional Control with Differentiable Router for Text-to-Image Generation
making based on the input latent representation. As shown in Figure 2, during inference, if the router outputs mask with value of "0", then the conditional mapping skips the next control block, and the injection of the control signal is stopped accordingly. Taking the 1-th control block as an example, the computation process can be formulated as:
We multiply the mask M and the output latent feature to zero out the corresponding control block and zero module. It can be seen from the above description that the mask M is learned from the latent variable h. Since timestep embedding is introduced into the blocks of the diffusion model during the generation of h, the output of the router is also affected by the sampled timesteps.
Fi (I O.) skip, (h/-1)
M = if = O,
h
(8)
Router for DiT-based architecture. For the router ap- plied in DiT, conduct feature analysis from multiple perspectives. Specifically, we perform both global and local feature encoding on the latent variable h E RNNC output by the Transformer block (Rao et al., 2021; 2023). The
where Fi (-) indicates the I-th control block with parameter @ h, is the output at timestep I, and skip, (.) is used to bypass the current block. Following the design of (Zhang et al., 2023b), we utilize the zero module to transform the latent feature by into conditional control:
detailed encoding process is as follows:
h ML.polobal (AVGdim=1 (h)):
(12)
ye =
Zz (be ()) N/A
if =1
(9)
h hocol MLPlocrl (AVG (h))
(13)
if M = 0.
From Equations (12) and (13), the encoding process for global and local features primarily consists of two steps. First, feature fusion is performed across all tokens and hid- den channels using the function AVG (-), which is imple- mented via average pooling along different dimensions of latent variable. This yields the global feature zolobal E IRC and local feature z ional E IR.. Second, the embedding di- mensions of goobal and z ziocul are aligned through an MLP layer and reduced to O, which is set to C/64 by default. Intuitively, the local feature captures token-specific informa- tion, while the global feature encodes potential relationships between tokens. We then merge these global and local fea- tures to form a new feature representation:
Here, y" denotes the conditional control which is incorpo- rated into the feature space of the diffusion backbone.
The above designed router is in fact lightweight, accounting for less than 1 1% of the parameters of the overall model. Since the skipped parameters are excluded from tensor computation during inference, FlexControl barely introduces computational burden by adaptively adjusting the number of active control blocks. See the detailed inference process in Algorithm 1.
Remark:
3.3. Router unit design
As illustrated earlier, the router unit is lightweight and plug- and-play to any diffusion architecture. However, given the differences between UNet and DiT, we will discuss the implementation of the router on these two commonly used architectures separately.
miz 01 h global + 02 h local
(14)
In the above equation, a1 and 0g are weight factors that balance the influence of global and local features, both set to 0.5 by default. The fused feature variable ha E RO is then passed through an MLP layer to produce K. At the end, IC is processed through a Sigmoid layer followed by the threshold discriminator described in Equation (11), resulting in the router mask M.
Router for UNet-based architecture. The output of UNet block is a multi-channel spatial feature. Given input h E RCCHXW the router unit first transforms the spatial feature into linear feature h' E RC through the downsam- pling layer, we use global average pooling (GAP) in imple- mentation, and then the MLP layer with weight W Rxx1 maps the linear feature into scalar Kc:
3.4. End-to-end training
Differentiable learning of router. To enable end-to-end training via gradient descent, we address the discrete, non- differentiable nature of the mask by incorporating Gumbel noise into the Sigmoid activation function. This allows the discrete mask . to be approximated by the differentiable Gumbel-Sigmoid version M during training:
K = MLP (b))
(10)
Henceforth, compute a new scalar K"' by restricting the value of I. to the interval 1) through the Sigmoid function. In order to convert K" into binary coding, we introduce a threshold discriminator to control the generation of the mask M by a preset threshold T (0.5 by default):
R2 (hi-1: + +GI-G2), (15)
Mi = Siigmiid
TP
where G₁, G2 ~ Gumbel(O, 1), TP denotes the temper- (11) ature hyperparameter (5 by default), R (-) denotes tensor computations in the router unit parametered by OR.
1 if >T T if ST.
M
5

---

FlexControl: Computation-Aware Conditional Control with Differentiable Router for Text-to-lmage Generation
To this end, we employ different mask schemes during the forward and backward passes:
2024)), segmentation mask (ADE20K, (Zhou et al., 2017)), and etc. Unless otherwise stated, ControlNet-LLarge is used as the base model by default.
Fi Mr + skip, (hg-1) : (1-Mi) if Forward Fi Mi + skip, (hi-1) (1 (1-Mi)) Mi)) if Backward. (16) Meanwhile, the computation process of the zero module is adjusted accordingly:
h
4.1. Quantitative comparison
Comparison of image quality. To evaluate the impact of dynamic controllable generation image quality, we com- pare the FID metrics of different methods across multiple conditional generation tasks (Table 1). We set to 0.5 to align FlexControl's FLOPs with ControlNet's, Our model achieves superior FID results across all conditions, outper- forming existing methods. We also examine ControlNet- Large, which replicates the entire SI) model tional control network. Although its larger parameter count enhances conditional feature extraction and control abil- ity, its performance remains inferior to FlexControl,=o.s- This confirms that adaptive control - selectively applying conditions instead of enforcing them across all blocks and timesteps - maximizes controllability. Beyond spatial con- ditions, we assess the influence of the text prompt using the _soPes metries. As shown in Table 1, Flex outperforms other methods, demonstrating that precise con- trol enhances spatially guided generation without compro- mising text- ext-ggided synthesis. Furthermore, we evaluate the performance of ControlNet and T2I-Adapter deployed on SDXL (Podell et al., 2023), revealing that a larger backbone does not necessarily improve the quality of the generated images.
2 (he; O²) M₁ if Forward
yc =
(17)
Z1 (br; 0!) M₁ if Backward.
Remark: As can be seen in Equations (16) and (17) during training, the blockwise routing differs from the inference process displayed in Equations (8) and (9): during train- ing, we retain all blocks to ensure proper back-pre pooek-ropagation, rather than skipping blocks done during inference.
addi-
Computation-aware training loss. Following standard controllable generation methods, our training dataset D contains triples of the original image æ, spatial conditioning image cs. and textual prompt The diffusion loss of FlexControl is formulated as:
LiIF = (xx....e..........)) [1ão (xx; Ct, Cs, - 2
(18) FlexControl aims to activate the optimal control blocks and inject conditional mapping into the backbone network for ef- ficient image generation. In addition to the regular diffusion loss DDIT. we introduce a cost loss Lc to regulate resource consumption to the desired sparsity 7, which measures the proportion of Hoating-point operations (FLOPs):
Comparison of controllability. We exam generation con- trollability in detail by comparing the results across dif- ferent spatial conditions. ControlNet and its variants gen- erally achieve stronger controllability than other existing methods. Within a similar computational budget, our Flex- Control further improves controllability across various con- trol conditions. Numerically, FlexControl reduces RMSE by 6.30% and 4.74% compared to ControlNet and Con- trolNet+ on the depth map task. For the canny edge and the segmentation mask task, FlexControl shows im- provements of 4.15% in SSIM and 9.87% in mIoU, respectively. Moreover, our method outperforms ControlNet-Large on both the depth map and segmenta- tion mask datasets, and achieves similar performance on the canny edge task. Similarly, we show the results of the SDXL-bbased ControlNet and T21-Adapter, which show only marginal improvements for specific tasks.
2
1
F. (d)
Lc =
)
(19)
|Dbsll
F. J.Basc (d)
dED
where Drs means the current batch samples, ta E [0, T] is the uniformly sampled timestep for sample d. F₂ (d) denotes FLOPs of the trainable branch at sampled timestep, and superscripts Flex and Base respectively denote FlexControl and baseline model (e.g., LonrrolNet-Large). We combine L and Le to bring out the final optimization goal,
La = LDif + Xc Lc,
(20)
where de is the hyperparameter that controls the force in- tensity of loss Lc. See the detailed training process in Algorithm 2.
4.2. Qualitative comparison
4. Experiment
In Figure 3, we compare different methods across depth map, canny edge, and segmentation mask tasks. Except for Con- trolNet (SDXL), the others use SD1.5 as the backbone. Flex- Control consistently outperforms others in visual quality and spatial / text alignment. For depth map, FontColtrol pro-
We evaluate FlexControl against state-of-the-art methods across different conditions: depth map (MultiGen-20n 20M, (Zhao et al., 2024a)), canny edge (L.LAVA-558K, (Liu et al.,
6

---

FlexControl: Computation-Aware Conditional Control with Differentiable Router for Text-to-Imare Generation
Image & Condition
FlexControl.yeng
LorrzolNet-Large
Co
ControlNet
121-Adapler
ControlNet (SDXL)
Depth map
SRETAN BOTTC
SRETAN BOFFC
SIETAN POZZ4!
SRETAN BOTTA!
SRETAN RAZTA!
SHELAAN OUCASE
REEAN
JOULA
Canny edge
SHLTER
Segementation mask
Figure Qualitative comparison controllable gemeration methods. FlexControl achieves higher fidelity and structure preservation across Depth Map. Canny Edge, and Segmentation Mask conditions, reducing distortions (boxes) seen in other methods. It better aligns with input conditions while maintaining visual quality. Captions: A luxarious villa with a swimming pool and furniture at dusk. Christmas greeting with wreath and red bauble, scandinavion christmas text. Two beds with blue bedding and wooden frames in bedroom.
Method
T2I Model
Depth Map
Canny Edge
Seg. Mask
#Average
FID
CLIP score
FID 4
CL.P score
FID 4
CL.P score
FID 4
CL.IP.ccore score
ControlNet (Zhang al., 2023b)
SDXL
19.90
0.3224
22.07
0.2657
26.95
0.2495
22.97
0.2792
T2I-Adapler (Mou el al., 2024)
SDXL
19.74
0.3197
0.2614
27.54
0.2501
23.40
0.2771
GLIGEN (Li et al., 2023)
SD14
18.36
0.3175
19.01
0.2520
23.79
0.2490
20,39
0.2728
T2I-Adapter (Mou et al., 2024)
SD1.5
22.52
0.3146
16.74
0.2598
24.65
0.2494
21.30
0.2728
ControlNet (Zhang al., 2023b)
SDL.5
17.76
0.3245
15.23
0.26113
21.33
0.2531
18.11
0.2796
ControlNet++ (Li el al., 2025)
SD1.5
16.66
0.3209
17.23
0.2598
19.89
0.2640
17.93
0.2816
ControlNet-Large
SD1.5
12.45
0.3492
12.92
0.2789
16.78
0.2796
14,05
0.3026
FlexControl,
SD1.5
11.65
0.3498
11.37
0.2778
14.80
0.2842
12.61
0.3039
Table Quantitative comparison of FlexControl with state-of-the-art methods. We report FID (L) and CLIP score (t) on different conditioning types: Depth Map. Canny Edge, and Sogmentation Mask. Lower FID indicates better image quality, while higher CLIP score reflects better alignment with textual prompts. The results are highlighted red, while the second-bed best results are shown blue. FlntColtrol achieves the best overall performance, demonstrating superior fidelity and semantic alignment.

---

FlexControl: Computation-Aware Conditional Control with Differentiable Router for Text-to- Image Generation
Method
T21 Model
Depth Map (RMSE )
Canny Edge
Seg, Mask (mloU 1)
Method
FID
CLIP score
mloo
Param.
FLOIS
Speed
(SSIM f)
ControlNet
21.33
0.2531
0.2761
0
233
5.23 its
ConteolNete-
19.89
0,2610
0.3435
0.36
233
5.23 its
ControlNet
SDXL
0.4001
0.4178
0.2058
ControlNee-Lage
16.78
0,2756
0.3688
0.72
4,02gam
T21-Adupler
SDXL
0.3976
0.3969
0.1912
FlexContral,
21.52
0.2995
0.73
112
5,38 105
GLIGEN
SD14
0,3882
0,4226
0.2076
FlesCintral,.. 14.80
FlexControl, -
17.21
0.2842
0,2713
0.3751
0.3572
0.73
0.73
230
168
5.21 los
5.642011 IbS
T2I-Adapter
SD1.
0,4840
0,4622
0.1839
FlesContral...
0.3775
0.35
393 G
4,94ua0n
ControlNet
SD1.5
0.2988
0.5197
0.2764
FlexContral.-. 15.59
0.2804
0.3695
0.73
448
4.82 Ibs
ControlNet++
SD1.5
0.2832
0,5436
0.3435
ControlNet-Laare
SI21.5
0,2372
0,5642
0,3668
Table 3. Quantitative comparison with existing methods on SD1.5. We compare image quality, controllability and efficiency. The speed is measured on single Nvidia RTX 2080 Ti GPU. The best and second-bex values are highlighted in red and blue. Flex- Control outperform original ControlNet with less computation, while increasing the blocks budgets observed performance increas- ing. Noticeable, ControlNet-I. activate all blocks yet not out- perform our methods, highlight effective of our dynamic strategy.
FlexControl,
SD1.5
0.2358
0,5612
0.3751
Table 2. Controllability comparison across different condition- ing types. We report RMSE (L) for Depth Map and SSIM (t) for Canny Edge and mloU (1) for Seg. Mask. The best and second- best results are highlighted in red and blue. FlexControl achieve similar but slightly better controllability than ControlNet- Largee with only half activation blocks.
Method
FID
CLIP-score
SSIM f
Puram.
FLOPS. L
Speed 1
ConmeiNer
27:21
0:2812
0.3249
1
328' 1
(20,88is syje-1 itss
duces smoother transitions and more natural textures. Under canny edge, it better preserves edge fidelity and fine details. For segmentation mask, it enhances mask reconstruction and visual consistency. These results demonstrate FlexControl's ability to selectively inject control information into rele- vant diffusion backbone blocks based on timestep and input characteristics, improving image fidelity. Finally, we com- pare against ControlNet and ControlNet-Large Although ControlNet- benefits from a larger control network for improved generation and condition alignment, FlexControl surpasses it in both accuracy and visual fidelity, showcasing the strength of our approach.
ContelNett Large
21,64
0.2690
0.4828
2.02
6.22 T
(16.82ia50e 3 12s
FleaCoatrel,
28.11
0.2524
0.3577
2,03
1.25 7
(26. aas0le 3 125
PlexCantrol....
24.39
0.2581
0.4266
2.03.
1.36 7
(34 oo-- lids
PeeeContrel....
22.47
0.2714
0.4598
2.03.
7
(21.661801 3 ids
FleaCentrol -
0.2714
0.4755
2.03.
4.35
(19
PlesCantral a.s
(18 118-90a8000-11 ins
Table 4. Quantitative comparison with existing methods on SD3.0. We compare image quality, controllability and efficiency. The speed is measured on single Nvidia RTX 2080 Ti GPU. The best and second best values are highlighted in red and blue. Flex- Control outperform original ControlNet with less computation, while increasing the blocks budgets observed performance increas- ing even more significant improvement than observed in SD1.5.
achieves a generation quality comparable to ControlNet++ and ControlNett-Large
4.3. Ablation study
In this section, we analyze how the proportion of activated control blocks impacts FlexControl. To better understand model complexity, we present the number of parameters, FLOPs, and diffusion speed- - the diffusion iterations per second (i.e., it/s). To measure the diffusion speed, we ran- domly select batch samples and compute the average single- step iteration time for each sample.
Results on DiT-based model. For the SD3.0-based back- bone, experiments are conducted on the LLAVA-558K dataset. As detailed in Table 4, FlexControl, and F0 outperform ControlNet with fewer FLOPs. Notably, while ControlNet has half as many blocks as the backbone, each control block's output is shared by two adja- cent backbone blocks, providing more conditional controls than FlexControl at all sparsity levels. FlexControl, achieves superior image quality and comparable control- lability to ControlNet-Largg while being more efficient. in Figure 5 further demonstrate the advantage of CoexControl in edge reproduction and image fidelity over ControlNet.
Results on UNed-based model. Recall the cost loss de- fined in Equation (19), we train FlexControl with different sparsity levels by adjusting the value of y. For the SD1.5- based backbone, experiments are conducted on the ADE20K dataset. At y = 0.3 ( (00% sparsity). FlexControl surpasses ControlNet and ControlNet++ in controllability and genera- tion quality but falls short of ControlNet-Large. Increasing y to 0.5 activates more control blocks, leading to perfor- mance that surpasses Lorrro Further increasing y to 0.7 does not yield significant performance gains (sug- gesting the dataset has already been saturated by the model capacity). For visual comparisons in Figure 4, FlexControl with 7 = 0.5 and 7 = 0.7 demonstrate superior structure preservation and mask information reconstruction. Mean- while, the more lightweight configuration with ? = 0.3
Visualization results
Expand lightweight architecture. ControlNeXt (Peng et al., 2024) is an improved architecture proposed en- hance the efficacy and efficiency of controllable generation based on ControlNet, which extracts control signals from conditional images through a lightweight neural network. We uniformly inject conditional controls into the blocks of diffusion backbone, and align spatial and channel dimen- sional by adaptive average pooling and xx 1 convolution.
8

---

FlexControl:
Computation-Awar Conditional Control with Differentiable Router for Text-to-Iman Generation
InagskContiion PloeCornool-an Flo
PloxCantndiooss ContriNNs-Lagee
ContnoiNatee
FirsControkuro CarssNaaslann
ComnoNa
Figure 4. Comparison of FlexControl and existing methods on SD1.5 for semantic consistency. FlexControl achieves better se- mantic alignment and structure preservation with varying sparsity levels, while ControlNet-bassee based methods show inconsistencies in segmentation accuracy (highlighted in yellow boxes). Captions: A stone building surrounded by a stone wall and a grassy lawn.
Figure 5. Comparison of FlexControl and existing methods on SD3.0 for edge preservation. FlexControl maintains better spatial consistency and object integrity across different sparsity levels, while Control based methods introduce distortions and incon- sistencies (highlighted in red boxes). Captions: A room with large windows, a gray sofa, table, and a TV stand.
Method
FID ||
CLir
mloU 1
FLOPS
Speed
Method
FID
CLIP "o.PPso
|SIM
FLOPSS
Speed
ContolNeXi (Pong ol., 2024)
24.16
0.2659
0.2825
S1.72G
5. 5.3400 ii
OniniCunzal (Tha 2024)
0.2630
0.4125
16,89 T
2.36.s Ib's
MexCoatroNNeKt....
25.22
0,2531
0,2644
FlesOmini Cuntmnninnted..-uu
36.62
0:2712
0.3123
10.76 7
3,42a iik
PesControNNeKt....
23.74
0,2664
FlesOminii Contrel.
26.65
0,2791
0.3668
11.45 T
3.28 lts
PlrCCantraNNNt.... 1.7
23.71
0.2674
0.2841
FlexOminiCCitre
22.61
0.2886
0.4123
13,08 7
3,08 it
PleCoantroNNei....
23.84
0,2632
0.2841
FlexOmini
0.4146
14.57
2.80
FlexOmiii Cintrool.........
22.27
(1286)
0.4153
15.40 7
2.6994as lol
Table 5. Extension on ControlNeXt architecture. All models build on SD1.5. We compare image quality, controllability and efficiency. The diffusion speed measured on single Nvidia RTX 2080 Ti GPU. The best and second- best values are highlighted in red and blue. After the optimization of the dynamic strategy. the performance of ControlNe has been further improved.
Table 6. Extension on OminiControl. All models built on FLUX.1 I and share similar number of parameters. We compare image quality, controllability and efficiency. The diffusion speed is measured on single Nvidia A100 (40G) GPU, The best and second-best values are highlighted in red and blue. OminiControl can also be optimized by our dynamic strategy.
As shown in Table 5, combined with the router unit, Con- trolNeXt has been further improved both in image quality and controllability. The results indicate that, not limited to the ControlNet architecture, our dynamic condition injec- tion strategy is a universal solution to improve controllable generative models. Since ControlNe reuses condition fea- tures, we just adapt the condition injection position rather than skipping the control blocks, so we are basically consis- tent with the baseline in the number of parameters, FLOPs, and diffusion latency. The visual comparison is presented in Figure 11.
ing process, we promptly add the attention mask to the attention map to shield the influence of spatial condition fea- tures when they are not needed. The quantitative results are shown in Table 6, our method can also effectively improve the efficacy and efficiency of OminiControl. The visual comparison can be found in Figure 13.
5. Conclusion
We presented FlexControl, a dynamic framework that reimagines controllable image generation by replacing heuristic block selection with a trainable, computation- aware, and alug-and-play gating mechanism. By adaptively injecting control signals into the latent space of each diffu- sion block during the denoising process, FlexControl clim- inates manual architectural tuning, reduces computational overhead, and maintains or improves image generation fi- delity across diverse tasks and architectures (e.g., UNet, DiT). Our experimental results demonstrate that flexibility and efficiency need not be mutually exclusive in controllable image generation - intelligent, data-driven block activation strategies can outperform rigid, hand-crafted designs. This work paves the way for future research into lightweight and generalizable control mechanisms for increasingly complex controlled generative pipelines. We also conduct a further investigation on the dynamic activation routes of the control blocks, which has been listed in Appendix C.
Expand to LoRA-based architecture. OminiControl (Tan et al., 2024) is representative LoRA-based DiT archi- tecture for efficient controllable image generation. Unlike common used dual-stean structure, OminiControl concate- nates noise latent variable and condition latent variable, extracts spatial condition features with the assistance of the LoRA component, and finally injects spatial condition into the latent feature space through the cross-attention mech- anism. Our adaptive conditional control framework still works seamlessly with OminiControl to inject control sig- nals into latent space as needed. We add the router unit to all Double Stream Blocks (DSB) and Single Stream Blocks (SSB)or of the FLUX.1 model (Labs, 2024), and use the noise variable as input of the router unit to generate the spatial conditional activation mask, Meanwhile, during the train-

---

FlexControl: Computation-Aware Conditional Control with Differentiable Router for Text-to-Image Generation
Impact Statement
Esser, P., Kulal, S., Blattmann, A., Entezari, R., Müller, J., Saini, H., Levi, Y., Lorenz, D., Sauer, A., Boesel, F. et al. Scaling rectified flow transformers for high- resolution image synthesis, march 2024. URL http:///aaxii org/abs/2403..mm 2abpbb/2243330206. 2024a.
This paper presents work whose goal to advance the field of controllable image generation. There are many potential socictal consequences our work, none of which we feel must be specifically highlighted here.
Esser, P., Kulal, S., Blattmann, A., Entezari, R., Müller, J., Saini, H., Levi, Y., Lorenz, D., Sauer, A., Boesel, F., et al. Scaling rectified flow transformers for high-resolutin image synthesis. In Forty-firsi International Conference on Machine Learning, 2024b.
References
Bao, F., Nie, S., Xue, K., Cao, Li,
C., Su, H., and Zhu, J. All are worth words: A vit backbone for diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 22669-22679, 2023a.
Fan, S., Jiang, X., Li, X., Meng, X., Han, P., Shang, S., Sun, A., Wang, Y., and Wang, Z. Not all layers of Ilms are necessary during inference. arXiv preprint arXiv: 444.00281,, 2024a.
Bao, F., Nie, S., Xue, K., Li, C., Pu, S., Wang, Y., Yue, G., Cao. Y.. Su, H., and Zhu, J. One transformer fits all distributions in multi-modal diffusion at scale. In International Conference Machine Learning, pp. 1692- 1717. PMLR, 2023b.
Fan, Y., Liu, C., Yin, N., Gao, C., and Qian, X. Adadiffsr: Adaptive region-awarr dynamic acceleration diffusion model for real-world image super-resolution. In European Conference on Computer Vision, pp. 396-413. Springer, 2024b.
Bolukbasi, T., Wang, J., Dekel, O., and Saligrama, V. Adap- tive neural networks for efficient inference. In Interna- tional Conference on Machine Learning, pp. 527-536. PMLR, 2017.
Fang, G., Ma, X., and Wang, X. Structural pruning for diffu- sion models. Advances in Neural Information Processing Systems, 2023.
Canny, J. A computational approach to edge detection. IEEE Transactions on puttern analysis and machine in- telligence, pp. 679-698, 1986.
Ganjdanesh, A., Shirkavand, R., Gao, S., and Huang, H. Not all prompts are made equal: Prompt-based prun- ing of text-to-image diffusion models. arXiv preprint arXiv:2406. 2tfi:2400..1042,, 2024.
Chen, M., Laina, I., and Vedaldi, A. Training-free layout control with ross-attention guidance. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision, pp.
2024a.
Han, Y., Pu, Y., Lai, Z.,, Wang, C., Song, S., Cao, J., Huang, W., Deng, C., and Huang. G. Learning weight sam- ples for dynamic early-exiting networks. In European conference on computer vision, pp. 362-378. Springer, 2022.
Chen, Z., Wu, J., Wang, W., Su, W., Chen, G., Xing, S., Zhong, M., Zhang, Q. Zhu, X., Lu, L.,, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24185-24198, 2024b.
Han, Y., Liu, Z., Yuan, Z., Pu, Y., Wang, C., Song, S., and Huang, G. Latency-aware unified dynamic networks for efficient image recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
Deng, Z., Zhou, K., Wang. F. and Mi, Z. Repcon- trolnet: Controlnet reparameterization. arXiv preprint arXiv:2408.092 09240, 2024.
Herrmann, C., Bowen, R. S., and Zabih, R. Channel selec- tion using gumbel softmax. In European conference computer vision. pp. 241-257. Springer, 2020.
Dhariwal, P. and Nichol, A. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8794 2021.
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Ding, X., Zhang, X., Ma, N., Han, J., Ding, G., and Sun, J. Repvgg: Making vgg-style convnets great again. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 13733-13742, 2021.
Ho, J., Jain, A., and Abbeel, Denoising diffusion proba- bilistic models. Advances in neural information process- ing systems, 33:6840-6851, 2020.
Elhoushi, M., Shrivastava, A., Liskovich, D., Hosmer, B., Wasti, B., Lai, L., Mahmoud, A., Acun, B., Agarwal, S., Roman, A., et al. Layerskip: Enabling early exit inference and speculaccu decoding. arXiv preprint arXiv:2404. 167 10, 2024.
Hsiao, Y.-T., Khodadadeh, S., Duarte, K., Lin, W.-A., Qu, H., Kwon, M., and Kalarot, Plug-and-play diffusion
10

---

FlexControl: Computation-Aware Conditional Control with Differentiable Router for Text-to-lmage Generation
distillation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 13743- 13752, 2024.
Li, M., Yang. T., Kuang, H., Wu, J., Wang, Z., Xiao, X., and Chen, C. Controlnet+++:: Improving conditional controls with efficient consistency feedback. In European Conference on Computer Vision, pp. 129-147. Springer, 2025.
Hu, M., Zheng, J., Liu, D., Zheng, C., Wang. C., Tao, D., and Cham, T.-J. Cocktail: Mixing multi multi-modality control for text-conditional image generation. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Li, Y., Liu, H., Wu, Q. Mu, E., Yang, J., Gao, J., Li, C., and Lee, J. Gligen: Open-set grounded text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22511 22521, 2023.
Huang. L., Chen, D., Liu, Y., Shen, Y., Zhao, D., and Zhou, J. Composer: Creative and controllable image synthesis with composable conditions. arXiv preprint arXiv:2302.09977 2023a.
Lipman, Y., Chen, R. T., Ben-Hamu, H., Nickel, M., and Le, M. Flow matching for generative modeling. In Interna- tional Conference on Learning Representations, 2023.
Huang. Z.., Wu, T., Jiang, Y., Chan, K. C.,, and Liu, Z. Re- version: Diffusion-based relation inversion from images. arXiv preprint arXiv:2303.13499 2023b.
Liu, H., Li, C., Wu, Q., and Lee, J. Visual instruction tun- ing. Advances in neural information processing systems, 36, 2024.
Jiang, R., Wang, C., Zhang, J., Chai, M., He, M., Chen, D., and Liao, Avatarcraft: Transforming text into neural human avatars with parameterized shape and pose control. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 14371-14382, 2023.
Liu, X., Gong, C., and Liu, Q. Flow straight and fast: Learning generate and transfer data with rectified flow. arXiv preprint 2r00002209.03003, 2022.
Liu, X., Zhang, X., Ma, J., Peng, J., et al. Instaflow: One step enough for high-quality diffusion- based text-to- image generation. In The Twelfth International Confer- ence Learning Representations, 2023.
Ju, X., Zeng, A., Zhao, C., Wang, J., Zhang, L., and Xu, Q. Humansd: native skeleton-guided diffusion model for human image generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15988-155 15998, 2023.
Mou, C.,, Wang, X., Xie, L.., Wu, Y., Zhang, J., Qi, Z.,, and Shan, Y. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. In Proceedings of the AAAI Conference Artificial Intelli- gence, volume 38, pp. 4296 4304,, 2024.
Ju, X., Liu, X.. Wang, X., Bian, Y., Shan, Y., and Xu, Q Brushnet: A and--an- image inpainting model with decomposed branch diffusion. arXiv preprint NKKv:2403306976 2024.
Nichol, A., Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., McGrew. B., Sutskever, I., and Chen, M. Glide: To- wards photorealistic image generation and editing with text-guided diffusion models. In International conference machine learning, 2022.
Kim, B.-K., Song, K...., Castells, T., and Choi, On archi- tectural compression of text-to-image diffusion models. arXiv preprint 2rXiv:2305.15500 /5798, 2023.
Kingma, D., Salimans. T., Poole, B., and Ho, J. Varia- tional diffusion models. Advances in neural information processing systems, 14:21696-21777,00 2021.
Peebles, W. and Xie, Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 4195-4205, 2023.
Labs, B. F. Flux: Official inference repository for flux.1 models, 2024. URL https: //gpt/uu///hhbbb/// com/
black-foreste wowestorretttttee lattt flux. Accessed: 2024-11- 12.
Peng, B., Wang, J., Zhang, Y., Li, W., Yang. M.-C., and Jia, J. Controlnext: Powerful and efficient control for image and video generation. arXiv preprint wxXiv:2408.0007 2024.
Lee, Y., Kim, J., Go, H., Jeong, M., Oh, S., and Choi, S. Multi-an architecture multi-expert diffusion models. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 38, pp. 13427-13436, 2024.
Podell, D., English Z., Lacey, K., Blattmann, A., Dockhorn, T., Müller, J., Penna, J., and Rombach, R. Sdxl: Im- proving latent diffusion models for high-resolution image synthesis. arXiv preprint
Li, C., Wang, G., Wang, B., Liang, X., Li, Z., and Chang, X. Dynamic slimmable network. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pp. 8607-8617, 2021.
20..23077.992.... 2023.
Pu, Y., Xia, Z., Guo, J., Han, D., Li, Q. Li, D., Yuan, Y., Li, J., Han, Y., Song, S., et al. Efficient diffusion transformer

---

FlexControl: Computation-Aware Conditional Control with Differentiable Router for Imxte Generation
with step dynamic attention mediators. In European Conference on Computer Vision, pp. 424 441. Springer, 2024.
Saharia, C., Chan, W., Saxena, S., Li, L.., Whang, Denton, E. I.., Ghasemipour, K., Gontijo Lopes, R., Karagol Ayan, B., Salimans, T., et al. Photorealistic text-to-image dif- fusion models with deep language understanding. Ad- vances in neural information processing systems, 35: 36479-36494, 2022.
Qin, C., Zhang, S., Yu, N., Feng, Y. Yang, X., Zhou, Y., Wang, H., Niebles, C., Xiong, C., Savarese, S., et al. Unicontrol: A unified diffusion model for con- trollable visual generation in the wild. arXiv preprint arXiv:2305. 11 147, 2023.
Schuhmann, C.,, Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T.,, Katta, A., Mullis, C., Wortsman, M., et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35: 25278 25294, 2022.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp. 8748-8788 8763. PMLR, 2021.
Sohl-Dickstein, J., Weiss, E., Maheswaranathhn, N., and Ganguli, S. Deep unsupervised learning using nonequi- librium thermodynamies. In International conference on machine learning, pp. 2256 2265. PMLR, 2015.
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21 (140):1-67, 2020.
Song, J., Meng, C., and Ermon, S. Denoising diffusion implicit models. In International Conference on Learning Representations, 2021.
Rajbhandari, S., Rasley, J., Ruwase, O., and He, Y. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Per- formance Computing, Networking, Storage and Analysis, pp. 1-16. IEEE, 2020.
Stability, Stable diffusion v1.5 model card. v1-5, 2022.
Tan, Z., Liu, S., Yang, X., Xue, Q., and Wang. X. Ominicon- trol: Minimal and universal control for diffusion trans- former. arXiv preprini arXiv:241 1.1509,, 2024.
Rao, Y. Zhao, W., Liu, B., Lu, J., Zhou, J., and Hsieh, C.-J. Dynamicvit: Efficient vision transformers with dynamic token sparsification. Advances in neural information processing systems, 34: 11111337--19444 2021.
Tu, Z., Talebi, H., Zhang, H., Yang, F., Milanfar, P., Bovik, A., and Li, Y. Maxvit: Multi-axis vision transformer. In European conference on computer vision, pp. 459-479. Springer, 2022.
Rao, Y., Liu, Z., Zhao, W., Zhou, J., and Lu, J. Dynamic spatial sparsification for efficient vision transformers and convolutional neural networks. IEEE Transactions on Puttern Analysis and Machine Intelligence, 45(9))10883- 10897, 2023.
Wang, X., Darrell, T., Rambhatla, S. S., Girdhar, R., and Misra, I. Instancediffusion:: Instance-level control for image generation. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 6232-6242 2024.
Ren, M. Delbracio, M., Talebi, H., Gerig, G., and Milanfar, P. Image deblurring with domain generalizable diffusion models. arXiv preprint arXiv:2212.01178,, 1, 2022.
Yang, L.,, Han, Y., Chen, X., Song. S., Dai, J., and Huang, G. Resolution adaptive networks for efficient inference. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2369-2262220 2378, 2020.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF con- ference on computer vision and pallern recognition, pp. 10684-10695, 2022.
Yang, X., Zeng, D., Wang, X., Wu, Y., Ye, H., 7hao, Q., and Li, S. Adaptively bypassing vision transformer blocks for efficient visual tracking. Pattern Recognition, 161: 11 1 1278, 2025.
Ronneberger, O., Fischer, P., and Brox, T. U-net: Con- volutional networks for biomedical image segmenta- tion. In Medical image computing and computer-assisted intervention-MICCAN 2015: 18th international confer- ence, Munich, Germany, October 5-9, 2015, proceedings, part 18, pp. 231. Springer, 2015.
Yang, Z.,, Wang, J., Gan, Z., Li, L., Lin, K., Wu, C.,, Duan, N., Liu, Z., Liu, C., Zeng, M., et al. Reco: Region- controlled text-to-image generation. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14246 14255, 2023.
12

---

FlexControl: Computation-Aware Conditional Control with Differentiable Router for Text-to-Image Generation
Zeng, D., Du, N., Wang. T., Xu, Y., Lei, T., Chen, Z., and
Cui, C. Learning skip for language modeling. arXiv
preprint arXiv:2311.154.. 2023.
Zhang, H., Lu, Y., Alkhouri, I., Ravishankar, S., Song, D., and Qu, Q. Improving efficiency of diffusion models via multi-stage framework and tailored multi-decoder architectures. arXiv preprint arXiv: 2tti::2312..188.. 2023a.
Zhang, I.., Rao, A., and Agrawala, M. Adding conditional control text-lo-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3836-3847, 2023b.
Zhang. T., Zhang, Y., Vineet, V., Joshi, N., and Wang, X. Controllable text-to-image generation with gpt-4. arXiv preprint arXiv:2305. 2023c.
Zhao, S., Chen, D., Chen, Y-C.., Bao, J., Hao, S., Yuan, L.., and Wong, K.-Y. K. Uni-controlnet: All-in-one control to text-to-image diffusion models. Advances in Neural Information Processing Systems, 36, 2024a.
Zhao, W., Han, Y., Tang, J., Wang, K., Song, Y., Huang, G., Wang, F., and You, Dynamic diffusion transformer. arXiv preprint arXiv:210.03355. 2024b.
Zhou, B., Zhao, H., Puig, X., Fidler, S., Barriuso, A., and Torralba, A. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 633 641, 2017.
Zhou, D., Li, Y., Ma, F., Zhang, X., and Yang, Y. Mige: Multi-instance generation controller for text-to-image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6818- 6828, 2024.
13

---

FlexControl: Compatation-Aware Conditional Control with Differentiable Router for Text-to- Generation
Supplementary Material
Algorithm 2 Training procedure
Input: dataset (a, C. Ct). hyperparameters (T, Ac). initialized FlexControl, frozen diffusion backbone.
The supplementary material presents the following sections
to strengthen the main manuscript:
Output: fully-trained FlexControl
1: /" Keep all the control blocks active "/
A. Pseudo-code of Our Algorithm.
2: Turn off the router units for warm-up up training
/" Set to learn conditional signal filtering *
B. Implementation Details.
Turn on the router units for end-to-end training while not converged do
C. Dynamic Routes Exploration.
Sample timestep Uniform (0,1)
Sample nosie ~ (0. I)
D. More Ablation Studies.
8: I" Based on Equation (2) or Equation (5) */
E. More Visualization Results.
9: Transfer image to noisy image X
10: I' Based on Equations (7), (16) and (17) */
11: Ypred: M = Êg (X,, Ct2 C2.t)
A. Pseudo-code of Our Algorithm
12:
LSD used optimize generation effect */
In this section, we give the pseudo-code algorithm of our FlexControl. The specific inference procedure is shown in Algorithm and the training procedure is shown in Algorithm 2.
13: Compute MSE loss Diift though Equation (18)
14:
1" LC is used to control sparsity */
15:
Compute cost loss Lc though Equation (19)
16:
1' Lo is used final optimization goal */
17: Compute final loss Le though Equation (20)
Algorithm 1 Inference procedure
18: Freeze the weight parameters of the backbone */
Input: noise x, visual condition image C,. textual prompt Ct. denoising steps T, fully-trained FlexControl model, pre-trained diffusion backbone.
19:
Ir Ve (Na, Y ppred:
20:
end while
Output: the denoised image X
out under various conditions, mainly including depth map, canny edge and segmentation mask. The following is a description of the experimental details.
1: for each t [T.1] do
2: 1" Denoising network predicts noise at timestep L*|
3: for each 1 E Blocks do
4: 1* "The value of Mr is adjusted with input hi-1" *
Training dateset. conditional maps:
The experiment involves three types of
5: Compute M₁ though router unit
6:
1 1
Extract latent features from conditions *
Depth map. In this application, use MultiGen-20M proposed by (Zhao al., 2024a) training data, which is subset of LAION-Aestheti (Schuhmann et al., 2022) and contains over 2 million depth-image-caplee pairs, and 5K test samples.
8:
Compute hy though Equation (8)
9:
1' Feature transformation by zero modules *
10:
Transform h, to y' though Equation (9)
11:
1' Inject modal information into latent space "/
12:
Inject y' to backbone though Equation (7)
13:
else
Canny edge. For the condition of the canny edge, use the LLAVA-558K (Liu et al., 2024) dataset to verify the model, which contains 558K image-caption pairs. A canny edge detector (Canny, 1986) is used to convert RGB images to edge images, and the low and high threshold of hysteresis procedure in this process are set 100 and 200, respectively.
Align the dimension of feature mapping" /
Bypass Fi and Z, though skip, (-)
16:
end if
17:
end for
18:
1" Guided by both visual and textual condition /
19:
Produce predicted result ypred at timestep
20:
1" Use appropriate scheduler to restore image *1
X;-1 scheduler.step Xt, Npred: f)
Segmentation mask. For the segmentation mask, we use the ADE20K (Zhou al., 2017) dataset for model training. This dataset contains a total of 27K segmen- tation image pairs, 25K for training and 2K for testing. VL2-2B (Chen et al., 2024b) is used to gener- ate captions for RGB images with instruction "Please use a brief sentence with as few words as possible to summarize the picture".
end for
B. Implementation Details
We implement FlexControl based on SD1.5 (Stability, 2022) and SD3.0 (Esser et al., 2024a). The experiments are carried
14

---

FlexControl: Computation-Aware Conditional Control with Differentiable Router for Text-to-Image Generation
Training settings. During the training procedure, we uni- formly use the AdamW optimizer with a learning rate of 1x10-5 For SD1.5- models, half-precision floating- point (Float16) is used for mixed precision training, original images and conditional images are resized to 512x512, and batch size and gradient accumulation steps are set to 4 and 32, respectively. When turning to SD3.0, we further use DeepSpeed (Rajbhandari et al., 2020) Zero-2 to acceler- ate the training process, the resolution of x024x 1024 is used, and the batch size and gradient accumulation steps are set to 4 and 8. We set the maximum training iterations to 50k and 25k for the models based on SD1.5 and SD3.0, 0., respectively. For the threshold parameter T required by the Gumbel-Sigmooe activation function in the router unit, we set it to 0.5, and the hyperparameter λe in the loss func- tion Co is set to 0.5, the value of 7 depends on the target sparsity. When training UNet- -Nsed ControlNet-laree and FlexControl, we remove the residual connection between the encoder blocks and the decoder blocks of the control network. For the problem of the weight dimension cannot be aligned when initializing the decoder blocks of the con- trol network using SD1.5's pre-trained weights caused by this operation, we solve it by reinitializing these weights. The models based on SD1.5 and SD3.0 are trained with 2 and 8 Nvidia-A 100 (40G) GPUs, respectively.
Benchmark and metries. For quantitative comparison, we present the Frechet Inception Distance (FID) (Heusel et al., 2017) and CLIP score (Radford et al., 2021) to as- sess the quality of the generated images. In addition, we calculate RMSE, SSIM and mloU on depth map, canny edge, and segmentation mask, respectively, to evaluate the controllability of image generation. Finally, we emphati- cally compare computational complexity. The results of depth map are tested on MultiGen-20M test set and the re- sults of canny edge and segmentation mask are tested on COCO validation set, which contains 5,000 samples and each sample contains five text descriptions, we randomly choose one text of each sample as input during testing. For sampling, we employ the DDIM (Song et al., 2021) and RFlow (Esser et al., 2024a) sampler, implementing 20 de- noising steps to generate images without incorporating any negative prompts. We generate five groups of images, and the average results are reported.
C. Dynamic Routes Exploration
In order to improve the parameter utilization of ControlNet in the application, we explore how the router unit activates the control block to generate conditional controls.
It can be seen from Figure I(c), both SD1.5 based on UNet and SD3.0 based on DiT, the activation of control blocks presents a sparse distribution in the early denoising stage and a dense distribution in the late stage. This means that the late denoising stage plays a more important role in con- trollable image generation. Since the early sampling is mainly responsible for generating the global structure and low-frequency information of the image (e.g. the approxi- mate shape of the object, the distribution of the components), while the late sampling is mainly responsible for generating high-frequency information and correcting complex details (e.g,, edge, texture). For this, more conditional control signals are necessary. Moreover, the generation deviation in the early stage is relatively small and can be rectified by subsequent sampling. If the sampling error in the later stage is large, the conditional consistency will be destroyed, resulting in a loss of control effect.
Our FlexControl follows the core design philosophy of (Zhang et al., 2023b), the trainable blocks are initialized with the pre-trained weight parameters of the SD model, and zero modules are added at the same time, which leads to the conditional mappings generated at the early training stage do not have the ability to control generation effectively. Therefore, we first fix mask M to 1 for warm-up training in the early training stage, e.g., 10K steps for SDI.5-based FlexControl and 5K steps for SD3-based FlexControl in our implementation, and then turn on the router unit to train together with the copy blocks. This helps the whole training procedure move in the right direction.
When it comes to the settings for applying the dynamic policy to ControlNeXt (Peng et al., 2024) and OminiCon- trol (Tan et al., 2024), we use the same settings as SD1.5- based FlexControl for FlexComtrolNext.. Since the FLOPs of FlexControINeXt does not change with sparsity 7, we slightly adjust the loss function when training FlexCon- trolNeXt. specifically, we change the ratio of FlexCon- trolNeXt's FLOPs to base model's FLOPs to the ratio of the number of backbone blocks injected with conditional controls to the total number of backbone blocks. For Flex- OminiControl, the resolution of 512x512 is used, and the learning rate is set to 1x 1x10- Except for the above settings, other settings are the same as those for SD3.0-base Flex- Control. We finetune the condition-LoRA on FLUX.1-dev (Labs, 2024), whose rank is set to 4 by default.
Based on the above findings, we can conclude that using unified control scheme in any case an inefficient control mode, which leads most of the conditional controls added in the early stage not playing the ideal role, and there will be insufficient conditional controls added in the late stage. Therefore, our dynamic control method can further release the performance of the controllable generation model by solving this problem. Next, we do a more detailed analysis of the different settings.
First, we test the activation time and position of the control block under different number of timestep settings with y
15

---

FlexControl: Computation-Aware Conditional Control with Differentiable Router for Text-to-Imare Generation
set 0.5 (i.e., approximately 50% sparsity). We set the timesteps to 10, 20 and 50 respectively. As shown in Fig- ure 6, it can be found that the pattern under different settings is basically the same as above. In the early stage, only a few blocks are activated, mainly concentrated in the head and tail. As the number of sampling steps increases, more blocks are activated. Until the middle stage of sampling, most of the blocks are activated.
Method
FID
CLIP score
mloU [
FLOPS
Speed
Random (front)
22.72
0.2516
0.2649
315 G
5.01 its
Random (middle)
21.34
0,2562
0.2803
G12G
5.02e0m it's
Random (buck)
20.64
0.2534
0.3217
314G
5.02 ltds
Uniform
19.14
0.2600
0.3024
323 G
4,95_om
FlexContrel, 17.21
0.2713
0.3572
168 G
1.64 it's
FlexContral,... 14.80
0.2842
0.3751
280 G
5.21gain
Table 7. Quantitative comparison with random selection and uni- form selection of control blocks. We compare image quality, con- trollability and efficiency. The diffusion iterations per second (i.e., it/s) is measured on single Nvidia RTX 2080 Ti GPU. The best and second-best values are highlighted in red and blue. FlexControl outperforms random selection and uniform selection under lower inference lateney.
Next, we test the activation of control blocks under different sparsity. We approximate 30%, 50%, and 70% sparsity by setting Y to 0.3, 0.5, and 0.7. It can be seen from Figure 7, when 30% sparsity is used, fewer blocks are activated at the late stage, and even some control blocks are not activated at all. When the sparsity increased to 70%, more middle blocks are activated in the early sampling period, and almost all blocks are activated in the late sampling period.
Method FID J
CLIP score
mloU
Speed
ControlNet
21.33
0.2531
0.2764
5.23 it/s
ControlNere+
19.89
0.2640
0.3435
5.23gnor it/s
FlexConirol (w.o. training)
19.86
0.2732
0.3295
5.24 its
In addition, we discuss the activation of various spatial con- ditions at each timestep. As shown in Figure 8, similar trend is found across different types of conditional maps, which proves the generalization of the above findings. In addition, there are some differences in activation details, which means that the router unit makes independent judg- ments on different conditional samples and plans specific activation routes for them. Due to the differences in feature distribution and information in the samples, this fine- -ra-irai control is particularly important for striking a balance be- tween performance and efficiency.
FlexControl
14.80
0,2842
0.3751
5.21 it/s
FlexControl (w.o. training)
16.56
0.2778
0.3665
4.86 it's
FlexControl
14.71
0.2840
0.3775
4.94gnor it/s
Table 8. Quantitative comparison with training-onces once-for-all all- sparsity strategy. We compare image quality, controllability and efficiency. The diffusion iterations per second (i.e., it/s) is mea- sured on single Nvidia RTX 2080 Ti GPU. The best values are highlighted in red. We adjust the inference latency by resetting the threshold T of Gumbel-Sigmoid activation function of router units.
back control blocks with 25%
Relying on the above findings, when we apply ControlNet or similar architectures in practice, only activating the head and tail blocks in the early stage, even activating ControlNet only in the late stage, can simply improve the inference efficiency, and no retraining is involved.
Random (back): sample the back control blocks with a sample probability of 50,, while the front and middle control blocks with 25%.
Uniform: start from the first control block, sequentially retain one block, and then skip the next block.
D. More Ablation Studies
Comparison with random selection. To further demon- strate that FlexControl's performance improvement is achieved by correctly positioning the injection location of conditional controls, we conduct ablation study on simpler alternatives to the random selection of control blocks. Since the relative positions of randomly sampled activation con- trol blocks will have a certain impact on the results, for a more comprehensive and fair evaluation, we partition the overall diffusion block into front, middle, and back parts, each part contains 30% of the blocks. We apply different sampling probabilities on each part:
We use the above division strategy to simulate the results produced by randomly selecting control blocks in a more comprehensive way. As shown in Table our adaptive selection strategy achieves significantly better results with lower FLOPs and latency. Additionally, scaling up the con- trol branch using our strategy by resetting y from 0.3 to 0.5 demonstrates a clear performance improvement.
Performance without training under assigned sparsity. Our methods usually require separate model training for target sparsity. Therefore, we try to train a model for ar- bitrary sparsity. To achieve this, we first set the threshold T in the router units to low level for model training to ensure that most control blocks can be activated. In addi- tion, we remove the cost loss in the optimization target and supervise the training process only through the diffusion loss. After completing the training, we inactivate part of the control block by raising the threshold T to convert it into a
Random (front): sample the front control blocks with sample probability of 50%, while the middle and back control blocks with 25%.
Random (middle): sample the middle control blocks with a sample probability of 50% while the front and
16

---

FlexControl: Computation-Aware Conditional Control with Differentiable Router for Text-to-Iman Image Generation
Timesteps
Timesteps 20
Timestape
steps
slepa
Blook Indee.
BR Indoo
Blook
SD1.5 b-baaeedd FlexContral
48.57%
blocka
adiivleed
Inference stops
Inference steps
Inference stops
inddec
Block indder.
SD3, O-bassed FlexsContral
47.89% blocks activated
50.00% blocks activated
7.4%% activated
Figure 6. The distribution of activated control blocks under different timesteps. The hyperparameter y is set to 0.5 to approximate the sparsity of 50%, and timestep set and 50 (i.e., the charts of columns one to threc), repectively. The charts the first and second rows show the results of the model based on SD1.5 and SD3.0, respectively. and denotes activated and inactivated control blocks, respectively.
sparse model. As shown in Table 8, the "training-once-for all" scheme achieves better performance than ControlNet similar inference speed and can be comparable to Con- trolNet++, but the performance does not fully match that of the y-aware trained version, indicating that explicit training with sparsity constraints remains crucial for achieving the trade-off.
optimal
E. More Visualization Results
17

---

FlexControl: Computation-Aware Conditional Control with Differentiable Router for Eex--to-mmag Generation
Act. -50%
Inference slops
92.85% blocks
Inference steps
44, 64% blocks
Inference steps
65. 18% blocks
Block inde:
Block
SD1 3-3ased FlexControl
steps
30.87% blocks activated
steps
50.00% blocks activated
Inference
+
aclivated
Block indox
Block
SD3. 1.3.0-baeed FlaxContral
Figure 7. The distribution of activated control blocks under different sparsity. The hyperparameter is set to 0.3, 0.5 and 0.7 to columns one to three), respectively. The timestep 20 is used, and the
approximate the sparsity 30%, 50%, 70% (i.e., charts in the first and second rows show the results of the model based on SDI.5 and SD3.0, respectively.
Depth Map
Canny Edge
Seg. Mask
Inference steps
48.75% blocks
Inference steps
48.39% blocks activated
Inference steps
44.64 44.64% blocks activated
Blook Indax
Block Index
Blook Index
Figure The distribution of activated control blocks under various conditional controls. The hyperparameter fy set to 0.5 to approximate the sparsity of 50%, and the timestep of 20 is used. The charts of columns three display the results depth map. canny edge, and segmentation mask datasets, respectively.
18

---

FlexControl: Computation-Awarr Conditional Control with Differentiable Router for Text-to-Image Generation
Image Condition
ControlNet-Lare
ControlNett+
ControlNet
T21-Adapter
ControlNet (SDXL)
Depth map
Canny edge
Segementation mask
Figure 9. Visualization comparison with state-of-the-art controllable generation methods on various spatial conditions. Except for
the last column ControlNet (SDXL), which uses SDXL the diffusion backbone, other models use SD1.5. Captions: white stallion horse gulloping furiously kicking up the dust behind it. Ingredients of curry, including onions, garlic, chili, and tomatoex. A group of people are observing an aquarium filled with colorful fish.
19

---

FlexControl: Computation-Aware Conditional Control with Differentiable Router for Text-to-Image Generation
Image & Couditon
FlexConinst
FlasConiru...o.
ControNNat-aage
ControlNatt+
Figure 10. Visualization comparison of FlexControl and existing methods SDL.5 for semantic consistency.
Image& Condition
RexCoerrooNee:s.... FexcCoerroNNee:t.... Pe FlenCootraNNeett-a
HeeCConrolNeee
Figure 11. Visualization comparison of FeexControlNeXtt and existing methods on SDL.5 for semantic consistency.
Image & Condition
FlexCoottol
FlexContzol
Fle
ControlNet-Large
ControlNet
Figure 12. Visualization comparison of FlexControl and existing methods on SD3.0 for edge preservation.
Image Concittion
FlexOminis--e
Flexthmintip-o.7
FlexOminsi
FlexOmini
OmrinCCentrol
Figure 13, Visualization comparison of Olexinniiootttoll and existing methods on FLUX.1 for edge preservation.
20


Reviews:
[{"review_id": "EqX5vdHUeC", "reviewer": "Reviewer_NbP9", "summary": "The paper addresses the limitations of existing ControlNet implementations in diffusion-based generative models, which often rely on ad-hoc heuristics for selecting control blocks. The authors employs a trainable gating mechanism to dynamically select which blocks to activate at each denosing step.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive feedback and support of our work. We hope to have answered all of your questions satisfactorily below. Please let us know if you see any further issues in the paper that must be clarified or addressed. \n\n> The ablation study … with simpler alternatives, such as random selection of control blocks…\n\nWe sincerely thank the reviewer for their suggestion to conduct ablation studies on simpler alternatives to the random selection of control blocks. In response, we have considered the following alternative sampling strategies.\n\nAs suggested, we first evaluate the simplest strategy—uniform sampling—where 50% of the control blocks are randomly selected, denoted as Uniform. \n\nThe experimental results corresponding to these sampling strategies are presented below.\n\n| Method             | Base Model |    FID    | CLIP_score |    mIoU    |   FLOPs   |        Speed       |\n|--------------------|:----------:|:---------:|:----------:|:----------:|:---------:|:------------------:|\n| Uniform            |    SD1.5   |   19.14   |   0.2600   |   0.3024   |   323 G   |   4.95±0.07 it/s   |\n| FlexControl(γ=0.3) |    SD1.5   |  _17.21_  |  _0.2713_  |  _0.3572_  | **168 G** | **5.64±0.12 it/s** |\n| FlexControl(γ=0.5) |    SD1.5   | **14.80** | **0.2842** | **0.3751** |  _280 G_  |  _5.21±0.12 it/s_  |\n\nNotably, compared to all random sampling strategies, our approach with $\\gamma$ = 0.3 achieved superior performance and higher inference speed. Moreover, the FID score improves from 17.21 to 14.80 when $\\gamma$ is increased from 0.3 to 0.5. This ablation study further demonstrates the effectiveness of our method, and we will incorporate these findings into the camera-ready version of our paper.\n\n>At the optimal performance setting ($\\lambda$=0.5)... regarding efficiency…\n\nWe would like to clarify that our method with  $\\gamma$ = 0.3 has already outperformed the ControlNet baseline in both SD1.5 and SD3.0 experiments, as demonstrated in Table 4 and Table 5 of the original paper, while also achieving significantly lower FLOPs, as shown in Table 3.\n\nOur choice of  $\\gamma$ = 0.5 for comparison in Table 1 and Table 2 is not because it represents the optimal value, but rather because it provides a more direct comparison with the ControlNet baseline in terms of computational cost (280G FLOPs vs. 233G FLOPs). In contrast, when  $\\gamma$ = 0.3, the FLOPs are substantially lower at just 168G. We decide to add all  $\\gamma$ value experiment results in Table 1 and Table 2 to increasing the clarity in camera ready version.\n\n> code links\n\nWe provided anonymous links for comparison and reproduction:【https://github.com/Anonym916/Anonymity】\n\n> In Table 3. why does the speed decrease from 0.3 to 0.5\n\nSince  $\\gamma$ represents the expected overall number of activated blocks, increasing  $\\gamma$ results in a higher number of active blocks, leading to increased computational cost (FLOPs) and consequently lower inference speed. This explains the observed decrease in speed from  $\\gamma$ = 0.3 to  $\\gamma$ = 0.5 in Table 3. We will refine our text in the camera-ready version to improve clarity.\n\n> How should ($\\lambda_C$) be chosen, and does its value affect performance?\n\n$\\lambda_C$​ is a scaling factor that balances the diffusion objective and the FLOPs constraint objective. Since these two loss functions operate on different scales, $\\lambda_C$​ is necessary to ensure proper weighting between them. We tune this value to achieve the target activation percentage while maintaining overall performance. Proper selection of $\\lambda_C$ ensures that the model activates the desired number of blocks without significantly degrading generation quality. We proposed to use $\\lambda_C$ = 0.5 in all our experiments as an experience value.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "no theoretical claims.", "experimental_designs_or_analyses": "Yes, the paper uses various metrics and baselines to support the effectiveness of the proposed method.", "relation_to_broader_scientific_literature": "The introduction of a computation-aware training loss aligns with prior research on optimizing computational efficiency in generative models.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths:\n\n1. The paper is well-written and organized.\n2. The paper introduces a novel dynamic control mechanism that enhances the adaptability of diffusion models, moving away from static, heuristic methods.\n3. The paper includes extensive experiments across multiple architectures (UNet and DiT) and various tasks, providing robust evidence of FlexControl's effectiveness.\n\nWeaknesses:\n\n1. The ablation study presented in the paper lacks rigor and comprehensiveness. The authors should investigate how the performance of FlexControl is affected by replacing the proposed gating mechanism with simpler alternatives, such as random selection of control blocks. \n\n2. At the optimal performance setting (\\lambda=0.5), both the number of parameters and the FLOPs (Floating Point Operations) are worse than those of ControlNet. This raises concerns about the validity of their claims regarding efficiency.\n\n3. Also no code provided here make it harder to evaluate the method.", "other_comments_or_suggestions": "None", "questions_for_authors": "1. In Table 3, why does the speed decrease when \\(\\lambda\\) changes from 0.3 to 0.5 with SD1.5?\n2. How should \\(\\lambda_C\\) be chosen, and does its value affect performance?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ymsir8XE0P", "reviewer": "Reviewer_KazD", "summary": "This paper studies the Computation-Aware ControlNet by proposing a dynamic routing strategy which dynamically selects blocks to activate at each denoising step. It aims at adjusting control blocks based on timestep and\nconditional information while maintaining (or even improving) generation quality. The experimental results show its effectiveness (higher score).\n\n### update after rebuttal\n\nThe response addressed my concerns partially. I would like to increase my initial rating, but I am also not against Rejection, as other reviewers an I both have some concerns on the performance (such as FLOPs, compuatational cost, the impact caused by hyperparameters.).", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We hope the answer below solves all the clarity issues. \n\n> The proposed method requires more parameters than the typical controller.\n\nYes, our method requires more parameters than standard ControlNet. However, compared to ControlNet-Large, **it achieves better generation quality and controllability while halving inference FLOPs and improving inference speed**. As discussed in Remark (page 4), the additional parameters have a negligible impact on GPU memory and inference performance.\nImportantly, **our focus is not on parameter efficiency within control blocks but on dynamic routing for adaptive efficiency. Our approach is also compatible with recent parameter-efficient control methods, such as ControlNext and OmniControl, as noted in our response to Reviewer eHec.**\n\n> not compare with ControlNet++ and controlnext on flops and parameters\n\nWe would like to clarify that our research does not focus on designing more efficient control block structures but rather on investigating how to integrate control blocks effectively and efficiently into pre-trained diffusion models. In this sense, our work is orthogonal to ControlNeXt methods, which is why we initially excluded it from Table 1 and Table 2. However, our approach can potentially be adopted by ControlNeXt to further enhance its performance.\n\nWe appreciate the reviewer's suggestion for additional experiments. To evaluate our method in the context of ControlNeXt, we refer the reviewer to our response to Reviewer eHec. Briefly, since ControlNeXt applies control to all blocks by default, our method achieves comparable performance to ControlNeXt at $\\gamma$  = 0.3 and outperforms it at $\\gamma$ = 0.5.\n\nRegarding ControlNet++, it primarily focuses on improving training strategies while maintaining the same inference cost as the standard ControlNet. In contrast, our work explicitly targets inference efficiency. Thus, a direct comparison is not applicable. Nevertheless, we acknowledge the relevance of these methods and will consider discussing them in future versions of our paper.\n\n> In Tab.5, what is the control signal? Why do not provide …\n\nTable 5 presents an ablation study on different control strategies under the \"Canny edge\" control signals. \nWe agree with the reviewer's suggestion and will reorganise the table in a camera-ready version to improve the clarity. \n\n> How to optimize the cost loss (Eq. 17).\n\nThe FLOPs loss is computed by referencing pre-computed FLOPs values from a lookup table, which are then combined with the diffusion loss using a scaling factor. This approach aligns with prior research[a,b,c] **(as referred to in response to reviewer eKoB)**, on dynamic and efficient neural network architectures that optimize computational cost while maintaining performance. We will add those parts in the related works. \n\n>There are many previous works for dynamic routing strategy…\n\nWe appreciate the reviewer’s suggestion to discuss prior work on dynamic routing strategies. While our method shares some conceptual similarities with existing approaches, its goal and implementation are fundamentally different. Below, we clarify these distinctions concisely:\n\n- [1] (Dynamic Routing Networks): Introduces a model with multiple branches, where a learned router selects the best path for each input to improve efficiency. Their method is trained from scratch for classification and focuses on reducing FLOPs. In contrast, our approach dynamically adjusts the influence of a fine-tuned ControlNet within a pre-trained diffusion model, aiming for controlled generation rather than computational savings.\n\n\n- [2] (SkipNet): Uses a gating mechanism to decide whether to skip certain convolutional layers, reducing computation for easier inputs. Unlike SkipNet, which skips layers within a single network, our method balances contributions between a fixed diffusion backbone and a fine-tuned control module. We modulate control strength from ControlNet to a pre-trained diffusion model for better adaptation.\n\n\n- [3] (DiT: Dynamic Token Routing): Dynamically routes image tokens within a Vision Transformer, deciding which tokens to process at each layer for efficiency. While DiT optimizes computation by selectively processing tokens, our method adjusts how much control blocks the ControlNet influences the final output, without altering token flow within the transformer.\n\n\n- [4] (RAPHAEL): A large-scale diffusion model using a mixture-of-experts (MoE) to assign different paths for different styles or concepts. RAPHAEL is trained from scratch on massive datasets, while our approach efficiently adapts a pre-trained diffusion model using dynamic routing, making it suitable for low-data settings.\n\n**These prior works focus on optimizing efficiency or designing new architectures, while our method adapts an existing pre-trained model for more flexible and controlled generation**. We appreciate the reviewer’s suggestion and will incorporate this discussion into the final version of our paper.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "No issues.", "relation_to_broader_scientific_literature": "It is helpful for designing an efficient controlnet for the image generation community.", "essential_references_not_discussed": "This paper proposes the dynamic routing strategy, which have been widely studied in computer vision community, like [1,2,3], even in text2image [4], while it does not discuss them.\n\n[1] Cai et al., Dynamic Routing Networks\n\n[2] Wang et al., SkipNet: Learning Dynamic Routing in Convolutional\nNetworks\n\n[3] Ma et al., DiT: Efficient Vision Transformers with Dynamic\nToken Routing\n\n[4] Xue et al., RAPHAEL: Text-to-Image Generation via Large Mixture of Diffusion Paths", "other_strengths_and_weaknesses": "Strengths:\n\nThe proposed method is efficient, achieving better performance with lower FLOPs.\n\n\nWeaknesses:\n\n- The proposed method requires more parameters than the typical controlnet.\n\n- In Tab.3, why do not compare with controlnet++. ControlNext should also be compared w.r.t. performance and parameters and flops.\n\n- In Tab.5, what is the control signal? Why do not provide the relevant results in Tab.1 for clear comparison?\n\n- How to optimize the cost loss (Eq. 17).\n\n- There are many previous works for dynamic routing strategy. The authors do not review them, and the proposed implementation also does not contain new thing.", "other_comments_or_suggestions": "-", "questions_for_authors": "See Weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "syHFjf8VmO", "reviewer": "Reviewer_eHec", "summary": "This paper proposes FlexControl, a novel method aimed at improving the computational efficiency of ControlNet, an important model for adding controllability in text-to-image generation tasks. Unlike the original ControlNet, which utilizes half of the diffusion architecture as its encoder, FlexControl introduces an additional, fully trainable encoder as a separate copy of the entire diffusion architecture. A differentiable router is trained alongside this encoder to dynamically activate only the necessary blocks required for each specific task.\nTo train this router, authors propose a computation-aware loss function that regularizes the model by matching a predetermined target ratio for reducing Floating Point Operations (FLOPs). The chosen ratio significantly influences both the performance and efficiency of FlexControl. The proposed method demonstrates improved results in terms of both performance and computational efficiency across various conditions, including depth maps, Canny edges, and segmentation masks.\n\n## Update after rebuttal\n\nThrough the rebuttal and discussion, I’ve come to understand that FlexControl can indeed achieve improved computational efficiency compared to baselines when gamma is properly selected. However, the method still requires careful hyperparameter tuning, which may present practical challenges.\n\nI am raising my score to a weak accept, though I still believe this paper sits on the borderline and could reasonably be rejected.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the detailed and constructive feedback. \n\n> ControlNext.. LoRA-based… \nWe appreciate the reviewer’s feedback regarding the comparison with other methods. While a direct comparison is not applicable (as our work focuses on control block integration rather than parameter fine-tuning), we have instead integrated our methods with two representative approaches: ControlNeXt and Omini-Control [1] (a recent popular LoRA-based control method). Specifically, for Omni-Control, instead of following Flux’s approach of concatenating control tokens into new tokens, it appends condition image tokens with noisy image tokens as a longer sequence and leverages LoRA to jointly process them.\n\nOur results show that, unlike methods that control all blocks by default, our approach achieves superior performance with fewer activated blocks, demonstrating its adaptability and broader applicability. Detailed comparisons follow below.\n\n*[1] Tan, Z., et,al. OminiControl: Minimal and Universal Control for Diffusion Transformer. arXiv e-prints, pp.arXiv-2411.*\n\n**On segmentation mask**\n| Method                 | Base Model |    FID    | CLIP_score |    mIoU    |  FLOPs  |      Speed     |\n|------------------------|:----------:|:---------:|:----------:|:----------:|:-------:|:--------------:|\n| ControlNeXt            |    SD1.5   |   24.16   |   0.2659   |  _0.2825_  | 51.72 G | 5.34±0.02 it/s |\n| FlexControlNeXt(γ=0.3) |    SD1.5   |   25.22   |   0.2531   |   0.2644   |    /    |        /       |\n| FlexControlNeXt(γ=0.5) |    SD1.5   | **23.74** |  _0.2664_  |   0.2819   |    /    |        /       |\n| FlexControlNeXt(γ=0.7) |    SD1.5   | **23.71** | **0.2674** | **0.2841** |    /    |        /       |\n| FlexControlNeXt(γ=0.8) |    SD1.5   |  _23.84_  |  _0.2662_  | **0.2841** |    /    |        /       |\n\n**On Canny**\n| Method                  | Base Model |    FID    | CLIP_score |    SSIM    |    FLOPs    |        Speed       |\n|-------------------------|:----------:|:---------:|:----------:|:----------:|:-----------:|:------------------:|\n| OminiControl            |   FLUX.1   |   22.84   |   0.2830   |   0.4125   |   16.89 T   |   2.36±0.00 it/s   |\n| FlexOminiControl(γ=0.2) |   FLUX.1   |   36.62   |   0.2712   |   0.3122   | **10.76 T** | **3.42±0.09 it/s** |\n| FlexOminiControl(γ=0.3) |   FLUX.1   |   26.65   |   0.2791   |   0.3668   |  _11.45 T_  |  _3.28±0.07 it/s_  |\n| FlexOminiControl(γ=0.5) |   FLUX.1   |   22.61   | **0.2886** |   0.4123   |   13.08 T   |   3.08±0.10 it/s   |\n| FlexOminiControl(γ=0.7) |   FLUX.1   |  _22.39_  |   0.2855   |  _0.4146_  |   14.57 T   |   2.80±0.09 it/s   |\n| FlexOminiControl(γ=0.8) |   FLUX.1   | **22.27** |  _0.2861_  | **0.4153** |   15.40 T   |   2.69±0.07 it/s   |\n\n> Table 4.. Gamma=0.3…Further exploration..gamma value…\n\nWe appreciate the reviewer’s observations on FlexControl’s performance across $\\gamma$ values. Even at $\\gamma$ = 0.3 our method surpasses standard ControlNet while being significantly more efficient. Though slightly behind the more computationally expensive ControlNet-Large, it achieves over three times the efficiency, highlighting its effectiveness. To provide further insights, we conducted additional ablation studies on segmentation and Canny tasks, analyzing $\\gamma$ values from 0.2 to 0.8. The results, detailed below, illustrate the trade-offs between efficiency and performance:\n\n**On segmentation mask**\n| Method             | Base Model |  FID  | CLIP_score |  mIoU  | FLOPs |      Speed     |\n|--------------------|:----------:|:-----:|:----------:|:------:|:-----:|:--------------:|\n| ControlNet |    SD1.5   | 21.33 |   0.2531   | 0.2764 | 233 G | 5.23±0.07 it/s |\n| FlexControl(γ=0.2) |    SD1.5   | 21.52 |   0.2584   | 0.2995 | 112 G | 5.98±0.09 it/s |\n| FlexControl(γ=0.3) |    SD1.5   | 17.21 |   0.2713   | 0.3572 | 168 G | 5.64±0.12 it/s |\n| FlexControl(γ=0.8) |    SD1.5   | 15.59 |   0.2804   | 0.3695 | 448 G | 4.82±0.06 it/s |\n\n**On Canny**\n| Method             | Base Model |  FID  | CLIP_score |  SSIM  |  FLOPs |      Speed      |\n|--------------------|:----------:|:-----:|:----------:|:------:|:------:|:---------------:|\n| ControlNet |    SD3.0   | 27.21 |   0.2512   | 0.3749 | 3.25 T | 48.34±1.78 s/it|\n| FlexControl(γ=0.2) |    SD3.0   | 28.11 |   0.2524   | 0.3577 | 1.25 T | 38.21±2.97 s/it |\n| FlexControl(γ=0.3) |    SD3.0   | 23.39 |   0.2581   | 0.4286 | 1.86 T | 40.83±3.09 s/it |\n| FlexControl(γ=0.8) |    SD3.0   | 20.72 |   0.2719   | 0.4816 | 4.97 T | 54.05±2.53 s/it |\n\nWe apologize for any confusion caused by our text and table presentation that may have led to misunderstandings. We will refine our wording to ensure greater clarity in the camera-ready version.\n\n> SD3 adaptation\n\nIn SD3 tasks, we select **all** transformer blocks as candidates and use our dynamic routing strategy to flexibly decide which transformer block to add control to. We copy all parameters in a block for both modalities.", "claims_and_evidence": "The claims presented in the paper regarding computational efficiency and performance gains through FlexControl require additional evidence. Specifically, a comparative analysis with the open-source community's LoRA baseline [A], which modifies only the input layer's channel count and trains only LoRA parameters, without additional models, should be included. Such baseline may be more efficient in parameter count, FLOPs, and inference speed. Moreover, comparisons with ControlNeXt are essential. \n\nThe authors note that FlexControl with Gamma values of 0.5 and 0.7 performs well and maintains similar speed to ControlNet at Gamma=0.5. However, Table 4 reveals that at Gamma=0.3, FlexControl underperforms relative to ControlNet, indicating that performance gains only occur when computational efficiency is equivalent to or less than ControlNet. This raises questions regarding the actual advantage of FlexControl over ControlNet, particularly when improved efficiency corresponds to reduced performance. Further exploration of the lower bound of the gamma value and its impact on model performance is also necessary for a comprehensive evaluation.\n\n[A] Black Forest Labs, https://github.com/black-forest-labs/flux/blob/main/docs/structural-conditioning.md", "methods_and_evaluation_criteria": "The evaluation used in this paper appropriately assess quality and fidelity across various tasks.", "theoretical_claims": "There are no theoretical claims.", "experimental_designs_or_analyses": "A deeper exploration into the effects of varying gamma values, especially their lower limits, would strengthen the experiments.", "relation_to_broader_scientific_literature": "Adding controllability to text-to-image generation models is a critical research topic within visual generative modeling. In this context, computational efficiency emerges as an essential aspect.", "essential_references_not_discussed": "No issues found.", "other_strengths_and_weaknesses": "A notable strength of FlexControl is its ability to maintain strong performance, comparable to ControlNet, particularly at gamma values around 0.5. However, its key weakness is evident when aiming for higher computational efficiency (gamma=0.3), where its performance significantly drops below ControlNet's baseline.", "other_comments_or_suggestions": "No issues found.", "questions_for_authors": "Regarding the SD3 adaptation, which part of the dual-stream block was specifically trained using a trainable copy in FlexControl? Parameters of both modalities? Or only the image modality?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TuA6MKOUqj", "reviewer": "Reviewer_eKoB", "summary": "This paper proposes FlexControl, a framework that introduces a novel gating mechanism for dynamically selecting blocks to activate in the control network, reducing computational overhead while preserving or improving image quality. The authors have conducted experiments on both UNet-based (SD1.5) and DiT-based (SD3.0) architecture across three tasks (depth, canny, seg.), demonstrating the effectiveness of the proposed method.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for acknowledging the novelty and performance of our paper. We hope the following answers reflect your questions. \n\n> Quantitative comparison and ablation on ControlNeXt..\n\nWe appreciate the reviewer’s concern regarding the need for additional comparisons with efficient control models. In response, we have conducted further experiments on ControlNext and Omini-Control[1], as mentioned in our reply to Reviewer eHec.\nIt is important to highlight that while optimizing control block efficiency is valuable, our focus is different: we propose a dynamic routing strategy that adaptively determines the most efficient control strategy across different time steps and samples. This approach complements existing efficient control methods rather than solely aiming to reduce the cost of control blocks.\nOur experiments confirm that integrating our method with these efficient control models further enhances their performance while improving efficiency. \n\n-[1] Tan, Z., Liu, S., Yang, X., Xue, Q. and Wang, X., 2024. OminiControl: Minimal and Universal Control for Diffusion Transformer. arXiv e-prints, pp.arXiv-2411.\n\n> The paper lacks explanation or ablation study on how to determine the hyperparameter $\\lambda_C$\n\nWe appreciate the reviewer’s request for further clarification on determining the hyperparameter $\\lambda_C$​ in Equation (18).\n$\\lambda_C$ serves as a scaling factor to balance different objectives: the diffusion loss optimizes image quality, while $L_C$ regulates and enforces control block sparsity. To ensure the trained gating mechanism achieves the desired sparsity while maintaining generation quality, $\\lambda_C$​ needs to be tuned empirically. While the optimal value may vary slightly across models, our experiments indicate that setting $\\lambda_C$ = 0.5 provides the best trade-off in practice.\nTo compute $L_C$​, we utilize a precomputed block-wise FLOPs lookup table, following methodologies from prior work ([a], [b], [c]). This approach ensures an efficient and structured way to regulate computational cost while preserving performance.\n\n-[a] Meng, L., Li, H., Chen, B. C., Lan, S., Wu, Z., Jiang, Y. G., & Lim, S. N. (2022). Adavit: Adaptive vision transformers for efficient image recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 12309-12318).\n\n-[b] Rao, Y., Liu, Z., Zhao, W., Zhou, J., & Lu, J. (2023). Dynamic spatial sparsification for efficient vision transformers and convolutional neural networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(9), 10883-10897.\n\n-[c] Han, Y., Liu, Z., Yuan, Z., Pu, Y., Wang, C., Song, S., & Huang, G. (2024). Latency-aware unified dynamic networks for efficient image recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence.\n\n>The desired sparsity γ needs to be specified before training. During inference, is it possible to manually adjust the number of activated blocks to achieve an efficiency-performance trade-off?\n\nWe appreciate the reviewer’s suggestion regarding an inference-time scaling mechanism. To explore this, we conducted additional experiments where a ControlNet-Large model was trained with all blocks activated on a segmentation mask control task. During inference, we dynamically adjusted the gating threshold to control the number of activated blocks.\nOur results confirm that scaling activation blocks at inference is feasible, leading to better performance than the ControlNet baseline while maintaining comparable FLOPs and inference speed. However, the performance does not fully match that of the $\\gamma$-aware trained version as we proposed in the paper, indicating that explicit training with sparsity constraints remains crucial for achieving the optimal efficiency-performance trade-off. Detailed results are presented below.\n| Method                     | Base Model |  FID  | CLIP_score |  mIoU  |      Speed     |\n|----------------------------|:----------:|:-----:|:----------:|:------:|:--------------:|\n| FlexControl(γ=0.5)         |    SD1.5   | 14.80 |   0.2842   | 0.3751 | 5.21±0.12 it/s |\n| FlexControl(w.o. training) |    SD1.5   | 19.86 |   0.2732   | 0.3295 | 5.24±0.11 it/s |\n| FlexControl(γ=0.7)         |    SD1.5   | 14.71 |   0.2840   | 0.3775 | 4.94±0.07 it/s |\n| FlexControl(w.o. training) |    SD1.5   | 16.56 |   0.2778   | 0.3665 | 4.86±0.09 it/s |\n\n> Issue of parameters\n\nWe acknowledge the reviewer’s concern about parameter count. Our goal is not just to reduce control block cost but to develop a dynamic routing strategy that optimally adapts efficiency across time steps and samples. As shown in our experiments and discussed in our response to Reviewer eHec, integrating our method with efficient control models further improves both performance and efficiency, reinforcing its broad applicability of our methods.", "claims_and_evidence": "The claims in the submission are well-supported by the experiments.", "methods_and_evaluation_criteria": "Yes, the proposed methods make sense for the problem. The evaluation metrics (FID, CLIP score, depth RMSE, canny SSIM, seg mIoU) are all common criteria in the area of controllable image generation.", "theoretical_claims": "This papar does not contain any theoretical claims or proofs.", "experimental_designs_or_analyses": "Yes, I've checked all the experiments. Some issues are listed as follows:\n\n1. Quantitative comparison: As one of the main contributions of this paper is to reduce the computational overhead, it lacks comparison on the image quality (Table 1) and controllability (Table 2) with efficient control models mentioned in the Related Work section, such as ControlNeXt[1].\n2. Ablation study: Similar to the first issue, this paper lacks computational complexity comparison (Table 3) with efficient methods. Adding these comparisons could help the readers understand the computational efficiency of the proposed method better.\n3. The paper lacks explanation or ablation study on how to determine the hyperparameter $\\lambda_{\\mathbf C}$ in equation (18).\n\n[1] Peng, Bohao, et al. \"Controlnext: Powerful and efficient control for image and video generation.\" *arXiv preprint arXiv:2408.06070* (2024).", "relation_to_broader_scientific_literature": "This paper falls into the area of controllable image generation. It addresses a key problem in this area that previous methods heavily rely on heuristic network design, and proposes a novel dynamic gating mechanism to solve this problem. This paper is also related to efficient control models, proposing a novel cost loss that controls the sparsity of the network.", "essential_references_not_discussed": "The essential related works are well-discussed and cited.", "other_strengths_and_weaknesses": "Strengths: \n\n1. The dynamic gating mechanism is novel in the area of controllable image generation.\n2. The paper is well-written, the presentation is clear and easy to follow.\n\nWeaknesses:\n\n1. The desired sparsity $\\gamma$ needs to be specified before training. It would be better if a single model can handle all possible $\\gamma$, further increasing the flexibility of the proposed method.\n2. Compared to other efficient control models, FlexControl only reduces computational overhead but does not decrease the number of parameters (actually doubles the parameters of the original ControlNet), which increases the burden of distributing and deploying the model.", "other_comments_or_suggestions": "I do not have other comments or suggestions.", "questions_for_authors": "1. How is the FLOPs in the cost loss $\\mathcal L_\\mathbf{C}$ computed, such that the gradients can be back-propagated to the parameters of the network?\n2. During inference, is it possible to manually adjust the number of activated blocks to achieve a efficiency-performance trade-off?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: After reading the authors’ rebuttal and discussing intensively, all reviewers come to the consensus of accepting this paper. The AC agrees with the reviewers that the new idea proposed in this paper to improve the computational efficiency of ControlNet is nice and this paper did make some valuable contributions to the community.
Decision: Accept (poster)