EVCtrl: Efficient Control Adapter for Visual Generation

## Abstract

Abstract Visual generation includes both image and video generation, training probabilistic models to create coherent, diverse, and semantically faithful content from scratch. While early research focused on unconditional sampling, practitioners now demand controllable generation that allows precise specification of layout, pose, motion, or style. While ControlNet grants precise spatial-temporal control, its auxiliary branch markedly increases latency and introduces redundant computation in both uncontrolled regions and denoising steps, especially for video. To address this problem, we introduce EVCtrl , a lightweight, plug-and-play control adapter that slashes overhead without retraining the model. Specifically, we propose a spatio-temporal dual caching strategy for sparse control information. For spatial redundancy , we first profile how each layer of DiT-ControlNet responds to fine-grained control, then partition the network into global and local functional zones. A locality-aware cache focuses computation on the local zones that truly need the control signal, skipping the bulk of redundant computation in global regions. For temporal redundancy , we selectively omit unnecessary denoising steps to improve efficiency. Extensive experiments on CogVideo-Controlnet, Wan2.1-Controlnet, and Flux demonstrate that our method is effective in image and video control generation without the need for training. For example, it achieves 2.16 × \mathbf{2.16\times} and 2.05 × \mathbf{2.05\times} speedups on CogVideo-Controlnet and Wan2.1-Controlnet, respectively, with almost no degradation in generation quality.Codes are available in the supplementary materials.

## 1 Introduction

Recent advances in the alignment of machine learning models for text-to-image [37, 32] and text-to-video synthesis [35, 36, 38, 34] have propelled a new wave of high-performance generators. Image models such as PixArt-$delta$[6], Stable Diffusion 3[10], Flux[1], Kolors[17], and Hunyuan-DiT[20], together with video models including Sora[25], CogVideoX[59], HunyuanVideo[18], and Qihoo-T2X[50], now deliver stunning visual fidelity and open up a wide spectrum of downstream applications.

However, current ControlNet-based[63] controlled generation methods for DiT have two major drawbacks. First, as an ”auxiliary” neural network model structure, ControlNet typically directly replicates half of the network architecture, such as PixArt-$\delta$[6], where a large number of additional parameters significantly increase the inference burden. Similarly, although OminiControl[46] only adds a limited number of parameters, it doubles the number of tokens involved in the attention and linear layers, resulting in an almost 70% increase in overall computational complexity. Second, during the entire diffusion process, the correlation between different time steps and the feature changes of conditional tokens is often overlooked, leading to low computational efficiency. To address these issues caused by computational complexity, this paper first examines the redundancy of conditional information in time and space:

Spatial Redundancy: We found that many spatial regions of conditional information are effectively empty, such as edgeless regions whose pixel values sit close to zero. Tokens produced in these zones offer almost no guidance, yet the model still processes them, driving up computation without improving control fidelity. This inefficiency becomes most acute when the condition itself is sparse.

Temporal Redundancy: Figure [2](https://arxiv.org/html/2508.10963v1#S0.F2) shows the distribution of the feature distances between adjacent time steps for different tokens, where a higher value indicates a lower similarity of the token between adjacent time steps. Observation reveals that conditional tokens C exhibit extremely high distances at most time steps during the diffusion process, while showing relatively low distances at individual time steps. This observation suggests that there is more significant temporal redundancy in Controlnet.

Motivated by these observations, we propose an Efficient Visual Control Adapter(EVCtrl). For spatial redundancy, EVCtrl presents Local Focused Caching(LFoC). LFoC identifies and stores only those tokens whose caching yields the highest benefit-to-cost ratio, prioritizing regions that encode salient edge cues. Concretely, during the denoising process, the adapter first computes and caches tokens across all spatial locations. In subsequent steps, tokens corresponding to edges are recomputed in full to preserve fidelity, whereas tokens associated with near-zero, edge-free regions are retrieved from the cache. This hybrid strategy enables the diffusion model to concentrate its capacity on edge-conditioned areas, sustaining control quality while substantially reducing computation over uniform regions.

Additionally, we observe temporal redundancy within the denoising processing: the conditioning signal evolves unevenly across timesteps. EVCtrl’s Denoising Step Skipping (DSS) quantifies the dynamic variation of the condition at each step and retains full computation for the few timesteps that contribute most to the final control. By concentrating resources on these pivotal steps, DSS simultaneously lowers computational load and improves output quality.
Numerous experiments in text-to-image and text-to-video generation have shown that EVCtrl is more effective than previous feature caching methods on CogVideo-Controlnet[47], Wan2.1-Controlnet, and Flux-Controlnet[57]. For example, on CogVideo-Controlnet[47], it can achieve$\mathbf{2.16\times}$ acceleration, preserving near-lossless quality across various evaluation metrics when compared to the original controlnet pipeline without cache acceleration.In general, our contributions can be summarized as follows:

- •
We emphasize the spatial and temporal redundancies inherent to image and video generation tasks, and propose EVCtrl, a novel and efficient control adapter for controllable image and video generation.
- •
Local Focused Caching (LFoC): To reduce spatial redundancy in the control signal, we propose spatial locality caching, which skips most computations over regions that receive no control . Then we identify the key carriers of fine-grained control information within DiT-ControlNet and construct a hierarchical attribution–sensitivity framework to further accelerate caching.
- •
Denoising Step Skipping (DSS): To reduce temporal redundancy, we exploit the higher redundancy of the control branch relative to the original denoising path and the uneven influence of individual denoising steps on the conditioning signal, adaptively selecting the few steps that most affect the control for full computation while maintaining a periodic cache.
- •
Extensive experiments have been conducted on CogVideo-Controlnet, Wan2.1-Controlnet, and Flux-Controlnet, and the results show that EVCtrl achieves a high acceleration ratio while maintaining nearly lossless generation quality.

## 2 Related Works

### 2.1 Controllable Diffusion Models

To achieve conditional control in pre-trained text-to-image diffusion models, there are two main methods for introducing controllable conditions into image [58, 56, 61, 65, 66] or video generation models [33, 31]: (1) training a new large model from scratch for multi-condition compliance [15] (2) freezing the large pretrained model and fine-tuning only a lightweight adapter. Recent studies have extended these ideas to diffusion transformers (DiTs) [45, 51, 4, 7, 5, 49, 12, 39, 60, 19, 43], exploiting the built-in multimodal attention (MM-attention)[41] to inject image or video conditions without further architectural change. T2I-Adapter[40] exemplifies the adapter route—it imposes minimal overhead yet yields weak control, limiting its use on complex prompts. In contrast, ControlNet[63] achieves stronger control effects by copying specific layers from pre-trained large models to guide image generation to align with control information, but it introduces a large number of additional parameters. Meanwhile, as the number of condition tokens increases, especially in controllable video generation, performing self-attention operations on the entire sequence of condition tokens leads to a significant increase in computational cost, imposing significant limitations on the controllable generation result rate from text to video.

### 2.2 Acceleration of Diffusion Models

Implementing low-latency and high-quality generation methods on diffusion models (DMs), including DiT, is an important research direction. Currently, there are mainly two types of acceleration methods for diffusion models: the first type is to reduce the number of sampling steps[24, 52, 26, 67, 8, 27, 44], and the second type is to accelerate the internal computation of diffusion models. Solutions to reduce computational complexity include model distillation and compression[55], token merging[3, 2, 11], token pruning[62], and layer-wise caching techniques[13, 54, 9, 22, 30, 29, 42]. However, layer-wise caching techniques use entire layers as the granularity and have difficulty perceiving the importance differences between tokens, resulting in key information being equally frozen or discarded. Toca[69] and Duca[70] instead use tokens as the minimum caching unit, accurately measure, and explicitly assign importance weights to each token with the help of a score map. In the subsequent computation stage, only high-weight tokens are refreshed according to a predetermined ratio, and the rest are directly reused from the cache, thus achieving lossless acceleration.

Figure: Figure 3: Overall framework.The pipeline takes initial noise and control conditions as inputs. Specifically, full caches are refreshed at fixed timestep intervals and at a handful of pivotal timesteps. For cached intermediate steps, only selected tokens within the mid-to-late fresh_blocks that capture local salient control cues are updated.
Refer to caption: x3.png

Unfortunately, the existing caching schemes are not specifically designed to adapt to ControlNet, which replicates specific layers in DM, and have the following issues: First, the existing caching and editing schemes often rely on explicit read and write operations of attention maps[69] or KV matrices[28, 68], which are incompatible with the mainstream Transformer acceleration pipeline and instead introduce additional latency. Additionally, there are differences in the correlation and importance of control information among different transformer layers in the existing diffusion transformer controlnet[5]. Meanwhile, the control conditions themselves provide prior information on the spatial distribution that should be focused on, but current methods fail to utilize these differences and information.

## 3 Method

### 3.1 Overall Framework

The pipeline of our method is show in Figure [3](https://arxiv.org/html/2508.10963v1#S2.F3).The core idea of EVCtrl is that, within each regular caching cycle, it performs full computation on an additional handful of denoising steps whose latent states critically determine the final output, while continuously updating the tokens that encode fine-grained control information, thereby mitigating errors introduced by repeatedly reusing stale control-signal features.In addition, existing caching strategies in deployed pipelines only cache the intermediate features of the MLP sub-layers inside a Transformer block. Unfortunately, the attention outputs themselves accumulate progressively larger errors when their outdated versions are reused, and these errors propagate downstream. Therefore, we extend the cache-update procedure to refresh the feature maps in both the attention and the MLP sub-layers simultaneously.

In the following, we first introduce the Spatial Locality Focused Caching to tackle the spatial redundancy within the Controlnet branch. Then we introduce the Temporal Denoising Step Skipping to tackle the temporal redundancy within the same branch.

### 3.2 Spatial Locality Focused Caching

**Table 1: Quality and efficiency benchmarking results of EVCtrl and other baselines**
| Type | Method | Quality | Efficiency |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| FID $\downarrow$ | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | VBench | Latency $\downarrow$ | Speedup $\uparrow$ |  |  |
| T2I | Flux-ControlNet (1024p) | – | – | – | – | – | $1.81\text{\,}\mathrm{s}$ | 1$\times$ |
| + Fora(N=2) | 47.97 | 27.65 | 0.92 | 0.053 | – | $1.05\text{\,}\mathrm{s}$ | $1.72\times$ |  |
| + Toca(N=2) | 55.82 | 26.98 | 0.90 | 0.065 | – | $1.22\text{\,}\mathrm{s}$ | $1.48\times$ |  |
| + Toca(N=4) | 64.78 | 25.43 | 0.88 | 0.134 | – | $1.09\text{\,}\mathrm{s}$ | $1.66\times$ |  |
| + Taylorseer(N=2,O=4) | 92.13 | 24.56 | 0.84 | 0.182 | – | $0.97\text{\,}\mathrm{s}$ | $1.87\times$ |  |
| + Ours(N=4) | 34.69 | 28.74 | 0.94 | 0.042 | – | $0.93\text{\,}\mathrm{s}$ | 1.95$\times$ |  |
|  | + Fora(N=4) | 56.73 | 26.12 | 0.89 | 0.101 | – | $0.87\text{\,}\mathrm{s}$ | $2.08\times$ |
|  | + Taylorsser(N=4,O=4) | 148.71 | 18.44 | 0.76 | 0.249 | – | $0.82\text{\,}\mathrm{s}$ | 2.21$\times$ |
|  | + Ours(N=8) | 42.34 | 27.88 | 0.91 | 0.057 | – | $0.85\text{\,}\mathrm{s}$ | 2.13$\times$ |
| T2V | CogVideoX-ControlNet (480p, 6s, 48 frames) | – | – | – | – | 82.6% | $81.2\text{\,}\mathrm{s}$ | 1$\times$ |
| + Fora(N=2) | 51.45 | 23.55 | 0.83 | 0.13 | 82.2% | $49.5\text{\,}\mathrm{s}$ | 1.64$\times$ |  |
| + Toca(N=2) | 62.97 | 20.93 | 0.78 | 0.18 | 81.6% | $57.8\text{\,}\mathrm{s}$ | 1.40$\times$ |  |
| + Toca(N=4) | 71.34 | 18.84 | 0.75 | 0.21 | 79.8% | $49.2\text{\,}\mathrm{s}$ | 1.61$\times$ |  |
| + Taylorseer(N=2,O=4) | 108.36 | 17.89 | 0.67 | 0.26 | 76.7% | $46.4\text{\,}\mathrm{s}$ | 1.75$\times$ |  |
| + Ours(N=4) | 51.37 | 23.62 | 0.87 | 0.11 | 82.2% | $43.4\text{\,}\mathrm{s}$ | 1.88$\times$ |  |
|  | + Fora(N=4) | 66.98 | 21.73 | 0.74 | 0.17 | 80.6% | $40.1\text{\,}\mathrm{s}$ | 2.02$\times$ |
|  | + Taylorsser(N=4,O=4) | 160.48 | 14.97 | 0.53 | 0.38 | 72.9% | $35.0\text{\,}\mathrm{s}$ | 2.32$\times$ |
|  | + Ours(N=8) | 64.83 | 22.13 | 0.81 | 0.15 | 81.3% | $38.5\text{\,}\mathrm{s}$ | 2.16$\times$ |

#### 3.2.1 Layer Sensitivity in DiT-ControlNet

To systematically characterise the layer-wise influence of DiT-ControlNet on both perceptual fidelity and conditional control accuracy, we propose a hierarchical attribution and sensitivity quantification framework that proceeds in three conceptually linked stages: norm-based observation, semantic mapping, and functional stratification. In the first stage, we extract every spatial token from the feature tensor produced by each DiT-ControlNet layer and compute its $L1$ norm. Global distributional properties are then estimated via histograms and kernel-density curves. In the second stage, a structure-similarity-driven alignment mechanism back-projects the indices of high-norm tokens into the original conditional domain, measuring their spatial overlap with the provided edge or pose guidance. This establishes an explicit correspondence between activation magnitude and fine-grained control semantics. Guided jointly by these two indicators, the network is stratified into two functional zones: (1) early layers exhibit peaked, low-variance distributions and are interpreted as robust encoders of low-frequency, global structural priors (2) mid-to-late layers display heavy-tailed distributions in which a minority of tokens obtain disproportionately large activations, thereby capturing high-frequency, local control cues such as edges, corners, and texture discontinuities. These high-energy tokens are identified as the primary carriers of fine-grained control information. This stratification paradigm is entirely training-free, providing an interpretable theoretical basis for subsequent selective caching strategies.

#### 3.2.2 Spatial Token-wise Feature Caching

Analogous to prior token-wise conservative caching schemes, ToCa offers two attention map-based selection strategies, whereas Duca directly retains tokens whose value matrices exhibit small norms as surrogates for those receiving high attention scores, thereby neutralizing the incompatibility between attention-score-driven token selection and mainstream attention optimizers such as FlashAttention and other memory-efficient variants. However, these two token selection paradigms remain insufficiently tailored to the DiT-ControlNet architecture; therefore, we propose a markedly superior alternative to select conditional tokens.

Figure [2](https://arxiv.org/html/2508.10963v1#S0.F2) empirically shows that tokens possessing large norms are strongly correlated with edge or pose conditioning signals.This observation motivates a caching strategy. It refreshes high-norm tokens more often within each interval. These tokens belong to mid-to-late layers that capture high-frequency local control cues and carry fine-grained information, thus strengthening conditional guidance. The entire procedure can be inserted into any DiT-ControlNet inference pipeline without parameter modification or retraining, realizing fine-grained caching that “refreshes frequently where critical and sparingly where redundant.” Concretely, let $P$ denote the proportion of critical conditional information; hence, the selection rule for $\mathcal{P}_{Cache}$ is defined as follows:

$$ $\mathcal{P}_{Cache}=\mathop{\arg\max}\limits_{\{i_{1},i_{2},\ldots,i_{n}\}\subseteq\{1,2,\ldots,N\}}\{\|t_{i_{1}}\|_{1},\|t_{i_{2}}\|_{1},\ldots,\|t_{i_{n}}\|_{1}\}$ (1) $$

where the $n$ is the total number of selected tokens, defined by $n=\lfloor P\%\times N\rfloor$

**Table 2: Quality and efficiency benchmarking results of EVCtrl and other baselines across diverse control conditions**
| Method | Canny | Hed |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Speed $\uparrow$ | FID $\downarrow$ | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | Speed $\uparrow$ | FID $\downarrow$ | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ |  |
| + Fora(N=4) | $1.98\times$ | 64.85 | 22.74 | 0.75 | 0.16 | $1.97\times$ | 64.67 | 22.56 | 0.75 | 0.17 |
| + Toca(N=4) | $1.55\times$ | 69.83 | 21.32 | 0.75 | 0.19 | $1.55\times$ | 69.74 | 21.25 | 0.76 | 0.18 |
| + Taylorseer(N=2) | $1.85\times$ | 104.68 | 19.94 | 0.69 | 0.24 | $1.86\times$ | 104.59 | 19.92 | 0.69 | 0.25 |
| + Ours(N=8) | $2.05\times$ | 62.24 | 23.26 | 0.80 | 0.12 | $2.02\times$ | 63.74 | 22.84 | 0.78 | 0.14 |
| Method | Pose | MLSD |  |  |  |  |  |  |  |  |
| Speed $\uparrow$ | FID $\downarrow$ | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | Speed $\uparrow$ | FID $\downarrow$ | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ |  |
| + Fora(N=4) | $1.98\times$ | 64.82 | 22.63 | 0.74 | 0.17 | $1.98\times$ | 64.79 | 22.68 | 0.75 | 0.16 |
| + Toca(N=4) | $1.56\times$ | 69.81 | 21.31 | 0.75 | 0.19 | $1.55\times$ | 69.75 | 21.46 | 0.75 | 0.18 |
| + Taylorseer(N=2) | $1.86\times$ | 104.47 | 19.86 | 0.70 | 0.22 | $1.87\times$ | 104.62 | 19.89 | 0.69 | 0.24 |
| + Ours(N=8) | $2.23\times$ | 56.64 | 24.76 | 0.82 | 0.09 | $2.08\times$ | 60.31 | 23.83 | 0.81 | 0.11 |

### 3.3 Temporal Denoising Step Skipping

#### 3.3.1 Denoising Redundancy and Criticality Detection

After addressing spatial redundancy, we focus on the temporal redundancy that naturally emerges during iterative denoising in the control branch. As shown in Figure [2](https://arxiv.org/html/2508.10963v1#S0.F2), we pairwise compare latent vectors across all timesteps through cosine similarity. Strikingly, tokens produced in consecutive denoising steps exhibit near-identical similarities, indicating an extremely slow evolution in latent space. This near-uniform behavior implies that substantial computational resources are repeatedly expended on nearly identical representations. Meanwhile, although most adjacent denoising steps are highly redundant, Figure [2](https://arxiv.org/html/2508.10963v1#S0.F2) also reveals abrupt drops in similarity at isolated timesteps. These sudden divergences mark rapid pivots in the denoising trajectory and inject details critical to final perceptual quality; consequently, these critical steps exert a disproportionately large influence on the final generation.

#### 3.3.2 Stride-Skipping with Critical-Step Preservation

To take advantage of the inherent similarity between adjacent time steps more efficiently, without altering the underlying model architecture, we set a cache interval $N$. Features are fully recomputed and cached every $N$ steps, then reused for the subsequent $N-1$ timesteps. To prevent overly aggressive skipping from inadvertently discarding steps that significantly affect quality, we introduce a complementary mechanism: within each cache interval, critical steps identified a priori are selectively restored to full computation, ensuring essential details are preserved.Concretely, given a set of $N$ adjacent timesteps$\{t,t-1,\dots,t-(N-1)\}$,we execute a full forward pass at the cycle’s leading timestep $t$ to compute and store the features in a cache as $C_{t}^{l}:=F(x_{t}^{l})$,where the superscript $l\in\{1,2,\dots,L\}$ denotes the layer index. Additionally, for a predetermined sequence of critical steps$\{k_{0},k_{1},\dots k_{m}\}$,we assume a timestep $k_{i}$ satisfies $t-1\leq k_{i}\leq t-(N-1)$,$0\leq i\leq m$.A full forward computation is executed at $k_{i}$ to update the cache $C_{k_{i}}^{l}:=F(x_{k_{i}}^{l})$. The cached representation is then reused for all remaining redundant timesteps so that $F(\cdot)$ is not invoked again.

## 4 Experiments

### 4.1 Experimental Setups

#### 4.1.1 Model Configurations

We conducted experiments with three commonly used DIT-based models equipped with ControlNet for various generative tasks, including Flux[57] for text-to-image generation, CogVideo[47], and Wan2.1[48] for text-to-video generation. The experiments were performed using an NVIDIA A800 80GB GPU. The CogVideo5B model is capable of generating 6-second videos with a resolution of 480p and a frame rate of 8fps. The Wan2.1-14B model can generate videos with a resolution of 720p or 480p, a frame rate of 16fps, and a duration of 5 seconds. Each model used its default sampling method: DPM-Solver++[27] with 20 steps for Flux, and DDIM[44] with 50 steps for both CogVideo and Wan2.1. For each model, we configure EVCtrl by setting the cache ratio $P$ according to the redundancy level of the conditional information, while also employing various average forced-activation periods $N$.

#### 4.1.2 Evaluation Protocol and Metrics.

We randomly selected 5,000 images and captions from the COCO-2017[21] dataset and extracted 100 videos and their descriptions from each of the 16 distinct categories of videos in the Vbench[16] dataset, totaling 1,600 videos. We evaluated the quality of the generated content using SSIM[53], LPIPS[64], PSNR[53], and FID[14]. Additionally, for text-to-video generation, we used video generation metrics within the Vbench framework as supplementary indicators.

Figure: Figure 4: The qualitative comparisons with existing methods.Visualization comparing acceleration methods on Wan2.1-ControlNet: while others sacrifice control-condition details and introduce visual distortion at high speed-up ratios, ours preserves fine-grained details and maintains high-quality generation.
Refer to caption: x4.png

### 4.2 Comparison with baselines

#### 4.2.1 Quantitative Results

As shown in Table [1](https://arxiv.org/html/2508.10963v1#S3.T1), we benchmark EVCtrl against three state of the art training free acceleration baselines: Fora[42], Toca[69], and Taylorseer[23], on both the Flux-ControlNet[57] and CogVideo-ControlNet[47] pipelines. Quantitative results demonstrate that EVCtrl outperforms all other baselines across every quality metric (FID $\downarrow$, PSNR $\uparrow$, SSIM $\uparrow$, LPIPS $\downarrow$) while achieving equal or even higher acceleration ratios. Specifically, on the COCO2017 validation set at a $2.13\times$ speedup, EVCtrl achieves the lowest FID of 42.34 and maintains SSIM above 0.9; on the VBench validation set, it delivers a $2.16\times$ speedup with SSIM = 0.81, and all remaining metrics also surpass those of competing baselines at the same speedup, without any retraining or extra memory cost.

It is also noteworthy that as the caching period lengthens and the overall acceleration ratio continues to climb, the baseline methods suffer a sharp quality drop due to the loss of critical control signals during strided propagation: FID and LPIPS deteriorate rapidly, while PSNR and SSIM decline in tandem, leading to obvious artifacts or structural distortions in the generated images. By contrast, EVCtrl maintains exceptional adaptability; even when the caching period increases from $N=4$ to $N=8$, it still exhibits an unrivaled balance between efficiency and fidelity.

In addition, as summarized in Table [2](https://arxiv.org/html/2508.10963v1#S3.T2), we evaluated EVCtrl in the Wan2.1-ControlNet pipeline under four distinct control modalities: Canny, HED, OpenPose, and MLSD. Quantitative results reveal that, regardless of the type of control, EVCtrl consistently outperforms all baselines on every quality metric (FID $\downarrow$, PSNR $\uparrow$, SSIM $\uparrow$, LPIPS $\downarrow$) while achieving equal or greater end-to-end acceleration. In particular, as the control signal becomes sparser, e.g., the thin edge maps produced by Canny or the minimal keypoint representation of OpenPose, the acceleration ratio of EVCtrl rises further, reaching up to $2.4\times$ on OpenPose without any quality degradation. This trend confirms that EVCtrl is especially effective when the conditioning information is low-density, enabling it to skip redundant computation while still preserving fine-grained structural fidelity.

**Table 3: Ablation study on different configurations of LFoC and DSS for controllable video generation**
| Method | Latency $\downarrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ |  |
| --- | --- | --- | --- | --- |
| LFoC-only | WA | 37.6s | 0.78 | 0.19 |
| WM | 36.6s | 0.74 | 0.22 |  |
| DSS-only | 35.8s | 0.66 | 0.27 |  |
| EVCtrl | 39.2s | 0.81 | 0.15 |  |

#### 4.2.2 Qualitative Results

We compare EVCtrl with prior accelerators on Wan2.1-ControlNet under four control conditions (Canny, HED, MLSD, pose), highlighting its ability to accelerate inference while preserving controllable detail and video quality. The results are shown in [4](https://arxiv.org/html/2508.10963v1#S4.F4). Across all conditions, EVCtrl consistently outperforms others in fidelity and detail: under Canny, it accurately reconstructs the distant mountains in the scene “a white sedan cruising along a sunlit sandy beach”; under HED, it retains the golden retriever’s tail in “golden retriever running in the rain”; under MLSD, it best matches the structural conditions in “gallery lined with displayed paintings”; and under pose guidance, the human hands exhibit both high quality and fidelity. In summary, EVCtrl leads existing methods in fidelity and detail.

Figure: Figure 5: The qualitative ablation study. We select the canny as the input condition to ablate the performance.
Refer to caption: x5.png

### 4.3 Ablation Study

In the subsequent section, we will analyze the effectiveness of spatial module LFoC and temporal module DSS.

#### 4.3.1 Effectiveness of Local Focused Caching(LFoC)

Table [3](https://arxiv.org/html/2508.10963v1#S4.T3) shows that removing LFoC lowers SSIM from 0.81 to 0.66 and raises LPIPS from 0.15 to 0.27, indicating a decline in generation quality. Moreover, comparing LFoC-with-attention(WA) and LFoC-with-MLP(WM) reveals that updating either the attention layers alone or the MLP layers alone leaves the cached key control tokens under-utilized. Only when LFoC refreshes both the attention and MLP layers simultaneously can the re-injected tokens fully unleash their expressive power, allowing the model to recover its original quality metrics. The visual results in Figure [5](https://arxiv.org/html/2508.10963v1#S4.F5) demonstrate that after removing LFoC the model struggles to control fine details, illustrating that LFoC better exploits spatially local, fine-grained prior knowledge and achieves superior controllability in controlled generation tasks compared with the aggressive cache approach and the approach [69][70] that only updates the MLP layers.

#### 4.3.2 Effectiveness of Denoising Step Skipping(DSS)

As shown in Table [3](https://arxiv.org/html/2508.10963v1#S4.T3), DSS effectively shortens inference time with almost no loss in quality.Additionally, Table [1](https://arxiv.org/html/2508.10963v1#S3.T1) demonstrates that DSS allows the skip interval $N$ to be increased to 8 for reduced computational cost while maintaining the same or better controllable-generation performance as other acceleration methods achieve at only half that skip interval. Moreover, the visual results in Figure [5](https://arxiv.org/html/2508.10963v1#S4.F5) reveal that removing DSS leads to a marked decline in both overall scene completeness and the fidelity of detail control, further validating the effectiveness and fidelity of our DSS strategy.

## 5 Conclusion

ControlNet-based controllable image and video generation usually incurs high computational costs due to spatial and temporal redundancies in diffusion models.To address this problem, we present EVCtrl, a training-free plug-and-play adapter that focuses computation on spatially local control-relevant tokens through spatial Local Focused Caching and prunes redundant denoising iterations through temporal Denoising Step Skipping. Extensive results demonstrate EVCtrl’s effectiveness for efficient controllable generation under diverse conditions. Our work offers valuable insights and directions toward future efficient, potentially real-time controllable image and video generation.

## References

- Black Forest Labs [2024]
Black Forest Labs.
Flux.
[https://github.com/black-forest-labs/flux](https://github.com/black-forest-labs/flux), 2024.
- Bolya and Hoffman [2023]
Daniel Bolya and Judy Hoffman.
Token merging for fast stable diffusion.
In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pages 4599–4603, 2023.
- Bolya et al. [2022]
Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman.
Token merging: Your vit but faster.
*arXiv preprint arXiv:2210.09461*, 2022.
- Cai et al. [2025]
Shengqu Cai, Eric Ryan Chan, Yunzhi Zhang, Leonidas Guibas, Jiajun Wu, and Gordon Wetzstein.
Diffusion self-distillation for zero-shot customized image generation.
In *Proceedings of the Computer Vision and Pattern Recognition Conference*, pages 18434–18443, 2025.
- Cao et al. [2025]
Ke Cao, Jing Wang, Ao Ma, Jiasong Feng, Zhanjie Zhang, Xuanhua He, Shanyuan Liu, Bo Cheng, Dawei Leng, Yuhui Yin, et al.
Relactrl: Relevance-guided efficient control for diffusion transformers.
*arXiv preprint arXiv:2502.14377*, 2025.
- Chen et al. [2024a]
Junsong Chen, Yue Wu, Simian Luo, Enze Xie, Sayak Paul, Ping Luo, Hang Zhao, and Zhenguo Li.
Pixart-$\delta$:fast and controllable image generation with latent consistency models, 2024a.
- Chen et al. [2024b]
Qihua Chen, Yue Ma, Hongfa Wang, Junkun Yuan, Wenzhe Zhao, Qi Tian, Hongmei Wang, Shaobo Min, Qifeng Chen, and Wei Liu.
Follow-your-canvas: Higher-resolution video outpainting with extensive content generation.
*arXiv preprint arXiv:2409.01055*, 2024b.
- Chen et al. [2023]
Siran Chen, Qinglin Xu, Yue Ma, Yu Qiao, and Yali Wang.
Attentive snippet prompting for video retrieval.
*IEEE Transactions on Multimedia*, 26:4348–4359, 2023.
- Chen et al. [2024c]
Siran Chen, Yue Ma, Yu Qiao, and Yali Wang.
M-bev: Masked bev perception for robust autonomous driving.
In *Proceedings of the AAAI Conference on Artificial Intelligence*, pages 1183–1191, 2024c.
- Esser et al. [2024]
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al.
Scaling rectified flow transformers for high-resolution image synthesis.
In *Forty-first international conference on machine learning*, 2024.
- Feng et al. [2025a]
Kunyu Feng, Yue Ma, Bingyuan Wang, Chenyang Qi, Haozhe Chen, Qifeng Chen, and Zeyu Wang.
Dit4edit: Diffusion transformer for image editing.
In *Proceedings of the AAAI Conference on Artificial Intelligence*, pages 2969–2977, 2025a.
- Feng et al. [2025b]
Kunyu Feng, Yue Ma, Xinhua Zhang, Boshi Liu, Yikuang Yuluo, Yinhan Zhang, Runtao Liu, Hongyu Liu, Zhiyuan Qin, Shanhui Mo, et al.
Follow-your-instruction: A comprehensive mllm agent for world data synthesis.
*arXiv preprint arXiv:2508.05580*, 2025b.
- Gao et al. [2024]
Jiayi Gao, Kongming Liang, Tao Wei, Wei Chen, Zhanyu Ma, and Jun Guo.
Dual-prior augmented decoding network for long tail distribution in hoi detection.
In *Proceedings of the AAAI Conference on Artificial Intelligence*, pages 1806–1814, 2024.
- Heusel et al. [2017]
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.
Gans trained by a two time-scale update rule converge to a local nash equilibrium.
*Advances in neural information processing systems*, 30, 2017.
- Huang et al. [2023]
Lianghua Huang, Di Chen, Yu Liu, Yujun Shen, Deli Zhao, and Jingren Zhou.
Composer: Creative and controllable image synthesis with composable conditions.
*arXiv preprint arXiv:2302.09778*, 2023.
- Huang et al. [2024]
Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, et al.
Vbench: Comprehensive benchmark suite for video generative models.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 21807–21818, 2024.
- Kolors Team [2024]
Kolors Team.
Kolors: Effective training of diffusion model for photorealistic text-to-image synthesis.
*arXiv preprint*, 2024.
- Kong et al. [2024]
Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al.
Hunyuanvideo: A systematic framework for large video generative models.
*arXiv preprint arXiv:2412.03603*, 2024.
- Li et al. [2025]
Jiajun Li, Yue Ma, Xinyu Zhang, Qingyan Wei, Songhua Liu, and Linfeng Zhang.
Skipvar: Accelerating visual autoregressive modeling via adaptive frequency-aware skipping.
*arXiv preprint arXiv:2506.08908*, 2025.
- Li et al. [2024]
Zhimin Li, Jianwei Zhang, Qin Lin, Jiangfeng Xiong, Yanxin Long, Xinchi Deng, Yingfang Zhang, Xingchao Liu, Minbin Huang, Zedong Xiao, Dayou Chen, Jiajun He, Jiahao Li, Wenyue Li, Chen Zhang, Rongwei Quan, Jianxiang Lu, Jiabin Huang, Xiaoyan Yuan, Xiaoxiao Zheng, Yixuan Li, Jihong Zhang, Chao Zhang, Meng Chen, Jie Liu, Zheng Fang, Weiyan Wang, Jinbao Xue, Yangyu Tao, Jianchen Zhu, Kai Liu, Sihuan Lin, Yifu Sun, Yun Li, Dongdong Wang, Mingtao Chen, Zhichao Hu, Xiao Xiao, Yan Chen, Yuhong Liu, Wei Liu, Di Wang, Yong Yang, Jie Jiang, and Qinglin Lu.
Hunyuan-dit: A powerful multi-resolution diffusion transformer with fine-grained chinese understanding, 2024.
- Lin et al. [2014]
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.
Microsoft coco: Common objects in context.
In *European conference on computer vision*, pages 740–755. Springer, 2014.
- Liu et al. [2024a]
Haozhe Liu, Wentian Zhang, Jinheng Xie, Francesco Faccio, Mengmeng Xu, Tao Xiang, Mike Zheng Shou, Juan-Manuel Perez-Rua, and Jürgen Schmidhuber.
Faster diffusion via temporal attention decomposition.
*arXiv preprint arXiv:2404.02747*, 2024a.
- Liu et al. [2025]
Jiacheng Liu, Chang Zou, Yuanhuiyi Lyu, Junjie Chen, and Linfeng Zhang.
From reusing to forecasting: Accelerating diffusion models with taylorseers.
*arXiv preprint arXiv:2503.06923*, 2025.
- Liu et al. [2022]
Xingchao Liu, Chengyue Gong, and Qiang Liu.
Flow straight and fast: Learning to generate and transfer data with rectified flow.
*arXiv preprint arXiv:2209.03003*, 2022.
- Liu et al. [2024b]
Yixin Liu, Kai Zhang, Yuan Li, Zhiling Yan, Chujie Gao, Ruoxi Chen, Zhengqing Yuan, Yue Huang, Hanchi Sun, Jianfeng Gao, Lifang He, and Lichao Sun.
Sora: A review on background, technology, limitations, and opportunities of large vision models, 2024b.
- Lu et al. [2022]
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu.
Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps.
*Advances in neural information processing systems*, 35:5775–5787, 2022.
- Lu et al. [2025]
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu.
Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models.
*Machine Intelligence Research*, pages 1–22, 2025.
- Lu et al. [2023]
Shilin Lu, Yanzhu Liu, and Adams Wai-Kin Kong.
Tf-icon: Diffusion-based training-free cross-domain image composition.
In *Proceedings of the IEEE/CVF International Conference on Computer Vision*, pages 2294–2305, 2023.
- Ma et al. [2024a]
Xinyin Ma, Gongfan Fang, Michael Bi Mi, and Xinchao Wang.
Learning-to-cache: Accelerating diffusion transformer via layer caching.
*Advances in Neural Information Processing Systems*, 37:133282–133304, 2024a.
- Ma et al. [2024b]
Xinyin Ma, Gongfan Fang, and Xinchao Wang.
Deepcache: Accelerating diffusion models for free.
In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pages 15762–15772, 2024b.
- Ma et al. [2022]
Yue Ma, Yali Wang, Yue Wu, Ziyu Lyu, Siran Chen, Xiu Li, and Yu Qiao.
Visual knowledge graph for human action reasoning in videos.
In *Proceedings of the 30th ACM International Conference on Multimedia*, pages 4132–4141, 2022.
- Ma et al. [2023]
Yue Ma, Xiaodong Cun, Yingqing He, Chenyang Qi, Xintao Wang, Ying Shan, Xiu Li, and Qifeng Chen.
Magicstick: Controllable video editing via control handle transformations.
*arXiv preprint arXiv:2312.03047*, 2023.
- Ma et al. [2024c]
Yue Ma, Yingqing He, Xiaodong Cun, Xintao Wang, Siran Chen, Xiu Li, and Qifeng Chen.
Follow your pose: Pose-guided text-to-video generation using pose-free videos.
In *Proceedings of the AAAI Conference on Artificial Intelligence*, pages 4117–4125, 2024c.
- Ma et al. [2024d]
Yue Ma, Hongyu Liu, Hongfa Wang, Heng Pan, Yingqing He, Junkun Yuan, Ailing Zeng, Chengfei Cai, Heung-Yeung Shum, Wei Liu, et al.
Follow-your-emoji: Fine-controllable and expressive freestyle portrait animation.
In *SIGGRAPH Asia 2024 Conference Papers*, pages 1–12, 2024d.
- Ma et al. [2025a]
Yue Ma, Kunyu Feng, Zhongyuan Hu, Xinyu Wang, Yucheng Wang, Mingzhe Zheng, Xuanhua He, Chenyang Zhu, Hongyu Liu, Yingqing He, et al.
Controllable video generation: A survey.
*arXiv preprint arXiv:2507.16869*, 2025a.
- Ma et al. [2025b]
Yue Ma, Kunyu Feng, Xinhua Zhang, Hongyu Liu, David Junhao Zhang, Jinbo Xing, Yinhan Zhang, Ayden Yang, Zeyu Wang, and Qifeng Chen.
Follow-your-creation: Empowering 4d creation through video inpainting.
*arXiv preprint arXiv:2506.04590*, 2025b.
- Ma et al. [2025c]
Yue Ma, Yingqing He, Hongfa Wang, Andong Wang, Leqi Shen, Chenyang Qi, Jixuan Ying, Chengfei Cai, Zhifeng Li, Heung-Yeung Shum, et al.
Follow-your-click: Open-domain regional image animation via motion prompts.
In *Proceedings of the AAAI Conference on Artificial Intelligence*, pages 6018–6026, 2025c.
- Ma et al. [2025d]
Yue Ma, Yulong Liu, Qiyuan Zhu, Ayden Yang, Kunyu Feng, Xinhua Zhang, Zhifeng Li, Sirui Han, Chenyang Qi, and Qifeng Chen.
Follow-your-motion: Video motion transfer via efficient spatial-temporal decoupled finetuning.
*arXiv preprint arXiv:2506.05207*, 2025d.
- Mao et al. [2025]
Chaojie Mao, Jingfeng Zhang, Yulin Pan, Zeyinzi Jiang, Zhen Han, Yu Liu, and Jingren Zhou.
Ace++: Instruction-based image creation and editing via context-aware content filling.
*arXiv preprint arXiv:2501.02487*, 2025.
- Mou et al. [2024]
Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan.
T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models.
In *Proceedings of the AAAI conference on artificial intelligence*, pages 4296–4304, 2024.
- Pan et al. [2020]
Zexu Pan, Zhaojie Luo, Jichen Yang, and Haizhou Li.
Multi-modal attention for speech emotion recognition.
*arXiv preprint arXiv:2009.04107*, 2020.
- Selvaraju et al. [2024]
Pratheba Selvaraju, Tianyu Ding, Tianyi Chen, Ilya Zharkov, and Luming Liang.
Fora: Fast-forward caching in diffusion transformer acceleration.
*arXiv preprint arXiv:2407.01425*, 2024.
- She et al. [2025]
D She, Mushui Liu, Jingxuan Pang, Jin Wang, Zhen Yang, Wanggui He, Guanghao Zhang, Yi Wang, Qihan Huang, Haobin Tang, et al.
Customvideox: 3d reference attention driven dynamic adaptation for zero-shot customized video diffusion transformers.
*arXiv preprint arXiv:2502.06527*, 2025.
- Song et al. [2020]
Jiaming Song, Chenlin Meng, and Stefano Ermon.
Denoising diffusion implicit models.
*arXiv preprint arXiv:2010.02502*, 2020.
- Tan et al. [2024]
Zhenxiong Tan, Songhua Liu, Xingyi Yang, Qiaochu Xue, and Xinchao Wang.
Ominicontrol: Minimal and universal control for diffusion transformer.
*arXiv preprint arXiv:2411.15098*, 2024.
- Tan et al. [2025]
Zhenxiong Tan, Songhua Liu, Xingyi Yang, Qiaochu Xue, and Xinchao Wang.
Ominicontrol: Minimal and universal control for diffusion transformer.
2025.
- TheDenk [2024]
TheDenk.
Cogvideo controlnet.
[https://github.com/TheDenk/cogvideox-controlnet](https://github.com/TheDenk/cogvideox-controlnet), 2024.
- Wan et al. [2025]
Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, et al.
Wan: Open and advanced large-scale video generative models.
*arXiv preprint arXiv:2503.20314*, 2025.
- Wan et al. [2024]
Zhen Wan, Yue Ma, Chenyang Qi, Zhiheng Liu, and Tao Gui.
Unipaint: Unified space-time video inpainting via mixture-of-experts.
*arXiv preprint arXiv:2412.06340*, 2024.
- Wang et al. [2024a]
Jing Wang, Ao Ma, Jiasong Feng, Dawei Leng, Yuhui Yin, and Xiaodan Liang.
Qihoo-t2x: An efficient proxy-tokenized diffusion transformer for text-to-any-task, 2024a.
- Wang et al. [2024b]
Jiangshan Wang, Yue Ma, Jiayi Guo, Yicheng Xiao, Gao Huang, and Xiu Li.
Cove: Unleashing the diffusion feature correspondence for consistent video editing.
*arXiv preprint arXiv:2406.08850*, 2024b.
- Wang et al. [2024c]
Jiangshan Wang, Junfu Pu, Zhongang Qi, Jiayi Guo, Yue Ma, Nisha Huang, Yuxin Chen, Xiu Li, and Ying Shan.
Taming rectified flow for inversion and editing.
*arXiv preprint arXiv:2411.04746*, 2024c.
- Wang et al. [2004]
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli.
Image quality assessment: from error visibility to structural similarity.
*IEEE transactions on image processing*, 13(4):600–612, 2004.
- Xiao et al. [2023]
Yicheng Xiao, Yue Ma, Shuyan Li, Hantao Zhou, Ran Liao, and Xiu Li.
Semanticac: semantics-assisted framework for audio classification.
In *ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*, pages 1–5. IEEE, 2023.
- Xie et al. [2024]
Enze Xie, Junsong Chen, Junyu Chen, Han Cai, Haotian Tang, Yujun Lin, Zhekai Zhang, Muyang Li, Ligeng Zhu, Yao Lu, et al.
Sana: Efficient high-resolution image synthesis with linear diffusion transformers.
*arXiv preprint arXiv:2410.10629*, 2024.
- Xiong et al. [2025]
Zhen Xiong, Yuqi Li, Chuanguang Yang, Tiao Tan, Zhihong Zhu, Siyuan Li, and Yue Ma.
Enhancing image generation fidelity via progressive prompts.
In *ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*, pages 1–5. IEEE, 2025.
- XLabs-AI [2024]
XLabs-AI.
x-flux.
[https://github.com/XLabs-AI/x-flux](https://github.com/XLabs-AI/x-flux), 2024.
- Yan et al. [2025]
Zexuan Yan, Yue Ma, Chang Zou, Wenteng Chen, Qifeng Chen, and Linfeng Zhang.
Eedit: Rethinking the spatial and temporal redundancy for efficient image editing.
*arXiv preprint arXiv:2503.10270*, 2025.
- Yang et al. [2024]
Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al.
Cogvideox: Text-to-video diffusion models with an expert transformer.
*arXiv preprint arXiv:2408.06072*, 2024.
- Yuluo et al. [2025a]
Yikuang Yuluo, Yue Ma, Kuan Shen, Tongtong Jin, Wang Liao, Yangpu Ma, and Fuquan Wang.
Follow-your-shape: Shape-aware image editing via trajectory-guided region control.
*arXiv preprint arXiv:2508.08134*, 2025a.
- Yuluo et al. [2025b]
Yikuang Yuluo, Yue Ma, Kuan Shen, Tongtong Jin, Wang Liao, Yangpu Ma, and Fuquan Wang.
Gr-gaussian: Graph-based radiative gaussian splatting for sparse-view ct reconstruction.
*arXiv preprint arXiv:2508.02408*, 2025b.
- Zhang et al. [2025a]
Evelyn Zhang, Jiayi Tang, Xuefei Ning, and Linfeng Zhang.
Training-free and hardware-friendly acceleration for diffusion models via similarity-based token pruning.
In *Proceedings of the AAAI Conference on Artificial Intelligence*, pages 9878–9886, 2025a.
- Zhang et al. [2023]
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala.
Adding conditional control to text-to-image diffusion models.
In *Proceedings of the IEEE/CVF international conference on computer vision*, pages 3836–3847, 2023.
- Zhang et al. [2018]
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.
The unreasonable effectiveness of deep features as a perceptual metric.
In *Proceedings of the IEEE conference on computer vision and pattern recognition*, pages 586–595, 2018.
- Zhang et al. [2025b]
Yinhan Zhang, Yue Ma, Bingyuan Wang, Qifeng Chen, and Zeyu Wang.
Magiccolor: Multi-instance sketch colorization.
*arXiv preprint arXiv:2503.16948*, 2025b.
- Zhu et al. [2024]
Chenyang Zhu, Kai Li, Yue Ma, Longxiang Tang, Chengyu Fang, Chubin Chen, Qifeng Chen, and Xiu Li.
Instantswap: Fast customized concept swapping across sharp shape differences.
*arXiv preprint arXiv:2412.01197*, 2024.
- Zhu et al. [2025a]
Chenyang Zhu, Kai Li, Yue Ma, Chunming He, and Xiu Li.
Multibooth: Towards generating all your concepts in an image from text.
In *Proceedings of the AAAI Conference on Artificial Intelligence*, pages 10923–10931, 2025a.
- Zhu et al. [2025b]
Tianrui Zhu, Shiyi Zhang, Jiawei Shao, and Yansong Tang.
Kv-edit: Training-free image editing for precise background preservation.
*arXiv preprint arXiv:2502.17363*, 2025b.
- Zou et al. [2024a]
Chang Zou, Xuyang Liu, Ting Liu, Siteng Huang, and Linfeng Zhang.
Accelerating diffusion transformers with token-wise feature caching.
*arXiv preprint arXiv:2410.05317*, 2024a.
- Zou et al. [2024b]
Chang Zou, Evelyn Zhang, Runlin Guo, Haohang Xu, Conghui He, Xuming Hu, and Linfeng Zhang.
Accelerating diffusion transformers with dual feature caching.
*arXiv preprint arXiv:2412.18911*, 2024b.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Discussions and contributions vs. existing methods, such as ToCA and Duca are expected.
2. What are the failure cases or limitations of the proposed method?
3. Ablation studies on the temporal consistency and perceptual coherence.
soundness: 3
strengths: **Strengths:**  
- Proposes a **clear and practical approach** that effectively targets spatial and temporal redundancy in controllable diffusion models.  
- **Training-free and plug-and-play**, making it highly practical for real-world deployment and easily integrated into existing ControlNet and DiT pipelines.  
- **Comprehensive experiments** across text-to-image and text-to-video tasks with multiple baselines (Fora, ToCa, Taylorseer) strongly support the claimed acceleration and quality preservation.  
- Introduces **interpretable token-sensitivity analysis** and hierarchical feature profiling, helping justify the LFoC design.  
- Ablation studies and qualitative visualizations effectively demonstrate the contribution of each module (LFoC, DSS).
summary: **Summary:**  
This paper proposes **EVCtrl**, a lightweight, training-free control adapter designed to improve the efficiency of controllable image and video generation based on diffusion transformers (DiTs). The method addresses the significant spatial and temporal redundancies in ControlNet by introducing two components: **Local Focused Caching (LFoC)** for spatial redundancy reduction and **Denoising Step Skipping (DSS)** for temporal redundancy pruning. EVCtrl selectively recomputes tokens containing fine-grained control cues while caching redundant ones, and dynamically skips diffusion steps with high similarity between adjacent states. Experiments on multiple image and video diffusion backbones (Flux, CogVideo, Wan2.1) show 2.0×‒2.4× acceleration with minimal degradation in FID, SSIM, and LPIPS scores.
weaknesses: Weaknesses:  
1. The paper’s novelty is **incremental**, mainly improving upon existing caching or skipping techniques (e.g., ToCa, Duca) rather than proposing a fundamentally new principle.  
2. The description of **critical-step detection** in DSS lacks clarity; how “critical timesteps” are selected or tuned is somewhat underexplained.  
3. Most experiments emphasize efficiency, but the **effect on temporal consistency and perceptual coherence** (especially in long video sequences) is not sufficiently analyzed.  
4. While claiming wide applicability, **the evaluation scope is limited** to a few DiT-based models; tests on other architectures (e.g., UNet-based diffusion models) would better demonstrate universality.  
5. The method requires **careful parameter tuning** (e.g., cache ratio P, skip interval N) that may vary across conditions, yet the paper provides limited guidance on how to set these hyperparameters.

---

confidence: 4
contribution: 3
initial_rating: 4
presentation: 2
questions: 1. Please clarify the discrepancy between Figure 1 T2V latency (37.5s) and Table 1 T2V latency (38.5s). Additionally, in Table 1, all quality metrics (FID, PSNR, etc.) of the "Original" Flux-ControlNet are missing, leaving the primary claim of "no quality degradation" unsubstantiated.
2. Please clarify the relationship between the parameters $P$ and $N$ in Equation (1) and the $P$ and $N$ used in the experiments (L268-270). Re-stating their precise definition is necessary as their role is fundamental to the method.
3. Given the ambiguity of "predetermined critical steps" (Weakness 1), could the authors provide a more concrete, adaptive online selection method for key steps (e.g., based on adjacent step cosine distance or gradient norm)?
soundness: 3
strengths: 1. The motivation of this paper is clear, i.e., addressing the temporal and spatial redundancy caused by utilizing ControlNet.
2. The method is training-free and easy to implement, requiring no retraining. The LFoC component attempts to explore the internal working principles of DiT-ControlNet by analyzing the functions of different layers via L1 norm, which is a promising approach to optimize ControlNet.
3. Extensive experiments have been conducted, both qualitatively and quantitatively, consistently demonstrating the effectiveness of EVCtrl.
summary: This work proposes EVCtrl, an efficient control adapter mainly addressing the spatial redundancy and temporal redundancy issues prevalent in current image and video generation methods that leverage ControlNet. This method has been experimented on the ControlNet branches of multiple models, achieving impressive acceleration effects without significant visual degradation in generation quality.
weaknesses: 1. The DSS (temporal acceleration) mechanism is insufficient explained, lacking crucial details for reproducibility. Terms like "identified a priori" and "predetermined sequence of critical steps" are used without explaining the specific criteria for screening these steps or specifying the quantity $m$. This missing information is vital.
2. The methodology is not very novel. The core idea of DSS is similar to prior work like FORA, where FORA has already verified the effectiveness of exploiting temporal redundancy in DiT through feature caching and reuse. The innovation of DSS is an application-specific migration of this known temporal acceleration strategy to the ControlNet branch, fine-tuned with the empirical design of "critical steps." Similarly, the essence of LFoC lies in token pruning or sparse computation, an approach already established by precursors such as ToCa and Duca.
3.The expression of this paper requires improvement. Furthermore, there are several typos, such as the inconsistent spelling of "Taylorseer" and "Taylorsser" in Table 1. More errors can be found in the Questions section.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: 1. Does the local caching window of LFoC and the skipping schedule of DSS remain effective for 2K/4K images or videos longer than 100 frames?
2. When control signals are time-varying or multi-modal, how does DSS ensure consistent denoising steps without introducing flickering or artifacts?
3. Compared with distillation, quantization, or NAS-based acceleration, what is the performance-to-cost ratio and theoretical upper bound of EVCtrl?
4. Can you provide a theoretical or statistical relationship between LFoC cache-hit rate and generation quality degradation?
soundness: 3
strengths: 1. High practicality: zero training cost, plug-and-play deployment.
2. Clear redundancy modeling: separates spatial (LFoC) and temporal (DSS) redundancy.
3. Comprehensive experiments: multiple models and control conditions, significant acceleration.
4. Quality preservation: maintains visual metrics comparable to the original ControlNet under 2.16 times speed-up.
summary: This paper introduces EVCtrl, an innovative, training-free, plug-and-play acceleration framework that tackles the low efficiency and high computational cost of ControlNet-based controllable image/video generation. By exploiting spatial redundancy via Local-Focused Caching (LFoC) and temporal redundancy via Denoising-Step Skipping (DSS), EVCtrl achieves 2.16 times speed-up on CogVideo-ControlNet without sacrificing quality. Extensive experiments on Flux, CogVideo, and Wan2.1 under four control conditions show superior SSIM, PSNR, LPIPS, and lower latency compared with existing training-free acceleration baselines. However, performance under high-resolution, long-video, or complex control scenarios has not been examined, and comparisons with non-ControlNet controllers are missing; consequently, the generality of the approach remains insufficiently validated. Although the novelty is somewhat conservative and lacks theoretical depth, the paper is of good overall quality. Strengthening theoretical analysis and presentation in the final version is expected to further increase its impact.
weaknesses: 1. Limited generalization: no tests on high-resolution, long-video, or complex control scenarios.
2. Narrow comparison: only evaluated against training-free acceleration baselines; Insufficient comparison with other types of controllers or acceleration methods.
3. Shallow theoretical analysis: lacks formal discussion on why and when LFoC and DSS work or fail.

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 2
questions: - It would be helpful to prove method significance to extend the evaluation to commonly used dense conditioning modalities such as depth, and show that the proposed method can compare favorably to baselines.
- Could the authors include the original control net model in tab 2 to serve as an upper bound?
- Please specify the unit of measure on the X axis for plots in Figure 1?
soundness: 3
strengths: - The method achieves higher speedup and better metrics with respect to the chosen baselines
- The small amount of showcased qualitative results shows some cases where the method is successful while baselines are failing.
- The analysis of token magnitudes, correlating high magnitude tokens to contour regions is interesting
summary: The authors propose EVCtrl, a training free method for speeding up inference of ControlNet based models. The method is based on the observation that when using sparse conditioning modalities mostly consisting of black pixels, only tokens corresponding to non-black regions are critical for updating. In addition, authors identify precise steps along the denoising trajectory where large changes happen inside the model, requiring full recomputation of features to preserve performance. Evaluation on CogVideo, Flux and Wan show positive performance and better or comparable speedups with respect to the baselines on the chosen metrics.
weaknesses: - The amount of provided qualitative results in the form of videos is limited to 6 samples in the supplementary power point presentation, making it difficult to qualitatively judge the quality of the method.
- The method relies on "Local Focused Caching (LFoC) to achieve its speedup and performance. LFoC relies on the assumption that control signals are sparse and mostly consisting of spatially uninformative regions (e.g. poses, edges). Usage of dense control signals such as depth maps should be considered and discussed, as it is of practical interest but might cause failure of the method. It is currently unclear how the method would perform on these control signals with respect to baselines.
- The method relies on the identification of critical steps, and on the identification of global and local model regions. The procedure needs be repeated for each method, requiring some effort from the user for correct application. The paper provides limited guidance on how this can be achieved in 3.3.2, which might make the presented method difficult to apply in practice.
- LL150-151 only present the case of retraining a large model from scratch, or use of control nets. Recent methods such as VACE achieve strong controllability through model fine-tuning. Several recent methods showcase strong conditioning capabilities through model fine-tuning, reducing significance of the work which is tailored to the specific case of ControlNets.
- Quality of writing is problematic with several typos that disturb reading, e.g. method names fused with author names, missing spaces, incorrect punctuation LL79, 81, 84, 105, 118, 119, 121, 170, 171, 172, 318, 322, 371, 372 ...