Fast and Memory-Efficient Video Diffusion Using Streamlined Inference
Authors: Zheng Zhan
Yushu Wu
Yifan Gong
Zichong Meng
Zhenglun Kong
Changdi Yang
Geng Yuan
Pu Zhao
Wei Niu
Yanzhi Wang
Venue: NeurIPS 2024 poster (2024)
Keywords: video diffusion models
training free
inference framework
Efficiency
Primary area: diffusion_based_models

Abstract: The rapid progress in artificial intelligence-generated content (AIGC), especially with diffusion models, has significantly advanced development of high-quality video generation. However, current video diffusion models exhibit demanding computational requirements and high peak memory usage, especially for generating longer and higher-resolution videos. These limitations greatly hinder the practical application of video diffusion models on standard hardware platforms. To tackle this issue, we present a novel, training-free framework named Streamlined Inference, which leverages the temporal and spatial properties of video diffusion models. Our approach integrates three core components: Feature Slicer, Operator Grouping, and Step Rehash. Specifically, Feature Slicer effectively partitions input features into sub-features and Operator Grouping processes each sub-feature with a group of consecutive operators, resulting in significant memory reduction without sacrificing the quality or speed. Step Rehash further exploits the similarity between adjacent steps in diffusion, and accelerates inference through skipping unnecessary steps. Extensive experiments demonstrate that our approach significantly reduces peak memory and computational overhead, making it feasible to generate high-quality videos on a single consumer GPU (e.g., reducing peak memory of Animatediff from 42GB to 11GB, featuring faster inference on 2080Ti).

Fast and Memory-Efficient Video Diffusion Using Streamlined Inference
Zheng Zhan'
Yushu Wu
Yifan Gong'
Zichong Meng' Zhenglun Kong" 12
Changdi Yang'
Geng Yuan
Pu Zhao
Wei Niu
Yanzhi Wang'
Northeastern University
HHarvard University
Uniiersity of Georgia
512x768 resolution
Animatediff Peak Mem: 29.7G Latency: 16.9s
Ours Peak Mem: 9.08G (3.3x less)
Latency: 10.7s (1.6x less)
full-sized man rides comically small motoraycle through residential neigh borhood
Figure 1: Our Streamlined Inference is training-free inference framework for video diffusion models that can reduce the computation and peak memory cost without sacrificing the quality.
Abstract
The rapid progress artificial intelligence-generane content (AIGC), especially with diffusion models, has significantly advanced development of high-quality video generation. However, current video diffusion models exhibit demanding computational requirements and high peak memory usage, especially for generating longer and higher-resolution videos. These limitations greatly hinder the practical application of video diffusion models standard hardware platforms. To tackle this issue, we present a novel, training-fiee free framework named Streamlined Infer- ence, which leverages the temporal and spatial properties of video diffusion models. Our approach integrates three core components: Feature Slicer, Operator Grouping, and Step Rehash. Specifically, Feature Slicer effectively partitions input features into sub-features and Operator Grouping processes each sub-feature with group consecutive operators, resulting significant memory reduction without sacrificing the quality or speed. Step Rehash further exploits the similarity between adjacent steps diffusion, and accelerates inference through skipping unnecessary steps. Extensive experiments demonstrate that our approach significantly reduces peak memory and computational overhead, making feasible to generate high-qualit videos on single consumer GPU (e.g., reducing peak memory of AnimateDiff from 42GB 11GB, featuring faster inference on 2080Ti
Introduction
Recent years have witnessed continual progress and advancements in artificial intelligence-genee -egeeertteeeee content (AIGC). Among them, diffusion models allow artists and amateurs create visual content
"Equal contributions.
I Corresponding author
"Code available at: https:://itthu: htt.tprr//.tttt..coo/ooosusutttt
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Baseline
With ours
Constraint
Time SVD
1024 video
40
41.7
40
39.3
Original
Ours
1.63x Less
Peek Memoryy (CB)
30
30G
30
Less
10
20
30
4090
20
20
Time for AnimateDiff to generate en 576 x 1024 video
2080TI
Original
1.61x Less
20
AnimateDiff 576#1024
SVD 576x1024
Peak Memory
Latency
Figure 2: Comparison on Animatediff and SVD inference using our Streamlined Inference. Memory requirement is crucial as "Out of Memory" enrors prevent the GPU from performing inference.
with text prompts, advancing the development of image and video generation in both academia and industry. For video diffusion models, the latest works such as SVD-XT [3), Gen2 [29]. Pika [20]. and notably the more advanced Sora (28), demonstrate impressive capabilities producing visually striking and artistically effective videos. Despite their great performance, video diffusion models also exhibit high computational requirements and substantial peak memory, particularly when generating longer videos with higher resolutions. For instance, SVD-XT generates 25 frames simultancously with resolution of 576 x 1024, while Sora expands these capabilities by supporting the generation of longer videos (over minute) at higher resolution of 1080 1920. Given the trends of generating longer videos with higher quality, the escalating memory and computation demands have impeded practical applications of these large-scale video diffusion models various platforms.
Existing model compression methods reduce peak memory and latency, such as weight pruning [8
12
typically require substantial retraining or fine-tuning the compressed model to recover performance. This process is costly, time-consuming and may raise data privacy concerns. Applying these methods in zero-shot avoids the expensive retraining, but leads to severe performance degradation. Furthermore, the variety and complexity of video diffusion architectures further complicate the model optimization. Therefore, it is challenging yet crucial to develop an effective and efficient video diffusion framework with reduced computations, smaller peak memory and less data (no re-training) requirements for its wide applications.
35
48
23
quantization [37 22 32), and distillation [18, TO
43 39
To address the above challenges, we first identify the sources of the high computation and memory cost, which scale up with the iterative denoising process and the simultaneous processing of multiple frames. We further observe that the feature maps of certain layers may exhibit high similarity between multiple consecutive denoising steps due to the temporal property of videos, enabling further optimizations for acceleration. Based on that, we propose training- -fre framework named Streamlined Inference, by leveraging the temporal and spatial characteristics of vidco diffusion models to effectively reduce peak memory and computational demands. Our framework contains three core components: Feature Slicer, Operator Grouping, and Step Rehash, which work together closely and comprehensively with different focuses peak memory reduction inference acceleration.
Our Feature Slicer performs lossless feature slicing in both temporal and spatial layers, raising the possibility of peak memory reduction through processing smaller features. However, the feature slicer alone is not able to decrease peak memory as we still need to store all intermediate results of one layer for all sliced features to form a complete intermediate feature map for the next layer. To reduce peak memory practically, we further propose Operator Grouping to group homogeneous and consecutive operators in the computational graph. Within each operator group, the intermediate result of one sliced feature can be directly sent to the next operator/layer without waiting for aggregation with all other intermediate results, achieving the full potential of Feature Slicer to reduce the peak memory. Furthermore, a pipeline technique is proposed to accelerate the computations of the same operator group for multiple sliced feature inputs, with improved parallelism.
Moreover, observing the high similarity of certain features between multiple consecutive denoising steps, we propose Step Rehash to reuse the generated features for a few following steps due to their high similarity, skipping the exact expansive and repetitive generation of similar features and thereby accelcrating the video diffusion significantly. With this framework, we can generate high-quality

---

videos in a fast and memory-effiiiee manner on single consumer GPU, as shown in Fig. 2 For example, the peak memory of AnimateDiff [[]] can be reduced significantly from 41.7GB IIGB, featuring inference on a typical consumer GPU 2080Ti. We summarize our contributions as follows:
We propose a novel training- free framework that can significantly reduce the peak memory and computation cost for the inference of video diffusion models by leveraging the spatial and temporal characteristics of video diffusion models.
Our approach can be seamlessly integrated into existing video diffusion models. Our extensive experiments on SVD, SVD-XT, and AnimateDiff demonstrate our effectiveness to reduce peak memory and accelerate inference without sacrificing quality.
Our approach offers new research perspective for fast and memory- -eficcee video diffusion, enabling the generation of higher quality and longer videos on consumer-gras -gaade GPUs.
2
Related Work
Video Diffuison Models. For video generation, various approaches have been proposed, with VDM (17) as a leading example. VDM transforms the conventional U-Net (30) architecture of image diffusion models into a 3D U-Net structure, employing joint training on both images and videos. Vidio [49] is the first work that introduces Latent Diffusion Model (LDM) for text-to-video (T2V) generation in latent space. LVDM [3] introduces a mask sampling technique that enhances its longer video generation capability. ModelScope 36 incorporates spatial-temporal convolution and attention into LDM. Video LDM 4 trains a T2V network composed of three training stages, enabling higher quality and longer video generation. Show-1 [45] first introduces the fusion of pixel-based and latent-based diffusion models for T2V generation. Recently, Stable Video Diffusion (SVD) 3 identifies three key stages for training video LDMs: text-to-image (T2I) pretraining, video pretraining, and high-quality video finetuning.
Architectural Efficiency of Video Diffusion Models. There are various research efforts exploring either architectural efficiency or model compression techniques for image/video generation. For example, ED-T2V
24 freezes parameters to reduce training costs and proposes attention mecha-
nism to ensure temporal coherence. SimDA 400 devises a parameter-efficie maintaining the parameter of the T2I model and uses two adapters to train it. For model compression, Diff-pruning
training approach by
[6] employs structural pruning techniques to reduce inference time at each sampling step. Additionally, the work (22 implements quantization on diffusion models using low-precision data types. However, these methods take substantial efforts to retrain or finetune the diffusion model recover performance, which is costly, time-consuming. and may raise data privacy concerns. Furthermore, applying post-training compression techniques in one-shot [8, 7 334 may save the retraining/fine-tuiing efforts, but suffers from significant performance degradation.
Sample-Eficiint Video Diffusion Models. To address the iterative denoising process in diffusion models and improve the sampling efficiency, two approaches proposed. The first approach [2] focuses on creating rapid solvers to resolve the differential equation associated with the
19 3 25
denoising process more effectively. The works Bi 27 21 utilize knowledge distillation methods to compress and simplify the sampling trajectory efficiently, thereby enhancing overall performance. Imagen video [15] is of the first methods to apply progressive distillation video diffusion models, incorporating guidance and stochastic samplers. Recent work Deepcache (26) proposes a novel training-free paradigm that accelerates diffusion models by reusing the high-level features.
Motivation
Peak memory and computation analysis. Existing open-source video diffusion models || 36 46) typically adopt a pretrained T2I 2D-UNet as backbone. Their temporal layers are seamlessly integrated into the backbone 2D-UNet, positioned after every spatial layer. Here, we use SVD as an example to demonstrate how peak memory and computational overhead scale with the number of frames. The SVD model is trained with two distinct configurations: regular SVD designed to generate 14 frames, while SVD-XT is tailored to produce 25 frames. To generate 14 or 25 video frames concurrently with SVD, its latent features require massive GPU memory and computation consumption, estimated to be approximately 14x or 25x higher than its base T2I model. This
3
3

---

Ours
Naive
Figure 3: The quality results of our method and naive slicing. Note that naïve slicing will incur unpleasant artifacts due to lack of temporal correction by fewer frames.
estimate does not account for the additional memory required by SVD's extra-temporal layers. More specifically, the SVD consumes 39.49G of peak memory for 576 x 1024 resolution output, whereas its image generation counterpart only requires 6.33G of memory at the same resolution. Furthermore, incorporating the classifier-free guidance [16] substantially enhances the generation quality but doubles the peak memory required during inference.
Consequently, video diffusion computationally demanding, but the challenge of memory con- sumption is more critical and demands immediate attention. Most consumer-grade GPUs do not have enough memory for video diffusion models and, therefore, suffer from the "Out of Memory" error, which prevents the GPU from generating high-quality videos. There is workaround without switching GPUs. Most users have to endure generating short and low-resolution videos.
Naïve Slicing. A Naive approach to reduce peak memory is to execute the video diffusion inference clip-by-clip. However, this strategy is hindered by the temporal layers, which are essential for maintaining temporal correlation in video diffusion models. Forcibly implementing this approach can generate random artifacts and cause motion vanishing in the output video, as detailed Fig. B Therefore, designing memory-efficient inference framework is challenging and non-trivial task.
4 Streamlined Inference Framework
To address the above massive peak memory and computation costs, in this section, we propose training-free framework named Streamlined Inference, which is composed of three core components: Feature Slicer, Operator Grouping, and Step Rehash. First, we discuss Feature Slicer, designed to partition input features of spatial and temporal layers, and enable the potential of massive peak memory reduction. Next, we introduce our Operator Grouping technique aggregate homogeneous and consecutive operators into the same group, achieving the full potential of Feature Slicer to reduce peak memory through reusing the memory of intermediate result from previous sliced feature. Finally, we discuss our Step Rehash method to reuse the same feature for few consecutive steps due to their high similarity. It accelerates the inference without increasing peak memory overhead it skips certain denoising steps with less computations.
4.1 Feature Slicer
Video diffusion models contain spatial and temporal layers which extract the corresponding infor- mation from their specific domains. On this basis, we propose feature slicer that consists of two components: Spatial-layer slicer and Temporal-layer slicer, to divide the feature map into multiple batches/sub-featuree, ensuring accurate computation without introducing additional operations. The slicer is further utilized for Operator Grouping to reduce peak memory cost.
Spatial layers slicer. Based our profiling (more details can be found in Appendix A for memory allocation of various video diffusion models, we find that performing slices at spatial layers can greatly reduce the memory footprint. The 5-D feature in the spatial layer X E x can be reshaped to 4-D feature X E Xx(BxT)xCxxHx> X where B, T, C. H, W are the batch size, number of frames, channels, height, and width, respectively. Thus, we slice into & sub-features, B x T/k] with [-] denoting the least integer greater than or equal to the input. If B T/k # B T/k, k. the dimension of the last sub-feature ny is different
(X: E IB=xXxxXXV H ii=1' where

---

Exec.
Done
13
14
15
(a) Original
Silcing
Grouping
(b) Operator Grouping with Feature sllcing
Pipeline
Figure 4: Overview of Operator Grouping with Pipeline our framework.
from others. The spatial layer slicer is applicable for most operations in spatial layers such as Conv2D, GroupNorm, LayerNorm, Attention, and Linear.
Temporal layers slicer. The input the temporal layer is a 5-D feature map with dimensions (butch, channels, frames, height, width). 3-D operations such as Conv3D are employed to extract temporal information from the 5-D feature. Differing from spatial layers, slicing along the temporal dimension may disrupt the extraction and processing of temporal information. Therefore, the temporal dimension untouched while slicing over other dimensions. Specifically, the 5-D feature j=k X RR. xxW can be sliced to k4 X kw sub-features (Xi, E RR Bxxxxxxxxxxxxxxx i=1-==1 where h; = [H/Ak, and w = [ After detailed profiling different configurations for temporal layer slicer, we discover that the configuration with k; max (H, 16) and k. max (W, 16) can result in promising peak memory reduction.
keep
4.2 Operator Grouping
Although Feature Slicer converts the original feature map into multiple smaller sub-features with reduced memory footprint, the peak memory can not be reduced since the intermediate results of multiple sliced features require re re-consolidation to send to the next layer/operator inputs. It still needs to store all intermediate outputs of sliced features to form the united/unsliced intermediate feature map without actual peak memory reduction. Therefore, to address this problem, propose Operator Grouping to group the operators accordingly in the computational graph, achieving the full potential of Feature Slicer with effective peak memory reduction due to less memory reserved for intermediate results. Furthermore, a pipeline technique is proposed to optimize the inference of operator groups with improved parallelism and practical acceleration.
4.2.1 Grouping Operators for Peak Memory Reduction
Operator Grouping directly re-uses existing operators by aggregating consecutive homogeneous operators into the group. Homogeneous operators indicate these operators extract features from coherent domains and dimensions. In video diffusion models, different operators can be grouped into GroupOPt (temporal operator groups) and GroupOP* (spatial operators groups) ensure the well-preserved semantics of sliced sub-features within each group. For example, in the SVD Model [3], consecutive GroupNorm, Conv2D, SiLU, and htp/DonSSample operators in the Spatial ResBlock can be aggregated to one group, as these operators all extract features from spatial domain and are deemed homogeneous. As shown in Fig. 4 when computing the output feature X" for an operator group (GroupOP), the input feature X is sliced into multiple sub-features X1, .e...., Xx with Feature Slicer. Each sub-feature X, goes through the operator group and their outputs are concatenated after all outputs are available, as shown in Eq. (L).
X Concat (GroupOP(X,)) GroupOP (X2),... GroupOP(X:))
(I)
where k is the number of slices, and Concat the concatenation operation.
Reducing peak memory cost. As shown in Fig. 4 the peak memory with the operator group determined by the memory footprint of the input feature, the output feature, and the intermediate
5

---

results. Without operator grouping, all intermediate results of all operators for sliced sub-features will allocate their own memory, hence failing to reduce memory consumption. Compared with the case above, grouped operators only need to allocate memory for intermediate results of a single sliced sub-feature and the final outputs, without the necessity to allocate full intermediate features corresponding to the original unsliced input feature, as shown in Fig. (a) and (b). Operator Grouping can effectively reduce peak memory cost, enabling successful inference of video diffusion models on one single consumer commercial GPU with low moderate available memory, as shown in Tab. [
Mitigating I/O intensity. As the original feature map is sliced into multiple sub-features to reduce peak memory cost, the computation may slow down due to multiple iterations corresponding to multi- ple inputs. However, we surprisingly observe that even with the naive basic for-loop implementation for each sub-feature as shown in Fig. (b), the overall runtime with Operator Grouping is around 10% slower than that of the original unsliced version. The marginal increase in runtime can be attributed to the memory bound of the GPU for video diffusion inference. Specifically, current video diffusion model inference suffers from the memory bound, where the I/O overhead of intermediate results is more notorious than their computation workload. The slicer provides solution to mitigate the I/O burden, thus balancing the computation and memory read/write to fully utilize the GPU capacity.
4.2.2 Pipelining with Improved Parallelism and Practical Acceleration.
With the proposed Feature Slicer and Operator Grouping, the peak memory will decrease significantly with marginal increase for the computation runtime (based on the basic for-loop implementation). With deeper investigation for the computation patterns, we find that the for-loop implementation cannot maximize GPU parallelism, and further employ the pipelining technique to optimize the for-loop implementation for faster inference without additional memory cost.
With Operator Grouping, there are multiple operators in one group to process one sliced sub- feature sequentially. With the naive for-lopp implementation, before feeding each sliced sub-feature into another operator group, it needs to wait until the last sub-feature finishes its computation within the group. The parallelism can be further improved with the proposed pipeline method. Specifically, in an operator group, after a sliced feature map is computed by the out-of-place computation operator (e.g., Conv, GroupNorm, Attention, etc.) and sent the next operator, its previous allocated memory is no longer required, but it is still reserved during inference, leading to resource waste. We can pipeline all operators in the same group to mitigate this issue. As shown in Fig. 4 c). once the Conv operator completes processing a sliced feature X; as described in Eq. T and its outputs are sent to the next operator Norm, the subsequent sliced feature X immediately piped into the same Conv operator, reusing the reserved memory of X₃. In this way, multiple operators are executed simultaneously with improved parallelism. No additional memory is required, as we only make use of previously reserved memory.
Acceleration performance. With the naive for-loop implementation, only one operator in an operator group is executed at a time. However, our pipeline method can simultancously execute multiple operators (such as Conv, Norm, and Activation as depicted in Fig. (c)) without incurring additional memory. Consequently, the inference speed can be further improved. Accordingly, integrating the pipeline within Operator Grouping can mitigate 10% speed degradation caused by feature slicing.
4.3
Step Rehash
In this section, we further introduce our step rehash method to optimize the iterative denoising steps for effective acceleration in video diffusion generation. Capitalized on the high similarity between adjacent steps. our approach accelerates the video generation, while ensuring both high quality and temporal consistency across video frames, without extra memory cost. Next we first discuss our observations for the high feature similarity and then explain details of our step rehash.
4.3.1 Similarity of Temporal Features between Steps
Similarity visualization. The denoising process of U-Net in diffusion models requires multiple steps and the features of different steps may share certain similarities with minor differences [26]. To explore this, we analyze the feature maps averaged over multiple images at different parts of the model and plot the similarity between features of different steps, with an example shown in Fig. 5
(and more results and details demonstrated in Appendix C). We find two key insights below:

---

The similarity between adjacent steps significantly depends on certain blocks and layers, and may change sharply after specific operations in video diffusion. The features do not always show high similarity. For example, neither deeper layers within the same block nor those in middle blocks consistently show higher similarity between adjacent steps.
The features between adjacent steps following the temporal layers and spatial layers in video diffusion usually exhibit remarkably higher similarity compared to outputs of other layers.
High similarity after temporal layers. Ex- isting video diffusion models typically em- ploy pretrained image diffusion models as their backbone. While these image mod- els are trained to produce a variety of im- ages, the addition of temporal layers is de- signed to improve the temporal continuity of latent features. This enhancement signif- icantly strengthens their correlation, thereby increasing similarity among the features.
20-
20-
Timestess
Similrry
15
Timest
10-
5-
0
Timestops
Timesteps
(a) SVD
(b) AnimateDiff
Figure 5: The high similarity of output features after temporal layers in U3 between each timestep.
Motivation and challenges for step rehash. Due to the high similarity between features of different steps, we propose the step rehash method to skip the computation of certain features by reusing previous generated features. However, we need to address the challenges of when and where to skip. Specifically, based the above insights, simply reusing features from deeper layers does not guarantee better results since deeper layers may not show high similarity. We need to carefully choose what layers blocks be skipped (where to skip) to make use of high similarities without significantly downgrading the generation performance. As shown in fig. [5] it exhibits high similarity between adjacent time steps, but the similarity pattern differs between video diffusion models. Thus, we need to determine which steps can use skip strategy (when to skip), and the remaining steps that require full computation are full computation steps.
4.3.2 Step Rehash
Xs
Xs+1
output features of temporal layer
Here we specify the details of our step rehash. The video diffusion models typically use U-Net architecture with 4 down-sampling and sp-sampling blocks, and their output features can be represented by Dā-, and Ug.3. respectively, with s denoting the current step number as shown in Fig. 6 Typically, Ug is obtained by feeding Dg and U into the 8th (b > 0) up-sampling block, and Uğ is the final output of the 8th step. Based on similarity analysis, in the next step + 1, we can directly reuse the output features of the temporal layer from the previous step without actual exact computations. Our insights into the similarity indicate that deeper and middle blocks do not consistently demonstrate high similarity. Reusing their features results in significant degradation of generation quality. There- fore, we rehash features of the temporal layer in the final up-sampling block. Specifically, to obtain U3+1 for step 8 1, feed the output features of the temporal layer from Uğ (current full compu- tation step) into the final up-sampling block. Note that we only compute part of U3++1 and do not need compute D for concatenation, since our reused temporal layer is deeper than the concat operator for features from D as shown in Fig. b We further propose a step search algorithm to solve the when to skip problem, algorithm details can be found in Appendix B
12g
0
D2
02
0
Figure Illustration of Step Rehash. Computation in grey areas are skipped.
Experimental results
5.1 Models, Datasets and Evaluation Metrics
We conduct the experiments on representative video diffusion models, including SVD 33 SVD-XT 3. and AnimateDiff [I].. For evaluation, we use the following evaluation protocols: The first frame of the video clips are extracted as the image condition for image-to-vide generation and their captions are considered as the prompts. All experiments are conducted on a NVIDIA A100 GPU.
7

---

Table I: Comparison of our Streamlined Inference with baseline methods in video visual quality (on UCFI01), PM (Peak Memory), and latency (measured with 50 runs with the average value).
512 x 512
576
1024
Model
Method
FVD1
CLIP-Scoret
Latency
PM
Latency
Original Naive Slicing
307.7
29.25
20.91G
10.23s
39.49G
23.29s
SVD #F=14
1127.5
26.32
8.12G
31.85s
10.72G
65,56s
Ours
340.6
28.98
13.67G
7.36s
23,42G
14.24s
Original 387.9
28.18
31,97G
17.05s
61.17G
40.77s
SVD-XT #F=25
Naive Slicing
2180.0
24.42
8.12G
59.86s
10.72G
121.82s
Ours 424.7
27.94
19.37G
12.10s
36.32G
25,47s
Original
758.7
28.89
21.83G
9.65s
41,71G
24.38s
AnimateDiff #F=16
Naive Slicing
2403.9
26.63
7.22G
19.98s
9.92G
38.69s
Ours
784.5
28.71
7.51G
7.08s
11,07G
15.15s
We sample clips from each categories of UCF-101 dataset, and gather subset with 1,000 video clips for evaluation. Their action categories are considered as their captions. For SVD and SVD-XT, our samples are generated at a resolution of 576 x 1024 (14 frames for SVD) and 25 frames for SVD-XT) and then resize to 240 x 320. For AnimateDiff, we generate samples with resolution 512 512 (16 frames).
Zero-shot UCF-101 [33]
Zero-shot MSR-VTT [1]: We generated video sample for each of the 9,940 development prompts. The samples are at resolution 320 576 then resized 240 x 426 for all models with different number of generated frames.
Metrics: We compute the FVD as outlined in (9] and CLIP-Score 14 using TetchMetrics measure the performance of generated samples.
Baseline: We use pretained weight for SVD (I2V) and AnimateDiff (T2V). We compare the pro- posed Streamlined Inference (use 13 full computation steps) with the original inference (use 25 full computation steps) and naïve slicing inference mentioned in Sec More specifically, for image-conditioned SVD model, we set each naïve slice with a frame size of and use the last frames of each generated slice as the image condition for the next slice. For AnimateDiff, we evenly generate 4 slices with a frame size of 4, then combine them into a full video clip.
5.2 Quantitative Evaluation
The results from Table [ demonstrate the effectiivnness of our proposed method in managing memory, computational resources, and performance. Our method significantly reduced peak memory and latency while maintaining competitive FVD and CLIP-Score values across all three models and resolutions compared to the original method. For SVD, our method achieved a notable reduction in peak memory and latency while maintaining competitive FVD and CLIP-Score, unlike Naive Slicing, which increased FVD and latency. For SVD-XT, our method improved over Naïve Slicing and balanced resource usage and performance. For AnimateDiff, our method significantly outperformed Naïve Slicing in FVD and latency, achieving nearly the same performance as the original method but with smaller latency and around 70% reduction peak memory.
5.3 Ablation Study
Our ablation study in Table demonstrates that our Step Rehash method consistently outperforms DeepCache with the same number of full computation steps. Step Rehash skips more computa- tions than DeepCache. For the SVD model, our method maintains competitive CLIP-Scores while slightly increasing FVD compared the original method (FVD of 307.7 and CLIP-Score of 29.25 on UCF101). DeepCache performs poorly. increasing FVD and reducing video quality. For the AnimateDiff model, our method maintains stable FVD (603.9 vs. 607.13) and CLIP-Score (29.29 vs. 29, 99,40) on MSR-VT VTT compared to the original method. DeepCache shows the worst performance on UCF101, with higher FVD and lower CL.P-Scores. Visual comparisons of our method with DeepCache are provided in Appendix D

---

Table 2: Ablation study of proposed method compared with DeepCache in video visual quality. Both our Step Rehash and DeepCache involve 13 full computation steps.
UCFI01
MSR-VI VTT
Model
Method
FVD!
CLIP-
FVD.
CLIP CLIP-Scoref
Original
307.7
29.25
373.6
26.06
SVD
DeepCache
394.0
28.57
463.6
25.30
Step Rehash
340.6
28.98
402.1
25.86
Original
758.7
28.89
607.1
29.40
AnimateDiff
DeepCache
840.2
28.15
615.8
29.06
Step Rehash
784.5
28.71
603,9
29.29
AnimateDiff
Ours
deseription of an owl and danger
AnimateDiff
Ours
boy and a girl walking along
SVD-XT
Ours
Figure 7: Quality evaluation of using our method baseline models. The results show that method can be generally applied to various video diffusion models and achieve competitive results.
5.4 Quality results
Fig. and Appendix B we present qualitative results comparing our method to the original model without using Streamlined Inference. We can see that our method produces vivid and high-quality samples aligned with the text descriptions. More importantly, these results demonstrate that our method can significantly reduce peak memory and computation without sacrificing quality.
6 Conclusion and Limitation
In this paper, propose novel training-free framework that significantly reduces peak memory and computation costs for video diffusion model inference by leveraging spatial and temporal charac- teristics. Our approach can be seamlessly integrated into existing models. Extensive experiments on

---

SVD, SVD-XT, and AnimateDiff demonstrate our method's effectiveness in reducing peak memory and accelerating inference without sacrificing quality. Our approach offers new perspective for fast, memory-efficied video diffusion, enabling the generation of higher quality and longer videos on consumer-ggee grrad GPUs. Though our method is general, the efficiency limited by baseline model architecture design.
Acknowledgments and Disclosure of Funding
This work is supported by National Science Foundation CNS-2312158. We would like to express our sincere gratitude to the reviewers for their invaluable feedback and constructive comments to improve the paper.
References
[1] Torchmetrics,
clip_score html.
[2] Fan Bao, Chongxuan Li, Jun Zhu, and Bo Zhang. Analytic-dpm: an analytic estimate of the optimal reverse variance in diffusion probabilistic models. arXiv preprint arXiv:2201.066 2022.
[3] Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Do- minik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127 2023.
[4] Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22563-22575, 2023.
[5] Tim Dockhorn, Arash Vahdat, and Karsten Kreis. Genie: Higher-order denoising diffusion solvers. Advances in Neural Information Processing Systems, 35:30150-30166 2022.
[6] Gongfan Fang, Xinyin Ma, and Xinchao Wang. Structural pruning for diffusion models. In Advances in Neural Information Processing Systems, 2023.
[7] Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned one-shot. In International Conference on Machine Learning, pages 00323-10337. PMLR, 2023.
[8] Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training compression for generative pretrained transformers. arXiv preprint arXiv:2210. 17323, 1, 2022.
[9] songwei Ge, Aniruddha Mahapatra, Gaurav Parmar, Jun-Yan Zhu, and Jia-Bin Huang. On the content bias in fréchet video distance. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2024.
[10] Yifan Gong, Zheng Zhan, Qing Jin, Yanyu Li, Yerlan Idelbayev, Xian Liu, Andrey Zharkov, Kfir Aberman, Sergey Tulyakov, Yanzhi Wang, al. E2gan: Effciiett training of efficient gans for image-to-mmage translation. arXiv preprint 4rXiv:2401.06127, 2024.
[11] Yuwei Guo, Ceyuan Yang, Anyi Rao, Yaohui Wang, Yu Qiao, Dahua Lin, and Bo Dai. Ani- matediff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint arXiv:2307.00472, 2023.
[12] Yang He. Ping Liu, Ziwei Wang. Zhilan Hu, and Yi Yang. Filter pruning via geometric median for deep convolutional neural networks acceleration. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[13] Yingqing He, Tianyu Yang, Yong Zhang, Ying Shan, and Qifeng Chen. Latent video diffusion models for high-fidelip long video generation. arXiv preprint arXiv:2211.132211 2022.
10

---

[14] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7514-7524 2021.
[15] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, al. Imagen video: High definition video generation with diffusion models. arXiv preprini arXiv:2210.0200
2022.
[16] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurlPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021.
[17] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. Advances in Neural Information Processing Systems, 35:8633-8646, 2022.
[18] Minguk Kang, Richard Zhang, Connelly Barnes, Sylvain Paris, Suha Kwak, Jaesik Park, Eli Shechtman, Jun-Yau Yan Zhu, and Taesung Park. Distilling diffusion models into conditional gans. arXiv preprint arXiv:2405.059677, 2024.
[19] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. Advances in Neural Information Processing Systems, 35:26565-2653 2022.
[20] Pika Labs. Pika labs ai text video generator, 2023.
[21] Muyang Li, Ji Lin, Chenlin Meng, Stefano Ermon, Song Han, and Yun-Yan Zhu. Effi- cient spatially sparse inference for conditional gans and diffusion models. arXiv preprint arXiv:221 1. 1.02048, 2022.
[22] Xiuyu Li, Yijiang Liu, Long Lian, Huanrui Yang. Zhen Dong, Daniel Kang, Shanghang Zhang, and Kurt Keutzer. Q-diffusion: Quantizing diffusion models. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 17535-17545, 2023.
[23] Yanyu Li, Pu Zhao, Geng Yuan, Xue Lin, Yanzhi Wang, and Xin Chen. Pruning-as-search: Efficient neural architecture search via channel pruning and structural reparameterization. arXiv preprint arXiv:2206.01 01 198, 2022.
[24] Jiawei Liu, Weining Wang, Wei Liu, Qian He, and Jing Liu, Ed-t2v: An efficient training framework for diffusion- text-to-video generation. In 2023 International Joint Conference on Neural Networks (IJCNN), pages 1-8, 2023.
[25] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 35:5775-5787, 2022.
[26] Xinyin Ma, Gongfan Fang, and Xinchao Wang. Deepeache: Accelerating diffusion models for free. In The IEEE/CVF Conference Computer Vision and Pattern Recognition, 2024.
[27] Chenlin Meng, Ruiqi Gao, Diederik P Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. On distillation of guided diffusion models. arXiv preprint 4rX11:2210.03142, 2022.
[28] OpenAl. Creating video from text, 2024.
[29] Runway Research. Gen-2: The next step forward for generative ai, 2023.
[30] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[31] Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv: 2rXiv:2202.00512. 2022.
[32] Xuan Shen, Zhenglun Kong. Changdi Yang, Zhaoyang Han, Lei Lu, Peiyan Dong, et al. Edgeqat: Entropy and distribution guided quantization-awii -wart training for the acceleration of lightweight Ilms the edge. arXiv preprint arXiv:2402.10087 2024.

---

[33] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint 1212.042, 2012.
[34] Mingjie Sun, Zhuang Liu, Anna Bair, and Zico Kolter. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306. 1669,, 2023.
[35] Haotao Wang, Shupeng Gui, Haichuan Yang. Ji Liu, and Zhangyang Wang. Gan slimming: All-in-one gan compression by unified optimization framework. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part IV 16, pages 54 73. Springer, 2020.
[36] Jiuniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, and Shiwei Zhang. Modelscope text-to-video technical report. arXiv preprint urXiv:2308. 0657711116,1 2023.
[37] Peiqi Wang, Dongsheng Wang. Yu Ji, Xinfeng Xie, Haoxuan Song, XuXin Liu, Yongqiang Lyu, and Yuan Xie. Qgan: Quantized generative adversarial networks. arXiv preprint orXiv:/901.08263 2019.
[38] Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. In Advances in neural information processing systems (NeurIPS), pages 2074-2082, 2016.
[39] Yushu Wu, Yifan Gong, Pu Zhao, Yanyu Li, Zheng Zhan, Wei Niu, Hao Tang. Minghai Qin, Bin Ren, and Yanzhi Wang. Compiler-awe aware neural architecture search for on-mobile real-time super-resolution. In European Conference on Computer Vision, pages 92-111. Springer, 2022.
[40] Zhen Xing, Qi Dai, Han Hu, Zuxuan Wu, and Yu-Gang Jiang. Simda: Simple diffusion adapter for efficient video generation. arXiv preprint X11:22008.09710, 2023.
[41] Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In 2016 IEEE Conference on Computer Vision and Pattern 2016.
Recognition (CVPR), pages
[42] Changdi Yang, Pu Zhao, Yanyu Li, Wei Niu, Jiexiong Guan, Hao Tang, Minghai Qin, Bin Ren, Xue Lin, and Yanzhi Wang. Pruning parameterization with bi-level optimization for efficient semantic segmentation on the edge. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15402-15412, 2023.
[43] Zheng Zhan, Yifan Gong, Pu Zhao, Geng Yuan, Wei Niu, Yushu Wu, Tianyun Zhang, Malith Jayaweera, David Kaeli, Bin Ren, al. Achieving on-mobile real-time super-resolution with neural architecture and pruning search. In Proceedings of the IEEF/CVF international conference on computer vision, pages 4821-4831 2021.
[44] Zheng Zhan, Zhenglun Kong, Yifan Gong, Yushu Wu, Zichong Meng, Hangyu Zheng, Xuan Shen, Stratis loannidis, Wei Niu, Pu Zhao, and Yanzhi Wang. Exploring token pruning in vision state space models. arXiv preprint arXiv:2409. 18962, 2024.
[45] David Junhao Zhang, Jay Zhangjie WI, Jia- Wei Liu, Rui Zhao, Lingmin Ran, Yuchao Gu, Difei Gao, and Mike Zheng Shou. Show-1: Marrying pixel and latent diffusion models for text-to-video generation. arXiv preprint arXiv:2309, 158188 2023.
[46] Shiwei Zhang, Jiayu Wang, Yingya Zhang, Kang Zhao, Hangjie Yuan, Zhiwu Qin, Xiang Wang, Deli Zhao, and Jingren Zhou. 12vgen-xl: High-quality image-to-video synthesis via cascaded diffusion models. arXiv preprint arXiv:2311 .4.14.. 2023.
[47] Yihua Zhang, Yuguang Yao, Parikshit Ram, Pu Zhao, Tianlong Chen, Mingyi Hong, Yanzhi Wang, and Sijia Liu. Advancing model pruning via bi-level optimization. Advances Neural Information Processing Systems, 35: 15:18999-1822 2022.
[48] Pu Zhao, Fei Sun, Xuan Shen, Pinrui Yu, Zhenglun Kong, Yanzhi Wang, and Xue Lin. Pruning foundation models for high accuracy without retraining. arXiv preprint 27X5 2024.
[49] Daquan Zhou, Weimin Wang, Hanshu Yan, Weiwei Lv, Yizhe 7hu, and Jiashi Feng. Magievideo: Efficient video generation with latent diffusion models. arXiv preprint arX1v:2211.11018, 2022.
12

---

Appendix
A Memory Snapshot during inference
3409
1622
200
400
wfo our method
(b) w/ our method
Figure A1: GPU memory snapshot of active cached segment timeline for Stable Video Diffusion with 14 frames 512 x 512
We provide memory snapshots under different configurations during inference, demonstrating the effectiivneess of memory reduction. An example is shown in Fig This example shows the memory reduction of our method on SVD with 512 512 resolution. The snapshot is collected following the tutoria
B Key Step Search for Step Rehash
Example of Step Rehash. For step s + 1, we only conduct part of the computations in the final up-sampling block, skipping most of the computations the U-net. Similarly, we can skip multiple steps. For example, if we skip both step + 1 and we feed the output features U into the final up-sampling block of step 2, where U3+ is also obtained from Uğ following the above reusing and skipping method.
to obtain the output U/ for step + 2,
Similarity patterns. The feature similarity between different steps exhibits certain patterns. As shown in Fig. 5a initial steps, the similarity is high (above 97%) across multiple steps such as from step 0 to step 13. In the middle steps, the high similarity only appears within a small step range. For example, the similarity between step 17 and step 19 is lower than 93%. In the final steps, the high similarity appears in slightly larger step range, such as from step 20 to step 22, with above 93% similarity.
Algorithm 1 Key step search in step re-
We address the where-to-ske to-sppp problem with a fixed strategy skip the computations from the specific blocks. Next, we address the when skip problem to choose what steps can be skipped based on the similarity map. Given the similarity map as shown in Fig. 5a the similarity value between step and j can be represented by S.y as shown in the similarity map. We develop a search method to find the key steps with feature rehash and skip the other steps.
hash
Require:
The similarity map S. the sim- ilarity threshold y, the maxi- mum step number
Ensure:
The set of key steps G
i t 0, j + 0, G - (.. G + Glli
while < K do
if S .y 2 then
The algorithm is shown in Algorithm| I We use a thresh- old to select the key steps. the similarity of multiple consecutive steps is above the threshold, we only select the start and end steps as key steps, and the middle steps can be skipped. Typically, a larger threshold leads to more key steps with high generation performance close to the original one, and smaller threshold leads to skipping more steps and, thus, computations with faster generation.
i
else
G Glli
j jti
G
return G
httpss:/pytorchhco hwwp:/ch..rggbogoggeeeeeeeee / copu-mmorr--1 1/
13

---

0.8
0.8
0.8
Timestaps
Similla riy
Timestaps
Similaa aili
Timestaps
Similarity llariityy
0.6
0.6
0.6
10
10
10
5
5
5
0.0
0.0
0.0
20
20
20
Timesteps
Timesteps
Timesteps
(a) wpl blockck..... O.resnets.....
(b) up_blocks .. O.resneta.1
(c) up_blocks O. rereneeta.
Figure A2: Similarity maps of different temporal layers in up_ blocks. O. resnets.
We provide sample PyTorch snippets for operation grouping and Step Rehash. The sample code effectively reduces the peak memory and accelerates the inference speed. However, the pipeline is released because it requires specific compilation support.
C Similarity map of middle layers
We illustrate the similarity map of several layers closer to the mid-block of the UNet, showing that similarity of these layers is relatively low compared to the results in Fig. S
D Visual comparison with DeepCache
We provide visual comparison of method with DeepCache in here. As see, method produces more vivid and detailed sample than DeepCache.
DeepCache
Ours
DeepCache
Ours
DeepCache
Ours
Figure A3: Visual comparison of our method with DeepCache.
14

---

E
More quality results
AnimateDiff
Ours
a boat sailing in the ocean with a rocky landmass in view
AnimateDiff
Ours
bus pulls up to curb then pulls off
AnimateDiff
Ours
the expensive audi car going very fastly the road at the center of the forest
AnimateDiff
Ours
lion is shown with another fion walking through the field Figure A4: Quality evaluation of using our method on baseline models.
15

---

AnimateDiff
Ours
movie trailer of james cameron titanic movie bunch of loud lamborghini driving up and down the street all at different times
AnimateDiff
Ours
2 cute parrots sitting nicely
AnimateDiff
Ours
AnimateDiff
Ours
a street with traffic shown first then a setting of custom made convertables are being displayed
Figure A5: Quality evaluation of using method bascline models.
16

---

AnimateDiff
Ours
band is performing and being shown a large screen
AnimateDiff
Ours
a man is standing on a car at a car tv show
AnimateDiff
Ours
campaign ad for hillary clinton
AnimateDiff
Ours
beautiful rhyme about the roadbus journey for children
Figure A6: Quality evaluation of using our method on baseline models.
17

---

AnimateDiff
black andi car is the road
assimated video playing the song twinkle twinkle little star
AnimateDiff
bbot lego game migsles
group of people harverting rice Pold
AnimateDiff
battle involving tanks is chown
young cail ground and in exercise wheel
AnimateDiff
with suit talking Front of the
harry ritchie talking about drama and what drama really
AnimateDiff
A woman with a yellow and gray checkad baokpask walke next small waterfall
the natural scenes the mountaing are very and wonderful to
Figure A7: Quality evaluation of using our method on baseline models.
18

---

SVD-XT
Ours
SVD-XT
Ours
SVD-XT
Ours
SVD-A
burs
SVD-XT
Ours
Figure A8: Quality evaluation of using our method bascline models.
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We explain method and summarize the contribution in introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitation included conclusion section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
20

---

Justification:
Our paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: This paper fully discloses all the information needed to reproduce the main experimental results.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
21

---

Answer: [Yes]
Justification: The datasets and models we used ppenes and we will open source our code soon.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurlPS code and data submission guidelines (https /pipss.cc/pulli//uuui guwde//ddee/eee Coc//ooodeeesiseeeeeee hwtpibbmsssiooooooooiooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have specified all the training and test details necessary to understand the results
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We report average results of multiple runs in our experimental section. Our paper does not report error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
22

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative emrr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We explain the computation resources in experiment section.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: Research is conducted in the paper conform with NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our paper is not highly related to societal impacts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
23

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper poses no such risks. Our work does not release a new model.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: CC-BY 4.0, and referenced the works that we used to implement our code.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "u5FTv2a5Dp", "reviewer": "Reviewer_mL5N", "summary": "The paper introduces a novel, training-free framework to optimize video diffusion models. This framework, consisting of Feature Slicer, Operator Grouping, and Step Rehash, significantly reduces peak memory usage and computational overhead while maintaining video quality. Extensive experiments demonstrate that the approach can cut memory usage by up to 70% and improve inference speed by 1.6 times compared to baseline methods. The framework is compatible with existing models like AnimateDiff and SVD, enabling high-quality video generation on consumer-grade GPUs. The research paves the way for more efficient video diffusion models, making advanced video generation accessible on standard hardware.", "questions": "- Are there any specific scenarios where the overhead introduced by slicing and grouping operations could outweigh the benefits?\n- Have you considered conducting user studies to provide a more comprehensive assessment of the generated video quality?", "limitations": "Authors have adequately addressed the limitations.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The introduction of a training-free framework that optimizes video diffusion models seems a novel approach.\n- The method is compatible with existing video diffusion models like AnimateDiff and SVD, ensuring broad applicability.\n- Comprehensive experiments and detailed analysis demonstrate the framework's effectiveness and robustness.", "weaknesses": "- Although the paper claims that video quality is maintained, the extent of quality degradation, if any, is not fully quantified. \n- Although some ablation studies are provided, a more detailed breakdown of the contributions of each component (Feature Slicer, Operator Grouping, and Step Rehash) would strengthen the understanding of their individual and combined impacts.\n- The evaluation relies heavily on FVD and CLIP-Scores, which, while useful, may not capture all dimensions of video quality and user satisfaction. Including additional metrics or user studies could provide a more holistic assessment of the generated video quality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer for recognizing the strengths of our papers and providing valuable feedback. We are happy to address the raised questions as below.\n\n---\n#### **W1. Concern about maintaining video quality.**\n\nPlease refer to Global rebuttal **A1** and **A4**. As discussed in Global rebuttal **A1**, the video quality can be **maintained** by slightly increasing the full computation steps of Step Rehash while still **keeping a good amount inference acceleration**. In our paper, we  use **fewer full computation steps** to **push the limits** of Step Rehash. Our quantitative results in Table 1 of Global rebuttal **A1** and visual demonstrations in Global rebuttal **A4** demonstrate that our method can lead to high quality video generation with reduced cost. \n\n---\n#### **W2. Need more breakdown to our method.**\n\nThank the reviewer for pointing out this part. We will provide more details and rephrase it for better clarity. We also provide more breakdown results for this question. First, we would like to kindly point out that:\n\nAs mentioned in “Line 165-167”, Feature Slicer cannot reduce peak memory if not combined with Operator Grouping.\n\nThe Operator Grouping can only be applied after slicing the features. It is a solution to reduce saving intermediate results after applying the Feature Slicer. We would also like to point out that our Operator Grouping, Pipeline and Step Rehash are general techniques.\n\nFrom the following table, we can see that Feature Slicer and Operator Grouping together can significantly reduce the peak memory.  Pipeline and Step Rehash lead to further accelerations. Step Rehash incurs more significant speedups with certain generation quality degradation. Pipeline does not affect the video quality.  \n\n---\n- Table 1. Breakdown of our method (the results here are aligned with results Table 1 from our paper)\n| **Model** | **Method** | **speed up** | **Peak Mem.  (576x1024)** | **FVD (UCF101)** | **CLIP-Score (UCF101)** |\n|:---:|:---:|:---:|:---:|:---:|:---:|\n| SVD | - | - | 39.49G | 307.7 | 29.25 |\n|  | + Feature Slicer | x0.95 | 39.49G | 307.7 | 29.25 |\n|  | + Feature Slicer + Operator Grouping | x0.98 | 23.42G (-40.7%) | 307.7 | 29.25 |\n|  | + Feature Slicer + Operator Grouping + Pipeline | x1.03 | 23.42G (-40.7%) | 307.7 | 29.25 |\n|  | + Feature Slicer + Operator Grouping + Pipeline + Step Rehash | x1.63 | 23.42G (-40.7%) | 340.6 | 28.98 |\n| AnimateDiff | - | - | 41.71G | 758.7 | 28.89 |\n|  | + Feature Slicer | x0.94 | 41.71G | 758.7 | 28.89 |\n|  | + Feature Slicer + Operator Grouping | x0.96 | 11.07G (-73.5%) | 758.7 | 28.89 |\n|  | + Feature Slicer + Operator Grouping + Pipeline | x1.03 | 11.07G (-73.5%) | 758.7 | 28.89 |\n|  | + Feature Slicer + Operator Grouping + Pipeline + Step Rehash | x1.61 | 11.07G (-73.5%) | 784.5 | 28.71 |\n\n---\n#### **W3. Need more evaluation metrics.**\n\nWe agree that it is important to provide a more comprehensive evaluation. Please refer to Global rebuttal **A2**, Table 2 and Table 3. We demonstrate more evaluation metrics such as PSNR, LPIPS, SSIM, IS and Vbench benchmark. Our method can outperform the DeepCache baseline under various metrics. \n\n---\n#### **Q1. Specific scenarios where the overhead introduced by slicing and grouping operations could outweigh the benefits.**\n\nWhen the generated video size is small enough like 64x64, the acceleration and memory reduction is not very obvious. However, in this case, even without any efficiency optimization methods, a consumer GPU is sufficient enough to handle this small size without difficulties. \n\n---\n#### **Q2. User studies for more comprehensive assessment.**\n\nWe agree that it is important to provide a more comprehensive evaluation. Please refer to Global rebuttal **A2**, Table 4.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7y3HoW56zm", "reviewer": "Reviewer_2uzE", "summary": "This paper presents a framework for reducing the computational demands of text-to-video diffusion models. The main idea involves dividing input features into subfeatures and processing them parallelly, thereby reducing peak memory usage during sampling. To address the increase in overall compute time caused by this partitioning, the authors propose a skip strategy that determines when and where to apply the skip operation based on feature similarities.", "questions": "What is the \"computational step\" in Table 2?", "limitations": "See the Weaknesses Section.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and motivated.\n2. The proposed method is intuitive.", "weaknesses": "While the paper is well-motivated for the important problem and proposes an intuitive and simple remedy, there are major concerns regarding the evaluation, scalability, and applicability of the proposed method, as follows:\n\n**Scalability and Applicability**\n- My major concern is the scalability of the proposed method. Since the method requires dividing the input features into patches, it inevitably increases the computation as the feature dimension gets larger, potentially introducing more patches to compute. While the proposed skip strategy might mitigate some of the increased computation time, it definitely harms the quality of the generated videos.\n\n- In terms of applicability, the proposed framework is only applicable to U-Net-based video diffusion models. However, recent models, such as those utilizing DiT backbones, are not addressed by this framework.\n\n**The evaluation is weak**\n- Lack of baselines: The authors only compare their method to naive hashing, which is the most simple baseline. To verify the effectiveness of the proposed method, please include existing method for efficient sampling that also reduce memory and sampling speed. The authors only compare with DeepCache. They should compare with more training-free baselines, even though these primarily demonstrate their efficiency in image diffusion models. If not applicable, discuss why extending these approaches to the text-to-video diffusion model is not straightforward in the Related Work section, and also discuss the differences with existing baselines.\n- Week evaluation with DeepCache: It is unclear why the authors made a comparison with DeepCache under the same computational step. The authors should define what they mean by “computational step” and compare with DeepCache in terms of efficiency metrics as thoes in Table 1. Also, the metric of MAC and GFlops should be included to measure efficiency.\n- Evaluation protocol: The current evaluation using FVD and CLIP score is insufficient. It would be beneficial to include more comprehensive video quality evaluation metrics, such as thoes proposed in VBench [1].\n- Additional experiments: Include more video diffusion backbones such ModelScope and VideoCrafter to extensively verify the effectiveness of proposed method under various video diffusion models.\n\n[1] Huang et al., VBench: Comprehensive Benchmark Suite for Video Generative Models, CVPR 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the feedback from the reviewer. First, we would like to kindly clarify some misunderstandings here: \n1. We do provide the definition of the “full computation step” in the “line 265, 266”.\n2. Our method can be applied to DiT backbones without any design changes.\n3. Our work takes an early step in fast and memory-efficient inference framework in a training free manner.\n\nWe address the raised questions as below. \n\n---\n**W1. Scalability concern.**\n\nWe would like to point out that our observation is that our method could actually mitigate I/O overhead which can handle the scalability issues, as stated in “Section 4.2.1 **Mitigating I/O intensity**”. Moreover, we introduce the pipeline to mitigate the “more patches” things as stated in “Section 4.2.2”. Overall, our experimental results use 3x576x1024 with up to 16 frames which is considered as large scale video generation for the current open-source SOTA video diffusion models.\n\nWe provide a detailed breakdown, as we can see, even at 576x1024 with 16 frames, our method without step rehash could still mitigate the overhead from the scale. And we use full computation steps=15 for maintaining the quality of the generated videos. Here are the results.\n- Table A, proposed method breakdown\n|Model|Method|speed up|Peak Mem. (576x1024)|FVD (UCF101)|CLIP-Score (UCF101)|\n|-|:-:|:-:|:-:|:-:|:-:|\n|SVD|-|-|39.49G|307.7|29.25|\n||+ Feature Slicer|x0.95|39.49G|307.7|29.25|\n||+ Feature Slicer + Operator Grouping|x0.98|23.42G (-40.7%)|307.7|29.25|\n||+ Feature Slicer + Operator Grouping + Pipeline|x1.03|23.42G (-40.7%)|307.7|29.25|\n||+ Feature Slicer + Operator Grouping + Pipeline + Step Rehash|x1.46|23.42G (-40.7%)|312.1|29.20|\n|AnimateDiff|-|-|41.71G|758.7|28.89|\n||+ Feature Slicer|x0.94|41.71G|758.7|28.89|\n||+ Feature Slicer + Operator Grouping|x0.96|11.07G (-73.5%) |758.7|28.89|\n||+ Feature Slicer + Operator Grouping + Pipeline|x1.03|11.07G (-73.5%)|758.7|28.89|\n||+ Feature Slicer + Operator Grouping + Pipeline + Step Rehash|x1.45|11.07G (-73.5%)|765.01|28.87|\n\nWe also provide results for scalability of batch_size to further demonstrate our method. Here are the scalability results of our method when batch size is not 1. Our slicing strategy will be accordingly adjusted based on the batch size and input size. Below is the performance benchmark on batch size = 1,2,4 for AnimateDiff with 512x512 output. We can see that when the batch_size becomes 4, the peak memory does not increase significantly compared with the batch_size of 1.  \n- Table B, scalability under batch size = 1,2,4\n|Model|Batch Size|Peak Mem|Latency|\n|-|:-:|-|-|\n|AnimateDiff+Ours|1|7.51G|7.08s|\n|AnimateDiff+Ours|2|8.30G|14.07s|\n|AnimateDiff+Ours|4|11.35G|27.05s|\n\n---\n**W2. Generalization to DiT backbones.**\n\nPlease refer to Global rebuttal **A3** for more detailed results.\n\nWe would like to kindly clarify that our method could be applied to DiT backbones without any design changes. \nThis is due to our general design of Feature slicer on spatial-temporal 3D blocks in video diffusion blocks. We would also like to point out that Operator Grouping, Pipeline and Step Rehash are general techniques. Our results on DiT backbones (OpenSora) demonstrates stable generalization on various backbones/architectures and can constantly improving efficiency while maintaining the video quality.\n\n---\n**W3. Lack of baselines.**\n\nWe would like to kindly clarify some misunderstandings here.  Our work tries to solve the memory bound issues of video diffusion due to users’ limited VRAM resources in a novel way. Current sampling methods focus on improving sampling efficiency, without any considerations of reducing the huge peak memory where \"Peak Memory\" is the **maximum** amount of memory which is used by the processes during the **entire time**. Also, DeepCache is the most recent training-free baselines that introduce a novel direction for improving inference speed **only** in image diffusion models. General techniques like Model compression for fast and memory-efficient models take substantial efforts to retrain or finetune the diffusion model to recover performance. To our best knowledge, our work takes an early step in fast and memory-efficient inference framework in a training free manner.\n\nWe survey several methods that can reduce diffusion inference step such as DPM-Solver and Euler solver. However, they can not reduce the peak memory during inference which we highlight as one of the main contributions of our method.\n\nWe would be thankful if the reviewer could give more details of “lacked baselines” about training-free baselines for memory efficiency in diffusion models so we could include more comprehensive experimental results.\n\n---\n#### **W4. Week evaluation with DeepCache.**\n\nWe would first like to clarify that we provide the definition of the “full computation step”  in the “line 265, 266” for steps that can't be skipped; other steps that use step rehash have most of their computation skipped as we stated in Section 4.3.2. Moreover, we would like to point out that our method can save more GFLOPS when under the same computational step as stated in “line 321, 322”. We provide detailed results below,\n- Table C, Computation Comparison\n|Model|Method|512x512|576x1024|\n|-|-|-|-|\n|SVD|Original|8.47T|20.40T|\n||DeepCache|5.68T|14.06T|\n||Ours|5.75T|13.25T|\n|AnimateDiff|Original|8.37T| 20.19T|\n||DeepCache|5.76T|14.09T|\n||Ours|5.53T|11.91T|\n\n---\n**W5. Evaluation protocol.**\n\nPlease refer to Global rebuttal **A2**, Table 2~4.\n\n---\n**W6. Additional experiments.**\n\nPlease refer to Global rebuttal **A3**, Table 5 and Table 7.\n\n---\n**Q1. What is the \"computational step\" in Table 2?**\n\nWe would like to point out that we provide the definition of the “full computation step” in the “line 265, 266” for steps that can't be skipped; other steps that use step rehash have most of their computation skipped as we stated in Section 4.3.2.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "c8YVZcZPbe", "reviewer": "Reviewer_M5Qw", "summary": "This paper proposes a training-free video diffusion inference acceleration method, which includes three processes: Feature Slicer, Operator Grouping, and Step Rehash. Compared to the baseline, the proposed method shows significant improvements in memory usage and inference speed.", "questions": "- From the images, it is clear that the quality of some of the synthesized images has decreased. Based on my experience, this decrease in quality may have a more severe impact on video synthesis tasks. The authors did not seem to provide video demos, so it is difficult to judge how severe the loss in performance is.\n\n- What is the relationship between the optimization method and batch size used during inference, and how do they affect the optimization speed and memory usage?", "limitations": "I think this paper addresses an important problem and appears to have significant improvements. However, the degree of loss in synthesis quality is not clear, which is an area of concern for me. The loss in quality seems to be less noticeable in SVD, but more significant in AnimateDiff (e.g., the owl in Figure 7, the bus in Figure A6, and the parrot in Figure A5).", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The method description is straightforward and easy to understand. \n- The proposed method is effective, simple, and easy to implement, and it shows significant improvements. \n- The problem addressed by the proposed method is critical.", "weaknesses": "- The experiments need to be improved. Step Rehash can greatly accelerate the inference speed, but it seems to be highly dependent on the weights of the video diffusion model. It is unclear whether the proposed method can be applied to most video diffusion models. Therefore, the authors need to compare more types of models and provide statistical information on the performance changes, such as mean and variance.\n- The proposed method shows a visible loss in video generation capability, as seen in Figure 7 where the owl's eyes disappear and the overall texture quality of the synthesized image decreases in the optimized version.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer for recognizing the strengths of our papers and providing valuable feedback. We are happy to address the raised questions as below.\n\n---\n#### **W1. The experiments need to be improved.**\n\nPlease refer to Global rebuttal **A3** and **A2** for more detailed results.\n\nWe agree that a more comprehensive evaluation could help demonstrate the generalization ability of our framework. Therefore, we conduct comprehensive experiments on various model architectures (Global rebuttal **A3**) and demonstrate more evaluation metrics results with statistical information (Global rebuttal **A2**). Our extended experiment results show that our framework could generalize to different backbones/architectures and significantly reduces peak memory and computation costs for video diffusion model inference.\nWe would also like to kindly point out that we already provide the mean of FVD, and FVD results do not support variance evaluation.\n\n---\n#### **W2. Concern about maintaining video quality.**\n\nPlease refer to Global rebuttal **A1** and **A4**  for more detailed results. As discussed in Global rebuttal **A1**, the video quality can be **maintained** by slightly increasing the full computation steps of Step Rehash while still **keeping a good amount inference acceleration**. In our paper,  we  use **fewer full computation steps** to **push the limits** of Step Rehash. Our quantitative results in Table 1 of Global rebuttal **A1** and visual demonstrations in Global rebuttal **A4** demonstrate that our method can lead to high quality video generation with reduced cost.\n\n---\n#### **Q1. Concern about maintaining video quality.**\n\nPlease refer to Global rebuttal **A1** and **A4**.\n\n---\n#### **Q2. Relationship between the optimization method and batch size.**\n\nThank the reviewer for raising this valuable question. We implement this part and find our method could handle the scalability in terms of batch size.\nOur slicing strategy will be accordingly adjusted based on the batch size and input size.\nBelow is the performance benchmark on batch size = 1,2,4 for AnimateDiff with 512x512 output.\n- Table A, when batch size = 1,2,4\n| **Model** | **Batch Size** | **Peak Mem** | **Latency** |\n|:-:|:-:|:-:|:-:|\n| AnimateDiff+Ours | 1 | 7.51G | 7.08s |\n| AnimateDiff+Ours | 2 | 8.30G | 14.07s |\n| AnimateDiff+Ours | 4 | 11.35G | 27.05s |\n\n---\n#### **L1. Concern about maintaining video quality.**\n\nPlease refer to Global rebuttal **A1** and **A4**.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tX5HDQmJ27", "reviewer": "Reviewer_5EX5", "summary": "In this paper, the authors focus on reducing the computational requirements and high peak memory usage for video diffusion models. Specifically, a train-free framework is proposed, which consists of three parts: Feature Slicer, Operator Grouping, and Step Rehash. Those three steps result in significant memory reduction and inference acceleration.", "questions": "My main question is about the generalization of the proposed framework. Based on my understanding, most contributions of this paper are “engineering”. It is important to verify the performance on various base models, such as diffusion models with DiT backbone and 3D diffusion models. \n\nIn addition, if I want to use this framework, do I need to modify a lot of content in the code?", "limitations": "N/A", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1.\tThe proposed framework has a great performance for reducing peak memory and accelerating the inference of video diffusion models. In particular， the peak memory of AnimateDiff can be reduced significantly from 41.7GB to 11GB, which can contribute to more practical applications.\n\n2.\tThe pipeline of the proposed framework is easy to understand and is well-documented.", "weaknesses": "1.\tMore base models with other frameworks of video diffusion should be involved for more comparisons. The two baselines used in experiments, i.e., SVD and AnimateDiff, are both based on T2I diffusion models and Unet backbone. Is the proposed framework suitable for other diffusion models with DiT backbone[1] or 3D diffusion, like Open-Sora Plan v1.1(can be found in Github).\n\n2.\tHuman-level metrics should be involved to compare the video visual quality in Table 2.  The FVD and CLIP-Score may not be enough to measure the performance. Based on the cases shown in Figure 1 and 7, I prefer to agree that the original base model has better performance. \n\n\n[1]: Peebles W, Xie S. Scalable diffusion models with transformers[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 4195-4205.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer for recognizing the strengths of our papers and providing valuable feedback. We are happy to address the raised questions as below.\n\n---\n#### **W1.  Generalization on other variants and backbones.**\n\nWe agree that a more comprehensive evaluation could help demonstrate the generalization ability of our framework. We evaluate DiT, ModelScope and VideoCrafter with our method, and demonstrate that our method is general and can be applied to multiple video diffusion models with different architectures.\nPlease refer to Global **A3** Table 5~8 for more detailed results. \n\n---\n#### **W2. More evaluation metrics and video quality.**\n\nWe agree that it is important to provide a more comprehensive evaluation. Please refer to Global rebuttal **A1**, **A2** and **A4** for more detailed results. We include human evaluation in Table 4 in Global rebuttal **A2**, other evaluation metrics results can be found in Table 2 and Table 3 in Global rebuttal **A2**. As discussed in Global rebuttal **A1**, the video quality can be **maintained** by slightly increasing the full computation steps of Step Rehash while still **keeping a good amount inference acceleration**. In our paper, we  use **fewer full computation steps** to **push the limits** of Step Rehash. Our quantitative results in Table 1 of Global rebuttal **A1** and visual demonstrations in Global rebuttal **A4** demonstrate that our method can lead to high quality video generation with reduced cost.\n\n---\n#### **Q1. Generalization of our method and contributions clarification.**\n\nFirst, we want to clarify that our work is trying to solve the memory bound issues of video diffusion due to users’ limited VRAM resources in a novel way. Previous training free works focusing on sampling efficiency without considering the huge peak memory requirement for diffusion models. General techniques like Model compression take substantial efforts to retrain or finetune the diffusion model to recover performance, which is costly, time-consuming, and may raise data privacy concerns. Furthermore, applying post-training compression techniques in one-shot may save the retraining/fine-tuning efforts, but suffers from significant performance degradation.\n  \nOur work leverages the **system and algorithm co-design** (where the feature slicer + operator grouping can collaborate with step rehash without any interference) to provide a new direction for providing a fast and memory-efficient diffusion **without training**. \n\nLast but not least, we would like to kindly clarify that our method could be applied to DiT backbones without any design changes. Please refer to Global rebuttal **A3** Table 6 and Table 8 for more detailed results.\n\n---\n#### **Q2. In addition, if I want to use this framework, do I need to modify a lot of content in the code?**\n\nWe have implemented our method on all spatial-temporal 3D blocks in `diffusers` and will release a Python wrapper which can automatically inherit original blocks with our fast and memory-efficient blocks in various video diffusion models. Our implementation extends the original blocks and can seamlessly replace them without changing the diffusion framework. It is easy to implement.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for acknowledging that our work importance and broad applicability (Reviewer mL5N, M5Qw), our method is novel, and high-performing (Reviewer mL5N, M5Qw, 5EX5), our experiments are comprehensive (Reviewer mL5N), and our paper is well-written (Reviewer 2uzE, 5EX5, M5Qw).

---
**A1. Concern about maintaining video quality.**

We would like to clarify that video quality can be **maintained** by slightly increasing the full computation steps of Step Rehash while still **keeping a good amount inference acceleration**. In our paper, the intention of our demonstrated results is using **fewer full computation steps** to **push the limits** of Step Rehash. To better illustrate our method's ability to maintain video quality, we provide additional results below and in **A4**.
- Table1. Quantitative results comparison under full computation step=15
|Model|Method|Full computation step|Speed Up|Peak Mem.(576x1024)|ucf101 FVD↓|ucf101 CLIP-score↑|MSR-VTT FVD↓|MSR-VTT CLIP-score↑|
|-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|
|SVD|-|25|-|39.49G|307.7|29.25|373.6|26.06|
||+DeepCache|15|x1.39|39.49G|385.4|28.89|412.4|25.73|
||+Ours|15|x1.46|23.42G(-40.7%)|312.1|29.20|382.8|25.99|
|AnimateDiff|-|25|-|41.71G|758.74|28.89|607.1|29.40|
||+DeepCache|15|x1.39|41.71G|810.93|28.72|608.2|29.16|
||+Ours|15|x1.45|11.07G(-73.5%)|765.01|28.87|599.1|29.39|

Compared to the baseline, our Step Rehash outperforms DeepCache in both quality and speed. Specifically, when using the same full computation steps, Step Rehash skips more computations than DeepCache with less performance drop.

Lastly, we highlight that our work is an **early step** towards a **fast** and **memory-efficient** inference framework in a **training-free** manner. It reduces peak memory and computational overhead, making it feasible to generate high-quality videos on a single consumer GPU (e.g., reducing peak memory of AnimateDiff from 42GB to 11GB, with faster inference on a 2080Ti).

---
**A2. More evaluation metrics.**

We finish evaluating PSNR, LPIPS, SSIM, IS and Vbench benchmark with our method under full computation step=13. Our framework demonstrates stable generalization ability and can constantly maintain the video quality across various evaluation metrics according to results in Table 2~4. Here are the results, 
- Table 2. More metrics
|Dataset|Method|PSNR↑|LPIPS↓|SSIM↑|IS↑|
|:-:|:-:|-|-|-|-|
|ucf101|SVD+Ours|22.44±3.60|0.27±0.08|0.72±0.12|27.11±0.54|
||SVD+DeepCache|17.07±3.14|0.41±0.09|0.52±0.16|25.69±0.33|
||AnimateDiff+Ours|16.94±2.28|0.54±0.08|0.51±0.09|27.89±0.68|
||AnimateDiff+DeepCache|16.31±3.14|0.54±0.12|0.50±0.12|27.62±0.47|
|msr_vtt|SVD+Ours|25.13±5.39|0.23±0.08|0.80±0.10|21.65±0.23|
||SVD+DeepCache|22.66±4.23|0.29±0.08|0.73±0.13|21.26±0.42|
||AnimateDiff+Ours|15.90±2.24|0.53±0.08|0.48±0.12|28.83±0.51|
||AnimateDiff+DeepCache|11.71±1.72|0.72±0.06|0.31±0.11|25.60±0.72|
- Table 3. Vbench Benchmark
|Dataset|Model|Method|Subject Consistency↑|Aesthetic Quality↑|Dynamic Degree↑|
|-|-|-|:-:|:-:|:-:|
|ucf101|SVD|-|0.92|0.39|0.45|
|||+Ours|0.89|0.36|0.33|
|||+DeepCache|0.88|0.34|0.62|
||AnimateDiff|-|0.94|0.50|0.77|
|||+Ours|0.93|0.49|0.80|
|||+DeepCache|0.92|0.49|0.80|
|msr_vtt|SVD|-|0.93|0.42|0.69|
|||+Ours|0.91|0.40|0.65|
|||+DeepCache|0.89|0.39|0.64|
||AnimateDiff|-|0.95|0.53|0.68|
|||+Ours|0.94|0.51|0.62|27.64±0.59|
|||+DeepCache|0.93|0.51|0.66|

We report human evaluation results in Table 4. via MTurk platform,
- Table 4. Human Preference Evaluation
|Method|Original (Win Rate)|Ours (Win Rate)|
|-|-|-|
|SVD|50.0%|50.0%|
|AnimateDiff|53.3%|46.7%|
|OpenSora|53.3%|46.7%|

---
**A3. Generalization on other variants and backbones.**

To demonstrate the generalization of our method, we evaluate DiT, ModelScope and VideoCrafter with our method. Our method is **general** and can be applied to other video diffusion variants. It **does not need any design change** for **adapting** to these **new** video diffusions. Specifically, we would like to point out that the generalization of Feature Slicer is achieved as all video diffusion models have temporal and spatial blocks for handling temporal and spatial information. Moreover, our Operator Grouping, Pipeline and Step Rehash **do not depend on the model architecture** which are **general techniques**. Our results in Table 5~8 demonstrates the stable generalization of our method on various backbones/architectures, and it can constantly improve efficiency while maintaining the video quality.
- Table 5. Quality on ModelScope and VideoCrafter2
|Dataset|Model|FVD↓|CLIP-score↑|
|-|:-:|-|:-:|
|ucf101|ModelScope|842.42|28.97|
||ModelScope+Ours|875.67|28.45|
||VideoCrafter2|823.11|29.02|
||VideoCrafter2+Ours|852.09|28.56|
|msr_vtt|ModelScope|846.90|25.65|
||ModelScope+Ours|868.01|25.14|
||VideoCrafter2|778.32|26.04|
||VideoCrafter2+Ours|810.66|25.54|
- Table 6. Quality on OpenSora
|Method|Subject Consistency↑|Aesthetic Quality↑|Dynamic Degree↑|
|-|:-:|:-:|:-:|
|OpenSora|94.45%|56.18%|47.22%|
|OpenSora+Ours|93.01%|54.82%|51.29%|
- Table 7. Efficiency results on ModelScope and VideoCrafter2.
|Model|Peak Mem.(576x1024)|Latency|
|-|:-:|-|
|ModelScope|12.51G|27.17s|
|ModelScope+Ours|8.30G|18.96s|
|VideoCrafter|Error*|Error*|
|VideoCrafter+Ours|14.16G|141.34s|

[*] Error occurred because VideoCrafter uses Xformer’s `memory_efficient_attention` which does not support the large memory scale when video size=576x1024. We replace the Attention with PyTorch `scaled_dot_product_attention`, the peak mem is 66.66G, and corresponding latency is 229.00s.
- Table 8, Efficiency results on OpenSora
|Model|Peak Mem (720P, 1:1)|Latency|
|-|:-:|-|
|OpenSora|59.91G|1230s|
|OpenSora+Ours|41.65G|952s|

---
**A4. Visualization results of videos.**

Due to the restriction of anonymous link, we provide a PDF contain every frame of generated videos (totally 30 videos with 15 video pairs) with our method on base models to better demonstrate that our method can lead to high quality video generation with reduced cost.

Meta-review: The paper introduces a novel, training-free framework aimed at reducing the computational and memory demands of video diffusion models. The approach shows substantial improvements in memory efficiency (up to 70% reduction) and inference speed (1.6x faster) when compared to existing methods. This is particularly valuable for enabling high-quality video generation on consumer-grade hardware.

Initially, reviewers had concerns regarding the scalability and applicability of the proposed method, as well as the comprehensiveness of the evaluation settings. However, the authors effectively addressed these concerns during the rebuttal, providing extensive additional experimental evidence. This led to a consensus among all reviewers, who ultimately rated the paper as a weak accept.

Given the innovation, the thoroughness of the experimental validation, and the successful rebuttal addressing the initial concerns, I recommend accepting this paper.
Decision: Accept (poster)