Fast and Memory-Efficient Video Diffusion Using Streamlined Inference

Authors: zheng  zhan
yushu  wu
yifan  gong
zichong  meng
zhenglun  kong
changdi  yang
geng  yuan
pu  zhao
wei  niu
yanzhi  wang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-12 22:25:06

Abstract: The rapid progress in artificial intelligence-generated content (AIGC), especially with diffusion models, has significantly advanced development of high-quality video generation. However, current video diffusion models exhibit demanding computational requirements and high peak memory usage, especially for generating longer and higher-resolution videos. These limitations greatly hinder the practical application of video diffusion models on standard hardware platforms. To tackle this issue, we present a novel, training-free framework named Streamlined Inference, which leverages the temporal and spatial properties of video diffusion models. Our approach integrates three core components: Feature Slicer, Operator Grouping, and Step Rehash. Specifically, Feature Slicer effectively partitions input features into sub-features and Operator Grouping processes each sub-feature with a group of consecutive operators, resulting in significant memory reduction without sacrificing the quality or speed. Step Rehash further exploits the similarity between adjacent steps in diffusion, and accelerates inference through skipping unnecessary steps. Extensive experiments demonstrate that our approach significantly reduces peak memory and computational overhead, making it feasible to generate high-quality videos on a single consumer GPU (e.g., reducing peak memory of Animatediff from 42GB to 11GB, featuring faster inference on 2080Ti).

Main paper: 
 introduction: 
 Recent years have witnessed continual progress and advancements in artificial intelligence-generated content (AIGC). Among them, diffusion models allow artists and amateurs to create visual content with text prompts, advancing the development of image and video generation in both academia and industry. For video diffusion models, the latest works such as SVD-XT [3] , Gen2 [29] , Pika [20] , and notably the more advanced Sora [28] , demonstrate impressive capabilities in producing visually striking and artistically effective videos. Despite their great performance, video diffusion models also exhibit high computational requirements and substantial peak memory, particularly when generating longer videos with higher resolutions. For instance, SVD-XT generates 25 frames simultaneously with a resolution of 576 × 1024, while Sora expands these capabilities by supporting the generation of longer videos (over a minute) at a higher resolution of 1080 × 1920. Given the trends of generating longer videos with higher quality, the escalating memory and computation demands have impeded practical applications of these large-scale video diffusion models on various platforms.Existing model compression methods to reduce peak memory and latency, such as weight pruning [38, 12, 35, 6, 48, 23, 42, 44, 47] , quantization [37, 22, 32] , and distillation [18, 10, 43, 39] , typically require substantial retraining or fine-tuning of the compressed model to recover performance. This process is costly, time-consuming, and may raise data privacy concerns. Applying these methods in zero-shot avoids the expensive retraining, but leads to severe performance degradation. Furthermore, the variety and complexity of video diffusion architectures further complicate the model optimization. Therefore, it is challenging yet crucial to develop an effective and efficient video diffusion framework with reduced computations, smaller peak memory and less data (no re-training) requirements for its wide applications.To address the above challenges, we first identify the sources of the high computation and memory cost, which scale up with the iterative denoising process and the simultaneous processing of multiple frames. We further observe that the feature maps of certain layers may exhibit high similarity between multiple consecutive denoising steps due to the temporal property of videos, enabling further optimizations for acceleration. Based on that, we propose a training-free framework named Streamlined Inference, by leveraging the temporal and spatial characteristics of video diffusion models to effectively reduce peak memory and computational demands. Our framework contains three core components: Feature Slicer, Operator Grouping, and Step Rehash, which work together closely and comprehensively with different focuses on peak memory reduction or inference acceleration.Our Feature Slicer performs lossless feature slicing in both temporal and spatial layers, raising the possibility of peak memory reduction through processing smaller features. However, the feature slicer alone is not able to decrease peak memory as we still need to store all intermediate results of one layer for all sliced features to form a complete intermediate feature map for the next layer. To reduce peak memory practically, we further propose Operator Grouping to group homogeneous and consecutive operators in the computational graph. Within each operator group, the intermediate result of one sliced feature can be directly sent to the next operator/layer without waiting for aggregation with all other intermediate results, achieving the full potential of Feature Slicer to reduce the peak memory. Furthermore, a pipeline technique is proposed to accelerate the computations of the same operator group for multiple sliced feature inputs, with improved parallelism.Moreover, observing the high similarity of certain features between multiple consecutive denoising steps, we propose Step Rehash to reuse the generated features for a few following steps due to their high similarity, skipping the exact expansive and repetitive generation of similar features and thereby accelerating the video diffusion significantly. With this framework, we can generate high-quality videos in a fast and memory-efficient manner on a single consumer GPU, as shown in Fig. 2 . For example, the peak memory of AnimateDiff [11] can be reduced significantly from 41.7GB to 11GB, featuring inference on a typical consumer GPU 2080Ti. We summarize our contributions as follows:• We propose a novel training-free framework that can significantly reduce the peak memory and computation cost for the inference of video diffusion models by leveraging the spatial and temporal characteristics of video diffusion models.• Our approach can be seamlessly integrated into existing video diffusion models. Our extensive experiments on SVD, SVD-XT, and AnimateDiff demonstrate our effectiveness to reduce peak memory and accelerate inference without sacrificing quality.• Our approach offers a new research perspective for fast and memory-efficient video diffusion, enabling the generation of higher quality and longer videos on consumer-grade GPUs.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes]Justification: We explain method and summarize the contribution in introduction.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as as long as it is clear that these goals are not attained by the paper. 
related work: 
 Video Diffuison Models. For video generation, various approaches have been proposed, with VDM [17] as a leading example. VDM transforms the conventional U-Net [30] architecture of image diffusion models into a 3D U-Net structure, employing joint training on both images and videos. MagicVideo [49] is the first work that introduces Latent Diffusion Model (LDM) for text-to-video (T2V) generation in latent space. LVDM [13] introduces a mask sampling technique that enhances its longer video generation capability. ModelScope [36] incorporates spatial-temporal convolution and attention into LDM. Video LDM [4] trains a T2V network composed of three training stages, enabling higher quality and longer video generation. Show-1 [45] first introduces the fusion of pixel-based and latent-based diffusion models for T2V generation. Recently, Stable Video Diffusion (SVD) [3] identifies three key stages for training video LDMs: text-to-image (T2I) pretraining, video pretraining, and high-quality video finetuning.There are various research efforts exploring either architectural efficiency or model compression techniques for image/video generation. For example, ED-T2V [24] freezes parameters to reduce training costs and proposes a attention mechanism to ensure temporal coherence. SimDA [40] devises a parameter-efficient training approach by maintaining the parameter of the T2I model and uses two adapters to train it. For model compression, Diff-pruning [6] employs structural pruning techniques to reduce inference time at each sampling step. Additionally, the work [22] implements quantization on diffusion models using low-precision data types. However, these methods take substantial efforts to retrain or finetune the diffusion model to recover performance, which is costly, time-consuming, and may raise data privacy concerns. Furthermore, applying post-training compression techniques in one-shot [8, 7, 34] may save the retraining/fine-tuning efforts, but suffers from significant performance degradation.Sample-Efficient Video Diffusion Models. To address the iterative denoising process in diffusion models and improve the sampling efficiency, two approaches are proposed. The first approach [2, 19, 5, 25] focuses on creating rapid solvers to resolve the differential equation associated with the denoising process more effectively. The works [31, 27, 21] utilize knowledge distillation methods to compress and simplify the sampling trajectory efficiently, thereby enhancing overall performance.Imagen video [15] is one of the first methods to apply progressive distillation on video diffusion models, incorporating guidance and stochastic samplers. Recent work Deepcache [26] proposes a novel training-free paradigm that accelerates diffusion models by reusing the high-level features.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: The limitation is included in conclusion section.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
motivation: 
 Peak memory and computation analysis. Existing open-source video diffusion models [11, 3, 36, 46] typically adopt a pretrained T2I 2D-UNet as backbone. Their temporal layers are seamlessly integrated into the backbone 2D-UNet, positioned after every spatial layer. Here, we use SVD as an example to demonstrate how peak memory and computational overhead scale with the number of frames. The SVD model is trained with two distinct configurations: regular SVD is designed to generate 14 frames, while SVD-XT is tailored to produce 25 frames. To generate 14 or 25 video frames concurrently with SVD, its latent features require massive GPU memory and computation consumption, estimated to be approximately 14× or 25× higher than its base T2I model. This More specifically, the SVD consumes 39.49G of peak memory for 576 × 1024 resolution output, whereas its image generation counterpart only requires 6.33G of memory at the same resolution. Furthermore, incorporating the classifier-free guidance [16] substantially enhances the generation quality but doubles the peak memory required during inference.Consequently, video diffusion is computationally demanding, but the challenge of memory consumption is more critical and demands immediate attention. Most consumer-grade GPUs do not have enough memory for video diffusion models and, therefore, suffer from the "Out of Memory" error, which prevents the GPU from generating high-quality videos. There is no workaround without switching GPUs. Most users have to endure generating short and low-resolution videos.Naïve Slicing. A Naïve approach to reduce peak memory is to execute the video diffusion inference clip-by-clip. However, this strategy is hindered by the temporal layers, which are essential for maintaining temporal correlation in video diffusion models. Forcibly implementing this approach can generate random artifacts and cause motion vanishing in the output video, as detailed in Fig. 3 . Therefore, designing a memory-efficient inference framework is a challenging and non-trivial task.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
streamlined inference framework: 
 To address the above massive peak memory and computation costs, in this section, we propose a training-free framework named Streamlined Inference, which is composed of three core components: Feature Slicer, Operator Grouping, and Step Rehash. First, we discuss Feature Slicer, designed to partition input features of spatial and temporal layers, and enable the potential of massive peak memory reduction. Next, we introduce our Operator Grouping technique to aggregate homogeneous and consecutive operators into the same group, achieving the full potential of Feature Slicer to reduce peak memory through reusing the memory of intermediate result from previous sliced feature. Finally, we discuss our Step Rehash method to reuse the same feature for a few consecutive steps due to their high similarity. It accelerates the inference without increasing peak memory overhead as it skips certain denoising steps with less computations.Video diffusion models contain spatial and temporal layers which extract the corresponding information from their specific domains. On this basis, we propose a feature slicer that consists of two components: Spatial-layer slicer and Temporal-layer slicer, to divide the feature map into multiple batches/sub-features, ensuring accurate computation without introducing additional operations. The slicer is further utilized for Operator Grouping to reduce peak memory cost.Spatial layers slicer. Based on our profiling (more details can be found in Appendix A) for memory allocation of various video diffusion models, we find that performing slices at spatial layers can greatly reduce the memory footprint. The 5-D feature in the spatial layer X ∈ R B×T ×C×H×W can be reshaped to a 4-D feature X ∈ R (B×T )×C×H×W , where B, T, C, H, W are the batch size, number of frames, channels, height, and width, respectively. Thus, we slice it into k sub-features,X i ∈ R ni×C×H×W k i=1, where n i = ⌈B × T /k⌉ with ⌈•⌉ denoting the least integer greater than or equal to the input. If ⌈B × T /k⌉ ̸ = B × T /k, the dimension of the last sub-feature n k is different from others. The spatial layer slicer is applicable for most operations in spatial layers such as Conv2D, GroupNorm, LayerNorm, Attention, and Linear.Temporal layers slicer. The input of the temporal layer is a 5-D feature map with dimensions {batch, channels, frames, height, width}. 3-D operations such as Conv3D are employed to extract temporal information from the 5-D feature. Differing from spatial layers, slicing along the temporal dimension may disrupt the extraction and processing of temporal information. Therefore, we keep the temporal dimension untouched while slicing over other dimensions. Specifically, the 5-D featureX ∈ R B×T ×C×H×W can be sliced to k h × k w sub-features X ij ∈ R B×T ×C×hi×wj i=k h ,j=kw i=1,j=1, where h i = ⌈H/k h ⌉ and w j = ⌈W/k w ⌉. After detailed profiling different configurations for temporal layer slicer, we discover that the configuration with k h = max (H, 16) and k w = max (W, 16) can result in promising peak memory reduction.Although Feature Slicer converts the original feature map into multiple smaller sub-features with reduced memory footprint, the peak memory can not be reduced since the intermediate results of multiple sliced features require re-consolidation to send to the next layer/operator as inputs. It still needs to store all intermediate outputs of sliced features to form the united/unsliced intermediate feature map without actual peak memory reduction. Therefore, to address this problem, we propose Operator Grouping to group the operators accordingly in the computational graph, achieving the full potential of Feature Slicer with effective peak memory reduction due to less memory reserved for intermediate results. Furthermore, a pipeline technique is proposed to optimize the inference of operator groups with improved parallelism and practical acceleration.Operator Grouping directly re-uses existing operators by aggregating consecutive homogeneous operators into the same group. Homogeneous operators indicate these operators extract features from coherent domains and dimensions. In video diffusion models, different operators can be grouped into GroupOP t (temporal operator groups) and GroupOP s (spatial operators groups) to ensure the well-preserved semantics of sliced sub-features within each group. For example, in the SVD Model [3] , consecutive GroupNorm, Conv2D, SiLU, and Up/DownSample operators in the Spatial ResBlock can be aggregated to one group, as these operators all extract features from spatial domain and are deemed homogeneous. As shown in Fig. 4 , when computing the output feature X o for an operator group (GroupOP), the input feature X is sliced into multiple sub-features X 1 , X 2 , . . . , X k with Feature Slicer. Each sub-feature X i goes through the operator group and their outputs are concatenated after all outputs are available, as shown in Eq. ( 1),EQUATIONwhere k is the number of slices, and Concat is the concatenation operation.Reducing peak memory cost. As shown in Fig. 4 , the peak memory with the operator group is determined by the memory footprint of the input feature, the output feature, and the intermediate results. Without operator grouping, all intermediate results of all operators for sliced sub-features will allocate their own memory, hence failing to reduce memory consumption. Compared with the case above, grouped operators only need to allocate memory for intermediate results of a single sliced sub-feature and the final outputs, without the necessity to allocate full intermediate features corresponding to the original unsliced input feature, as shown in Fig. 4 (a) and (b). Operator Grouping can effectively reduce peak memory cost, enabling successful inference of video diffusion models on one single consumer or commercial GPU with low or moderate available memory, as shown in Tab. 1.Mitigating I/O intensity. As the original feature map is sliced into multiple sub-features to reduce peak memory cost, the computation may slow down due to multiple iterations corresponding to multiple inputs. However, we surprisingly observe that even with the naive basic for-loop implementation for each sub-feature as shown in Fig. 4 (b), the overall runtime with Operator Grouping is around 10% slower than that of the original unsliced version. The marginal increase in runtime can be attributed to the memory bound of the GPU for video diffusion inference. Specifically, current video diffusion model inference suffers from the memory bound, where the I/O overhead of intermediate results is more notorious than their computation workload. The slicer provides a solution to mitigate the I/O burden, thus balancing the computation and memory read/write to fully utilize the GPU capacity.With the proposed Feature Slicer and Operator Grouping, the peak memory will decrease significantly with a marginal increase for the computation runtime (based on the basic for-loop implementation). With a deeper investigation for the computation patterns, we find that the for-loop implementation cannot maximize GPU parallelism, and further employ the pipelining technique to optimize the for-loop implementation for faster inference without additional memory cost.With Operator Grouping, there are multiple operators in one group to process one sliced subfeature sequentially. With the naive for-loop implementation, before feeding each sliced sub-feature into another operator group, it needs to wait until the last sub-feature finishes its computation within the group. The parallelism can be further improved with the proposed pipeline method. Specifically, in an operator group, after a sliced feature map is computed by the out-of-place computation operator (e.g., Conv, GroupNorm, Attention, etc.) and sent to the next operator, its previous allocated memory is no longer required, but it is still reserved during inference, leading to resource waste. We can pipeline all operators in the same group to mitigate this issue. As shown in Fig. 4 (c), once the Conv operator completes processing a sliced feature X i as described in Eq. ( 1) and its outputs are sent to the next operator Norm, the subsequent sliced feature X i+1 is immediately piped into the same Conv operator, reusing the reserved memory of X i . In this way, multiple operators are executed simultaneously with improved parallelism. No additional memory is required, as we only make use of previously reserved memory.Acceleration performance. With the naive for-loop implementation, only one operator in an operator group is executed at a time. However, our pipeline method can simultaneously execute multiple operators (such as Conv, Norm, and Activation as depicted in Fig. 4 (c )) without incurring additional memory. Consequently, the inference speed can be further improved. Accordingly, integrating the pipeline within Operator Grouping can mitigate 10% speed degradation caused by feature slicing.In this section, we further introduce our step rehash method to optimize the iterative denoising steps for effective acceleration in video diffusion generation. Capitalized on the high similarity between adjacent steps, our approach accelerates the video generation, while ensuring both high quality and temporal consistency across video frames, without extra memory cost. Next we first discuss our observations for the high feature similarity and then explain details of our step rehash.Similarity visualization. The denoising process of U-Net in diffusion models requires multiple steps and the features of different steps may share certain similarities with minor differences [26] .To explore this, we analyze the feature maps averaged over multiple images at different parts of the model and plot the similarity between features of different steps, with an example shown in Fig. 5 (and more results and details demonstrated in Appendix C). We find two key insights below:• The similarity between adjacent steps significantly depends on certain blocks and layers, and it may change sharply after specific operations in video diffusion. The features do not always show high similarity. For example, neither deeper layers within the same block nor those in middle blocks consistently show higher similarity between adjacent steps. • The features between adjacent steps following the temporal layers and spatial layers in video diffusion usually exhibit remarkably higher similarity compared to outputs of other layers. High similarity after temporal layers. Existing video diffusion models typically employ pretrained image diffusion models as their backbone. While these image models are trained to produce a variety of images, the addition of temporal layers is designed to improve the temporal continuity of latent features. This enhancement significantly strengthens their correlation, thereby increasing similarity among the features.Motivation and challenges for step rehash. Due to the high similarity between features of different steps, we propose the step rehash method to skip the computation of certain features by reusing previous generated features. However, we need to address the challenges of when and where to skip. Specifically, based on the above insights, simply reusing features from deeper layers does not guarantee better results since deeper layers may not show high similarity. We need to carefully choose what layers or blocks can be skipped (where to skip) to make use of high similarities without significantly downgrading the generation performance. As shown in fig. 5 , it exhibits high similarity between adjacent time steps, but the similarity pattern differs between video diffusion models. Thus, we need to determine which steps can use skip strategy (when to skip), and the remaining steps that require full computation are full computation steps. , as shown in Fig. 6 . We further propose a step search algorithm to solve the when to skip problem, algorithm details can be found in Appendix B. 
models, datasets and evaluation metrics: 
 We conduct the experiments on representative video diffusion models, including SVD [3] , SVD-XT [3] , and AnimateDiff [11] . For evaluation, we use the following evaluation protocols: The first frame of the video clips are extracted as the image condition for image-to-video generation and their captions are considered as the prompts. All experiments are conducted on a NVIDIA A100 GPU. Zero-shot UCF-101 [33] : We sample clips from each categories of UCF-101 dataset, and gather a subset with 1,000 video clips for evaluation. Their action categories are considered as their captions.For SVD and SVD-XT, our samples are generated at a resolution of 576 × 1024 (14 frames for SVD and 25 frames for SVD-XT) and then resize to 240 × 320. For AnimateDiff, we generate samples with resolution 512 × 512 (16 frames).Zero-shot MSR-VTT [41] : We generated a video sample for each of the 9,940 development prompts.The samples are at resolution 320 × 576 then resized to 240 × 426 for all models with different number of generated frames.We compute the FVD as outlined in [9] and CLIP-Score [14] using TorchMetrics [1] to measure the performance of generated samples.Baseline: We use pretained weight for SVD (I2V) and AnimateDiff (T2V). We compare the proposed Streamlined Inference (use 13 full computation steps) with the original inference (use 25 full computation steps) and naïve slicing inference as mentioned in Sec.3. More specifically, for image-conditioned SVD model, we set each naïve slice with a frame size of 2 and use the last frames of each generated slice as the image condition for the next slice. For AnimateDiff, we evenly generate 4 slices with a frame size of 4, then combine them into a full video clip.The results from Table 1 demonstrate the effectiveness of our proposed method in managing memory, computational resources, and performance. Our method significantly reduced peak memory and latency while maintaining competitive FVD and CLIP-Score values across all three models and resolutions compared to the original method. For SVD, our method achieved a notable reduction in peak memory and latency while maintaining competitive FVD and CLIP-Score, unlike Naïve Slicing, which increased FVD and latency. For SVD-XT, our method improved over Naïve Slicing and balanced resource usage and performance. For AnimateDiff, our method significantly outperformed Naïve Slicing in FVD and latency, achieving nearly the same performance as the original method but with smaller latency and around a 70% reduction in peak memory.Our ablation study in Table 2 demonstrates that our Step Rehash method consistently outperforms DeepCache with the same number of full computation steps.Step Rehash skips more computations than DeepCache. For the SVD model, our method maintains competitive CLIP-Scores while slightly increasing FVD compared to the original method (FVD of 307.7 and CLIP-Score of 29.25 on UCF101). DeepCache performs poorly, increasing FVD and reducing video quality. For the AnimateDiff model, our method maintains stable FVD (603.9 vs. 607.13) and CLIP-Score (29.29 vs. 29.40) on MSR-VTT compared to the original method. DeepCache shows the worst performance on UCF101, with higher FVD and lower CLIP-Scores. Visual comparisons of our method with DeepCache are provided in Appendix D. The results show that our method can be generally applied to various video diffusion models and achieve competitive results.In Fig. 7 and Appendix E, we present qualitative results comparing our method to the original model without using Streamlined Inference.We can see that our method produces vivid and high-quality samples aligned with the text descriptions. More importantly, these results demonstrate that our method can significantly reduce peak memory and computation without sacrificing quality. 
conclusion and limitation: 
 In this paper, we propose a novel training-free framework that significantly reduces peak memory and computation costs for video diffusion model inference by leveraging its spatial and temporal characteristics. Our approach can be seamlessly integrated into existing models. Extensive experiments on SVD, SVD-XT, and AnimateDiff demonstrate our method's effectiveness in reducing peak memory and accelerating inference without sacrificing quality. Our approach offers a new perspective for fast, memory-efficient video diffusion, enabling the generation of higher quality and longer videos on consumer-grade GPUs. Though our method is general, the efficiency is limited by baseline model architecture design. We provide memory snapshots under different configurations during inference, demonstrating the effectiveness of memory reduction. An example is shown in Fig. A1 . This example shows the memory reduction of our method on SVD with 512 × 512 resolution. The snapshot is collected following the tutorial2 . 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] Justification: Our paper is not highly related to societal impacts.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our paper poses no such risks. Our work does not release a new model.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: CC-BY 4.0, and we referenced the works that we used to implement our code.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: Example of Step Rehash. For step s + 1, we only conduct part of the computations in the final up-sampling block, skipping most of the computations in the U-net. Similarly, we can skip multiple steps. For example, if we skip both step s + 1 and s + 2, to obtain the output U s+2 3 for step s + 2, we feed the output features U s+1 3 into the final up-sampling block of step s + 2, where U s+1 3 is also obtained from U s 3 following the above reusing and skipping method. Similarity patterns. The feature similarity between different steps exhibits certain patterns. As shown in Fig. 5a , at initial steps, the similarity is high (above 97%) across multiple steps such as from step 0 to step 13. In the middle steps, the high similarity only appears within a small step range. For example, the similarity between step 17 and step 19 is lower than 93%. In the final steps, the high similarity appears in a slightly larger step range, such as from step 20 to step 22, with above 93% similarity.Algorithm 1 Key step search in step rehash Require: The similarity map S, the similarity threshold γ, the maximum step number K Ensure: The set of key stepsG i ← 0, j ← 0, G ← {}, G ← G∥i while i < K do if S ij ≥ γ then i ← i + 1 else G ← G∥i j ← i G ← G∥K -1 return GWe address the where-to-skip problem with a fixed strategy to skip the computations from the specific blocks. Next, we address the when skip problem to choose what steps can be skipped based on the similarity map. Given the similarity map as shown in Fig. 5a , the similarity value between step i and j can be represented by S ij as shown in the similarity map. We develop a search method to find the key steps with feature rehash and skip the other steps.The algorithm is shown in Algorithm 1. We use a threshold to select the key steps. If the similarity of multiple consecutive steps is above the threshold, we only select the start and end steps as key steps, and the middle steps can be skipped. Typically, a larger threshold leads to more key steps with high generation performance close to the original one, and a smaller threshold leads to skipping more steps and, thus, computations with faster generation. We provide sample PyTorch snippets for operation grouping and Step Rehash. The sample code effectively reduces the peak memory and accelerates the inference speed. However, the pipeline is not released because it requires specific compilation support.We illustrate the similarity map of several layers closer to the mid-block of the UNet, showing that the similarity of these layers is relatively low compared to the results in Fig. 5 .We provide visual comparison of our method with DeepCache in here. As we can see, our method produces more vivid and detailed sample than DeepCache. NeurIPS Paper Checklisthttps://pytorch.org/blog/understanding-gpu-memory-1/ 


Decision: The paper introduces a novel, training-free framework aimed at reducing the computational and memory demands of video diffusion models. The approach shows substantial improvements in memory efficiency (up to 70% reduction) and inference speed (1.6x faster) when compared to existing methods. This is particularly valuable for enabling high-quality video generation on consumer-grade hardware.

Initially, reviewers had concerns regarding the scalability and applicability of the proposed method, as well as the comprehensiveness of the evaluation settings. However, the authors effectively addressed these concerns during the rebuttal, providing extensive additional experimental evidence. This led to a consensus among all reviewers, who ultimately rated the paper as a weak accept.

Given the innovation, the thoroughness of the experimental validation, and the successful rebuttal addressing the initial concerns, I recommend accepting this paper.

Reviews: review: paper_summary: paper_summary: The paper introduces a novel, training-free framework to optimize video diffusion models. This framework, consisting of Feature Slicer, Operator Grouping, and Step Rehash, significantly reduces peak memory usage and computational overhead while maintaining video quality. Extensive experiments demonstrate that the approach can cut memory usage by up to 70% and improve inference speed by 1.6 times compared to baseline methods. The framework is compatible with existing models like AnimateDiff and SVD, enabling high-quality video generation on consumer-grade GPUs. The research paves the way for more efficient video diffusion models, making advanced video generation accessible on standard hardware.
strength_weakness: strength_weakness: - Although the paper claims that video quality is maintained, the extent of quality degradation, if any, is not fully quantified. 
- Although some ablation studies are provided, a more detailed breakdown of the contributions of each component (Feature Slicer, Operator Grouping, and Step Rehash) would strengthen the understanding of their individual and combined impacts.
- The evaluation relies heavily on FVD and CLIP-Scores, which, while useful, may not capture all dimensions of video quality and user satisfaction. Including additional metrics or user studies could provide a more holistic assessment of the generated video quality.
questions: questions: - Are there any specific scenarios where the overhead introduced by slicing and grouping operations could outweigh the benefits?
- Have you considered conducting user studies to provide a more comprehensive assessment of the generated video quality?
limitations: limitations: Authors have adequately addressed the limitations.
paper_summary: paper_summary: This paper presents a framework for reducing the computational demands of text-to-video diffusion models. The main idea involves dividing input features into subfeatures and processing them parallelly, thereby reducing peak memory usage during sampling. To address the increase in overall compute time caused by this partitioning, the authors propose a skip strategy that determines when and where to apply the skip operation based on feature similarities.
strength_weakness: strength_weakness: While the paper is well-motivated for the important problem and proposes an intuitive and simple remedy, there are major concerns regarding the evaluation, scalability, and applicability of the proposed method, as follows:

**Scalability and Applicability**
- My major concern is the scalability of the proposed method. Since the method requires dividing the input features into patches, it inevitably increases the computation as the feature dimension gets larger, potentially introducing more patches to compute. While the proposed skip strategy might mitigate some of the increased computation time, it definitely harms the quality of the generated videos.

- In terms of applicability, the proposed framework is only applicable to U-Net-based video diffusion models. However, recent models, such as those utilizing DiT backbones, are not addressed by this framework.

**The evaluation is weak**
- Lack of baselines: The authors only compare their method to naive hashing, which is the most simple baseline. To verify the effectiveness of the proposed method, please include existing method for efficient sampling that also reduce memory and sampling speed. The authors only compare with DeepCache. They should compare with more training-free baselines, even though these primarily demonstrate their efficiency in image diffusion models. If not applicable, discuss why extending these approaches to the text-to-video diffusion model is not straightforward in the Related Work section, and also discuss the differences with existing baselines.
- Week evaluation with DeepCache: It is unclear why the authors made a comparison with DeepCache under the same computational step. The authors should define what they mean by “computational step” and compare with DeepCache in terms of efficiency metrics as thoes in Table 1. Also, the metric of MAC and GFlops should be included to measure efficiency.
- Evaluation protocol: The current evaluation using FVD and CLIP score is insufficient. It would be beneficial to include more comprehensive video quality evaluation metrics, such as thoes proposed in VBench [1].
- Additional experiments: Include more video diffusion backbones such ModelScope and VideoCrafter to extensively verify the effectiveness of proposed method under various video diffusion models.

[1] Huang et al., VBench: Comprehensive Benchmark Suite for Video Generative Models, CVPR 2024
questions: questions: What is the "computational step" in Table 2?
limitations: limitations: See the Weaknesses Section.
paper_summary: paper_summary: This paper proposes a training-free video diffusion inference acceleration method, which includes three processes: Feature Slicer, Operator Grouping, and Step Rehash. Compared to the baseline, the proposed method shows significant improvements in memory usage and inference speed.
strength_weakness: strength_weakness: - The experiments need to be improved. Step Rehash can greatly accelerate the inference speed, but it seems to be highly dependent on the weights of the video diffusion model. It is unclear whether the proposed method can be applied to most video diffusion models. Therefore, the authors need to compare more types of models and provide statistical information on the performance changes, such as mean and variance.
- The proposed method shows a visible loss in video generation capability, as seen in Figure 7 where the owl's eyes disappear and the overall texture quality of the synthesized image decreases in the optimized version.
questions: questions: - From the images, it is clear that the quality of some of the synthesized images has decreased. Based on my experience, this decrease in quality may have a more severe impact on video synthesis tasks. The authors did not seem to provide video demos, so it is difficult to judge how severe the loss in performance is.

- What is the relationship between the optimization method and batch size used during inference, and how do they affect the optimization speed and memory usage?
limitations: limitations: I think this paper addresses an important problem and appears to have significant improvements. However, the degree of loss in synthesis quality is not clear, which is an area of concern for me. The loss in quality seems to be less noticeable in SVD, but more significant in AnimateDiff (e.g., the owl in Figure 7, the bus in Figure A6, and the parrot in Figure A5).
paper_summary: paper_summary: In this paper, the authors focus on reducing the computational requirements and high peak memory usage for video diffusion models. Specifically, a train-free framework is proposed, which consists of three parts: Feature Slicer, Operator Grouping, and Step Rehash. Those three steps result in significant memory reduction and inference acceleration.
strength_weakness: strength_weakness: 1.	More base models with other frameworks of video diffusion should be involved for more comparisons. The two baselines used in experiments, i.e., SVD and AnimateDiff, are both based on T2I diffusion models and Unet backbone. Is the proposed framework suitable for other diffusion models with DiT backbone[1] or 3D diffusion, like Open-Sora Plan v1.1(can be found in Github).

2.	Human-level metrics should be involved to compare the video visual quality in Table 2.  The FVD and CLIP-Score may not be enough to measure the performance. Based on the cases shown in Figure 1 and 7, I prefer to agree that the original base model has better performance. 


[1]: Peebles W, Xie S. Scalable diffusion models with transformers[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 4195-4205.
questions: questions: My main question is about the generalization of the proposed framework. Based on my understanding, most contributions of this paper are “engineering”. It is important to verify the performance on various base models, such as diffusion models with DiT backbone and 3D diffusion models. 

In addition, if I want to use this framework, do I need to modify a lot of content in the code?
limitations: limitations: N/A

Comments: comment: We sincerely thank the reviewer for recognizing that the concerns have been thoroughly addressed and increasing the score! We will add all these constructive suggestions in the final version of our paper.
Thank you very much for the detailed responses. My concerns have been thoroughly addressed, and I have accordingly increased my score.
Dear Reviewer,

As we enter the final day of the Reviewer-Author discussion, please take a moment to review the authors' rebuttal and consider the comments from other reviewers. If you have any additional questions, now is the time to ask them to help you better evaluate the paper. Thanks!
We sincerely thank the reviewer for recognizing that the concerns have been addressed and raising the score! We will add all these constructive suggestions in the final version of our paper.
We sincerely thank the reviewer for acknowledging our responses and improving the score! We will add all these constructive suggestions in the final version of our paper.
Thanks for your responses, which has addressed some of my concerns. I will raise my score.
Thanks for your responses. I'm glad to improve my score.
Dear Reviewer,

Thank you very much for taking the time to review our paper and for acknowledging the performance of our framework. Since the discussion will end very soon, we sincerely hope that you have found time to check our detailed response to your previous questions/comments. If you have any further questions, please feel free to let us know. We will try our best to reply to you before the discussion deadline.

Thank you very much,

Authors
Dear Reviewer,

Thank you very much for taking the time to review our paper and for acknowledging the contributions we've made. Since the discussion will end very soon, we sincerely hope that you have found time to check our detailed response to your previous questions/comments. If you have any further questions, please feel free to let us know. We will try our best to reply to you before the discussion deadline.

Thank you very much,

Authors
Dear Reviewer,

Thank you very much for spending time reviewing our paper. Since the discussion will end very soon, we sincerely hope that you have found time to check our detailed response to your previous questions/comments. If you have any further questions, please feel free to let us know. We will try our best to reply to you before the discussion deadline.

Thank you very much,

Authors
Dear Reviewer,

Thank you very much for taking the time to review our paper and for acknowledging the novelty and applicability of our work. Since the discussion will end very soon, we sincerely hope that you have found time to check our detailed response to your previous questions/comments. If you have any further questions, please feel free to let us know. We will try our best to reply to you before the discussion deadline.

Thank you very much,

Authors