Ca2-VDM: Efficient Autoregressive Video Diffusion Model with Causal Generation and Cache Sharing
Authors: Kaifeng Gao
Jiaxin Shi
Hanwang Zhang
Chunping Wang
Jun Xiao
Long Chen
Venue: ICML 2025 poster (2025)
Keywords: Video Generation
Video Diffusion Model
Primary area: applications->computer_vision

Abstract: With the advance of diffusion models, today's video generation has achieved impressive quality. To extend the generation length and facilitate real-world applications, a majority of video diffusion models (VDMs) generate videos in an autoregressive manner, i.e., generating subsequent clips conditioned on the last frame(s) of the previous clip. However, existing autoregressive VDMs are highly inefficient and redundant: The model must re-compute all the conditional frames that are overlapped between adjacent clips. This issue is exacerbated when the conditional frames are extended autoregressively to provide the model with long-term context. In such cases, the computational demands increase significantly (i.e., with a quadratic complexity w.r.t. the autoregression step). In this paper, we propose **Ca2-VDM**, an efficient autoregressive VDM with **Ca**usal generation and **Ca**che sharing. For **causal generation**, it introduces unidirectional feature computation, which ensures that the cache of conditional frames can be precomputed in previous autoregression steps and reused in every subsequent step, eliminating redundant computations. For **cache sharing**, it shares the cache across all denoising steps to avoid the huge cache storage cost. Extensive experiments demonstrated that our Ca2-VDM achieves state-of-the-art quantitative and qualitative video generation results and significantly improves the generation speed. Code is available: https://github.com/Dawn-LX/CausalCache-VDM

Ca2-VDM:
Efficient Autoregressive Video Diffusion Model with Causal Generation and Cache Sharing
Kaifeng Gao 12
Jiaxin Shi
Hanwang Zhang*
Chunping Wang
Jun Xiao
Long Chen 6
Abstract
With the advance of diffusion models, today's video generation has achieved impressive qual- ity. To extend the generation length and facilitate real-world applications, a majority of video dif- fusion models (VDMs) generate videos in an au- toregressive manner, i.e., generating subsequent clips conditioned on the last frame(s) of the previ- ous clip. However, existing autoregressive VDMs are highly inefficient and redundant: The model must re-compute all the conditional frames that are overlapped between adjacent clips. This issue is exacerbated when the conditional frames are extended autoregressively to provide the model with long-term context, In such cases, the compu- tational demands increase significantly (i.e., with quadratic complexity w.r.t the autoregression step). In this paper, we propose Ca2-VDM, an efficient autoregressive VDM with Causal gen- eration and Cache sharing. For causal gener- ation, it introduces unidirectional feature com- putation, which ensures that the cache of con- ditional frames can be precomputed in previous autoregression steps and reused in every subse- quent step, eliminating redundant computations. For cache sharing. it shares the cache across all denoising steps to avoid the huge cache stor- age cost. Extensive experiments demonstrated that our Ca2-VDM achieves state-of-the-ar quan- titative and qualitative video generation results and significantly improves the generation speed. Code is available: https:/ //gi wti.ghtthhhhioooo com/
Bidirectional Generation
Bidirectional
Bidirectional
[a] Existing Autoregressive VDMs (w/ Extendable Condition)
Cache Sharing across denaisnng steps
Cache
Causal Gen.
Coche Writing
Causal Gen.
Coche We Writing
Causal Gen.
Writing
2
KV-cache
KV-rache
KV-cache
(b) Ca2-VDM w/ (Ours)
T-stap Denolsing
Figure (a): Existing autoregressive VDMs with bidirec- tional generation. The conditional frames can be fixed- length (Henschel al., 2025; Zheng et al., 2024) extend- able. (b): Our Ca2-VDM, which uses causal generation to enable VV-cache and introduce cache sharing across all de- noising timesteps. Cache writing stands for partial model forward on the denoised frames (i.e., at timestep = 0) until the K V-caches of every layer are computed.
1. Introduction
Video diffusion models (VDMs) (Guo et al., 2024b; Ren et al., 2024; Lu et al., 2024; Ma et al., 2025) have made significant advancements by benefiting from the powerful diffusion techniques (Ho et al., 2020; Song et al., 2021a;b) and prior studies on image generation (Rombach al., 2022: Peebles & Xie, 2023; Chen et al., 2024a). In contrast images, VDMs need capture interactions across multi- ple frames and generate all frames simultaneously (e.g., 16-frame clip). This is usually facilitated by the tem- poral attention in prevailing UNet- or Transformer-ban based VDMs (Wang et al., 2023b; Ma et al., 2025). They in- troduce interdependencies during the bidirectional atten- tion computation. Consequently, the training and inference lengths must be aligned, extremely restricting the flexibility of VDMs in real-world applications such long-term (Hen- schel et al., 2025) or live-stream (Alonso et al., 2024) video generation. Meanwhile, simply scaling the clip length at inference time breaks the alignment and leads to poor gener- ation quality (e.g., Figure (bb) in (Qiu al., 2024)), unless
Eqqual contribution 'Zhejiang University, Hangzhou, China "Manycore Tech Inc., Hangzhou, China 'X Ltd., Beijing, China "Nanyang Technological University, Singapore Finvolution Group, Shanghai, China 'hhe Hong Kong University of Science and Technology, Hong Kong, China. Correspondence to: Long Chen <longehen ust.hk>.
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Title Suppressed Due to Excessive Size
one undertakes time-consuming retraining or fine-tuning.
Clean Latent (condition)
Noisy Latent
Clean KV
Noisy KV
To address this issue, an effective and prevalent solution is autoregressive VDMs (Blattmann et al., 2023a; Henschel et al.. 2025; Lu et al., 2024): They are capable of autore- gressively generating subsequent clips conditioned on last frames of previous clip, as shown in Figure (a). However, the autoregression process of existing VDMs is highly inef- ficient and redundant: The conditional frames constitute the overlapping frames between adjacent autoregression chunks and they are re-computed at each step. This issue is exacer- bated when the conditional frames are extended autoregres- sively to provide the model with long-term context. In such cases, the model must re- ro-compute all the conditional frames concatenated by the previously generated chunks, with a quadratic computational demand w.r.t. the autoregressive step (cf. Figure 6 in Sec. 4.3).
70
20
7
7
Zt
Z
KV-cache
write
Layer
cache
i
Layer
(t=0)
KV-cache
2
write
Layer E+1
cache
Layer ë+1
(t=0)
(a) Bidirectional Attention
(b) Causal Attention w/ KV-cache
Figure 2: Comparison of bidirectional attention (a) and causal attention (ours) (b). Our design addresses the cache computation and cache storage issues.
To overcome the above limitations, we propose to cache the intermediate features (specifically, the keys and values of every attention layer) at each autoregression (AR) step, and reuse them in subsequent AR steps, as shown in Figure 1(b). In this way, the model 1) eliminates the redundant com- putations in temporal attention blocks, and 2) reduces the processing length to constant for other temporal-parallel blocks (e.g., spatial attention and visual-text cross attention) while maintaining the extendable long-term context. To successfully implement the VV-cache in VDMs, two key factors must be carefully considered:
former ensures each generated frame only depends on its prefix frames, and the latter enhances the guidance from the prefix frames. As a result, the cache to be used in sub- sequent autoregression steps can be precomputed at early steps. For cache storage, we propose cache sharing. leverages the advantages of causal generation: The cache determined only by the non-noisy preceding (conditional) frames and unaffected by the subsequent noisy frames (i.e., independent of the timestep t). Thus, by using a distinct timestep embedding indexed by = 0 for the conditional frames in both training and inference, we enable the cache to be shared across all the denoising steps.
Cache Computation. In existing VDMs, the tempo- ral attention is bidirectional, shown in Figure 2(a). The frames zi 3,4 denoised conditioned on =0 0,1,2 and key/value features of 2 0,1,2 are also computed condi- tioned on 3,4 at every diffusion timestep / (highlighted by the red box and arrows). It's impossible to precom- pute and cache the keys and values of 2 0,2,, at previous AR steps, since x
Equipped with causal generation and cache sharing, we pro- pose to store the KV-cache in queue so that the model can exploit the long-term context while maintaining afford- able computation and storage cost. To support this queue design, the training samples are partially noised to keep clean prefix frames (with random length) as the condition, and the maximum condition length covers the length of KV-cache queue at inference time. Meanwhile, sinusoidal spatial and temporal positional embeddings (i.e., SPEs and TPEs) are added to the frame sequence following Vision Transformer (ViT) (Dosovitskiy et al., 2020). During infer- ence, the TPEs are assigned chunk-by-chine by-chunk as the autore- gression progresses. To ensure TPES are correctly assigned when the cumulatively generated video exceeds the train- ing length, we carefully design a cyclic shift mechanism: Cyclic-TPEs
3,4 are not yet available.
Cache Storage. During inference, the VDM repeat- edly called in the denoising process each AR step, where each call is taken with a different timestep t. All most all Existing VDMs (Lu et al., 2024; Ren et al., 2024) use the same timestep embedding (indexed by t) for both conditional and noisy frames. This requires each denoising step to have its own cache, i.e., caching the key/value features for all denoising steps will con- sume huge GPU memory.
We evaluated our Ca2-VDM on multiple public datasets including MSR-VTI VTT (Xu et al., 2016), UCF-101 (Soomro et al., 2012), and Sky Timelapse (Zhang et al., 2020) for both text-to-videy and video prediction tasks. The results
In this paper, we propose efficient autoregressive VDM boosted by causal generation and cache sharing, termed Ca2- VDM, to handle both challenges. For cache computation, we propose causal generation: We replace the full temporal attention in each block of the VDM with causal temporal at- tention, and propose prefix-enhancee spatial attention. The
Originally, TPEs are re-assigned from scratch at each AR step. However, when KV-cache is enabled, early TPEs have been bound to previous KV-caches. They can not he re-assigned (ef. Figure 4(c) for more details).
2

---

Title Suppressed Due to Excessive Size
show that our model achieves significant inference speed im- provement while maintaining comparable quantitative and qualitative performance state-of-the-art VDMs. In sum- mary, we make three contributions in this paper: 1) A causal generation structure that allows the intermediate features of conditional frames can be cached and reused in every autoregression step, eliminating the redundant computation. 2) A cache sharing strategy implemented on the KV-eache queue and facilitated by Cyclic-TPES, It allows the model to acquire extendable context while significantly reducing the storage cost. 3) Our Ca2-VDM achieves comparable performance with SOTA VDMs at much less computation demand and a high inference speed.
malization (Voleti et al., 2022; Lu et al., 2024), cross-
attention (Zhang et al., 2023b; Lu et al., 2024; Henschel et al., 2025), and explicitly concatenating to the noisy latent along the temporal-axis (Harvey et al., 2022; Lu et al., 2024) or channel-axis (Chen et al., 2024b; Girdhar et al., 2024; Zeng et al., 2024). Some works (Weng et al., 2024; Guo et al., 2024a) also inject conditional frames by adapter-like subnets (e.g., T2I-adapter (Mou et al., 2024) or Control- Net (Zhang al., 2023a)). In contrast to existing works, our Ca2-VDM avoids the redundant computation of condi- tional frames by causal generation and cache sharing, and significantly improves the generation speed.
3. Method
2. Related Work
3.1. Preliminaries and Problem Formulation
Video Diffusion Models (VDMs) have shown impressive generation capabilities, building on the success of latent dif- fusion models in image generation applications (Rombach et al., 2022; Peebles & Xie, 2023; Chen et al., 2024a). Some works (Lu et al., 2023; Khachatryan et al., 2023; Hong et al., methods for zero-shot video generation based on pretrained image diffusion models (e.g., Stable Diffusion (Rombach et al., 2022)). To leverage video training data and improve the generation quality, many works (Ge et al., 2023; Guo et al., 2024b; Wang et al., 2023b; Ren et al., 2024; Dai et al., 2023) extend the 2D Unet in text-to-image diffusion models with temporal attention layers or temporal convolution layers. Recent studies (Ma al., 2025: Lu et al., 2024) also build VDMs based spatial-temporal Transformers due to their inherent capability of capturing long-term temporal depen- dencies. We build our Ca2- Ca2-VDM based on -ettalltemppral Transformers following prior structures.
Preliminaries. Diffusion Models (Sohl-Dickstein al.. 2015; Ho al., 2020) are generative models that model tar- get distribution æo ~ ((x)) by learning a denoising process with arbitrary noise levels. To do this, a diffusion process is defined to gradually corrupt 2p with Gaussian noise. Each diffusion step is g(2; = - [Mxx;; /1 -, 3,I), where / = .,....?7 T and 9r E (0, 1) the variance schedule. By applying the reparameterizatioo trick (Ho et al., 2020), each H, can be sampled H, = vā + VV a,c, where E, ~ NO I) and ā II (1 (1- B.). Given the dif- fusion process, diffusion model is then trained to approxi- mate the denoising process. Each denoising step is param- eterized as po (1 (x)) = (xx Mo (1, t), Ex t)), where @ contains learnable parameters.
2023; Zhang et al., 2024) develop training-
Problem Formulation. Following existing mainstream VDMs (Guo et al., 2024b; Lu et al., 2024; Ma et al., 2025), we develop Ca2-VDM based on latent diffusion models (Rombach et al., 2022) to reduce the modeling com- plexity of high dimensional visual data. This is achieved by using a pretrained variational autoencoder (VAE) encoder E to compress Do into a lower-dimensional latent represen- tation, i.e., 2 = {(xu). Consequently, the diffusion and denoising processes are implemented in the latent space, for- mulated as ( and pa (zt-1 2x), respectively. The denoised latent &, is decoded back to the pixel space by the pretrained VAE decoder D, i.e., Zo = D(2o).
Tuning-free Video Extrapolation. Prior studies have explored autoregressively extrapolating videos using pre- trained short video diffusion models without additional fine- tuning. These methods usually consist of initializing noise sequence based on the DDIM inversion (Song et al., 2021a; Mokady et al., 2023) of previously generated frames (Oh et al., 2024), co-denoising overlapped short clips (Wang et al., 2023a), or iteratively denoising short clips with noise- rescheduling (Qiu et al., 2024). However, their generation quality opper-bounded by the pretrained VDMs. Mean- while, the lack of finetuning also leads to temporal inconsis- tencies between short clip transitions.
In our setting, the model takes as input a VAE encoded latent sequence² 20-4 = [= 26 2-11 1] E R XC where L is the number of frames. H x W is the downsampled resolution, and C' is the number of channels. Then, it aims to generate future frames conditioned on past frames, by learning a distribution Po ( 1=0-")) Here the first P pre- fix frames serve as condition (referred to as clean prelix), and the remaining L- P frames are those to be denoised
Past-frame Conditioned Video Prediction. To enhance generation quality and temporal consistency, a popular paradigm is training VDMs conditioned on past frames to predict future frames, enabling video extrapolation through autoregressive model calls. Recent works of autoregres- sive VDMs have studied variety of design choices for injecting conditional frames, such as adaptive layer nor-
Throughout this paper, we use "a: 6" to denote a half-open interval ranging from a (inclusive) to b (exclusive)
3

---

Title Suppressed Due to Excessive Size
autorograssion direction
Training
tEmb(o)
tÉmb(t)
Random Mask
00O11
AR-step k
AR-step
(0.))
PrePetl
Pheitlettt
Inference
Danoising
tmmb(t)
Cache Writing
tEmb(0)
2 Cyclic-TPEs
SPEs
Clean Prefix Z.0
P.P
Denoising Target 2
tEmb(o)
tEmb(T)
Causal Generation
tEmb(0)
Cache Writing
(Emb(T) 4
Causal Generation
Layer Norm
autooeggr rssion steps
Prefix Enhanced
tEmb(0)
tEmblt)
Cache Sharing
Cache Sharing
Spatial Attn
Cache Update
tEmb(t) 2. Pytt
tEmb(t)
Causal
Temporal Attn
Causal Generation
Causal Generation
Causal Generation
Temporal KV
Temporal KV
Visual-Text
Cross Attn
Spotial
Spatial
Layer Norm
20
Pic
Denolsing Stage
2
Pe:Pxtl
Zg Pkes
text
Linear
(a) Training
Loss
Writing Stage
(b) Autoregressive Inference
Pan=BAt (c) Causal Generation Block
Figure 3: Overview of the Ca2-VDM pipeline. (a): During training, we randomly set P frames clean prefix, and set distinctive timestep embeddings, i.e., tEmb(0) for the clean prefix and tEmb(t) for the denoising target. (b): During inference, in each autoregression (AR) step, the model denoises an I-frame chunk conditioned on the spatial/temporal KV-caches shared across all timesteps (denoising stage), and then computes the keys/values of denoised chunk to update the KV-caches (cache writing stage). (c): Causal generation block. We further illustrate the details of causal temporal attention with Cyclic-TPEs in Figure and the prefix-enhanced spatial attention is left in the Appendix (cf. Figure 9).
(referred denoising target). The model parameterized by 0 is denoted as 60(2, t).
2020), the model can be trained by simplified objective:
Lsimple (0) E [| (z,, - <l E e~ ~0,10, (I)
The overall pipeline of Ca2-VDM is shown in Figure 3. We first illustrate the causal generation in the training stage (Sec. 3.2), as well as the training objectives. Then, we introduce the KV-cache realization combined with the cache sharing mechanism in the autoregressive inference stage (Sec. 3.3), and the queue structure for temporal KV-cache supported by Cyclic- (cf. Figure 4).
In our setting, each sample is partially noised. We randomly keep P consecutive frames uncorrupted as the clean pre- fix, and the remaining frames are treated as the denoising target, as shown in Figure 3(a). We use different timestep embeddings for the clean prefix (i.e., tEmb(0)) and the de- noising target (i.e., tEmb(tt). rather than unified timestep embedding for the whole video clip in many existing VDMs (Lu et al., 2024; Ma et al., 2025). This ensures the cache from the clean prefix can be correctly shared across each denoising timestep & at inference time (since the clean prefix is always assigned with tEmb(O)). Consequently, the simplified objective function for our model is
3.2. Causal Generation and Training Objectives
We first introduce the training objectives, followed by the causal generation block (cf. Figure 3()). Here we focus on the causal temporal attention and prefix-enhanced spatial attention layers. For the visual-text cross attention, it is widely used in VDMs for text-to-video generation (Rom- bach al., 2022; Chen et al., 2024a). And is optional for pure video prediction (Lu al., 2024). We refer readers to related works (Chen al., 2024a) for more details.
Example (0) [[](60((2) 0:P , 2. E) 2 (2) z.c,t
where [.,-] stands for concatenation along the temporal axis, and is the timestep vector with t; = t if 2 P else 0. m E (0, 1 a loss mask to exclude the clean prefix part, i.e., with m, if 2 P clse O. In practice, we train the model with learnable covariance Eg by optimizing combination of Lsimple and Lyb (with the same loss mask) following (Nichol & Dhariwal, 2021; Peebles & Xie, 2023). More details are left in Sec. B.
Training Objectives. Existing diffusion models (Ho et al., 2020; Peebles & Xie, 2023) are trained with the variational lower bound of 20 's log-likelihood, formulated as Lvm (0) =
2o) sp)l (2: (z)),, where D is determined by the mean μ4 and covariance Eg. By re-parameterizing pao as noise prediction network Eg and fixing Eg as a constant variance schedule (Ho et al.,
- -log 21) + E, D (q((((((11)))
Causal Temporal Attention. To introduce the causality, we mask the attention map to force each frame to only

---

Title Suppressed Due to Excessive Size
attend to its preceding frames, shown in Figure 4(a). Specifically, the input to each layer is first permuted by treating the spatial resolution H x W as the batch dimension, and then linearly projected to query, key, and value features as Q, K, V E Lx (for every spatial grid). The causal attention is computed as
TPE iandom
TPE ProPett
Attention Map Ix (P,+0
OKV Linear
Attention Map
OKV Linear
Q
Pg
Cache Writing
K
cached
V
cached
QKT VC +M) V, (3)
Attention
Attention
KV-cache queue (bound with TPE)
CausalAttn( Q, K, =)=Smmxxx
(a) Training
(b) Inference (denoising cache writing)
where M E IR 1xL is a lower triangular attention mask with M = 00 if < j else Note that we only describe one attention head and omit the diffusion step for brevity.
Paxxe
Paxx
[0:]]
AR step
K. dequeue
cyclic shift
[0:1]
[Pa:P +
bound
[t:0]
[0:
re-assign
Prefix-Enhanced Spatial Attention. In analogy to causal temporal attention, integrating the clean prefix and denois- ing target into one attention sequence helps enhance the guidance of conditional information. Inspired by prior works (Hu, 2024; Ren et al., 2024), we do this via spatial- wise concatenation (ef. Figure 9 in the Appendix). Let h.:. E R xC' be the hidden input to each layer, where the number of frames I, is treated as batch dimension and H x Wix is flattened for attention calculation. We take sub-prefix of length P' and concatenate it to the denoising target. Specifically, for hi from the i-th frame, the query is ((1 = hi The prefix-enhanced key is
AR step
1:21
A
P
K, V
[:21]
[Pees Pao1:
w/o KV-cache, TPE is re-assigned to input
w/ KV-cache, TPE bound KV
(c) Original TPES' reassignment (left) vs. Cyclic-PPss (right)
Figure 4: Illustration of causal temporal attention (a) & (b) and the temporal K KV-cache queue with Cyclic-TPEs (c). In (c), Luain Paux + I and Pk+1= Pk + I. We show the state that autoregressive inference reaches Ps Paax.
at AR step k; each denoising step samples Pa:Pa-l 2t-1
(= Pe Pt 1 Iz Pu+t 20.P)). Here =o ,..Pk serves as the clean prefix and 2, 2. is the denoising target. Benefiting from the causal generation, the feature computation is uni- directional. This means z Pu:Pet is denoised conditioned on 2 0:P while the cache of 2 0:P% could be precomputed in previous autoregression steps without referring to 2, 2.2xPx++ In the cache writing stage, the denoised 2)" PP:PPe+ is input to the model again to compute its clean spatial and temporal KV-caches, which will be used in the next AR step.
W P-P" ;...; ho -1; :hi
W [ W K [h: ...; hd]
ifiz P ifi<p'
((1)-
(4)
where [::-] stands for concatenation along the spatial dimen- sion, and hij is broadcasted by self-repeat Ir times for every i < P (i.e., the clean prefix part). We do the same opera- tion to obtain the prefix-enhance enhanced value V. Consequently, for every frame, the prefix- wrefx-ennhnneed spatial attention is computed as Attention( Q, K, V) with an attention map of shape (HW) x ((P' + H)) In practice, P' is relatively small (e.g., P' 3). as the computational cost scales pro- portionally with P', while adjacent prefix frames tend to exhibit similar appearances. We empirically show that prefix enhancement improves the generation quality (cf. Table 4).
Temporal KV-Cache. Suppose that there are Ps generated frames (i.e., the clean prefix) at AR step k. In the denois- ing stage. the query, key, and value features at timestep are Q Pe+ K. Pe-P V.Pa E Rixx' (consider- ing only one spatial grid). The model reads the clean key and value caches as K. Vo0.% E PRx'" Then, they are concatenated to the noisy ones as K(k,t) and t(k, [v.0:P VV., P.PPPPtt Fi-
3.3. Autoregressive Inference with Cache Sharing
O TKO.P. K.P.Petii nally, the causal temporal attention is computed as:
We first introduce an overview of the autoregressive infer- ence equipped with cache sharing, as shown in Figure 3(b). Then for each autoregression step, we illustrate the temporal KV-cache queue and cyclic temporal positional embeddings (Cyclic-TPEss. Finally, we introduce the spatial KV-eache for prefix- weefi-eenhanced spatial attention.
CausalAutn(( Q Pr K(k, (k.
(5)
where the attention map has a shape of x (P& + 1). as shown in Figure 4(b). During denoising, the clean KV- cache Ki and D--.% are shared for every timestep t. In the cache writing stage, the clean temporal keys and values are computed as K K. and P.P.P, They are then updated into the KV-cache queue, resulting in K0 0P. and 00+ which will be used in AR step k + 1 (Le., Ps41 = P& + 1). As the autoregression progresses, the earliest KV-cache will be dequeued when the length of the
Autoregressive Inference. The model starts from given first frame and generates an I-frame chunk per AR step. Each AR step consists of denoising stage and a cache writing stage. The spatial and temporal KV-caches are shared across every denoising timestep (i.e., cache shar- ing). In the denoising stage, given Ps generated frames
5

---

Title Suppressed Due to Excessive Size
clean prefix Pk reaches a predefined mmax (i.e., maximum number of conditional frames), as shown in Figure 4(c).
4. Experiments
4.1. Experimental Setup
Cyclic-TPEs: Assume that the model was trained on video clips with a maximum length of Luain = maax + I (i.e., with Pmn.. frames clean prefix and 7 frames denoising tar- get). Liain is also the maximum length of TPE sequence during training. As the autoregressive inference progresses till Px= = P'max. the TPEs are used up. When KV-cache is disabled (ef. Figure 4(c)-left), to align the training pattern, we can re-assign the TPES from scratch after the earliest clean frames are dequeued. However, when KV-cache is enabled (ef. Figure 4(c)-right), the TPES were bound to keys and values at previous AR steps and had been stored in preceding KV-cache chunks. As a result, we cannot do reassignment to match the training pattern of TPEs, Here we introduce a cyclic shift mechanism, where the denois- ing target will be assigned those TPEs indexed from the beginning. To support the training/inference alignment of Cyclic-TPEs, in the training stage, each sample is assigned a TPE sequence that is cyclically shifted with a random offset.
Model Details and Baselines. We built Ca2-VDM based on spatial-temporal Transformer following (Ma et al., 2025; Chen et al., 2024a) and initialized it with Open-Sora v1.0 (Zheng al., 2024). Following PixArt-a (Chen et al., 2024a), we used T5 (Raffel et al., 2020) as the text en- coder and used the VAE from StableDiffusion (Rombach et al., 2022). The length of the clean prefix was randomly sampled according to the multiples of chunk length I, i.e, P E {1, 1 + ,,...,I 1 + nl) and Pmux = 1 + nl. We used training videos of various lengths with Luain = P L. As comparisons, we built two bidirectional baselines (cf. Fig- ure I(a)) based on the same Open-Sora v1.0: One was trained with fixed-length conditional frames (denoted as OS-Fix), where P is fixed as P = m 2 in training and inference. The other was trained with autoregressively ex- tendable conditional frames using the same training configs as Ca2-VDM (denoted as OS-Ext).
Spatial KV-Cache. Let h Pl be the input to the prefix- enhanced spatial attention at AR step k. In the denois- ing stage, the keys and values from the denoising target are enhanced by the spatial KV-cache (a sub- sub-prefix of P' frames) via spatial-wise concatenation. In the cache writ- ing stage, the denoised latent frames are first enhanced via self-repeat and then computed to obtain the clean spatial keys and values. These operations are aligned with the prefix-enhancement in Eq. (4) of the training stage. Since P' is relatively small (P' < I), the prefix enhancement for the current denoising target h Pa only depends on spa- tial KV-cache from the most recent generated chunk (i.e., h h PP--I:P)). ). Thus, in contrast to the queue structure for tem- poral KV-cache, we only store the spatial KV-cache for one chunk and overwrite it at every AR step.
Training Details We conducted training on the text-to- video (T2V) generation and video prediction (i.e., with- out text prompt) tasks. For T2V generation, we trained OS-Fix and Ca2-VDM on a large-scale video-text dataset Intern Vid (Wang et al., 2024), by filtering it to a sub-set of 4.9M high-quality video-text pairs. The models were trained video clips at resolution 256x256 with /=16 and P 1 + 3/ = 49. For video prediction, we trained OS-Fix, OS-Ext, and Ca2- VDM on the SkyTimelapse (Zhang et al., 2020) dataset at resolution 256x256 with 1=8. OS-Ext and Ca2-VDM both used Pmax= = 1 + 31 = 25. OS-Fix used a fixed P= 8. More hyperparameters are left in Sec. C.
Evaluation Datasets and Metrics. We used MSR-VTT (Xu et al., 2016). UCF101 (Soomro et al., 2012), and SkyTime lapse (Zhang et al., 2020) datasets at resolution 256x256 and reported Fréchet Video Distance (FVD) (Unterthiner et al., 2019) following previous works (Zeng et al., 2024; Ge et al., 2023; Chen et al., 2024b). More details about choosing text prompts and computing FVD scores on these datasets are left Sec. D
Discussion. It's worth noting that our KV-cache queue for autoregressive VDMs is not trivial extension of the KV- cache techniques from large language models (LLMs): 1) LLMs predict the next token at each AR step, and the KVs are computed and cached simultaneously in each forward call. For VDMs, however, the model is repeatedly called during denoising (with different t). This brings the cache computation and storage issues as introduced in See. 1. Our implementation solves these two issues, sharing the cache across every denoising step. 2) Caching visual KVs costs much more storage than KVs for text since each token in our setting corresponds to HW visual grids. The queue structure for KV-cache is essential for VDMs considering this heavy storage cost. Early KVs can be safely dequeued as the appearance and motion of new frames are primarily influenced by the most recent KVs. Meanwhile, we propose Cyclic-TPE TPEs to facilitate this mechanism.
4.2. Evaluation for Generation Quality
We first compared the in-chunk generation quality of Ca2- VDM with SOTA VDMs. Then, we evaluated the temporal consistency of the autoregressive generation. Finally, we conducted ablation studies on Ca2-VDM's design choices.
In- In-Chunk Generation Quality. We evaluated the zero- shot text-to-video (T2V) FVD scores on MSR-VTT (Xu et al., 2016) and UCF101 (Soomro et al., 2012), as shown in Table 1. We compared Ca2-VDM to of T2V models including two groups: 1) Text conditioned: Mod- elScope (Wang et al., 2023b), VideoComposer (Wang et al.,
6

---

Title Suppressed Due to Excessive Size
Table 1: Zero-shot FVD scores on MSR-VTT (Xu et al.. 2016) Table 2: Finetuned FVD scores on UCF-101 (Soomro and UCF101 (Soomro et al., 2012) test sets. All methods generate video at resolution of x6x256x 256. C: condition. T and I are
et al., 2012) test set. Methods with were trained on both train and test sets.
text and image conditions, respectively.
Method
Res.
FVD
Method
C
MSR-VTT
UCFI01
MCVD (Voleti et al., 2022)
644
1143
ModelScope (Wang et al., 2023b)
T
550
410
VideoComposer (Wang et al., 2023c)
T
580
VDT (Lu et al., 2024)
64
225.7
Video-LDM (Blattmann et al., 2023b)
T
550.6
DIGAN" (Yu et al., 2022)
128²
577
PYoCo (Ge et al., 2023)
T
355.2
TATS (Ge et al., 2022)
128
420
Make-A- Video (Singer al., 2023)
T
367.2
VideoFusion (Luo et al., 2023)
128?
220
Animale-nythinng (Dai al., 2023)
T+I
443
LVDM' (He ct al., 2022)
256
372
PixelDance (Zeng el al., 2024)
T+I
381
242.8
PVDM (Yu et al., 2023)
256²
343.6
SEINE (Chen et al., 2024b)
T+I
181
Latte (Ma al., 2025)
256²
333.6
Ca2-VDM
T+1
181
277.7
CD2-VDM
256
184.5
Table 3: FVD results on MSR-VTT test set.
Table 4: Ablations of Pmmx and prefix-enhancement (PE) on Sky SkyTimelapse (Zhang et al., 2020). Each variant of Ca2- VDM generated 48 frames by AR steps. The results were divided into three 16-frame chunks for FVD evaluation.
Method
FVD between AR step and
i
i=2
i ==
3
i=
i
5
i i=6 6
GenLV
282,8
291.4
299.0
318.2
310.3
StreamT2V
317.5
434.7
478.2
462.0
512.4
Chunk Id
OS-Fix
182,9
210.6
260.8
284.3
315.1
Panx
PE
2
Ca2-VDM
160.6
206.5
262.8
281.3
304.7
25
x
274.8
244.5
275.1
25
/
257.4
216.5
238.5
41
187.3
209.3
263.2
2023c), Video-LDM (Blattmann et al., 2023b), PYoCO (Ge et al., 2023), and Make-A A- Video (Singer et al., 2023). 2) Text with extra image conditioned, e.g., for image-to-video- AnimateAnything (Dai et al., 2023), PixelDance (Zeng et al., 2024) and video transition: SEINE (Chen et al., 2024b). We also finetuned Ca2-VDM on UCF101 at res- olution 16x256x and reported the FVD scores in Ta- ble 2. We compared it with SOTA video generation models: MCVD (Voleti et al., 2022), VDT (Lu et al., 2024), DI- GAN (Yu et al., 2022), TATS (Ge et al., 2022), LVDM (He al., 2022), PVDM (Yu al., 2023), and Latte (Ma al., 2025). The FVD results in both Table and Table 2 show that our Ca2- RD has competitive T2V performance with SOTA models. More qualitative examples are left in Sec. E.
V
185.0
202.9
240.5
the others. This indicates that extendable (long-term) con- dition helps to improve the temporal consistency. We also show qualitative examples in Figures 5. It shows content mutations in consecutive frames from the results of fixed- length condition methods, e.g., the 24th and 25th frames in GenLV, and the 65th and 66 frames in Stream We further compared Ca2-VDM with the condition extendable baseline, i.e., OS-Ext (cf. Figure 7). We see that Ca2-VDM shows comparable results with OS-Ext (while being more computationally efficient as demonstrated in Sec. 4.3). We conducted further comparisons between Ca2-VDM and OS- Ext in terms of video quality and long-tern content drift. The results are left in Sec. E of the Appendix.
Temporal Consistency. We compared Ca2-VDM with the two baselines (i.e,, OS-Fix and OS-Ext) and existing SOTA autoregressive VDMs. To the best of our knowledge, existing autoregressive VDMs all use fixed-length condi- tional frames (similar to OS-Fix). We used Gen-L-Video (GenLV) (Wang et al., 2023a) and StreamT2V (Henschel et al., 2025). Specifically, GenLV utilizes a base model AnimateDiff (Guo et al., 2024b) and conducts co-denoising for overlapped 16 16-frame clips. We implemented it with an overlapping length (i.e., the condition length) of 8 frames. StreamT2V is based on Stable Video Diffusion (Blattmann et al., 2023a) and finetunes it conditioned on preceding frames to generate subsequent frames. It also generates 16 frames at each AR step, with 8 frames as the condition.
Ablation Studies. We studied the effectiveness of longer condition length and the prefix-enhancement (PE) in spatial attention (cf. Eq. (4)). We trained variants of Ca2- VDM with different Pmax or without PE. The results are reported in Table 4. Each model was called with 6 AR steps to generate 49 frame video (with the given first frame) and evaluated by the FVD scores of three 16-fram f6-frame chunks (exclude the first frame) w.r.t. the 16-frame ground-truth videos. We can see that both increasing Pmax and using PE are beneficial in improving the generation quality.
4.3. Evaluation for Autoregression Efficiency
We evaluated the FVD scores of each autoregression (AR) chunk w.r.t. the first chunk, as shown in Table 3. We can ob- serve that Ca2-VDN has relatively lower FVD scores than
We evaluated the efficiency in two aspects: 1) time cost for autoregressive generation, and 2) detailed computational costs for each component in the Transformer blocks.
7

---

Title Suppressed Due to Excessive Size
GenLy
OS-Ext, 25
OS-Ext. Pum =41
120
Ca2-VDM, Piie
Accumulated Time (s)
Ca2- Ca2-VDM, Pinxx
Stream
100
OS-Fix, P
60
OS-
40-
20-
Cx2-VOM
Frame ids
Figure 6: Accumulated time cost frame
Figure 5: Qualitative examples from GenLV (Wang al., 2023a),
ids. We show OS-Ext and Ca2-VDM with Paax 25 and 41, and OS-Fix with a fixed p
StreamT2V (Henschel et al., 2025), OS-Fix (Zheng et al., 2024), and Ca2-VDM. Yellow arrows highlight consecutive frames having mutations.
Table 5: Time cost for generating 80 frames at resolution 256 256. OS-Fix used P=8. OS-Ext and
OS-Ext
Ca2- VDM used P Pmax =25. Ext.C.
Ca2 VDM
means extendable condition.
Method
Ext.C. Time (s)
S2ream
150
OS-ExL
130.1
Figure 7: Results from OS-Ext and Ca2-VDM, They have comparable quality, while Ca2-VDM is more computationally efficient, as evidenced in Table 5, Figure 6 and 8.
OS-Fix
77.5
Ca2-V VDM
V
52.1
08-Fi
16/44
see that Ca2-VDM sigmificantly improved over OS-Fix, OS- Ext, and St (Henschel al., 2025), while being compatible with extendable condition. We further evaluated the accumulated time cost till each AR step, as shown in Figure 6. We observe that: 1) Compared to OS-Fix, the time cost in Ca2-VDM has a clear reduction since it does not have redundant computations. 2) As the condition extends, the time cost of OS-Ext grows quadratically (before P Pmax is reached), while the time cost of Ca2-VDM only grows linearly. 3) the Puux grows incorporate longer condition, the increase of time cost for OS-Ext is significant, while it is relatively slight for Ca2-VDM.
Sparal Asset on
2.108 388 1.75
Ca2-VDM
32.304
2458 Lons
Feax
02-Ext
Swar
2,000 2353 1015
Cx2.VOM
O8-Ext
35:377
101)
Figure 8: Number of floating-pon point operations (FLOPs) for generating 56 frames (7 AR steps). All results were com- puted by conducting only one denoising step for simplicity.
Computational Cost. We counted the floating -pinnt opera- tions (FLOPs) of temporal, spatial, and visual-text attention layers the Transformer blocks (cf. Figure 8). As the Pmsx grows, the increased computations are seen in all three types of attention layers for OS-Ext. In contrast, for Ca2-VDD VD-M. the number of FL.OPs only slightly increases in the temporal attention, while keeping constant in other operations. This is because the extended conditional frames only participate in the computation as temporal K KV-caches.
Time Cost. We first show the cumulative time cost of autoregressive generation in Table 5. Our models were tested on a single NVIDIA A100 GPU to generate 80 frames at resolution 256x using improved DDPM (Nichol & Dhariwal, 2021) with 100 denoising steps. The result of StreamT2V (Henschel et al., 2025) is from its GitHub page, which was tested on the same device and resolution. We can
Memory Cost. We conducted empirical GPU memory
8

---

Title Suppressed Due to Excessive Size
Table 6: GPU memory usage comparison between Live2diff (Xing et al., 2024) and Ca2-VDM. The comparisons are not strictly aligned since Live2diff is based. The resolution of the generated video is 256 x 256. is the number of generated frames at each auto-regression step. and W are after 8x VAE down sampling. The values of N'w' and C' vary across blocks due to the down-sampling and up-sampling in Unet. PE means arefix-enhancement (ef. E4.(
Denoising Steps (T)
Model Forward Shape
KV-cache Shape
KV-cache
Total
Method
(B, C, I, H, W)
(T, Tegoad low, C*)
Memory Cost
Memory Cost
Live2diff
4
(4, 32,
(4, 16, h'w', C*)
1.42 GB
10,90 GB
Live2diff
50
(50, 4, 1, 1, 32) (1,4, 8, 32)
(50, 16, W'w, C*)
17.70 GB
29.46 GB
Ca2-VDM w/ PE
50
(1, 25, hw,, C)
0.86 GB
4.79 GB
Ca2-VDM w/o PE
50
(1, 4, 8, 32,
(1, 25, hw, C)
0.77 GB
3,95 GB
statistics, as shown in Table 6. We compared Ca2-VDM
ported by the 2024-2025 Grant for Pursuing Outstanding Doctoral Dissertations of Zhejiang University.
with a concurrent work, Live2diff (Xing et al., 2024). It stores KV-cache for every denoising step (with different noise levels t and thus different KV features), which costs much more GPU memory than ours. Note that Live2diff uses a batch size that is equal the number of denoising steps, f.e., B = T. This is because it uses pipeline denois- ing following StreamDiffusiio (Kodaira et al., 2023), which puts frames with progressive noisy levels into a batch and generates one frame each autoregression step. Benefited from cache sharing, 2a2-sDM's memory cost is indepen- dent of denoising steps, as its fixed shape (1, 25, hw, C)) ensures constant memory usage. In contrast, Live2diff's memory cost scales with T (e.g,, from 1.42 GB 7 = 4 to 17.70 GB at T' = 50), confirming that cache sharing saves Tx GPU memory. As a result, Ca2-VDM requires only 0.86 GB (w/ PE) or 0.77 GB (w/ PE), with the difference due to spatial KV-cache for prefix-enhancement (PE).
Impact Statement
Our Ca2-VDM is a generic fast video generation paradigm. It is potentially powerful to boost existing VDMs to gen- erate high-quality live-stream videos. The live-stream (or real-time) video generation techniques have a revolutionary impact on the field of content creation industry, and have great potential commercial values. Mcanwhile, it's neces- sary to note that Ca2- also has the inherent risks of common image/video generation models, such as generating videos with harmful or offensive content, being used by malicious actors for generating fake news. We can use some watermarking technologies (e.g., (Lukas & Kerschbaum, 2023)) to avoid the generated videos being abused.
References
5. Conclusions
Alonso, E., Jelley, A., Micheli, V., Kanervisto. A., Storkey, A. J., Pearce, T., and Fleuret, F. Diffusion for world modeling: Visual details matter in atari. NeurIPS, 37: 58757-58791, 2024.
In this paper, we present an efficient autoregressive video diffusion model, ie., Ca2-VDM. It has two key designs: causal generation and cache sharing. The former eliminates the redundant computations of conditional frames. The latter significantly reduces the storage cost. Our model shows comparable generation quality with existing SOTA VDMs with existing bidirectional attention while achieving notable speedup for the autoregressive generation.
Blattmann, A., Dockhorn, T., Kulal, S., Mendelevitch, D., Kilian, M., Lorenz, D., Levi, Y., English, Z., Voleti, V., Letts, A., et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint MrXiv:2311.15/22 2023a.
Acknowledgements
Blattmann, A., Rombach, R., Ling, H., Dockhorn, T., Kim, S. W., Fidler, S., and Kreis, K. Align your latents: High- resolution video synthesis with latent diffusion models. In CVPR, pp. 22565-22575 2023b.
This work was supported by the National Key Research & Development Project of China (2024YFB3312900) Key R&D Program of Zhejiang (2025C01 128), an Fundamental Research Funds for the Central Universities. Long Chen was supported by the Hong Kong SAR RGC Early Career Scheme (26208924), the National Natural Science Foun- dation of China Young Scholar Fund (62402408), Huawei Gift Fund, and the HKUST Sports Science and Technology Research Grant (SSTRG24EG04). Kaifeng Gao was sup-
Chen, J., Jincheng, Y., Chongjian, G., Yao, L., Xie, E., Wang. Z., Kwok, J., Luo, P., Lu, H., and Li, Z. Pixart-o: Fast training of diffusion transformer for photorealistic text-to-image synthesis. In ICLR, 2024a.
Chen, X., Wang. Y., Zhang, L., Zhuang, S., Ma, X., Yu, J., Wang, Y., Lin, D., Qiao, Y., and Liu, Z. Seine: Short-to-

---

Title Suppressed Due to Excessive Size
long video diffusion model for generative transition and prediction. In ICLR, 2024b.
text-to-video generation. arXiv e-prinis, pp. arXiv-2305, 2023.
Dai, Z., Zhang, Z., Yao, Y., Qiu, B., Zhu, S., Qin, L.,, and Wang, W. tnin Fine-grained open domain image animation with motion guidance. arXiv e-prints, pp. arXiv-2311, 2023.
Hu, Animate anyone: Consistent and controllable image- to-video synthesis for character animation. In CVPR, pp. 8153 81338163,22 2024.
Huang. Z., He. Y., Yu, J., Zhang, F., Si, C., Jiang, Y., Zhang, Y., Wu, T., Jin, Q., Chanpaisit, N., Wang, Y., Chen, X., Wang, L., Lin, D., Qiao, Y., and Liu, Z. Vbench: Com- prehensive benchmark suite for video generative models. In CVPR, 2024.
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenbom,
D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M.,, Heigold, G., Gelly, S., et al. An image is worth 16x words: Transformers for image recognition at scale. In ICLR, 2020.
Khachatryan, L., Movsisyan, A., Tadevosyan, V., Henschel, R., Wang, Z., Navasardyan, S., and Shi, H. Text2video- zero: Text-to-image diffusion models are zero-shot video generators. In ICCV, pp. 15954-15964, 2023.
Ge, S., Hayes, T., Yang, H., Yin. X., Pang. G., Jacobs, D., Huang, J.-B., and Parikh, D. Long video generation with time-agnostic vqgan and seneisenii transformer. In ECCV, pp. 102-118. Springer, 2022.
Kodaira, A., Xu, C., Hazama, T., Yoshimoto. T., Ohno, K., Mitsuhori, S., Sugano, S., Cho, H., Liu, Z., and Keutzer, K. Streamdiffusion: A pipeline-level solu- tion for real-time interactive generation. arXiv preprint arXiv:2312. 122491 2023.
Ge, S., Nah, S., Liu, G., Poon, T., Tao, A., Catanzaro, B., Jacobs, D., Huang, J.-B., Liu, M.-Y., and Balaji, Y. Preserve your own correlation: A noise prior for video diffusion models. In ICCV, pp. 22930-22944 2023.
Loshchilov, and Hutter, Decoupled weight decay regu- larization. ICLR, 2019.
Girdhar, R., Singh, M., Brown, A., Duval, Q., Azadi, S., Rambhatla, S. S., Shah, A., Yin, X., Parikh, D., and Misra, I. Emu video: Factorizing text-to-video generation by explicit image conditioning. In ECCV, pp. 205-224, 2024.
Lu, H., Yang, G., Fei, N., Huo, Y., Lu, Z., Luo, P., and Ding, M. Vdt: General-purpose video diffusion transformers via mask modeling. In ICLR, 2024.
Guo. Y., Yang, C., Rao, A., Agrawala, M., Lin, D., and Dai, B. Sparsectrl: Adding sparse controls to text-to-video diffusion models. In ECCV, pp. 330-348, 2024a.
Lu, Y., Zhu, L., Fan, H., and Yang, Y. Flowzero: Zero-shot text-to-video synthesis with llm-driven dynamic scene syntax. arXiv preprint arXiv:2311.15813 2023.
Guo, Y., Yang, C., Rao, A., Liang, Z., Wang, Y., Qiao, Y., Agrawala, M., Lin, D., and Dai, B. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. In ICLR, 2024b.
Lukas, N. and Kerschbaum, F. Ptw: Pivotal tuning wa- termarking for pre-trained image generators. In 32nd USENIX Security Symposium (USENIX Security 23), pp. 2241-22555 2023.
Harvey, W., Naderiparizi, S., Masrani, V., Weilbach, C., and Wood, F. Flexible diffusion modeling of long videos. In NeurIPS, volume 35, pp. 27953-27965. 2022.
Luo, Z., Chen, D., Zhang, Y., Huang, Y., Wang, L., Shen, Y., Zhao, D., Zhou, J., and Tan, T. Videofusion: Decomposed diffusion models for high-quality video generation. In 2023 IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10209-10218, IEEE Computer Society, 2023.
Hc, Y., Yang, T., Zhang, Y., Shan, Y., and Chen, Q. La- tent video diffusion models for high-fidelity long video generation. arXiv preprint arXiv:2211. 132211 2022.
Ma, X., Wang, Y., Chen, X., Jia, G., Liu, Z., Li, Y.FF, Chen, C., and Qiao, Y. Latte: Latent diffusion transformer for video generation. TMLR, 2025.
Henschel, R., Khachatryan, L., Hayrapetyan, D., Poghosyan, H., Tadevosyan, V., Wang, Z., Navasardyan, S., and Shi, H. Streamingt2v: Consistent, dynamic, and extendable long video generation from text. In CVPR, 2025.
Mokady. R., Hertz, A., Aberman, K., Pritch, Y., and Cohen- Or, D. Null-text inversion for editing real images using guided diffusion models. In CVPR, pp. 8038-6043 2023.
Ho. J., Jain, A. and Abbeel, P. Denoising diffusion proba- bilistic models. In NeurlPS, volume 33, pp. 2020.
Mou, C., Wang, X., Xie, L.. Wu, Y., Zhang, J., Qi, Z., and Shan, Y. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. In AAAI, volume 38, pp. 2296-4304 2024.
Hong, S.. Sco, J., Hong, S., Shin, H., and Kim. Large language models are frame-level directors for zero-shot
10

---

Title Suppressed Due to Excessive Size
Nichol, A. Q. and Dhariwal, P. Improved denoising diffusion probabilistic models. In ICML, pp. 8162-8171. PMLR, 2021.
Voleti, V., Jolicoeur-Martnnaau, A., and Pal, C. Mevd masked conditional video diffusion for prediction, gen- eration, and interpolation. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), NeurIPS, volume 35, pp. 23371-23385 Curran Asso- ciates, Inc., 2022.
Oh, G., Jeong, J., Kim, S., Byeon, W., Kim, J., Kim, S., Kwon, H., and Kim, S. Mtvg: Multi-text video generation with text-to-video models. In ECCV, 2024.
Peebles, W. and Xie, S. Scalable diffusion models with transformers. In ICCV, pp. 4195-4205, 2023.
Wang, F.-Y., Chen, W., Song, G., Ye, H.-J., Liu, Y., and Li, H. Gen-l-video: Multi-text to long video generation via temporal co-denoising. arXiv preprint arXiv:2305. 18264, 2023a.
Qiu, H., Xia, M., Zhang, Y., He, Y., Wang, X., Shan, Y., and Liu, Z. Freenoise: Tuning-free free longer video diffusion via noise rescheduling. In ICLR, 2024.
Wang, J., Yuan, H., Chen, D., Zhang, Y., Wang, X., and Zhang, S. Modelscope text-to-vides viveo technical report. arXiv preprint (tth:2308.06577/, 2023b.
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li. W., and Liu, P. J. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research, 21 (140):1-67, 2020.
Wang, X., Yuan, H., Zhang. S., Chen, D.. Wang. J., Zhang.
Y., Shen, Y., Zhao, D., and Zhou, J. Videocomposer: Compositional video synthesis with motion controllabil- ity. NeurlPS, 36, 2023c.
Ren, W., Yang, H., Zhang, G., Wei, C., Du, X., Huang, W., and Chen, W. Consisti2v: Enhancing visual consistency for image-to-video generation. Transactions on Machine Learning Research, 2024. ISSN 2835-8856.
Wang. Y., He, Y., Li, Y., K., Ma, X., Li, X., Chen, G., Chen, X., Wang, Y., et al. Internvid: A large- scale video-text dataset for multimodal understanding and generation. In ICLR, 2024.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, High-resolution image synthesis with latent diffusion models. In CVPR, pp. 10684 10695, 2022.
Weng, W., Feng, R., Wang, Y., Dai, Q., Wang, C., Yin, D., Zhao, Z., Qiu, K., Bao, J., Yuan, Y., et al. Art-v: Auto- regressive text-to-video generation with diffusion models. CVPR, pp. 7395-7405, 2024.
Singer, U., Polyak, A., Hayes, T., Yin, X., An, J., Zhang, S., Hu, Q. Yang, H., Ashual, O., Gafni, O., et al. Make-a- video: Text-to-video generation without text-video data. In ICLR, 2023.
Xing, Z., Fox, G., Zeng. Y., Pan, X., Elgharib, M., Theobalt, C., and Chen, Live2diff: Live stream translation via uni-directional attention in video diffusion models. arXiv preprint 240.::2007.0800,,, 2024.
Skorokhodov, I,, Tulyakov, S., and Elhoseiny, M. Stylegan- v: A continuous video generator with the price, image quality and perks of stylegan2. 2022 ieee. In CVPR, 3616-3626, 2021.
Xu, J., Mei, T., Yao, T., and Rui, Y. Msr-vtt: A large video description dataset for bridging video and language. In CVPR, pp. 5288-5296 2016.
Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., and Ganguli, S. Deep unsupervised learning using nonequi- librium thermodynamics. International conference on machine learning, pp. 2256-2265 PMLR, 2015.
Yu, S., Tack, J., Mo, S., Kim, H., Kim, J., Ha, J.W.., and Shin, J. Generating videos with dynamics-awan implicit generative adversarial networks. In ICLR, 2022.
Song. J., Meng, C., and Ermon, S. Denoising diffusion implicit models. In ICLR, 2021a.
Yu, S., Sohn, K., Kim, S., and Shin, J. Video probabilistic diffusion models in projected latent space. In CVPR, pp. 18456-18466, 2023.
Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Er- mon, S., and Poole, B. Score-based generative modeling through stochastic differential equations. In ICLR, 2021b.
Zeng, Y., Wei, G., Zheng, J., Zou, J., Wei, Y., Zhang, Y., and Li, H. Make pixels dance: High-dynamic video generation. In CVPR, pp. 8850-8860, 2024.
Soomro, K., Zamir, A. R., and Shah, M. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint 1rXiv: 2012.
Unterthiner, T., van Steenkiste, S., Kurach, K., Marinier, R., Michalski, M., and Gelly, S. Fvd: A new metric for video generation. In ICLR 2019 Workshop DeepGenStruct, 2019.
Zhang, J.. Xu, C., Liu, L., Wang, M., Wu, X., Liu, Y., and Jiang, Y. Dtvnet: Dynamic mapelapse video generation via single still image. In ECCV, pp. 305-315. Springer, 2020.

---

Title Suppressed Due to Excessive Size
Zhang, L.., Rao, A., and Agrawala, M. Adding conditional control to text-to-image diffusion models. In CVPR, pp- 3836-3847, 2023a.
Zhang, S., Wang, J., Zhang, Y., Zhao, K., Yuan, H., Qin, Z.,, Wang, X., Zhao, D., and Zhou, J. 12ggen-xl: High-quality image-to-video synthesis via cascaded diffusion models. arXiv preprint arX4::231.004145, 2023b.
Zhang, Y., Wei, Y., Jiang, D., ZHANG, X., Zuo, W., and Tian, Q Controlvideo: Training-free free controllable text-to- video generation. In ICLR, 2024.
Zheng, Z., Peng, X., Yang. T., Shen, C., S.,, Liu, H., Zhou, Y., Li, T., and You, Y. Open-sora: Democratizing efficient video production for all, March 2024. URL https: //github. www.coo/////..ee haam//pcaatech/peee. Sora.
12

---

Title Suppressed Due to Excessive Size
Appendix
clean prefix
denoising target
Sec. A: Illustration of Prefix-enhanced Spatial Atten- tion
TSP
Sec. B: Detailed Training Objectives
CKV Unear
OKV Linear
HEW
Attentinn
RW
[ceched] K
Sec. C: Training Details and Hyperparameters
[cached]
reseat
(P 13HW
(M' 13RW
Sec. D: Evaluation Details
Train: 1 P Infer: cache writing stage
Train: 2 P. Infor: donolsing stage
Sec. E: More Experiment Results
Figure 9: Illustration of eeefi-eenhance spatial attention. For i 2 P, the left part of K, V is from clean prefix (in training) or cached K, V (in the denoising stage of infer- ence).
Sec. Limitations and Possible Future Directions
A. Illustration of Prefix-enhanced Spatial Attention
We provide more details of Prefix-enhanced Spatial Atten- tion (cf. Eq. (4)) in Figure 9.
4.9M high-quality video-text pairs with resolution 256x256. For 2D2-VDM, the training consists of two stages. We first train the causal modeling ability without the clean prefix (i.e., without conditional frames) on 32- 22-frame videos. Then we use longer videos of 65 frames to train the model with the clean prefix, i.e., with = 16, Pmax = 1 + 3/ = 49 and max [Tmain) = Pmax + 1 65. In the first stage, the model was trained with a batch size of 288 for 32k steps. In the second stage,
B. Detailed Training Objectives
Recall that (ef. Sec. 3.2 in the main text) existing diffusion models (Ho et al., 2020; Nichol & Dhariwal, 2021; ; Peebles & Xie, 2023) are trained with the variational lower bound of zoos log-likelihood, formulated as
trained with batch size of 144 for 21k steps. For OS-Fix, it was trained with Luain = 32 frames and P = 1 = Lusin/ / = 16 frames, i.e. the prefix length is fixed. It was trained with a batch size of 288 for 20k steps
(0) = - log log.ol (z)lzz)
+ Σ DKL zo) (l0((((())))) (6)
Video Prediction Training. We trained OS-Fix, OS-Ext, and Ca2-VDM on the SkyTimelapse (Zhang et al., 2020) dataset at resolution 256 256 with | 8. OS-Ext and Ca2-V DDD both used Pauux 1 + 37 25 (i.e., Turain 33). OS-Fix used fixed P 8 and Lirain 16. All three models were trained with a batch size of 8 for 11k steps 4
Since q and po are both Gaussian, DRr. r. is determined by the mean 1so and covariance Eg. By re- p--arammttrriingg 140 as noise prediction network and fixing Σg as a constant variance schedule (Ho et al., 2020), the model can be trained using a simplified objective function:
Hyperparameters. For all the training, we used the DDPM (Ho et al., 2020) schedule with T = 1000, 31 = 10-4, and Br 0.02. The models were trained using AdamW (Loshchilov & Hutter, 2019) optimizer with learn ing rate of 2e-5. At the inference stage, used the im- proved DDPM schedule (Nichol & Dhariwal, 2021) with 100 steps. For text-to-video, we set the classifier-fre guid- ance scale as 7.5.
Cximple (0) = IE [z).) - <l c N(,,
(7)
z.c.t
In our setting, the simplified objective function is
Eximple (0)= z.c.! [[( 0.PP 2,P P.L.) 1, - c) mml?? (8)
Following prior works (Nichol & Dhariwal, 2021; Peebles & Xie, 2023), we train the model with learnable covariance Eg to improve the sampling quality. This is achieved by optimizing the full DEL term in Lvb. resulting in an Lvb in our setting, i.e., applied with the same timestep vector t and loss mask m. Then, the model is optimized by a combined loss function Eximple + Lvlb-
D. Evaluation Details
D.I. Datasets
MSR-VTT (Xu et al., 2016). we used its official test split which contains 2990 videos, with 20 manually annotated
C. Training Details and Hyperparameters
OSS- Fix converges faster than Ca2-VDM. since it only needs to learn fixed-length conditional frames.
Text-to-Viae Video (T2V) Training. We trained Ca2-VDM and the OS- baseline on a large-scale video-text dataset In- tern Vid (Wang et al., 2024), by filtering it to sub-set of
'In contrast text-to-video, the video prediction task the Sky Timelaaee dataset has less diversity and converges faster. So we used smaller batch size and training steps.
13

---

Title Suppressed Due to Excessive Size
GenLV
First Frame
OS-Fix StreamT2V
First Frame
Text prompt:
Text prompt: "A hedgehog walking through field of wildflowers and dandelions
"A white tiger in 200
swimming in lake...
Ca2-VDM
Figure 10: Qualitative examples generated by GenLV (Wang al., 2023a), StreamT2V (Henschel al., 2025), OS-Fix, and Ca2-VDM. We sampled 32 frames with an interval of 8 frames for display. Note that GenLV does not strictly follow the given first frame, since it was not finetuned on explicitly injected conditional frames. In the implementation of GenLV, we used DDIM inversion to build the initial noise based on the first frame.
captions for each video. Following prior works (Ren et al., 2024; Zeng et al., 2024) and for fair comparisons, we ran- domly selected a caption for each video and generated 2990 videos for evaluation.
For the autoregressive generation results (e.g., the results in Table and Table 4), we calculated the chunk-wise FVD, Specifically, for Table 3, each model generated 48 frames with 6 AR steps and I = 8. Since the I3D model accepts evaluated the FVD scores of three 16-
least 16 frames, frame chunks (i.e., 2 AR steps in each) w.r.t. the 16-frame ground-trute videos. For Table 4, each model generated 96 frames with 6 AR steps and = 16. We evaluated the FVD scores of the generated 16- 16-frame chunk from each AR step w.r.t. the first AR step. Each model generated 512 videos for FVD calculation.
UCF101 (Soomro et al., 2012). As it only contains label names, we employed the descriptive text prompts from PY- oCo (Ge et al., 2023), and generated 2048 samples with uniform distribution for each category following (He et al., 2022; Ge et al., 2023; Ren et al., 2024).
Sky AkyTeellppse (Zhang et al., 2020). is time-lapse dataset showing dynamic sky scenes (e.g., cloudy sky with moving clouds). We used it for video prediction (i.e., without text input). Its training set contains 997 long timelapse videos, which are cut into 2392 short videos. Its test set contains 111 long timelapse videos, which are cut into 225 short videos. We trained the models on training set and evaluated them on its set.
E. More Experiment Results
In Figure 10 and Figure 11, we show more qualitative exam- ples from GenLV (Wang et al., 2023a). StreamT2V (Hen- schel et al., 2025), OS-Fix (Zheng al., 2024), and Ca2- VDM. We
that Ca2-VDM has comparable genera- tion quality to existing SOTA models.
D.2. Quantitative Evaluation
In Table 7, we evaluated Ca2-VDM and OS-Ext on the VBench (Huang et al., 2024) benchmark. VBench is pri- marily designed for text-to-video evaluation. For our assess- ment, we selected four metries: aesthetic quality, imaging quality, motion smoothness, and temporal flickering. The first two measure spatial (appearance) quality, and the last assess temporal consistency. The results in Table 7 show that Ca2-VDM achieves comparable performance in both appearance quality and temporal consistency.
Fréchet Video Distance (FVD) (Unterthiner et al., 2019) measures the similarity between generated and real videos based on the distributions on the feature space. We followed prior works (Blattmann et al., 2023b; Ge et al., 2022; Ren et al., 2024) to use a pretained I3D model to extract the features. We used the codebase from StyleGAN-V (Sko- rokhodov et al., 2021) to compute FVD statistics.
pectps://gitiin com http://owwwigg///ee /TASe//AA///A STATS/blob. /blbb/ http://fd//dd__deereeee
In Figure 12, we further compared the long-term content drift (i.e., error accumulation) between Ca2-VDM and the
14

---

Title Suppressed Due to Excessive Size
GenLV
StreamT2V
OS-Fix
Ca2-VDM
Figure 11: Qualitative examples from GenLV (Wang al., 2023a), StreamT2V (Henschel al., 2025), OS-Fix, and Ca2-VDM. Yellow arrows highlight the consecutive frames having mutations.
Table 7: VBench (Huang et al., 2024) evaluation on Sky- Timelapse (Zhang et al., 2020) test set. The resolution of the generated video is 256 256. Both models were evaluated with Paax= 25 and 6 autoregression steps.
temporal attention layers might have inherent gaps. Due to the limited computational resources, we did not con- duet causal pretraining. Pretraining the VDM's temporal modules from scratch (using causal attention) might have potential improvements.
Aesthetic Quality
Imaging
Motion Smoothness
Temporal Flickering
Method
Quality
Training Efficiency Trade-off. Ca2-VDM uses extendable conditional frames and cyclic TPEs. These designs require the model to learn all the possible situations during training, Compared to fixed-length conditional frames and conven- tional TPEs, the model needs more time to achieve training convergence. Meanwhile, the longer maximum condition length (i.e., Pmax) we use, the more training is required. On the other hand, once the model is trained, it is more powerful for integrating long-term context. Consequently. it's also potentially beneficial for long-term autoregressive video generation.
OS-Ext
44.39
50.74
98.93
98.57
Ca2-VDM
44.30
50.55
97.59
97.14
OS-Ext baseline. As result, they show comparable visual quality. Both models exhibit similar degree of error accu- mulation over time. Given our primary focus on efficiency, we conclude that Ca2-VDM matches the bidirectional base- line while being more efficient in both computation and storage for autoregressive video generation.
Quality Degradation in Long-term Generation. As common challenge, VDMs in long-term autoregressive gen- eration suffer from frame appearance changes and quality degradation. Some works (Henschel et al., 2025; Zhang et al., 2023b) mitigate this issue by providing the VDM with the global appearance information extracted from the initial frame. However, during the long-term generation, video content may change and not all frames commit the same global appearance. In our setting, the long-term extendable context (i.e., early context from the KV-cache queue) helps mitigate the quality degradation, demonstrated by the re- sults in Table 3 and Table 4. Further research on approaches addressing quality degradation is warranted and may hold potential significance for long-tew ter- video generation.
F. Limitations and Possible Future Directions
We analyze the limitations of the current work and propose some possible directions for future work.
Causal Modeling in Pretraining. Currently, all the pre- trained weights for video diffusion models (either UNet- based, e.g., ModelScore-Tie T2V (Wang et al., 2023b), Ani- mateDiff (Guo et al., 2024b), or Transformer-based e,g., Open-Sora (Zheng al., 2024)) use bidirectional attention in their temporal modules. Our Ca2-VDM is built upon Open-Sora which was also pretrained using bidirectional attention. However, finetuning these bidirectionally pre- trained temporal modules using causal attention might sub-optimal. The weights between bidirectional and causal
15

---

Title Suppressed Due to Excessive Size
16
24
32
40
48
OS-EXt
16
24
32
40
48
Ca2 Ca2-VDM VDM
72
80
104
OS- Extt
56
64
72
80
88
96
104
Ca2-VDM
16
24
32
40
4B
OS Ext
24
32
40
48
Ca2- Ba2-VDM
56
64
72
80
88
96
104
OS OS-Ext
56
72
BB
96
104
Ca2 Ca2-VDM -DD
Figure 12: Comparison between OS-Ext and Ca2-VDM in terms of long-term content drift (i.e., long-term quality degradation). Both models were trained on Sky-Timelapse (Zhang et al., 2020). Frame IDs are labeled at top-left corner.
16


Reviews:
[{"review_id": "TPNCyy2yRJ", "reviewer": "Reviewer_19uZ", "summary": "This paper introduces Ca2-VDM, an autoregressive video diffusion model tailored for generating long videos efficiently. The core idea is to eliminate redundant computation of conditional (overlapped) frames when chaining multiple short clips. To achieve this, the model applies causal generation that replaces standard (bidirectional) temporal attention layers with causal temporal attention and cache sharing that implements a queue-like caching system that holds key/value features from prior frames. Empirical results on text-to-video and video prediction tasks (e.g., MSR-VTT, UCF-101, SkyTimelapse) show that Ca2-VDM achieves SOTA generation quality.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Anonymous link for additional experiment results\n\nhttps://anonymous.4open.science/r/additional-exp-results-for-anonymous-github-F6EB/readme.md This includes: Table_R1, Table_R2, Figure_R1, and Figure_R2\n\n$~$\n\n## Q1: Unsatisfactory supplementary video quality\n\nWe acknowledge that the current qualitative results are not ideal. However, our primary focus is on improving the generation efficiency of VDMs, providing a computational and storage efficient framework. \n\nDue to limited computational resources and the tight schedule of the rebuttal period, we regret that we can not conduct additional large-scale training to further improve the qualitative results in the supplementary material. Instead, we conducted experiments on SkyTimelapse, as in Figure_R1. It shows that Ca2-VDM achieves comparable qualitative performance to OS-Ext, both in terms of single-frame quality and long-term visual content drift (with a similar degree of error accumulation).\n\nIn addition, we can still apply distillation techniques to improve the quality, e.g., distilling from a bidirectional teacher model to enhance the visual quality of Ca2-VDM (as the student). Also, as discussed in the \"Future Directions\" of Appendix (Sec. F), pretraining the causal attention from scratch might have potential improvements.\n\n\n$~$\n\n\n\n## Q2: Essential References Not Discussed\n\n> Modular approaches (ControlNet or T2I-Adapter style methods) for domain shift\n\nControlNet and T2I-Adapter are methods designed to introduce additional conditional signals to VDMs. They provide structural guidance like edges, depth maps, or segmentation maps.\n\nHowever, our work does not focus on domain adaptation or structure-guided generation. Instead, we generate future frames autoregressively conditioned on previous frames, rather than conditioning on external guidance to synthesize realistic frames from sketches or animate realistic videos.\n\n\n$~$\n\n\n\n## Q3: Highlight the differences between existing works about cache sharing\n\nIn our work, \"cache sharing\" specifically refers to sharing kv-cache across denoising steps, not for \"sharing the cache across autoregressive steps\". **To this end, we are the first to introduce cache sharing to autoregressive VDMs.**  \n\nTo the best of our knowledge, there is also a concurrent work Live2diff [1] that introduces kv-cache to autoregressive VDMs. However, it stores kv-cache for every denoising step (with different noise level $t$ and thus different KV features), which costs much more GPU memory than ours. \n\nIn contrast, Ca2-VDM enables cache sharing and only stores the KVs of clean frames. This is boosted by our \"clean prefix\" conditional frames,  the corresponding timestep sampling strategy, and the training objectives (cf. Eq.2).\n\nWe further provide GPU memory comparison in Table_R1. More detailed analysis can be found in the Q4 of Reviewer-7MWe.\n\n\n$~$\n\n\n## Q4: The design of Ca2-VDM compared to existing techniques\n\nWe would like to clarify our original contribution. Our model is the first one to introduce 1) kv-cache queue boosted with cyclic-TPE and 2) cache sharing to VDMs. \n\n1. KV-cache queue with cyclic-TPE:  It's non-trivial to apply LLMs' kv-cache to VDMs. In LLMs, all tokens from the beginning are maintained during all autoregression steps. However, in VDMs, early conditional frames can be (or should be) removed due to the large memory cost for visual data, as the appearance and motion of new frames are primarily influenced by the most recent frames. Due to the kv-cache queue, early TPEs have been bound to previous KV-caches and can not be reassigned (as discussed in Figure 4(c)). This motivates us to propose Cyclic TPEs to keep training-inference alignment while enabling generation beyond the training length.\n\n2. Cache sharing: As answered in Q3.\n\n\n$~$\n\n\n## Q5: Exploration on distributed generation strategies\n\nThank you for your suggestion. We surveyed some related works. E.g., Video-Infinity [2] offers a distributed approach across multiple devices, where spatial modules operate independently, whereas temporal modules synchronize context via collaborative communications.\n\nWe plan to incorporate this kind of method into Ca2-VDM as future work to improve the scalability of video generation.\n\n\n$~$\n\n\n\n## Q6: More evaluations for visually consistent transitions\n\nWe evaluated Ca2-VDM and OS-Ext on the Skytimelapse test set using VBench evaluation, as shown in Table_R2.  The last two metrics (motion smoothness and temporal flickering) measure consistent transitions. It shows that Ca2-VDM achieves comparable performance with OS-Ext.\n\n\n$~$\n\n\n\n## Q7: Writing improvement\n\nThank you for your suggestion. We will revise the paper to improve the writing and incorporate the discussions and clarifications in the above answers.\n\n$~$\n\n[1] Xing, Zhening, et al. Live2diff: Live stream translation via uni-directional attention in video diffusion models. 2024.\n\n[2] Tan, Zhenxiong, et al. Video-infinity: Distributed long video generation.  2024.", "claims_and_evidence": "Overall, the evidence is consistent with the main claims. The ideas and methods are straightforward to understand.", "methods_and_evaluation_criteria": "The proposed approach makes sense for scenarios where longer or continuous video generation is needed.", "theoretical_claims": "No particularly novel theoretical results (in terms of closed-form proofs or new sampling theorems) are emphasized. The standard diffusion framework is used (Ho et al., 2020 style training objectives; re-parameterization with 𝜖, 𝜃, ϵ, θ), and the model architecture reconfigures attention mechanisms in a causal manner.", "experimental_designs_or_analyses": "The primary design tests text-to-video (MSR-VTT, UCF-101) and unconditional video prediction (SkyTimelapse). The experimental designs are convincing. The analysis is straightforward and aligns well with prior video diffusion literature. The chosen metrics (FVD, speed) are standard for generation tasks. Repeated runs or more diverse metrics (like SSIM or user study) could add to robustness, but the chosen metrics are acceptable.", "relation_to_broader_scientific_literature": "This work is situated in the line of latent diffusion models (Rombach et al., 2022) and follows expansions to video (e.g., Imagen Video, Make-A-Video, LVDM, and so on). It closely resembles other methods that attempt to handle long video generation by chaining short segments (e.g., Gen-L-Video, StreamT2V).", "essential_references_not_discussed": "It might be beneficial to include more discussion of modular approaches (e.g., “ControlNet” or “T2I-Adapter” style methods adapted to video) if relevant for controlling domain shift.", "other_strengths_and_weaknesses": "Strengths:\n\n1. The causal generation approach plus key/value caching substantially reduce time and GPU cost for longer sequences.\n2. Extending the conditional prefix in an autoregressive way often yields more stable transitions between chunks.\n3. Multiple standard datasets, thorough runtime analysis, and FVD-based evaluation.\n\nWeaknesses: The paper is difficult to follow. The writing needs significant improvement. While I appreciate the authors’ efforts in implementing and evaluating Ca2-VDM, The design appears to be ad-hoc/incremental to existing techniques.", "other_comments_or_suggestions": "Further exploration into concurrency or distributed generation strategies might be helpful, especially for generating extremely long or high-resolution videos.\n\nMore user-centric evaluations (e.g., user preference tests or other human metrics) could highlight how visually consistent transitions are perceived.", "questions_for_authors": "Cache sharing is a popular design for video generation framework, could you highlight/discuss the differences between existing works?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hexbaN4Oj2", "reviewer": "Reviewer_dKiu", "summary": "Ca2-VDM is an autoregressive video diffusion model designed to generate long videos more efficiently. The paper identifies that existing autoregressive video diffusion models (VDMs) suffer heavy redundant computation when generating videos in chunks: overlapping frames between successive clips are repeatedly processed, leading to quadratic time complexity as more clips are generated​​. To address this, Ca2-VDM introduces two key innovations: (i) a causal generation mechanism with unidirectional temporal attention, so each frame only attends to previous frames. This allows the model to cache intermediate features (keys/values) of past (conditional) frames and reuse them in subsequent generation steps, eliminating redundant computations​. (ii) a cache sharing strategy that uses a fixed-size KV-cache queue along with cyclic temporal positional embeddings (Cyclic-TPEs) to recycle the cache across all diffusion denoising steps​. By sharing cached features rather than storing separate caches per denoising step, memory usage is kept in check. Using these techniques, Ca2-VDM can autoregressively extend video length with only a slight increase in computation per step (linear in the number of steps, rather than quadratic)​. The model is built on a spatial-temporal Transformer (initialized from Open-Sora v1.0, a pre-trained video diffusion model) and is evaluated on both text-to-video generation and video prediction tasks. Experiments show that Ca2-VDM maintains state-of-the-art video generation quality while significantly improving inference speed, enabling generation of longer videos (dozens of frames) more practically​. In summary, the paper’s primary contributions are: (1) a causal generation architecture for VDMs that enables reusing past frame features via caching, (2) a cache-sharing mechanism (with a cyclic positional encoding scheme) that provides long-term context without prohibitive memory growth, and (3) an implementation that achieves comparable or better video quality than prior state-of-the-art models but at much lower computational cost and faster inference​.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Anonymous link for additional experiment results\n\nhttps://anonymous.4open.science/r/additional-exp-results-for-anonymous-github-F6EB/readme.md This includes: Table_R1, Table_R2, Figure_R1, and Figure_R2\n\n$~$\n\n## Q1: Additional metrics for per-frame perceptual evaluation \n\nWe conducted additional evaluations on the VBench (https://github.com/Vchitect/VBench) benchmark. It is primarily designed for text-to-video evaluation. For our assessment, we selected four metrics: aesthetic quality, imaging quality, motion smoothness, and temporal flickering. The first two measure spatial (appearance) quality, and the last two assess temporal consistency. We compared Ca2-VDM and OS-Ext on the Skytimelapse test set,  as shown in Table_R2.\n\nThe results show that Ca2-VDM achieves comparable performance in both appearance quality and temporal consistency. Given our primary focus on efficiency, we conclude that Ca2-VDM matches the bidirectional baseline while being more efficient in both computation and storage for autoregressive video generation.\n\n$~$\n\n## Additional evaluation on temporal consistency\n\nWe also conducted an additional evaluation in terms of frame-differencing, as shown in Figure_R2. The results show that our method has good temporal consistency, while the other three have periodic content mutations (at the edge of each autoregression chunk), especially GenLV.\n\n$~$\n\n## Q2: GPU memory usage for w/ cache sharing vs. w/o cache sharing\n\n\n\nCache sharing is integral to Ca2-VDM due to the clean prefix condition design, meaning it cannot be evaluated without cache sharing. Instead, we compared Ca2-VDM with a concurrent work, Live2diff [1], which also uses kv-cache during autoregressive video generation. We conducted empirical GPU memory statistics, as shown in Table_R1\n\nLive2diff stores the kv-cache for every denoising step (with different noise level $t$ and thus different KV features) , which costs much more GPU memory than ours. Live2diff uses StreamDiffusion [2]'s pipeline denoising, which puts frames with progressive noisy levels into a batch and generates one frame each autoregression step. So, its batch size in the model forward shape is equal to the denoising steps, i.e., $B=T$. \n\nBenefited from cache sharing, Ca2-VDM’s KV-cache memory cost is independent of denoising steps, as its fixed shape $(1,25, hw, C)$ ensures constant memory usage. In contrast, Live2diff's memory scales with $T$ (e.g., from 1.42 GB at $T=4$ to 17.70 GB at $T=50$), confirming that **cache sharing saves $T \\times$ GPU memory.** As a result, Ca2-VDM requires only 0.86 GB (w/ PE) or 0.77 GB (w/o PE), with the difference due to spatial KV-cache for prefix-enhancement (PE).\n\nWhile Live2diff uses distillation (e.g., LCM) to reduce $T$, existing few-step acceleration methods still require at least 4 steps to generate frames with acceptable quality. This means we can save at least 4 x GPU memory. \n\nIn addition,  Live2diff cannot run 50-step denoising with KV-cache when GPU memory is limited or for high-resolution videos. This prevents proper evaluation of the teacher model before distillation, further restricting its applicability.\n\n\n\n[1] Xing, Zhening, et al. Live2diff: Live stream translation via uni-directional attention in video diffusion models. 2024.\n\n[2] Kodaira, Akio, et al. Streamdiffusion: A pipeline-level solution for real-time interactive generation. 2023.\n\n\n$~$\n\n## Q3: Unsatisfactory qualitative results of supplementary videos.\n\nWe acknowledge that the current qualitative results are not ideal. However, our primary focus is on improving the generation efficiency of video diffusion models, providing a computational and storage efficient framework. The visual quality improvement over the bidirectional attention baseline (OS-Ext) is not claimed as our contribution. \n\nDue to limited computational resources and the tight schedule of the rebuttal period, we regret that we could not conduct additional large-scale training to further improve the qualitative results in the supplementary material. Instead, we conducted experiments on the Sky-Timelapse dataset, as shown in Figure_R1. We can observe that Ca2-VDM achieves comparable qualitative performance to OS-Ext, both in terms of single-frame quality and long-term visual content drift (with a similar degree of error accumulation during long-term video generation).\n\nNevertheless, we can still apply distillation techniques to improve the quality, e.g., distilling from a bidirectional teacher model to enhance the generation results of our causal generation model (as the student). Additionally, as discussed in the \"Future Directions\" section of the Appendix (Sec. F), pretraining the causal attention from scratch might have potential improvements.", "claims_and_evidence": "The claims in the paper are generally well-supported by empirical evidence. The authors claim that Ca2-VDM eliminates redundant computation and achieves a significant speedup in autoregressive video generation. This is convincingly demonstrated by detailed runtime comparisons. For example, to generate an 80-frame video at 256^2 resolution (with 100 denoising steps), Ca2-VDM required only ~52 seconds on an A100 GPU, whereas a comparable baseline with extendable context (OS-Ext) took 130 seconds and a prior streaming method took 150 seconds​. This confirms a 2.5–3× speedup in practice, matching the claim of improved efficiency. The claim of reduced computational complexity (from quadratic to linear scaling with the number of autoregressive steps) is backed by analysis of FLOPs: as the number of autoregressive steps grows, baseline models see dramatic increases in computation at each new step, while Ca2-VDM’s computation grows only slightly in temporal attention and stays constant in other parts​. The paper also claims state-of-the-art video generation quality (or at least comparable to SOTA). This is supported by quantitative results on standard benchmarks: for text-to-video on MSR-VTT, Ca2-VDM achieves an FVD of 181, matching the best prior model (SEINE) and substantially better than older models (e.g. PixelDance 381, ModelScope 550)​. For video prediction on UCF-101, after fine-tuning, Ca2-VDM attains an FVD of 184.5, outperforming previous methods like VDT (225.7) and VideoFusion (220) by a large margin​. These results substantiate the claim that efficiency was achieved without sacrificing quality. In fact, Ca2-VDM often slightly improves quality in long video settings due to its extended temporal context: the paper shows it yields lower FVD between early and later video chunks compared to baselines, indicating better temporal consistency over long generations​. The qualitative examples (Figure 7) further illustrate that Ca2-VDM avoids the serious frame discontinuities (“mutations” between consecutive frames) that occur in other methods at clip boundaries​. All these observations give clear and convincing evidence for the paper’s claims.", "methods_and_evaluation_criteria": "The proposed methods are well-chosen for the problem of long video generation. The use of causal (unidirectional) temporal attention is a natural solution to avoid re-computation: it’s analogous to how Transformer language models cache past token embeddings to generate long sequences efficiently. Adapting this idea to video diffusion is appropriate and non-trivial – the authors had to modify the video UNet/Transformer to mask temporal attention so that each frame attends only to previous frames​. This ensures that features for past frames (the “conditional” context frames) can be computed once and reused. The introduction of a KV-cache queue for those features, combined with cyclic temporal positional embeddings, is an effective architectural innovation to handle unlimited sequence length. The cyclic positional encoding scheme addresses the fact that positional indices will grow beyond the model’s training length; by wrapping around and randomizing the offset during training, the model learns to handle very long sequences without divergence​. This is an appropriate solution to maintain alignment between training and inference when extending to longer video than seen during training. Additionally, the paper introduces a “prefix-enhanced spatial attention” (a method to feed the past frames’ information into spatial attention layers) – this is a sensible design to strengthen temporal continuity, and the ablation study confirms it modestly improves performance​. Overall, the methodology directly tackles the identified inefficiencies in a principled way and is well-grounded in Transformer design practices. The evaluation criteria and experiments are appropriate for the application domain. The authors evaluate on standard video generation benchmarks for both text-to-video (MSR-VTT, UCF-101 with text prompts) and unconditional video prediction (Sky Timelapse for extrapolating future frames) – covering both major use cases. They use FVD as the primary quantitative metric, which is standard for generative video quality, and they follow common practice by computing FVD with a pre-trained I3D model​. This choice is appropriate, as FVD captures both visual fidelity and temporal coherence. The paper also reports inference time and breakdowns of computation (FLOPs) to support the efficiency claims, which are crucial given the paper’s focus. The combination of quality metrics (FVD), speed measurements, and qualitative examples covers the relevant evaluation angles for a generative model. One could argue for additional metrics like text-to-video relevance (e.g., CLIP score for text alignment) or human evaluation for visual quality, but FVD and the provided visuals are generally accepted indicators for this task. The evaluation protocol is fair: Ca2-VDM is compared against strong baselines, including existing SOTA models (e.g. ModelScope, VideoFusion, SEINE) and carefully constructed ablations (OS-Fix, OS-Ext) that isolate the impact of their contributions. The authors generate a sufficient number of samples (e.g., 2990 videos for MSR-VTT, 2048 for UCF101) for reliable FVD estimation​, and even evaluate temporal consistency by measuring FVD across chunks within a generated video​, which is a thoughtful evaluation of long-term quality. In summary, the methods are well-suited to the problem and the evaluation methodology is rigorous and appropriate for validating both the performance and the efficiency of the proposed approach.", "theoretical_claims": "The paper does not heavily focus on new theoretical derivations。", "experimental_designs_or_analyses": "The experimental design in this paper is solid and thorough. The authors conduct experiments on multiple datasets covering different scenarios, which strengthens the validity of their findings. For text-conditioned video generation, they evaluate on MSR-VTT (a large benchmark with diverse videos and captions) and UCF-101 (using class names or prompts from prior work for text, ensuring a broad range of actions)​. For unconditional video prediction, they use the Sky Timelapse dataset​, which tests the model’s ability to continue a given video. These choices cover both open-domain content (MSR-VTT) and structured motion (Sky Timelapse’s moving skies), demonstrating the model’s generality. The experiments are designed to isolate the effects of the proposed innovations. Specifically, they compare Ca2-VDM against two controlled baselines built on the same architecture: OS-Fix (which uses fixed-length context like conventional models) and OS-Ext (which allows extended context but without caching, akin to a naive autoregressive extension)​. This is a critical comparison because it shows what benefits come purely from the new design. Indeed, Ca2-VDM vs. OS-Ext reveals the efficiency gains at similar quality, and Ca2-VDM vs. OS-Fix reveals the benefit of having extendable context. They also compare to external baselines like Gen-L-Video (a tuning-free method using overlapping denoising) and StreamingT2V (another autoregressive diffusion approach), which demonstrates where Ca2-VDM stands relative to contemporary solutions​. The inclusion of these baselines indicates a conscientious experimental design aimed at covering all relevant alternatives. The analysis of results is detailed and convincing. The paper reports quantitative metrics (FVD scores) for each dataset and method, and breaks down results by scenario (zero-shot vs. fine-tuned) to ensure fairness​. Notably, the authors include an ablation study (Table 3) on the Sky Timelapse video prediction task, examining the effect of the prefix-enhanced spatial attention and different maximum prefix lengths​. This ablation shows, for instance, that using a longer prefix (more past frames) and the prefix enhancement yields better FVD in later chunks of the video, validating those design choices. They also specifically analyze temporal consistency using their chunk-wise FVD metric (Table 4)​: this analysis revealed that methods with fixed context (OS-Fix, GenLV) or naive streaming can accumulate error and cause noticeable distribution shift in later frames (higher FVD against the first chunk), whereas Ca2-VDM (with extended context) maintains lower drift​. Such an analysis directly addresses the long-term quality claim of the paper. Additionally, the authors present runtime analysis: a table of cumulative inference time per autoregressive step and a FLOPs breakdown by component​. The time analysis (also visualized in Figure 10) clearly demonstrates the linear vs. quadratic growth of time cost, which is central to the paper’s thesis​. It’s commendable that they tested on the same hardware and settings for all methods, even quoting StreamingT2V’s numbers from its GitHub under the same conditions​ – this lends credibility to the comparisons. Overall, the experiments are sound and well-controlled. The paper uses large sample sizes (e.g., generating 512 videos for each condition in consistency tests) to ensure statistical reliability​. Where direct comparison wasn’t feasible (e.g., some baselines not available for certain datasets), the authors either cite published results or reasonably adapt the methods. No obvious flaws or missing analyses were noted. One might have wanted to see a qualitative user study or per-frame perceptual scores, but given the consistency of FVD and the provided visuals, the conclusions seem trustworthy. The only slight issue is that memory usage (GPU memory) for caching vs. not caching is not quantified in the main text – the authors qualitatively claim huge memory savings due to cache sharing​, which is logical. It could have been interesting if they had reported actual VRAM usage for generating long videos with and without cache sharing. However, this omission does not undermine the results; the focus was clearly on computation time, which they addressed thoroughly. In summary, the experimental design is comprehensive, and the analyses directly support the paper’s conclusions without glaring omissions or confounding factors.", "relation_to_broader_scientific_literature": "This work is well-situated in the context of the broader literature on video generation and diffusion models.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "The biggest weakness of the paper is its scale. This is inevitable considering not all researchers have sufficient GPU resources, but one always may question the results, especially the videos authors provided in the supplementary material --- they are not really convincing, firstly there are only four of them, secondly they all appear blurry and have significant artifacts. Quantitatively, the paper is sound and complete, but these qualitative results do not convince me.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ke7e3EDI6H", "reviewer": "Reviewer_7MWe", "summary": "This work propose an optimized ar video diffusion model, Ca2-VDM, which aims to enhance the efficient long-term, and real-world video generation. In Ca2-VDM, **causal generation** is proposed to reduce the redundant computations of previous conditional frames, and **cache sharing** is proposed to reduce the storage cost during inference. The effectiveness of Ca2-VDM is validated on three different benchmarks, compared with two different conditional generation baselines.\n\n## update after rebuttal\n\nI have read the response from the authors. My concerns are addressed. Some other reviewers are still concerned about the overall poor generation results. I will keep my initial rating.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Anonymous link for additional experiment results\n\nhttps://anonymous.4open.science/r/additional-exp-results-for-anonymous-github-F6EB/readme.md This includes: Table_R1, Table_R2, Figure_R1, and Figure_R2\n\n$~$\n\n## Q1: Essential References Not Discussed\n\n> Yin, Tianwei, et al. \"From slow bidirectional to fast causal video generators.\" CVPR 2025.\n\nThank you for your suggestion. This work also introduces causal attention into autoregressive video diffusion models (AR-VDMs). They use distributed matching distillation to align the visual quality of causal student model to the bidirectional teacher model.\n\nThis referenced work is a concurrent work with our paper.  I.e., Our initial version of Ca2-VDM was developed concurrently with it.\n\nOur approach distinguishes itself through the kv-cache queue (with cyclic-TPE) and cache sharing mechanisms that provide a both computational and storage efficient framework for (AR-VDMs). We will include this referenced work and add additional discussions in our revised paper.\n\n$~$\n\n## Q2: Any negative impact or instability observed during finetuning from bidirectional VDM ?\n\nWe did not observe any obvious negative impact or instability during our fine-tuning.\n\nIndeed, as verified by the aforementioned reference work (Yin, Tianwei, et al, 2025), distilling the knowledge from a pre-trained bidirectional teacher model to the causal student model brings quality improvement over directly fine-tuning.  Additionally, as discussed in our \"Possible Future Directions\" section of the Appendix  (Sec. F), pretraining the causal attention from scratch might have potential improvements. We will explore these two strategies in our future work.\n\n$~$\n\n## Q3: Content drift over longer generated videos\n\nAs discussed in our Introduction section, bidirectional models are also capable of generating longer videos beyond the training video length, and this is not a unique strength of causal models. Also, we did not claim long video quality improvement over the bidirectional baseline (OS-Ext) as our contribution.\n\nWe conducted additional experiments to show long-term content drift. As shown in Figure_R1, OS-Ext and our Ca2-VDM show comparable visual quality. Both models exhibit a similar degree of error accumulation over time.\n\nNevertheless, long-term video generation remains an open problem. Both models face challenges in this regard. We can apply the aforementioned distillation technique to enhance the visual quality.\n\n$~$\n\n## Q4: GPU memory usage between Ca2-DM and baselines\n\nWe conducted empirical GPU memory statistics, as shown in Table_R1\n\nWe compare Ca2-VDM with a concurrent work, Live2diff [1]. It stores the kv-cache for every denoising step (with different noise level $t$ and thus different KV features) , which costs much more GPU memory than ours. Live2diff uses StreamDiffusion [2]'s pipeline denoising, which puts frames with progressive noisy levels into a batch and generates one frame each autoregression step. So, its batch size in the model forward shape is equal to the denoising steps, i.e., $B=T$. \n\nNote that cache sharing is an inherent property of Ca2-VDM, and it cannot be evaluated without cache sharing. Instead, we demonstrate that Ca2-VDM’s KV-cache memory cost is independent of denoising steps, as its fixed shape $(1,25, hw, C)$ ensures stable memory usage. In contrast, Live2diff's memory scales with $T$ (e.g., from 1.42 GB at $T=4$ to 17.70 GB at $T=50$), confirming that cache sharing saves $T \\times$ GPU memory. As a result, Ca2-VDM requires only 0.86 GB (w/ PE) or 0.77 GB (w/o PE), with the difference due to spatial KV-cache for prefix-enhancement (PE).\n\nWhile Live2diff uses distillation (e.g., LCM) to reduce $T$, existing few-step acceleration methods still require at least 4 steps to generate frames with acceptable quality. This means we can save at least 4 x GPU memory. \n\nIn addition,  Live2diff cannot run 50-step denoising with KV-cache when GPU memory is limited or for high-resolution videos. This prevents proper evaluation of the teacher model before distillation, further restricting its applicability.\n\n\n\n[1] Xing, Zhening, et al. Live2diff: Live stream translation via uni-directional attention in video diffusion models. 2024.\n\n[2] Kodaira, Akio, et al. Streamdiffusion: A pipeline-level solution for real-time interactive generation. 2023.\n\n$~$\n\n## Q5: How much is the model sensitive to cyclic TPEs\n\nThe cyclic TPE in Ca2-VDM is specifically designed to enable the model to generate videos that exceed the training length. In other words,  a direct comparison of model performance with and without cyclic TPEs is not feasible. Also, it is not intended to enhance visual quality.\n\nIn terms of the model's sensitivity to cyclic TPE, we discussed the impact of training with cyclic TPE in the  \"Limitations\" of Appendix (Sec. F). It requires the model to learn all the possible situations during training and making it difficult to converge.", "claims_and_evidence": "Yes, the proposed Ca2-VDM is validated by thorough experimental results on different benchmarks.", "methods_and_evaluation_criteria": "Yes, the evaluation is the commonly used metrics in generation community like FVD.", "theoretical_claims": "No theoretical claims in this work.", "experimental_designs_or_analyses": "Yes,", "relation_to_broader_scientific_literature": "The core contributions of this work, i.e., causal ar temporal generation with cache sharing to significantly improve efficiency of ar video diffusion models, which are deeply related to several lines of research in the broader scientific literature.", "essential_references_not_discussed": "- From Slow Bidirectional to Fast Autoregressive Video Diffusion Models", "other_strengths_and_weaknesses": "Strenghts:\n\n1. This work identifies the limitation of existing ar vdms, the repeated computation of conditional frames, which clearly motivate the proposed work to address real-world applications of video generation like long-term video generation and live-stream scenarios.\n2. The proposed causal attention and cache sharing is both innovative for long term ar video generation.\n3. The paper is well-written, and figures help clarify the proposed method.", "other_comments_or_suggestions": "No", "questions_for_authors": "1. The model is initialized from open-sora v1.0, which is a bidirectional video generation model and there exists mismatch for the causal video generation. Did the authors observe any negative impact or instability during finetuning? \n2. One of the causal video generation strengths is that it can be generalized to longer-time video generation, there is no discussion about the this point in this work. The evaluation is conducted under the settings same as the training. Dose the content drift over longer times, like hundreds of frames? \n3. The authors claim the cache sharing helps avoid the huge cache storage cost in previous works, but it remains unclear how much actual memory is saved empirically. Could the authors provide detailed metrics of memory footprint comparisons between Ca2-DM and baselines or other methods?\n4. The proposed cycle TPEs are interesting contribution, but the paper dose not fully explore its importance separately. Dose the model sensitive to tpe and how much sensitive to the embeddings?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8V8U0LLr4K", "reviewer": "Reviewer_jMMe", "summary": "This paper introduces a diffusion-based method for video generation via causal transformers. The main idea is to apply kv-caching (technique widely used for AR transformers in NLP) to a causal diffusion transformer. This leads to faster video generation and potentially enables streaming scenarios. The method is based on a OSS text/image2video model (opensora) and results are demonstrated on two video generation benchmarks.\n\n## Update after rebuttal\nI am inclined to keep my original score (weak reject). Evaluation is limited, and qualitative results are unsatisfactory (e.g. 19uZ shares this viewpoint). \nSimilarly, evaluation of the cyclic TPE is not complete: I am not convinced by the argument that comparison to model w/o cyclic TPEs is infeasible - one can probably come up with a smaller-scale experiment to validate it.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Anonymous link for additional experiment results\n\nhttps://anonymous.4open.science/r/additional-exp-results-for-anonymous-github-F6EB/readme.md This includes: Table_R1, Table_R2, Figure_R1, and Figure_R2\n\n$~$\n\n## Q1:  Evaluation on VBench\n\nVBench is primarily designed for text-to-video evaluation. For our assessment, we selected four metrics: aesthetic quality, imaging quality, motion smoothness, and temporal flickering. The first two measure spatial (appearance) quality, and the last two assess temporal consistency. We compared Ca2-VDM and OS-Ext on the Skytimelapse test set.\n\nAs shown in Table_R2, Ca2-VDM achieves comparable performance in both appearance quality and temporal consistency. Given our primary focus on efficiency, we conclude that Ca2-VDM matches the bidirectional baseline while being more efficient in both computation and storage for autoregressive video generation.\n\n$~$\n\n## Q2: The claim on applicability of the method to long-term or live-stream video generation is not validated.\n\nLong-term video generation remains an open problem, with numerous challenges to be solved, including short-term frame quality, long-term content consistency, and generation efficiency.\n\nFor our Ca2-VDM, as claimed in the introduction, **the primary focus is to improve the generation efficiency (both computation and storage) of auto-regressive video generation**. Providing superior long-term video generation performance (surpassing SOTAs) is not the primary focus of Ca2-VDM.\n\nFor the visual quality, we achieve comparable visual quality to SOTA methods, as evidenced by the FVD results on MSR-VTT and UCF-101 datasets  (Tables 1 and 2), as well as the results on VBench (Table_R2). \n\nFor the long-term quality, we conducted additional experiments to show long-term content drift, as in Figure_R1.  In fact, OS-Ext and our Ca2-VDM show comparable visual quality. Both models exhibit a similar degree of error accumulation over time.\n\n$~$\n\n## Q3: Essential References Not Discussed\n\n>  Hoogeboom, Emiel, et al. \"Autoregressive Diffusion Models.\" in ICLR, 2022\n\nThank you for the suggested reference. This work proposes Autoregressive Diffusion Model (ARDM), which is built on autoregressive models and incorporates techniques from **discrete diffusion models**. It offers efficient training without the need for causal masking and parallel generation at test time.\n\nIn contrast, our Ca2-VDM is built on **continuous latent diffusion models** for video generation, and enables efficient autoregressive generation with causal attention and cache sharing. We will add more discussion in the revised paper.\n\n$~$\n\n## Q4: Clarity: Confusion in the Introduction about the temporal attention\n\nThank you for your suggestion. We agree that temporal attention does not necessarily have to be bidirectional. Our introduction aims to highlight that, in current video diffusion models, temporal attention is commonly used and is typically applied in a bidirectional manner, both in UNet and DiT structures. We will improve the writing of the introduction to avoid any potential misleadings in the revised paper.\n\n$~$\n\n## Q5: Quality: Unsatisfactory temporal consistency, limited qualitative results.\n\nWe would like to clarify that the causal attention is not intended to improve the temporal consistency. The causal attention is designed to combine with the kv-cache queue and cache sharing to improve the generation efficiency. \n\nFor temporal consistency, as evidenced in Table 3 and Figure 5, our Ca2-VDM offers extendable conditions and achieves better temporal consistency than fixed-length condition SOTA models (StreamT2V, Gen-L-Video, and OS-Fix). We also conducted an additional evaluation in terms of frame-differencing, as shown in Figure_R2. The results show that our method has good temporal consistency, while the other three have periodic content mutations (at the edge of each autoregression chunk), especially GenLV.\n\nDue to limited computational resources and the tight schedule of the rebuttal period, we regret that currently we cannot conduct additional large-scale training to further improve the quality of videos in the supplementary material. \n\n$~$\n\n## Q6: Originality: Concerns about the contribution\n\nWe would like to clarify that applying auto-regressive modeling and kv-cache to VDMs is not trivial (as acknowledged by Reviewer-dKiu).  Our original contribution includes: 1) kv-cache queue boosted with cyclic-TPE and 2) cache sharing\n\n1. KV-cache queue with cyclic-TPE:   It addresses the cases where positional indices grow beyond the training length while keeping the training-inference alignment. More detailed analysis can also be found in the **Q4 of Reviewer-19uZ**.\n2. Cache sharing: It enables the model to store only KVs of clean frames, leading to much less GPU memory usage than concurrent work (e.g., Live2diff). More detailed analysis can also be found in the **Q4 of Reviewer-7MWe**.", "claims_and_evidence": "- The claim on applicability of the method to long-term or live-stream video generation is not validated.", "methods_and_evaluation_criteria": "- The set of benchmarks is quite limited (UCF101, MSR-VTT), there are more recent and relatively widely adopted benchmarks: https://github.com/Vchitect/VBench", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "- Validation on longer sequence generation or streaming seems missing. \n- Benchmarks are limited.", "relation_to_broader_scientific_literature": "This paper is similar in spirit to LVDM-like AR diffusion but applies a common solution for transformers to improve speed.", "essential_references_not_discussed": "- Auto-regressive diffusion models [Hoogeboom'22] is one of the first works to combine AR and DM into one model.", "other_strengths_and_weaknesses": "- Clarity: there is a bit of confusing text in the intro about transformers-based VDMs - temporal attention can certainly be causal and must not necessarily be bidirectional - and in practice it is a matter of applying a different mask? \n- Quality: The resulting videos do not look like particularly temporally consistent - thus making it hard to understand whether using causal attention (in contrast to dense attention which is more frequently used in recent models such as moviegen).\n- Originality: the main contribution of this work is applying auto-regressive modeling and kv-caching which have been widely used in transformer architectures. It is unclear if applying those to VDMs this is a significantly original contribution, especially given that the visuals do not look convincing.\n- Limited results: 3 videos seems like not enough to really understand the quality of the model.", "other_comments_or_suggestions": "- Might be worth clarifying the intro as it reads like temporal attention is inherently bidirectional. \n- More results would be useful to understand the quality, on longer sequences in particular.", "questions_for_authors": "- One of the issues with the auto-regressive approach is error accumulation. Given that the claim of the paper is ability to generate longer-term videos, did authors try applying their model on a longer context?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: 3x weak accept, 1x weak reject. This paper introduces an autoregressive video diffusion model that leverages causal generation and cache sharing to improve efficiency in long video generation. The reviewers agree on the (1) innovative causal generation mechanism with cache sharing that substantially reduces redundant computation and GPU cost, (2) thorough experimental evaluation on standard benchmarks demonstrating competitive video quality alongside significant speedup, and (3) clear potential for practical applications in extended video generation scenarios. However, they note the (1) clarity and writing issues that render the paper difficult to follow, (2) unsatisfactory and limited qualitative results in the supplementary material, and (3) concerns about the incremental nature of the contributions with insufficient evaluation on long-term generation. The authors have provided detailed rebuttals addressing cache sharing, cyclic TPE, and additional quantitative evaluations, so the AC leans toward accepting this submission.
Decision: Accept (poster)