Frame-wise Conditioning Adaptation for Fine-Tuning Diffusion Models in Text-to-Video Prediction
Authors: Zheyuan Liu
Junyan Wang
Zicheng Duan
Cristian Rodriguez-Opazo
Anton van den Hengel
Venue: Accepted by TMLR ()

Abstract: Text-video prediction (TVP) is a downstream video generation task that requires a model to produce subsequent video frames given a series of initial video frames and text describing the required motion.
In practice TVP methods focus on a particular category of videos depicting manipulations of objects carried out by human beings or robot arms.
Previous methods adapt models pre-trained on text-to-image tasks, and thus tend to generate video that lacks the required continuity.
A natural progression would be to leverage more recent pre-trained text-to-video (T2V) models.
This approach is rendered more challenging by the fact that the most common fine-tuning technique, low-rank adaptation (LoRA), yields undesirable results.
In this work, we propose an adaptation-based strategy we label Frame-wise Conditioning Adaptation (FCA).
Within the module, we devise a sub-module that produces frame-wise text embeddings from the input text, which acts as an additional text condition to aid generation.
We use FCA to fine-tune the T2V model, which incorporates the initial frame(s) as an extra condition.
We compare and discuss the more effective strategy for injecting such embeddings into the T2V model.
We conduct extensive ablation studies on our design choices with quantitative and qualitative performance analysis.
Our approach establishes a new baseline for the task of TVP.
Our code is open-source at https://github.com/Cuberick-Orion/FCA.

Published in Transactions on Machine Learning Research (11/2025)
Frsme-wise Conditioning Adaptation for Fine-Tuning Diffu- sion Models in Text-to-Video Prediction
Zheyuan Liu
zheyuan.liu wiw.lluuaaaae www.liiiaaaaddee..mm Center for Augmented Reasoning, Australian Institute for Machine Learning, University of Adelaide
Junyan Wang
jungaan. wang www. Madegaddeedddeeeeeeoeee eew.elllaaddeeed..a... Center for Augmented Reasoning, Australian Institute for Machine Learning, University of Adelaide
Zicheng Duan
zicheng. duan Oadelaide. Center for Augmented Reasoning, Australian Institute for Machine Leorning, University of Adeloide
Cristian Rodriguez-Opazo
cristian, oristian.rodriguezoo row.iigoeeoocoooaaaaaaaaee edu.au Center for Augmented Reasoning, Australian Institute for Machine Learning, University of Adelaide
Anton van den Hengel
anton. www.:oonnnneegge@deeliiddedddddd Center for Augmented Reasoning, Australian Institute for Machine Learning, University of Adelaide
Reviewed on OpenReview: https:// https://operresiewwwue forum ? ? HSAJ4LUHK
Abstract
Text-video prediction (TVP) downstream video generation task that requires a model to produce subsequent video frames given series of initial video frames and text describing the required motion. In practice TVP methods focus depicting manipulations of objects carried out by human beings or robot arms. Previous methods adapt models pre-trained on text-to-image tasks, and thus tend to generate video that lacks the required continuity. A natural progression would be to leverage more recent pre-trained text-to ti-eeo (T2V) models. This approach rendered more challenging by the fact that the most common fine-tuning technique, low-rank adaptation (LoRA), yiclds undesirable results. In this work, we propose an adaplation-baset strategy we label Frame- wise Conditioning Adaptation (FCA). Within the module, devise sub-module that produces frame-wise text embeddings from the input text, which acts additional text condition to aid generation. We use FCA to fine-tune the T2V model, which incorporates the initial frame(s) as an extra condition. We compare and discuss the more effect strategy for injecting such embeddings into the T2V model. We conduct extensive ablation studies on our design choices with quantitative and qualitative performance analysis. Our approach establishes a new baseline for the task of TVP. Our code open-source litps: Cetihuub.coo/Cuuuriccckooo
particular category of videos
1 Introduction
Text-video prediction (TVP) Gu et al., 2023 takes as input an initial frame frames, and text describing particular motion (Figure On this basis it generates subsequent frames wherein the existing content undergoos the described motion. As downstream video generation task, its configuration enables the study of instruction adherence, as well as temporal consistency in the context of video extension given conditioning frames, particularly a fine-tuning setup. The established benchmark and datasets emphasize motions carried out by human beings or robot arms in manipulating objects.
Given the relatively limited training data for TVP, methods for this task typically involve fine-tuning a pre- trained generative model. The challenge, therefore, lies in guiding the pre-trained model in comprehending

---

Published in Transactions on Machine Learning Research (11/2025)
and adhering to the multi-modal conditions. While current methods (e.g., (Gu et al. 2023 have made admirable progress, they still fall short in generation quality, partly because they build upon text-to-image models (Rombach et al. 2022). Given the recent success of large-scale pre-trained text-to-video generative models (Hong et al. 2022 Yang et al. 2024b Kong et al., 2024) particularly based on diffusion transformers (DiT) (Peebles & Xiel 2023). we propose to adapt such models towards the task of TVP. However, we find this goal non-trivial. The reasons are twofold. First, we empirically find that low-rank adaptation (LoRA) (Hu al. 2022a], the standard method for fine-tuning generative models domain-specific data, yields undesired results TVP. Second, text-to-video models alone, whereas the task of TVP involves conditioning the initial frames as well. Specifically, it requires the model to reason over the input frames and extend its future predictions upon them, while jointly taking the text prompl into consideration.
pre-trained to align to text prompt
"Holding brown case."
Module
Tokens 1y X
teet, initial hrames
Frame-wfse Text Conditioning Module
Gate
DiT Block
Feed Forward
Attention Block
Φ (frozen
Scale, Shift
Insetted
Initial frames
Pre treetrained T2V
FCA
Gate, Scale
Gate
Y2
Kinit
Figure 1: Overview. Tixt-video prediction (TVP) models generate subsoquent video frames on the ba- sis of the initial frame(s) and natural language description of the required motion. Our method leverages pre-trained text-to-video (T2V) diffu- sion transformer (DiT) model, while introducing an effective adaptation method (FCA) for fine-tuning. Our method integrates the initial frames, as well as Trame-wise lext conditions aid the generation.
Concat
Multi-Head
3D
Virame
Scale, Shift
Frame-vise
Scale, Shilt
Initial Fraie
Conditioning
Input by x,]
Expert AdaLN
Text; learnable tokens
b:
Pre Pre-trained DìT block
Figure 3: An illustration of Frame-wise Con- ditioning Adaptation (FCA) on diffusion trans- former (DiT). Right: an arbitrary pre-trained DiT block (Peebles & Xie 2023 Left: the proposed FCA module, introduced in Section 3.2 which in- corporates frame-wise text conditioning discussed in Section 3.3. We only show one DiT layer here for clarity, but note that we separately apply the same modules to every layer. y, X2 denote the text tokens and noisy latent for a DiT, respectively; [ denotes concatenation. Xinit represents the la- tent of the initial frames introduced in Section 3.2 and yframes is the frame-wise text conditioning beddings. A frame-wist attention mask is applied within the multi head cross-attention module, which omitted in this figure, see Section 3.3 a, 8, are the learned parameters of the adaptive Layernorm (adaLN) (Peebles & Xie (023). The trainable mod- ule is marked with fire symbol. See Figure 2 for details of the frame-wise text conditioning module.
Trame-wise Text Conditioning
Feed-Forward
Dx
Initial
VIT
Gross-lltnttion
Self Seff-Attention
Cross Attention
Multi-Heed
Block
Frame
Text; learnable tokens
Figure 2: Details of the frame-wist wise text con- ditioning module inspired by Q-Former (Li et al., 2023a), and its integration with FCA. We only show one DiT layer here for clarily, but note that we separately apply a frame-wise text condi- tioning module to every layer. In total, initialize and train D such modules for the D layers of the DiT. This figure complements Figure 3 (FCA mod- ule, bottom-left block). ViT stands for the Vision Transformer (Dosovitskiy et al. 2020
The proposed approach requires an adaptation strategy tailored to the task setup. We have thus developed an effective design based on DiT model, shown in Figure]] 1 Specifically, inject parallel attention modules
2

---

Published in Transactions on Machine Learning Research (11/2025)
into the layers of the DiT, whose output is added back in residual manner. Meanwhile, we freeze the pre- trained DiT to avoid disrupting its ability to process video. To integrate the initial frames as a condition, we propose to use a frozen copy of the DîT' model to extract the latent of such frames, and incorporate this information into video generation through cross-attention. The strategy is similar to some zero-shot subject- driven personalized image generation methods (Jia el. al. 2023; Purushwalkarn et al. 2024 Duan et al. 2024) in that their approach to capture the visual characteristics of the subjects from the user-provided images. We qualitatively validate that our design yields superior performance over standard fine-tuning techniques such as LoRA; and demonstrate the effectiveness of our approach in integrating the initial frames in comparison to well-established practices of enabling additional control, such as ControlNet (Hu & Xu, 2023). We report empirical findings and training tricks to aid future work in fine-tuning such DIT models via adaplation.
We additionally seek to apply frame-wise text conditioning to our method, which has been empirically demonstrated beneficial for TVP (Gu et al., 2023). The reason is thought to be that the raw text conditions in the TVP datasets (e.g., "pick something up" or "move away from the camera") are not specific enough to develop
specify the intermediate stages of carrying out motion. Without collecting extra annotations, learnable modules that infer such frame-wise conditions through fine-tuning. The Icarned conditioning embeddings are inserted back to the parallel attention layers we introduced, where a frame-wise attention mask is applied. We experiment with varions designs and share our insights on how to construct and apply such conditions efficiently. Collectively, our method, termed wise Conditioning Adaptation (FCA), surpasses existing work on TVP, thereby establishing a strong baseline for future research.
To summarize, we propose to apply recent large-scale pre-trained text-to-video generative models to the task of TVP. To accommodate the task requirements while maintaining a reasonable training cost, we introduce an effective adaptation-based fine-tuning strategy that incorporates frame-wise text conditions via learned modules to assist the learning. Our design additionally integrates the initial frames into the condition. We conduct extensive ablation studics to validate our method, while documenting empirical findings and training tricks for future research. Our method, termed FCA, improves on previous work both quantitatively and qualitatively on standard benchmark datasets. Specifically, we achieve a 40% reduction (relative) in the FVD metric on both Somnehing-SSomethingg V2 (Goyya et al. 2017) and EpicKitchen-100 (Damen et al. 2020 and impressive 60% FVD reduction (relative) on Bridge Data Ebert et al. 2021
2 Related Work
Pre-trained Video Generation Model. The impressive performance of diffusion models (Ho et al.
2020 Song et al. 2020b) has spurred a recent trend in developing large-scale pre-trained video generative models (Ho et al. 2022 Hong ct al. 2022), which are viewed as a natural progression from the image generalive task [Ramesh el al.] 2022 Saharia et al. 2022 Rombach el al. (022). Text is most commonly used as the conditioning signal for generating videos, i.e., text-to-video (2V),, analogous to what is seen in
toxt-to-image (T2I). Architecturo-wisc pioncer work (Singer et al. 2022 Wu ct al. 2023b Blattmann al.
Wang et al. 2023a Zheng et al. 2024 often adopts the U-Net architecture Ronneberger et al. 2015 and injects temporal modules alongside the existing spatial modules, thereby inflating an image generation model for video generation. A line of recent work (Yang et al. 2024b Kong et al. 2024g adopts the diffusion transformer (DiT) (Peebles & Xie 2023) with better scalability. In this work, we adopt CogVideoX (Yang) et al. 2024b) for our base model, which utilizes 3D full attention that alleviates the need for separate spatial and temporal attention modules.
2023
Although T2V is the most common paradigm for pre-trained video generation models, several recent work (Yang et al. 2024b Kong et al. 2024 Blattmann et al., 2023) have also included variant of their proposed T2V base model, that additionally supports an image for conditioning as the first frame. To adapt a pre-trained T2V model into image-to-video (I2V), the common practice (Girdhar el al. 2024] is to concatenate the extracted representation of the initial frame to the input noisy latent along the channel dimension, and perform further training. We note that the task of text-vidco prediction (TVP) is similar to I2V regarding the setup of input conditions. However, we opt for developing based on T2V model, we refer readers to Section 4.1 for discussions.
3

---

Published in Transactions on Machine Learning Research (11/2025)
Incorporating Extra Conditions. Multiple works seek to inject extra conditions to guide the generation Various categories of conditions exist for different applications, including edges or depth maps (Guo et al.
2024 Chen et al. 2023), optical flow (Zhang et al.
2024
Liang et al.
Yang et al, 2024). Another video can also serve as Xu ct al.]
2024), camera movements
2024al and pose information (Hu 2024 Ma ct al.
2024
guidance for video-Lo-video generalion (Hu & Xu
2023
Zhao et al.
2024 Wu et al. 2023ch. Notably, a
line of work exists that addresses the concept of " "motion" (Shen et al.
2024 Shi et al., 2024 Zhang et al.,
2024 Li et al. 2023b Hu & Xu 2023; Zhao et al., 2024 Dai et al. 2023), however, they either focus on the movement of rigid body (e.g., car moving) or human being (e.g., riding a bicycle). As such, they are different from the concept of motion seen in the task of text-video prediction (TVP), i.e., the manipulation of objects. We note that different categories of conditions often require specific designs to inject into the generalive model, as they differ in format and semantics.
Video Prediction (TVP). TVP conditions the generation on both the initial frames and the text, which differs from the conventional task of video prediction (Feichtenholer et al. 2016) that conditions solely on the previous frames (Ye & Bilodeau 2024 Höppe et al.) 2022 Voleti et al., 2022 Harvey et al. 2022). It is recently proposed by Gu et al. (2023) alongside a mothod termed Socr, where a pre-trained, U- Net (Ronneberger et al. 2015) T2I model is adapted for producing videos by inserting additional temporal attention layers, as discussed above. The authors discovered that frame-wise text conditions benefit. the fask, and introduced text decomposition module built into the proposed architecture. In this work, we further advance the task of TVP by leveraging more suitable, dideo-generative pre-trained models, while also incorporating frame-wise conditions but with tailored design to fit the architecture.
3 Method
The task of text-video prediction (TVP) takes in as imput the initial k frame(s) of a video {f f alongside the conditional text and aims at producing the subsequent frames ,nlk We note the entire video as the collection of all frames F [_]]_ In this work, we leverage a pre-trained text-lo-video (T2V) diffusion transformer (DiT) (Peebles & Xie 2023) as our base model, where we propose our Frame- wise Conditioning Adaptation (FCA) design. We note the number of transformer layers in DiT as D,
seen in Figure 2
3.1 Preliminaries
Denoising Diffusion Probabilistic Models. Diffusion models (Ho et al. 2020 Song et al. 2020a;
Dhariwal & Nichol 2021 Song et al.) 20205) are a type of generative model that performs bi-direction operations. The forward direction takes in sample Xo and itcratively adds Gaussian noise to it until a maximum step T is reached. Opposite to it, the generative process starts with the Gaussian noise x7, where the model gradually estimates the noise to be subtracted, and ultimately produces the sample Xg- The training objective of the model to predict the to-be-subtracted noise:
L = Ex,.c.e.......,..ll 6o (Xe, c, 1)1?:
(1)
where the diffusion model is denoted as Eg parameterized by 8 t E [0, represents the time step of the process, and x, = 0gX0 + 6,e is the noisy data at step-t with ag a, being functions of t. Here, e represents various conditions that can guide the generation.
Once trained, the diffusion model can be used for predicting samples from the Gaussian noise. Pioneer work (Dhariwal& & Nichol, (2021) adopts classifier guidance that uses the gradients of trained image classifier to control the generated samples. Subscquently, classifier-free guidance (Ho & Salimans] 2021) eliminates the need for separate classifier model. Its sampling stage involves linear combination of conditional and unconditional predictions, where the latter term replaces the condition with an empty embedding Ø, as:
Eo c, Xeo(x,, c, + (1- Xx,, 8,t),
with A being the guidance scale. A noticeable characteristic of the diffusion model is the disparity between its training and inference operations. Specifically, one can apply one of many samplers for generating samples.
4

---

Published in Transactions on Machine Learning Research (11/2025)
For instance, samplers such as DDIM (Song et al. 2020a enable faster generation by requiring only small number of steps.
Recent work often adopts the latent diffusion model (LDM) (Rombach et al. 2022 which operates on the latent space as opposed to the raw pixel space, and therefore requires less computational resources. The encoding and decoding operation is through a frozen variational autoencoder (VAE). We omit the VAE the notations for brevity and directly view X₁ as the latent embedding below.
Diffusion Transformer. While conventional diffusion models often adopt the U-Net (Ronneberger et al, 2015) architecture, recent variants leveraging diffusion transformer (DiT) (Peebles & Xie 2023) have in- creasingly gained popularity thanks to its scalability. The DiT replaces the convolution layers the U-Net with multiple layers of transformer modules, where each layer performs multi-head attention (Vaswani al. 2017) over a sequence of tokens.
The sequence of tokens is effectively concatenation between the input noisy latent and the conditioning embeddings, such as text embeddings obtained through pre-trained text encoder. Here, the input noisy latent of an image is patchified along the spatial dimensions to form a sequence of tokens. In the case of video gencrative models, multiple frames are treated as consecutive images to form a longer sequence. Positional embeddings Dosovilskiy et al. 2020 Su al. 2024) are oflen injected, similar lo what is seen the Vision Transformer (ViT) (Dosovitskiy et al. 2020 The different types of tokens separately go through adaptive Layernorm (adaLN) (Peebles & Xie, 2023), which involves scaling and shifting the embeddings bridge the modality gap. Without harming clarity, omit the patchifying and adaLN operations, and denote the token sequence received by each layer of the transformer as Z = concat [y;x.]
Within each transformer layer of the DIT, the concatenated token sequence Z performs self- attention (Vaswani et al. 2017), as:
QK vd
Z' = Attention (Q, V) = Softmax
V
(3)
where Q - ZW, g. K ZW,, V ZW, are the query, key, and value matrices computed by multiplying the token sequence Z with separate learnable weight matrices. Here, Z serves as the token sequence for query, key, and value simultaneously. The output Z' is then added to Z in residual manner (He et al.) 2016). followed by feed-forward layer.
3.2 Adapting Pre-trained T2V Model for TVP
In this work, we adopt an effective adaptation-baser fine-tuning strategy for DiT-based T2V model. As shown in Figure 3 within each layer of the transformer architecture, we insert an attention block parallel to the pre-trained attention block, whose output added back to the pre-trained branch. In practice, we find it crucial to initialize a small learnable scaling factor (c.g., 0.1) and multiply it with the attention output. We freeze the pre-trained branch to best preserve its learned knowledge while training the inserted modules.
Unlike in the pre-trained branch where each attention block performs self-attention over the concatenated text and video tokens (Z- concat/y: x,]). in the inserted attention blocks, we modify the key and value token sequences to perform a cross attss-atteniion Since the query sequence remains as Z, the dimensionality of the output sequence is unchanged and can be added back. Specifically, we construct the key and value token sequences as the concatenation among the text token y, the initial frames latent ((xinit (see below), and the frame-wist text conditions Yirames (see Section 3.3
Intograting the Initial Frames. Since TVP requires the predicted frames to extend on the initial frames, it is necessary for us to inject the initial frames as conditions to the T2V model. As illustrated in Figure 3 (top-left), we append the latent embeddings of the initial frames to the key and value token sequences of the FCA attention layers. Consequently, the model learns attend the information within such embeddings through fine-tuning. Inspired by recent work on personalized image generation (Jia et al., 2023; Purushwalkam et al. 2024), we obtain such embeddings for each FCA layer using frozen copy of the pre-
5

---

Published in Transactions on Machine Learning Research (11/2025)
trained DiT model Φ(-), where we feed the latent of the initial frames Xait as imput and perform forward pass, thereby extracting its intermediate layer-wise wise embeddings, noted as Xinit = {
To obtain Xinit, we treat the initial frames as a short video of only one or two frames, and encode via the VAE Yuu et al. 2023a). As the DiT model is trained to receive noisy latent, rather than a noise-free latent imput, follow Duan et al. (024 by adding a small noise (one step) to Xinit before imputting it into the DiT. This ensures that the latent embedding adheres to the expected input data distribution while introducing minimum loss to its information.
3.3 Frame-wise Text Condition
We explore the use of frame-wise text conditions, which previous work (Gu et al. 2023 has demonstrated to be beneficial, as the text conditions text-video prediction (TVP) datasets (e.g., "taking remote out of pen stand") are often too concise to depict the intermediate stages of carrying out a motion.
As in the previous work (Gu et al., 2023), without collecting additional lext annotations, we seek to infer frame-wise text embeddings and use them to complement the vanilla text during generation. In essence, given the text tokens y E R44xd, we aim to derive Virames E Rfxlxd such that f, i.c., its frame dimension corresponds to that of the video tokens X Here, 4 represents the length of the text token coming out of the text encoder while d is the channel dimension; we omit the batch dimension for brevity.
Frame-wise Text Conditioning Module Design. Our Trame-wise text conditioning module is inspired by Q-Former in BLIP-2 (Li al. 2023a). As shown in Figure 3 (bottom-lcft), we initialize a sequence of learnable tokens for Ytrames: which are concatenated with the tokens of lext y obtained via tokenization and illustrates the detailed architecture. Specifically, the two perform self-attention first, followed by cross-attention layer where we introduce the initial frame fi. Vframes is obtained following final feed-forward layer.
projection. Figure 2
The frame-wise text conditioning module is trained alongside the FCA attention layers. We find it ideal to initialize separate such module per attention layer, instead of employing one module and feeding its output all attention blocks of FCA (Figure 2) Our intuition is that every DiT layer learns different semantics; therefore, so should the conditioning embeddings. We denote the i-th layer frame-wise text conditioning module (-), therefore, y Virames r', fi).
Applying Frame-wise Text Conditioning. Given Virames that corresponds to the i-th FCA attention layer, we append it to the key and value token sequences to perform cross-altention, analogous to our approach of integrating the initial frames in Section 3.2 We learn a pre-oormaliaation to Fraaes mimicking the adaptive Layernorm (adaLN) (Poebles & Xie 2023 seen in DiT, which is empirically beneficial.
To guide the frame-wise text conditioning modules to learn Virames that serves as frame-wise condition, we construct an identity matrix for the attention mask on the frame dimension between Virames and the video tokens x,. Recall that Virames and x, share the same length in frame. Effectively, the mask prevents video tokens of frame-p from attending to text conditions of frame-g when p # q. Meanwhile, we still include the vanilla text condition y in the cross-attention, and allow all video tokens to attend to it.
3.4 Training and Inference
During training, we follow the standard diffusion objective in Equation to optimize the inserted FCA blocks. We incorporate the FCA attention layers into Eg parameterized by 8, while separately denoting the frame-wise text conditioning modules as 7-)). We freeze other parameters of the pro-trained DiT. The training objective is formulated as:
L = Ex. X r Eg (Xt, X y, fi,
(4)
where y denotes the conditioning text embeddings, and Xinit represents the extracted latent of the initial frames. We perform no dropouts to the initial frames text find it not beneficial for fine-tuning.
6

---

Published in Transactions on Machine Learning Research (11/2025)
SSv2
BridgeData
Epic100
Methods
Pre-trained with Text Condition
FVD
KVD
FVD
KVD
FVD
KVD
TATS Ge al.
2022
video
no
428.1
2177
1253
6213
920.0
5065
2
MCVD
Voleti
al.
n1
1407
3.80
1427
2.50
4804
5.17
2022
3
SimVP
Gan et al.
2022
10
537.2
0.61
681.6
0.73
1991
1.34
MAGE
Hu et al.
20226
video
yes
1201.
1.64
2005
3.19
1358
1.61
PVDM
Yu al
20235
11
502.4
61.08
490.4
122.4
482.3
104.8
FideoFusion
Lib et al.
text-video
yes
163.2
0.20
501.2
1.45
349.9
1.79
2023
7
Tume- A- Video
(Wi et al.
text-image
y78
291.4
0.91
515.7
2.01
365.0
1.98
2023a
Secr
[Gu et al.
2023
text-image
yt8
112.9
0.12
246.3
0.55
271.4
1.40
9
FCA
(Ours)
text-video
yt8
67.7
-0.18
96.5
0.43
164.6
0.45
Table 1: Quantitative results of video generation. We report on Something- Soitiiinnngotetiigg-gggg -2 (SSv2), BridgeDaaa and EpicKitchen-100 (Epic100). We follow Gu et al. (2023) to report FVD and KVD (L the lower the better). For cach method, we also report the data type used for pro-training (if any) and whether it supports text conditioning, The best numbers are in bold black. Our method is shaded in gray. Rows 1-8 are cited from (Gu al. 2023
In inference, we use the DDIM (ong el al. 2020a sampler to accelerate the process while leveraging the classifier-free guidance (CFG) per Equation 2 For the unconditional prediction, we follow the standard practice (Ho & Salimans 2021 by replacing the text prompt with an empty string and passing it to the text encoder to obtain y. However, we do not drop the initial frames as it empirically harms the generation quality.
4 Experiments
Datasets. We follow Gu et al. (2023) and test on three text-to-video datasets. Something-Sometiinn V2 (SSv2) (Goyal al.) 2017) contains 168,913 training samples on daily human behaviors of interacting with common objects. Priide-Data-Vi (Ebert et al., 2021 includes 20,066 training samples in kitchen environments where robot arms manipulate objects. Epie-Kitchen-100 (Epic100) DDamnn et al. 2020
consists of 67,217 training samples of human actions in first-person (egocentric) views. All datasets include short text descriptions of motions per video. In validation, we follow the implementatior[ by Gu et al. (2023)
for a fair comparison. For SSv2, we select the first 2,048 samples in the validation set; for BridgeData, we split the dataset by 80% and 20% and use the latter for validation; finally, for Epic100, we directly adopt its validation split.
Implementation Details. We leverage the CogVideoX-2B Yang et al. 2024b T2V model in this work. We use its pre-trained weights for initializing the 3D VAE, the T5 text encoder (Colin 2020), and the DIT, while freezing these modules. We initialize the FCA attention layers from scratch for fine-tuning. Regarding
the frame-wiss wise text conditioning modules, we initialize its ViI (Dosovitskiy et al.) 2020 vision encoder, as well as the CLIP [Radford et al., 2021) text tokenizer and projection layer from BLIP-2 For cach DiT layer, we initialize a frame-wise text conditioning module of one layer from scratch and train them alongside the FCA attention blocks. two initial frames SSv2, and one initial frame on BridgeData and Epic100, as in GGu
For DIT, et al. 2023). For the frame-wise text conditioning module, due to its architectural design that accepts only one image through cross-attention, we input the first frame regardless of the dataset. We train our model for 100,000 steps on SSv2, 25,000 steps on BridgeData, and 50,000 steps on Epic100. In inference, we set the CFG guidance scale X = 6.0 and a sampling step of 50, following (Yang et al. 2024b). Further details are elaborated in Section B.

---

Published in Transactions on Machine Learning Research (11/2025)
Text prompt: "Holding helmet."
SEBR
QURS
GT
Figure 4: Qualitative examples on video generation. We compare our method with Seer Gu et al, 2023), where we also show the ground truth (GT). For inference on Seer, we adhere to its configuration with a frame number of 12; while ours is of 16 (split into two rows). For illustration purposes, we replace the conditioning frames (first two) with the ground truth. See qualitative examples on other datasets in Section C.6 Note that the aspect ratios of the figures are slightly adjusted for demonstration purposes.
Evaluation Settings. Following Gu el al. 2023 we assess our method on both machine and human evaluations. For machine evaluation, we include Fréchet Video Distance (FVD) and Kernel Video Dis- tance (KVD) (Unterthiner et al.) 20118). Both metrics are computed with a Kinetics-400 pre-trained I3D model (Carreira & Zisserman 2017]. See Section
C.2 for discussions on human evaluation.
We additionally assess on VBench (Huang et al., 2024). which is a recent, well-regarded evaluation benchmark for video generation. However, we note that certain adaptations are required for us to benchmark on this metric. We refer readers to Section C.I for details and our results.
All ablation studies are conducted on SSv2 with the FVD and KVD metrics. To more efficiently compare various setups, we perform all ablation studies for 20,000 steps.
4.1 Quantitative Results
Comparison with Existing Methods. As shown in Table I our method (row 9) surpasses previous methods in both the FVD and KVD metrics. Notably, compared to Seer (Gu et al., 2023 (row 8), we achieve an approximately 40% reduction in the FVD score on SSv2 and 60% reduction on BridgeData. This suggests that our model generates predictions that are visually closer to the ground truth videos, which implies that FCA succeeds in harnessing the generative abilities of the pre-trained T2V model. On Epic100, also acquire a better performance compared to the previous methods, albeit having FVD score of above 150, which is noticeably higher than scores on the other two datasets. We note that compared to SSv2 and BridgeData. where the camera angle is mostly fixed throughout a video, Epic100 is arguably the most challenging dataset among the three, as it contains egocentric views where the camera attached to the performer (human being), thereby including more substantial view changes and movements of the environments (see Section C.6 for qualitative examples). This characteristic renders the video frames harder to predict.
Comparison with Image-to-Vided Video (12V) Models. Even though pre-trained video generative mod- cls are traditionally text-based; most recently, contemporary work has begun to support image conditions alongside the text. As such, we include discussions on the image-to-video (I2V) model in comparison to our method. We take CogVideoX 11..:oXX.1.--5B-2[[ for the assessment. Compared to our base model (CogVideoX-2B), this model is of a larger parameter size and includes architectural improvements as noted by *1.5". It has
hhttps:/ http:///wwggigeeeeee////tttttooo
8

---

Published in Transactions on Machine Learning Research (11/2025)
Pre-trained with
Methods
FVD
KVD
Yang al. 2024b
image-video
198.0
0.20
Vido LoRA
image-video
352.6
0.45
FCA (Ours)
text-video
67.7
-0.18
Table 2: Comparison with pre-trained image-to-video (12V) models. We report CogVideoX1..56BB 12V Yang et al., 2024bb with and without LoRA (Hu et al. 2022a fine-tuning, Experiments on SSv2. The best numbers are bold black. Our method is shaded gray
(n) Xideo TVV + LoRA (prepend initial Frames)
(c) Cideo 1--00XX11-------- 5-5B-I2V direct inferencn
(b) Ours
(d) Xo 1-V
LoRA
Figure 5: Comparison with LoRA Hu et al,, 2022a) fine-tuning and pre-trained 12V model (CogVideoX1.555B--1 We replace the conditioning initial frames (first two) with the ground truth. Each sample of 16 frames, split into two rows. Text prompt: "Pouring something out of something".
been further trained from T2V model to support a conditional frame in addition to the text (see Sec- tion As shown in Table (row 3 us. 1), our method achieves much better performance in FVD and KVD metrics, which is impressive since we use model with a smaller, two billion parameter size, while competing with a more advanced architecture. We show that the I2V model often cannot predict the correct motions (Figure 5 c). Granted, this is not an entirely fair comparison because the 12V model is not trained on SSv2. However, we note that the performance does not improve, even degrades with LoRA fine-tuning (see discussions on LoRA in Section 443), as shown in Table 2 (row 2) and Figure (d). This suggests the necessity of fine-tuning with capable adaptation design for this task.
4.2 Qualitative Analysis
Figure 4 qualitatively compare our method with existing work. Compared to Seer (Gu et al. 2023), our method exhibits stronger ability in temporal consistency and fidelity, demonstrating that succeed in harnessing the power of the pre-trained video generative model. The examples also show that the model aligns well with the text. Interestingly, we illustrate case where the generated contents differ from the ground truth, yet remain consistent with the input prompt. Such cases may not be accurately measured by the standard FVD and KVD metrics, which compute the similarity between the prediction and the ground truth. This partially demonstrates the necessity of extra quantitative evaluations, to this end, we refer readers the analysis on human evaluation well as VBench (Huang al.) 2024) (Section
4.3 Ablation Studies
In this section, we analyze the effects our key design choices, as well as training tricks that are critical to the performance, as mentioned in Section 3
Fine-tuning Strategy (FT Strat). We begin with validating our fine-tuning strategy, i.e., FCA, by comparing with low-rank adaptation (LoRA) (Hu et al., (022a)) as the standard fine-tuning approach on generative models. Table 3 (rows) 1, 2) shows our design achieves a much better performance in generation, where the FVD scores reduced by over 100%. Note that both experiments conducted without
9

---

Published in Transactions on Machine Learning Research (11/2025)
FT Strat.
FTC
Attn. Mask
FVD
KVD
LoRA [Ilu ot al.
2022a
354.9
0.78
2
FCA
X
152.3
0.18
3
FCA
uniform
X
157.2
0.34
FCA
uniform-R
X
155.2
0.20
5
PCA
layer-wise
X
117.6
0.14
FCA
layer-wise
94.1
0.11
Table 3: Ablation studies of our design choices. Here, FT Strat. stands for fine-tuning strategy; FTC for frame-wist text conditioning design; Attn. Mask for attention mask. For FTC variants: "uniform" one FTC module for all FCA attention layers; "+R" further refine the embeddings through attention; "layer-wise" one FTC module per FCA attention layer. We refer readers Section [B.4 for architectural illustrations of these variants. Experiments are conducted on SSv2 for 20,000 steps. The best numbers are in bold black. Our method is shaded in gray.
introducing other trainable modules into the pipeline for controlling variables. For LoRA fine-tuning, we append the latent of the initial frames to the token sequence.
The benchmark is corroborated with qualitative results in Figure (a), where show that LoRA fine- tuning often fails at fully capturing visual cues in the initial frames, which renders the predicted frames merely "look similar in colors" to the ground truth. Mcanwhile, the generated content is often blurry, contains distortions, and does not adhere to the text conditions. Inferior results are also observed for the more advanced CogVideoX1 1.gVideoX1.5-5B-122 model with LoRA fine-tuning (d), where the model fails to learn meaningful motions, and merely attempts to repeat the conditioning frame. We note that this particular be contributed by other training samples, as SSv2 contain assorted,
case, the color shift is conjectured and widely different color peatterns/saturatie caaattioonbbggthtte /brightnessess across video samples. This suggests that LoRA 's limited parameter space (Hu et al. 2022a) struggles at learning valid patterns from the sophisticated and large amounts of training data. As result, it fails at adapting the pre-trained T2V model to this task. This observation in turn supports our choice of a more capable adaptation strategy that yields much better predictions (Figure b).
(a) ControlNet. duplicating initial frame
(a) No scaling factor
(b) With scaling factor, initialized
Figure 6: Inserting initial frames. We visual- ize ControlNet [Zhang et al. 2023a us. our ap- proach (see Section 3.2). Experiments conducted without frame-wise text conditioning. Text prompt: "Spreading margarine onto bread".
Figure 7: FCA output integration. We exem- plify the significance of scaling factors. Text prompt: "Holding polato next to Vicks VapoRub bollle".
Inserting Initial Frames. Regarding approach of inserting the initial frames into T2V model
we again validate it by comparison. Here, we showcase ControlNct (Zhang et al. 2023a as a baselinn the widely adopted method of injecting spatial conditions inlo text-based generative models (haa
(Section 3.2
et al. 2023 Hu & Xu 2023). In Figure G we qualitatively demonstrate the results. Since ControlNet was originally developed on image generative models rather than vidco ones, we test two variants of the input inspired by recent work Chen al.) 2024; Ye et al., 20233 Blattmann et al., (023):: either duplicating the initial frame along the frame dimension; or padding the initial frames with zero, i.e., black images. We
10

---

Published in Transactions on Machine Learning Research (11/2025)
Fine-tuning
Attn. Mask
Pre- Norm
with
FTC
FVD
KVD 1
FCA
layer-wise
x
105.4
0.13
FCA
layer-wise
94.1
0.11
Table 4: Ablation studies on pre-normalizing the frame-wise text conditioning Ytrames Experi- mented on SSv2 for 20,000 steps. The best numbers are in bold black. Our method is shaded in gray.
discover that in both cases, the model struggles to deviate from the conditioning visual cues in subsequent frames, which greatly hinders its ability to predict motions. As an example, we demonstrate the case of duplicating the imitial frame in Figure t
(a). Our intuition is that ControlNet strongly influences the latent embeddings via direct addition. This is acceptable for video-Lo-video editing or style transfer where the input condition is frame-wiss wise matched to the intended output. However, it is not suitable for inserting the initial frames. We additionally note that one can observe mosnic-pattern artifacts - aside from poor motion predictions by zooming into the ControlNet results, which is also undesired. These finding justifies the need for an approach more tailored to the task setup, and hence, our FCA module design (Figure G b).
FCA Output Integration. In Figure 7 we empirically illustrate the effect scaling the FCA output before adding it back the DiT branch, as discussed in Section 3.2 Note the training would fail without initializing the scaling factor to a relatively small value (i.e., 0.1). We conjecture that down-scaling effectively tunes down the influences of the added embeddings, thereby avoiding drastic changes that harm the training.
Frame-wise Text Conditioning (FTC). Table 3 rows 3-6 ablate our design choices of the frame-wise text. conditioning module. Our key discovery that, separate, layer-wise conditioning modules perform better than using one such module to generate the frame-wise conditioning emboddings for the attention blocks of all FCA layers, even though the latter design more commonll used with cross-altention (e.g., LLi et al. 2022 2023a ). We note these two types of architectures as "layer-wise" and "uniform" in Table 3 and illustrate the architectural differences in Section B.5 (Figure S For the "uniform" architecture, we include two variants, with the difference being whether further to refine the frame-wise text embeddings throughout the FCA atlention layers. As shown in Table 3 (rows 3 5), the layer-wise design outperforms others by a large margin. Notably, both "uniform" designs adversely harm the performance when compared to not adding the framo-wise text conditioning module (row 2), whereas "layer-wise" does not. The results fit our inluition in Section 3.3 that each DiT and FCA allention layer learns different semanties, and so should the conditioning embeddings.
Frame-wise Attention Mask (Attn. Mask). In Table (rows 6 us. 5) we show that the model learns better with the framo-wise attention mask, which enforces the frame-wise text conditioning modules producing text conditions per frame, as introduced in Section 3.3 For a fair comparison, the two experiments adopt the same length in the text conditioning embeddings Yframes-
Fre-Normalizing Frame-wise Text Condition. In Table 4 we show the need for pre-normalizing the frame-wise text embeddings before injecting it into the FCA cross-attention layer. Our intuition that the operation mimics the rrennormalization, i.e., expert adaptive Layernorm (Peebles & Xie 2023) performed in DiT layers, which helps bridge the potential modality gap between embeddings from different modules.
5 Limitations
When comparing with common adaptation strategies, we identify the main trade-offs of our method the result of the increased parameter size introduced by the additional adaptation modules, which is more powerful in its capabilitics, yet undoubtedly heavier than the conventional parameter-offiiient adaptation methods, such as LoRA. This has brought two major implications. First, our method requires more compu- tation resources, specifically, VRAM, which is a direct effect of the parameter count (Section B.2 Second, our method takes longer to train (Section B.I), as well as to inference in practice, we observe the infer- ence time increases by 21% when compared to typical implementation of LoRA (Table We provide the detailed hyperparameters related to the parameter count in Table
11

---

Published in Transactions on Machine Learning Research (11/2025)
In comparison with previous work, note that our method adopts more recent, however, relatively heavier text-to-video base model as opposed to, e-g., Seer (Gu et al. 2023), who adapted a pre-trained text-lo-image model into video generation. While leveraging video generative models aligns with the community's pro- gression, the text-to-video base model contains more parameters than the text-to-image model, for example, to learn additional temporal consistency across frames. Therefore, it also contributes to our method being heavier than the previous model.
6 Conclusion
We propose Frame-wise Conditioning Adaptation (FCA) for text-video prediction, which enables exploiting a pre-trained text-to-video generation model for text-video prediction. The FCA architecture is based on a parallel attention module applied to a diffusion transformer, alongside a frame-wise text conditioning module. This enables FCA to condition on both initial video frames and natural language. We have shown that the method greatly surpasses existing work, both in qualitative and quantitative results, and shared detailed practical findings and tricks to aid future research in this direction. Our ambition is that FCA might provide an indication of a potential avenue of exploration for scalable practical video generation methods.
References
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable vidco diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint or 2rXiv:2311.15127, 2023.
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics datasel. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6299 6308, 2017.
Junsong Chen, Chongjian Ge, Enze Xie, Yue Wu, Lewei Yao, Xiaozhe Ren, Zhongdao Wang, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-E: Weak-to-strong training of diffusion transformer for 4k text-to- image generation. Xiv preprint ar arXie:2403.00 04692, 2024.
Weifeng Chen, Yatai Ji, Jie Wu, Hefeng Wu, Pan Xie, Jiashi Li, Xin Xia, Xucfeng Xiao, and Liang Lin. aovdrol-a-video: Controllable text-to-video diffusion models with motion prior and reward feedback learn- ing. arXiv preprint Xrv::2205..1 13840, 2023.
Raffel Colin. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21, 2020.
Zuozhuo Dai, Zhenghao Zhang, Yao Yao, Bingxue Qiu, Siyu Zhu, Long Qin, and Weizhi Wang. Animatoany- thing: Fine-grained open domain image animation with motion guidance. arXiv preprint arXiv:2311.128 128886 2023.
Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. The epic-kitchens dataset: Collection, challenges and baselines. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(1):44125 4141, 2020.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780 8794, 2021.
Alexcy Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohna Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference Learning Rep- resendations, 2020.
Zicheng Duan, Yuxuan Ding, Chenhui Gou, Ziqin Zhou, Ethan Smith, and Lingqiao Liu. Ezigen: Enhancing zero-shot personalized image generation with precise subject encoding and decoupled guidance. arXiv preprint Xiv: 08091, 2024.
12

---

Published in Transactions on Machine Learning Research (11/2025)
Frederik Ebert, Yanlai Yang, Karl Schmeckpeper, Bernadette Bucher, Georgios Georgakis, Kostas Danillidis, Chelsea Finn, and Sergey Levine. Bridge data: Boosting generalization of robotic skills with cross-domain datasets. arXiv preprint arXiv:2109. 2021.
Christoph Feichtenhofer, Axel Pinz, and Andrew Zisserman. Convolutional two-stream network fusion for video action recognition. In Proceedings of the IEEE conference on compuler vision and pallern vecognilion, pp.
2016.
Zhangyang Gao, Cheng Tan, Lirong Wu, and Stan Z Li. Simvp: Simpler yet better vidco prediction. Proceedings of the IEEE/CW ECVF//V conference on computer vision and pollern recognition, pp. 3170 3180, 2022.
Songwei Ge, Thomas Hayes, Harry Yang, Xi Yin, Guan Pang, David Jacobs, Jia-Bin Huang, and Devi Paríkh. Long video generation with time-agnostic vggan and time-sonsitive transformer. In European Conference on Computer Vision, pp. 102 118. Springer, 2022.
Rohit Girdhar, Mannat Singh, Andrew Brown, Quentin Duval, Samaneh Azadi, Sai Saketh Rambhatla, Akbar Shah, Xi Yin, Devi Parikh, and Ishan Misra. Factorizing text-to-video generation by explicit image conditioning, In European Conference Computer Vision, pp. 205 224. Springer, 2024.
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, He- una Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, et al. The' something something' video database for learning and evaluating visual common sense. In Proceedings of the IEEE international conference on computer vision, pp. 5842-5850, 2017.
Xianfan Gu, Chuan Wen, Weirui Ye, Jiaming Song, and Yang Gao. Seer: Language instructed video predic- tion with latent diffusion models. In The Twelfth International Conference on Learning Representations, 2023.
Yuwei Guo, Ceyuan Yang, Anyi Rao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Sparsectrl: Adding sparse controls to text-to-video diffusion models. In European Conference on Computer Vision, pp. 330-30 348. Springer, 2024.
modeling of long videos. Advances in Newral Information Processing Systems, 35:27953 -79965, 2022. William Harvey, Saeid Naderiparizi, Vaden Masrani, Christian Weilbach, and Frank Wood, Flexible diffusion
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770 100-778 2016.
Jonathan Ho and Tim Salimans. Classifier-free diffusson guidance. In NewrIPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 6840-6851. Curran Associates, Inc., 2020.
Jonathan HO, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. Advances in Neural Information Processing Systems, 35:8633 8646, 2022.
Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvidoo: Largo-scale pretraining for text-to-video generation via transformers. arXiv preprint arXiv:2205. 15868, 2022.
Tobias Höppe, Arash Mehrjou, Stefan Bauer, Didrik Nielsen, and Andrea Dittadi. Diffusion models for video prediction and infillin. arXie preprint arXin:2206. 07696, 2022.
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022a. URL http://ppnevvvieeeeeor/ru//////a
13

---

Published in Transactions on Machine Learning Research (11/2025)
Li Hu. Animate anyone: Consistent and controllable image-to-video synthesis for character animation. In Proceedings of the IEEE/CV Conference on Compuler Vision and Pollern Recognition, pp- 2024.
Yaosi Hu, Chong Luo, and Zhenzhong Chen. Make move: controllable image-to-video generation with text descriptions. Proceedings of the IEEE/CVP Conference on Compuler Vision and Pullern Recognition, pp. 12219, 2022b.
Zhihao Hu and Dong Xu. Videocontrolnet: A motion-guided video-to-video translation framework by using diffusion model with controlnct. arXiv preprint arXi::2307.14073, 2023.
Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, et al. Vbench: Comprehensive benchmark suite for video generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 21807-21818, 2024.
Xuhui Jia, Yang Zhao, Kelvin CK Chan, Yandong Li, Han Zhang, Boqing Gong, Tingbo Hou, Huisheng Wang, and Yu-Chuan Su. Taming encoder for zero fine-tuning image customization with text-to-image diffusion models. arxiv preprint Xiv:23304, 02642, 2023.
Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, Kathrina Wu, Qin Lin, Aladdin Wang, Andong Wang, Changlin Li, Duojun Huang, Fang Yang, Hao Tan, Hongmei Wang, Jacob Song, Jiawang Bai, Jianbing Wu, Jinbao Xue, Joey Wang, Junkun Yuan, Kai Wang, Mengyang Liu, Pengyu Li, Shuai Li, Weiyan Wang, Wenqing Yu, Xinchi Deng, Yang Li, Yanxin Long, Yi Chen, Yutao Cui, Yuanbo Peng, Zhentao Yu, Zhiyu He, Zhiyong Xu, Zixiang Zhou, Zumnan Xu, Yangyu Tao, Qinglin Lu, Songtao Liu, Dax Zhou, Hongfa Wang, Yong Yang, Di Wang, Yuhong Liu, Jie Jiang, and Caesar Zhong, Hunyuanvideo: A systematic framework for large video generative models, 2024. URL htttps:/aaxxvv.org///a http://bb/2212..055 2412.03603
Jumnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified Dision-language understanding and generation. In International conference on machine learning, PMLR, 2022.
pp.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping languageimage pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 19730-19747 PMLR, 2023a.
Pengxiang Lí, Kai Chen, Zhili Liu, Ruiyuan Gao, Lanqing Hong, Guo Zhou, Hua Yao, Yan Yeung, Huchuan Lu, and Xu Jia. Trackdiffusion: Tracklet-conditionee video generation via diffusion models. arXiv preprint 2t4t::332200655. 2023b.
Feng Liang, Bichen Wu, Jialiang Wang, Licheng Yu, Kunpeng Li, Yinan Zhao, Ishan Misra, Jia- Bin Huang, Peizhao Zhang, Peter Vajda, et al. Flowvid: Taming imperfect optical flows for consistent video-to-video synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8207-8216, 2024.
Yang Liu, Chien-Yo Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Tingg Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. In Forty-first International Con- ference on Machine Learning, 2024.
Zhengxiong Luo, Dayou Chen, Yingya Zhang, Yan Huang, Liang Wang, Yujun Shen, Deli Zhao, Jingren Zhou, and Tieniu Tan. Videofusion: Decomposed diffusion models for high-quality video generation. arXiv preprint erx: 083200, 2023.
Yue Ma, Yingqing Hc, Xiaodong Cun, Xintao Wang, Siran Chen, Xiu Li, and Qifeng Chen. Follow your pose: Pose- dose-guided text-to-video generation using prse-free videos. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 4117-4125, 2024.
14

---

Published in Transactions on Machine Learning Research (11/2025)
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVP International Conference Computer Vision, pp. 4195-4205, 2023.
Senthil Purushwalkam, Akash Gokul, Shafiq Joty, and Nikhil Naik. Bootpig: Bootstrapping zero-shot
personalized image generation capabilities in pretrained diffusion models. arXiv preprint or orXiv:2401.13974 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748 8763, PmLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hicrarchical text-conditional image generation with clip latents. arXiv preprint ar Xiv: 06120. 1(2):, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10684 10695, 2022.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer- assisted Intervention-MICCAA. 2015: 18th inter- national conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pp. 234-241. Springer, 2015.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text- to-image diffusion models with deep language understanding. Advances in neural information processing systems, 35:36479-366 36494, 2022.
Cuifeng Shen, Yulu Gan, Chen Chen, Xiongwei Zhu, Lele Cheng, Tingting Gao, and Jinzhi Wang. Decouple content and molion for conditional image-to-video generation. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pp. 4757 4765, 2024.
Xiaoyu Shi, Zhaoyang Huang, Fu- Yun Wang, Weikang Bian, Dasong Li, Yi Zhang, Manyuan Zhang, Ka Chun Choung, Simon See, Hongwei Qin, et al. Motion-i2v: Consistent and controllable image-to- toviccoo generation with explicit motion modeling. In ACM SIGGRAPH 2024 Conference Papers, pp. 1 1-11, 2024.
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data, In The Eleventh International Conference on Learning Representations, 2022.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International
Conference on Learning Representattins, 2020a.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stelano Ermon, and Ben Poole. Score- based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2020b.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhancod transformer with rotary position embedding. Neurocomputingg 568:12706; 2024.
Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Syl- vain Gelly. Towards accurate generative models of video: A new metric & challenges. arXiv preprint arXiv:1812.01111 2018
Ashish Vaswani, Noam Shazoer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
15

---

Published in Transactions on Machine Learning Research (11/2025)
Vikram Voleti, Alexia Dolicoeu-Maartneen and Chris Pal. Mevd-masked conditional video diffusion for prediction, generation, and interpolation. Advances in newral information processing systems, 35:23371- 23385, 2022.
Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen- Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, et al. Wan: Open and advanced large-scale video generative models. arXio preprint arXiv:2503.2003 20314, 2025.
Jiuniu Wang, Hangje Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, and Shiwei Zhang. Modelscope text-to-video technical report. arXiv preprint arXiv:230..668 2023a.
Yi Wang, Yinan He, Yizhuo Li, Kunchang Li, Jiashuo Yu, Xin Ma, Xinhao Li, Guo Chen, Xinyuan Chen, Yaohui Wang, et al. Internvid: A largo-scale video-text datasct for multimodal undorstanding and gener- alion. In The Twelfth International Conference on Learning Represenlations, 2023b.
Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 7623-7633, October 2023a.
Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tunc-a-video: One-shot tuning of image diffusion models for text-to-video generation. In Proceedings of the IEEE/CVI International Conference Computer Vision, pp. 2233-663,,2 2023b.
Ruiqi Wu, Liangyu Chen, Tong Yang, Chunle Guo, Chongyi Li, and Xiangyu Zhang. Lamp: Learn a motion pattern for few-shot-basse based video generation. arXiv preprint arXiu:2310 0.10769, 2023c.
Zhongcong Xu, Jianfeng Zhang, Jun Hao Liew, Hanshu Yan, Jia-Wei Liu, Chenxu Zhang, Jiashi Feng, and Mike Zheng Shou. Magicaninate: Temporally consistent human image animation using diffusion model. In Proceedings of the IEEE/CVE Conference on Compuler Vision and Pallern Recognition, pp. 1481-1490) 2024.
Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. Videogpt: Video generation using vq-vae and transformers. arXiv preprint 2rXiv:2104.1015 2021.
Shiyuan Yang, Liang Hou, Haibin Huang, Chongyang Ma, Pengfei Wan, Di Zhang, Xiaodong Chen, and Jing Liao. Direct-a-video: Customized video generation with user-directed camera movement and object motion. In ACM SIGGRAPH 2024 Conference Papers, pp. 1-12, 2024a.
Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, al. Cogvideox: Text-to-video diffusion models with an expert transformer. arxiv preprint arXiv:2408.000 06072, 2024b.
Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-iman image diffusion models. arXiv preprint arXiv:2308.0677 067221 2023.
Xi Ye and Guillaume-Alexaadrre Bilodeau. Stdiff: Spatio-temporal diffusion for contimous stochastic video prodiction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 6666-6674, 2024.
Lijun Yu, Jose Lezama, Nitesh Bharadwaj Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Agrim Gupta, Xiuye Gu, Alexander G Hauptmann, et al. Language model beats is key to visual generation. In The Twelfth International Conference on Learning Hepresentations, 2023a.
Sihyun Yu, Kihyuk Sohn, Subin Kim, and Jinwoo Shin. Vidco probabilistic diffusion models in projected latent space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 18456 18466, 2023b.
16

---

Published in Transactions on Machine Learning Research (11/2025)
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedingg of the IEEE/CVF international conference on computer vision, pp. 9836-3847, 2023a.
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-effiient fine-tuning, In The Eleventh International Conference on Learning Representations, 2023b.
Zhenghao Zhang, Junchao Liao, Menghao Li, Zuozhuo Dai, Bingxue Qiu, Siyu Zhu, Long Qin, and Weizhi Wang. Tora: Trajectory-oriented diffusion transformer for video generation. arXiv preprint arXiv:2407.21 20705 2024.
Rui Zhao, Yuchao Gu, Jay Zhangjic Wu, David Junhao Zhang, Jia- Wei Liu, Weijia Wu, Jussi Keppo, and Mike Zheng Shou. Motiondirector: Motion customization of text-to video diffusion models. In European Conference on Computer Vision, pp. 773-200. Springer, 2024.
Shihao Zhao, Dongdong Chen, Yen-Chun Chen, Jianmin Bao, Shaozhe Hao, Lu Yuan, and Kwan- Yee K Wong. Uni-controlnet: All-in-one control to text-to-image diffusion models. Advances in Neural Informa- tion Processing Systems, 36:11127 11150, 2023.
Zangwei Zheng, Xiangyu Peng. Tianji Yang, Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. Open-sora: Democratizing efficient video production for all. arXiv preprint arXiv:24120 12.20404 2024.
A Appendix
B Additional Implementation Details
B.1 Training and Inference
In Table 5 we report our detailed papammerrseeters for training. We note that our training configuration mostly inherits from the CogVideos codebasd
We conduct our experiments four NVIDIA A100 40G GPUs. In practice, we observe the VRAM usage approximately 40GiB per GPU. The training takes approximately 24 hours per 10,000 steps with our batch size noted in Table 5 on SSv2. For inference, we leverage the diffusers's pipeline" without altering its configurations. Specifically, we use the DDIM Song et al. (2020a) sampler with the guidance scale À set to 6.0, and the sampling step being 50, The evaluation on SSv2 (first 2,048 samples) takes around six hours on four NVIDIA A100 40G GPUs.
Our method is fine-tuned on video resolution of 720 480 with the number of frames per video downsampled to 16 for all datascts. We follow the CogVideoX implementation in resizing given frame, while using the Seer Gu et al. (2023) codebase for casting an arbitrary raw video to a fixed nunber of frames. Our choice of resolution is the lowest recommended by the pre-trained CogVideoX- 2B model, we empirically confirm that sctting the resolution to lower will irrecoverably disrupt the generation quality. Mcanwhile, CigVidooX requires the frames to be divisible by 8. We note that this differs from the settings of previous work Gu et al. 2002)), where the resolution is 256 256, with SSv2 being further downsampled that our choice of a higher resolution and number of frames introduces greater challenge to our method. However, it does not affect benchmarking the FVD and KVD metrics, whose details are introduced in Section
frames. We note
In Table 6 we quantitatively compare FCA vs LoRA inference time obtained from the validation split. We observe that FCA increases the inference time per sample by 21%, which is to be expected given our increased parameter size (see Section B.2
http::///ttaabboo///UUU//////// https://wwgggaeeee///////////eea cogvideexx
httpp:///it http:///dddd....mm/eeeeeeeee/e//
17

---

Published in Transactions on Machine Learning Research (11/2025)
Parameter
Value
Height
480
Width
720
Number of frames
16
VAE temporal downsampling rate
4
Effective number frames
16/4
DIT text token length (T5 Colin 2020
226
FTC
Number of layera each
Number of queries
226 x Eff. No. frames 904
Batch size per GPU
1
Number of GPUs
Gradient, accumolation
Effective botch size
x
Traiuing steps (SSv2)
100, 000
TRAINING
Training steps (BridgeData)
25,
Training steps (Fpic100)
50,
Scheduler
cosine restart
Number of eycles
Warmup steps Learning rate
200
0.001
Optimizer
Adamm (G = 0.9, 0g
0.95)
Gradient clip Precision
1.0
bfloat16
Table 5: Hyper-parameters for fine-tuning. FTC: frame-wise text conditioning module. Values com- puted from other parameters shaded in gray.
Inference time for test split (2048 samples) (min)
Per-sample time (s)
Methods
1 LoRA
1,172
34.34
2 FCA (Ous)
1,424
41.72
Table Inference efficiency. Quantitatively, we observe that FCA increases the inference time per sample by 21%.
B.2 Parameter Size Comparison
Table demonstrates the parameter count for the stale-of-the-art model (Seer GGu et al, 2023 preceding our method, as well as the parameter count for ours.
Mcanwhile, note that the remaining baseline methods shown in Table are derived from Gu al. 20223 where cannot verify their training details or adaptations to this task. Nevertheless, present the estimated total parameter counts for VideoFusion (Luo et al. 2023 and Tune-A-Video (Wu et al. 2023b based on the publicly available knowledge, as they are two of the relatively recent methods that bear decent performance compared to Seer and ours.
Table shows the trainable parameter count for LoRA and FCA (our module). We note that for LoRA on CogvideoX, we tested various rank values and our reported results (Section 2 are based on a rank of 256, which we discovered to be optimal (lower rank yields worse results, while higher ranks do not improve the performance).
B.3 FVD and KVD Evaluation
We follow Gu et al. (023)) and adopt the evaluation code implemented by VideoGPT Yan et al. (2021) The benchmark requires pairs prediction and ground truth videos with an equal number of frames, as well as the same resolution. Note that the prediction and ground truth are firstly resized into 224 x 224 regardless of the original resolution. This is because the evaluation based on a pre-trained I3D model on Kinetics-400 Carreira & Zisserman (017 that expects such inputs Unterthiner et al. (2018).
18

---

Published in Transactions on Machine Learning Research (11/2025)
Pre-trained with
Trainable Params
Base Model
Total
Methods
Params
Params
VideoFusion (Luo al,, 2023)*
text-video
2.59B
Tune-A A-Video (Wu et al., 2023a)*
text-image
890M
Seer (Gu al,, 2023)1
text-image
405.90M
890M
405.90M
890M 1.3B
Ours!
text-video
1.0B
2B 3.0B
Table 7: Comparison of parameter counts across different video generation models. * Values derived from publicly available knowledge. Specifically, for VideoPusion, pre-trained base model 2B + residual gencrator 0.59B 2.59B; for Tune-A-Video, based on Stable Diffusion v1.5 890M. f Both methods apply additional trainable modules, i.e., additional parameters in addition to the base model parameters. The best-performing configuration (ours) is shaded in gray.
Row numbers follow Table 1.
Trainable Parameter Count
Methods
1 LoRA (c.f. Table row 2)
0.27B
2 PCA (Ours)
1.04B
Table 8: Comparison of trainable parameter counts. FCA uses larger but more expressive parameter subset than LoRA. The best-performing configuration (ours) is shaded in gray.
B.4 Frame-wise Text Conditioning Design
In the main text Section 4.3 and Table 3 have ablated various designs the frame-wise text conditioning
module (FTC). We introduce the implementation details follows.
Layer-wise Design. We empirically determine this as the superior way of integrating the frame-wise text conditioning embeddings into the FCA attention layers, which is noted as "layer-wise" in Table 3 and demonstrated in Figure along with the detailed architecture. In practice, each frame-wise lext conditioning module is of one layer as reported in Table 5 where each layer contains a self-attention, a cross attentioenn and a foed-forward block Li al.) (2022). We are unable to experiment with such modules of multiple layers under this setting due to the VRAM limitation.
Uniform Design. In Table 3 (rows 3 3-6), we inchude results two variants of the frame-wiss text condi- tioning design which yield inferior results, we label them as "uniform" (i.e., uniformly using one module for all layers) and "uniform+R" (further refining the module output through FCA attention layers), as shown in Figure S For these "uniform" " designs which use only one such module in lotal, we are less constrained by the VRAM limitation compared to the above. To this end, we experiment with various numbers of layers. However, we empirically find this not critical for the performance. Indeod, as noted in the main text, Table 3 such " "uniform" designs adversely harm the training, rendering it ineffectual in general.
B.5 Implementation Details of Ablation Studies
LoRA Fine-tuning. In the main text Section and 4.3 fine-tuning. These LoRA experiments have been conducted on CogVideoX-22 (12V) and CogVideoX1.55 5B-I2V. We adhere to the training configurations released in the official codebase and make no changes to the parameters.
we have compared our method against LoRA
we have conducted experiments on using ControlNet to inject the initial frames. We note that since ControlNet Hu & Xu (2023) was originally proposed on the U- Net Ronneberger et al. (2015) architecture, modifications
ControlNet on DIT. In the main text Section 4.3
needed for it to be used on CogVideox: which is based on DiT Peebles & Xie (2023) We experimented with two variants of the architect applicable
however, both yielded very similar inferior results, as demonstrated in the main text, Figure
httte///chhubb.mm/////aaa///oooe controllet
htlps://ghthubb.cmm/ Tewp:///eeeeeeeeeeneen http://hhwhbb..mmmmmmmmm/aaaaaaa
19

---

Published in Transactions on Machine Learning Research (11/2025)
Multi-Head Attention
Block
Multi-ttedd Cross-Attention
Block
Frame-wise Text Conditioning
Multi-Head Crass Attention
Multi-eead CoossAAtentionn
DIT Block
Frame wiise
Conditioning
Figure Additional variants of designs inject frame-wise text conditions into the FCA attention block. This figure complements Figure 2 and results in the main text, Table 3 (rows 3 and 4). Left: Using one frame-wise text conditioning module for all FCA attention layers; note this "uniform" in the main text. Right: Further refining the embeddings through the FCA attention; note this "uniform+R"" in the main text.
Background Imaging Consistency Quality
Motion
Overall
Subject
Temporal
Methods
Smoothness
Consistency Consistency Flickering
Seer Gu et al. 2023
0.8635
0.3765
0.9172
0.1038
0.7301
0.8955
FCA (Ous)
0.8769
0.4279
0.9570
0.1965
0.8098
0.9461
Table 9: VBench Huang et al. (2024) evaluation results. We compare with Seer Gu et al. (0023
Results are reported on SSv2. Note that we use custom prompts (i.e., from the SSv2 validation set) for the evaluation, which differs from the standard VBench evaluation setup. Values are the higher the better. The best numbers are in bold black. Our method is shaded in gray.
C Additional Experimental Results
C.1 VBench Evaluation
We additionally evaluate our model using VBench Huang al. (2024), a recently introduced comprehen- sive benchmark designed for video generative models. Unlike standard video generation benchmarks that primarily evaluate unconditional video synthesis, VBench enables a more fine-grained analysis that directly aligns with our goal of producing temporally stable and semantically coherent video sequencos.
Evaluation Details. We specificaly focus on six key dimensions: background consistency, imaging quality, motion smoothness, overall consistency, subject consistency, and temporal flickering. These dimensions are highly relevant to the text-video prediction task by collectively capturing both spatial quality and temporal stability.
We note that the standard VBench setup is designed for unconditional video generation, which assesses a model on a list of pre-defined prompts. While well-suited for benchmarking pre-trained general-purpose video generative models, does not align directly with our fine-tuning task. As such, we opt for using custom prompts, i.., the prompts from the validation set SSv2 for the evaluation.
Results. As shown in Table 9 our method outperforms Seer Gu al. 2023 across all selected VBench dimensions. These results validate the effectiveness of our approach in improving both temporal coherence and per-frame quality in video prediction tasks. The substantial gains in consistency-related scores and motion smoothmess indicate that our method successfully learns to maintain global coherence, while higher subject consistency confirms improved local stability. Meanwhile, we note that our method better at adhering the text prompt, as overall consistency powered by ViCLIP (Wang et al. (223b)) specifically assess text-visual alignment. These results highlight the robustness of our approach to produce high-quality, temporally consistent video predictions.
20

---

Published in Transactions on Machine Learning Research (11/2025)
C.2 Human Evaluation
Evaluation Details. Following previous work Gu et al. (2023), we conduct a human evaluation to assess the generation quality further. Specifically, we evaluate the SSv2 datasel, with 24 participants and 50 samples on three methods. We compare across our model, Seer Gu et al. (2023). and the more advanced CogVidcoX1.5-5B-I1 Vang et al. (2024b). For each sample, we ask the participant to rate integer, from to (inclusive, the higher the better) on three aspects: text alignment, temporal consistency, and visual fidelity. Text alignment focuses the model's ability to generate motions that adhere to the input text. Temporal consistency assesses if the sample is valid extension of the initial frames, and if the subsequent generated frames are consistent. Here, we replace the initial frames with the ground truth frames, thereby assessing if the immediale predicted frames seamlessly extend from these initial frames. Visual fidelity addresses the appearances of the predicted objects, as well as e.g., human hands throughout the video.
We supply the ground truth vidco to the participants as reference compare, and direct the participants refrain from using the ratings 0 or frequently, unless the sample of extremely low or high quality. We allow the participants to review each sample multiple times and compare across each method.
Average Ratings (0 to 5)
3.91
3.58
3.42
3.44
3.28
2.83
2.69
2.40
2.16
Ours
Seer
CogVideoX1.5-300 5---12
Text Alignment
Temporal Consistency
Visual Fidelity
Figure 9: Human evaluation results. We report the average ratings (from to 5 inclusive, the higher the better) among method, Scer Gu et al. (2023), and CogVideoX11.-5BB-22 Yang et al. (2024b).
Results. Figure presents the average ratings for each method on the three aspects. Our method surpasses both Seer and CogVideoX1.5-5BBB2 on text alignment and visual fidelity, suggesting the effoctiveness of our approach in fitting the pre-trained generative model to the task via fine-tuning. On text alignment, note that CogVideexXI.--5B-I2V receives the lowest rating, which is reasonable given that it has not is more favored than
been fine-tuned on the datasct. However, visual fidelity, Cog CogVidcoX1.555B... Seer, which demonstrates the power of its pre-training. Our method receives a similar rating compared to Cog CogVideoX1.. V..iioe111.-5B-22 on temporal consistency, while both surpass Seer. We note that the high rating of Cog 5--B-IV in this aspect is partly because the model often cannot render meaningful motions, and simply repeats the initial frames, thereby appearing consistent across time. This effect is discussed the main text. (Section 4.1 and demonstrated in Figure (c). Overall, the human evaluation indicates the strength of our method across the three aspects.
C.3 Additional Experiments on Wan2.1-1.3B
To demonstrate the generalizability of our adaptation stratogy, we additionally test our method on Wan2.1- 1.3B (Wan et al. 2025) and compare it with LoRA (Hu al. 2022a) fine-tuning and the pre-trained model, shown in Table
21

---

Published in Transactions on Machine Learning Research (11/2025)
Additional Fine-tuning
Methods
FVD
KVD
Wan2.1-1.38 12V
N/A (pre-trained inference)
155.12
0.10
Wan2.1-1.38 12V
LoRA
94.51
-0.06
Wan2.1-1.3B T2V
FCA
72.17
-0.11
Comparison of Wan2.1-1.3B (Wan et al., 2025) variants with different fine-tuning methods. We report results for pre-trained inference and fine-tuned models using LoRA and FCA. The best scores are highlighted bold, and our method is shaded in gray.
Table 10:
Additional Fine-tuning
Methods
FVD 4 KVD
CogVideoX1.5-5B 12V Li 12V
DoRA
Liu al. 2024
349.11
0.89
AdaLoRA
Zhang
20233
305.83
0.70
Table 11:
Comparison of CogVidcoX1.55 T2V with different fine-tuning adapters. We compare DoRA (Liu el al. 2024) and AdaLoRA (Zhang et al. 2023b) fine-tuning on the same base model. This table complements Table row 2.
Note that we enlist the 12 version for pure inference and LoRA fine-tuning, because the task TVP requires one input the initial frame as a condition alongside the text. Meanwhile, for our method, which is designed adapt a T2V model, we apply on the T2V pre-trained mode
Quantitatively, note that method is more effective than LoRA fine-tuning plain inference of the pre-trained model, which corroborates our findings in our main paper and the performance of our method on DIT.
C.4 Additional Experiments on variants of LoRA
We additionally test on variants of LoRA, namely, DoRA and AdaLoRA, which demonstrated in Table I1 We note that these variants LoRA perform similarly to the vanilla LoRA on the TVP task, in the sense that the generation quality is also less optimal.
C.5 Quantitative Results ControlNet Experiments
Table 12 demonstrates the quantitative results on the ControlNet-related experiments, which complements our discussion in Section 4.3
C.6
Additional Qualitative Examples
We include additional qualitative examples SSv2 in Figure 10
BridgeData in Figure 12 and Figure 13
and EpicKitchen-100 (Epic100) in Figure 14 and Figure 15
Instruction Manipulation. In Figure I1 we showcase method's ability to cope with instruction manipulation. Specifically, we perform inference on the fine-tuned model but manually alter the text prompt, e.g. "turning the camera left. to "turning the camera down "; or "putting egg in bowl" to "putting tomato in bowl". We show that our method can handle such variations and generate plausible predictions, indicating superior generalizability.
C.7 Limitations and Failure Cases
Figure 10 (h), Figure (a), well as Figure 15 illustrate several failure cases on different datasets.
http:://aangingfeeeeo/o//lllllla Wan.p Fun- 133 Control
htlpp: http:///wggggnngacooooaaaaaaaaa Wan-
22

---

Published in Transactions on Machine Learning Research (11/2025)
Methods
Padding Strategy
FVD 1 KVD 1
ControlNet
Padding with (black) frames Duplicating the initial frame
1073.43
2.18
ControlNet
529.65
1.38
Table 12: Results of different padding strategies on ControlNet performance. Padding with
duplicated initial frames siguificantly improves both FVD and KVD metrics. These results complement Sec- tion
On SSv2, note that the model may fail when facing object transformations or deformations, e-g-, folding mattress. Understandably, this poses a greater challenge compared to other common types of prompts, such moving objects or the camera angle along certain direction.
For BridgeData, we note that imperfection would sometimes arise when cortain object physically makes contact with another object (e.g,, put something a pot), thereby causing artifacts to rendered in the scene. Such imperfections are often minor and occur at the ending frames, however, they are indeed failed attempts at maintaining the physical shape of object.
On Epic100, we note that the prediction quality is usually worse than the other two datasets, which is corroborated by the quantitative metries reported in the main text, Table As shown in the failure cases, the generated contents are often blurry or contain artifacts. We note that this is partly because Epic is a dataset with egocentric views, which often includes frames with drastic camera motions (Figure 15) Combined with the fact that the kitchen environments are very diverse, this dataset. is arguably the most challenging among the threc.
Potential Social Impacts
Our work is on vidco generation, which involves producing visual content based on user prompts. Our model is trained on samples involving the manipulation of objects by human beings or robot arms. This setup bears great potential in domains where such manipulations are of interest such action anticipations planning in robotics and virtual/and /angmented reality. However, note that method could also be misused, which is concern shared by most existing research on generative models. In our particular case, as a part of the training data (in SSv2 and EpicKitchen-100) portrays real-life human beings carrying out motions, our model might have learned to recreate specific human identitics within such data. In addition, since our method is based on a large-scale, pre-trained lext-to-video generative model, shares many of the capabilities with the pre-trained model that could be used maliciously in creating and disseminating harmful untrue content.
23

---

Published in Transactions on Machine Learning Research (11/2025)
(a) Text prOmpt: "Turning the camera left while filming wall mounted fan."
QuRS
(b) Text prompt: "Moving eraser and sharpner closer to each other."
OTTS
(c) Text prompt: "Pushing bottle from left to right."
Ouns
XXI
MI
11
M
M
UTI
(d) Text prompt: "Covering a dcodorant with towel."
Ouns
Text prompt: "Moving the spoon away from the fork."
Quns
(f)
Text prompt: "Pretending open bottle without actually opening it."
Omiss
(g)
Text prompt: "Putting egg into bowl."
Ouns
(h) Text prompt: "Folding wool mat."
GT OuRS
(1) TTxt prompt: "Moving wallet down."
O0is Orrs
(j) Text prompt: "Pushing orange bowl from right."
Figure 10: Additional qualitative examples on Sorrehhinn-SSmehiigg-.... V2 (SSv2). Each sample is of 16 frames. We replace the initial frames (first two) with the ground truth.
24

---

Published in Transactions on Machine Learning Research (11/2025)
(a) Text prompt: "Turning the camera left while filming wall mounted fan."
Text. prompt: "Turning the camera. right while filming wall mounted fan."
Text prompt: "Turning the camera. up while filming wall mounted fan."
Text prompt: "Turning the camera down while filming wall mounted fan."
(b) Text prompt: "Putting egg into bowl."
Text prompt: "Putting tomato into bowl."
V2 (SSv2). We manipulate the text prompt inference to see if the model can predict diverse motions a given initial frame. All frames are obtained from our model. ORI the original text prompt. The modified phrases are shaded in red.. Each sample is of 16 frames, split into two rows. We replace the initial frames (first two) with the ground truth.
Figure 11: Instruction manipulation on
25

---

Published in Transactions on Machine Learning Research (11/2025)
Text prompt: "Flip pot upright sink."
(b) Text prompt: "Put lid stove."
Qmts
(c) Text prompt: "Put eggplant into pan."
Figure 12: Qualitative examples on BridgeData. Each sample is of 16 frames, split into two rows for
casier viewing of details.
26

---

Published in Transactions on Machine Learning Research (11/2025)
(d) Text prompt: "Put detergent from into drying rack."
(e) Text prompt: "Put pepper in pot pan."
(f) Text prompt: "Put sweet potato in pot."
Outs
Figure 13: Qualitative examples on BridgeData (cont.). Each sample is of 16 frames, split into two
rows for easier viewing of details.
27

---

Published in Transactions on Machine Learning Research (11/2025)
Text prompt: "Put down saucepan." ?
Ouns
(b) Text prompt: "Tura off tap."
Orts
(c) Text prompt: "Open fridge."
Quns
Figure 14: Qualitative examples on EpicKitchen-100 (Epic100). Each sample is of 16 frames, split into two rows.
28

---

Published in Transactions on Machine Learning Research (11/2025)
(a) Text prompt: "Pick up mug."
Oras
(b) Text prompt: "Close kitchen door."
Quna
Figure 15: Qualitative examples on EpicKitchen-100 (Epic100) (cont.). We show several failure cases. Note the drastic change view due to camera movements the ground truths (appearing at the end of the video clips), making learning motions challenging. Each sample of 16 frames, split into two rows.
29


Reviews:
[{"review_id": "bIppfrWnyE", "reviewer": "Reviewer_Yjeu", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors introduce a novel fine-tuning method (and architecture) for improving sample quality of text-to-video (T2V) generation models, specifically focusing on diffusion transformer-based models. They're adaptation fine-tuning (adaptation) approach, named Frame-wise Conditioning Adaptation (FCA) uses an auxiliary attention-based architecture to provide frame-wise text embeddings for each frame, which improves upon existing techniques for fine-tuning T2V generation models. The authors also provide a thorough empirical evaluation of their newly proposed approach to validate its usefulness for fine-tuning T2V generation models. Through this empirical evaluation, the authors show that FCA (their approach) is a dominant fine-tuning approach for improving video generation quality, in comparison to the baselines considered in this work.  \n\n**Strengths:**\n\n- The authors propose a novel fine-tuning method, FCA, for T2V diffusion transformer models. The method is elegant and can incorporate conditions from both video frames and natural language within its fine-tuning procedure.   \n- The authors provide a comprehensive empirical evaluation and demonstrate that their proposed fine-tuning method, FCA, improves sample quality of T2V generation, and outperforms all counterpart baseline approaches (which were considered in this work).\n- The manuscript is generally well written and easy to follow, making it easy to understand the central contributions of this work.\n\n**Weaknesses:**\n\n- My understanding is that the authors consider only 1 pre-trained text-to-video model in their experiments (CogVideoX-2B). Since this work proposes (and claims) a general method for fine-tuning pre-trained T2V diffusion models, it would be useful to see experiments/results for 1-3 other models. Possibly, it is sufficient to pick a single experiment and show performance for different models. \n- No explicit mention of the limitations in the main text. For example, it sounds like FCA requires the fine-tuning of an additional attention-based architecture. How does this compare to existing approaches? It is valuable to mention some of the trade-offs (if any) and limitations regarding the increased complexity of fine-tuning the proposed method in the main text/conclusion. \n- Only a minor weakness, some portions of the manuscript could use further clarification and improved organization. See questions and minor comments below.", "claims_and_evidence_yn": "Yes", "claims_explanation": "In general, yes. The authors conduct a thorough empirical evaluation of their proposed approach by considering various experimental settings, several baselines, and show both quantitative and qualitative results. Because of this, I believe the claims made in this submission are sufficiently supported.", "audience_yn": "Yes", "audience_explanation": "Yes, I believe the findings of this work would be of interest to TMLR's vision and video generation audience. This paper tackles the problem of improving fine-tuning of pre-trained diffusion transformers for T2V generation. I believe this is an interesting a relevant problem in the domain.", "requested_changes": "Apart from minor clarifications (see comments below), my requested changes are not pertinent. However, I believe they could help improve the overall quality of this work. Specifically, applying FCA to an additional pre-trained T2V model and adding a more explicit discussion of limitations and future work in the main text (conclusion).", "broader_impact_concerns": "I have no broader impact concerns.", "additional_comments": "**Questions:**\n\n- In several of the figures showing qualitative results, the authors compare to \"ground truth\" frames. How are the ground truth frames acquired in this setup? Is this based on the training data, i.e. for each text prompt, e.g. like in Figure 4, \"Holding helmet\", is there a corresponding set of frames (a video) that the model uses during training? \n- How does FCA affect inference-time (if at all)?\n\nMinor comments:\n\n- Figure 1 is never referenced in the text. \n- It would be helpful for the reader to move figures closer to where they are referenced in the text (e.g. Figure 2 and Figure 3).", "extra_scores": {}, "extra_text": {}}, {"review_id": "6FgKjobqkn", "reviewer": "Reviewer_bhJ4", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose a fine-tuning method that extends text-to-video models to perform text-video prediction. In particular, they introduced Frame-wise Conditioning Adaptation (FCA), which can be inserted into a DiT block to allow initial frames as extra condition.", "claims_and_evidence_yn": "Yes", "claims_explanation": "**Claim 1:** \n- Existing TVP methods adapted from text-to-image (T2I) models lack continuity and quality.\n- Evidence: Table 1 quantitatively shows that methods like Seer or Tune-A-Video can yield higher FVD/KVD (temporal consistency metrics) compared to FCA.\n\n\n**Claim 2:** \n- Standard fine-tuning with LoRA fails for TVP on T2V models.\n- Evidence: Ablation experiments (Table 3, Figure 5) demonstrate LoRA produces blurry, distorted outputs that poorly adhere to text, with FVD scores significantly worse than FCA\n\n**Claim 3:** \n- The proposed Frame-wise Conditioning Adaptation (FCA) is an effective fine-tuning strategy for T2V models.\n- Evidence: FCA integrates frame-wise conditioning modules and parallel attention while freezing the main DiT. Ablation studies show FCA outperforms LoRA and ControlNet baselines (Figures 5–6).\n\n**Claim 4:** \n- Incorporating initial frames as conditions improves temporal consistency in TVP.\n- Evidence: Using frozen DiT embeddings of initial frames (Figure 2) leads to smoother video predictions. Comparisons against ControlNet baselines (Figure 6) show FCA handles motion prediction better, while ControlNet struggles with overfitting to static inputs", "audience_yn": "Yes", "audience_explanation": "I believe the text-video prediction problem is interesting and the paper establishes useful insights and baselines for the task.", "requested_changes": "- Can the authors add parameter count for each model in Table 1?\n- Can the authors add trainable parameter count for FCA and LoRA?\n- For TVP task, I think that both video generation quality and text alignment should be equally important. However, the majority of the paper just discusses the first metric. On the other hand, the evaluation seems to be weak. I think an evaluation of more than 15 samples is needed to make it more persuasive. Also, I even think this section should be moved to the main text, not the Appendix.\n- I like the ControlNet comparison, but I think it can be stronger with quantitative results.", "broader_impact_concerns": "None", "additional_comments": "- “This is probably because the limited amount of tunable parameters of LoRA is insufficient to cope with the quantity and complexity of the training data”. This claim is weakly supported. In my opinion, FCA is more complex than LoRA and so the fact that it performs better than LoRA might tell us that FCA is more suitable for the task, but does not tell us about why LoRA fails.", "extra_scores": {}, "extra_text": {}}, {"review_id": "xsBd4TzNQT", "reviewer": "Reviewer_sKjJ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "finetuning pre-trained text-to-video models using LoRA yields undesirable results. To address this, the authors propose an adaptation-based strategy, Framewise Conditioning Adaptation (FCA) to inject more context into the models embedding space. This strategy generates frame-wise text embeddings from input text as well as utilize the initial video frames, acting as an additional condition for generation.\n\n\nStrengths:\n- Method is well motivated and presented in a simple manner\n- Empirical results are strong (large FVD/KVD improvements vs Seer et al.) and ablations credibly support the design choices\n- Ablation studies provide a better understanding about FCA's contribution\n\nWeaknesses:\n- Generality beyond the object manipulation task is unclear from the results. performance on broader motion regimes is unclear.\n- While baseline coverage is reasonable, comparisons with more recent T2V/DiT variants from 2024-2025 would strengthen claims, given the fast-moving landscape.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Yes, the claims made in the paper are supported. The authors provide qualitative and quantitative results showing LoRA inadequacy for Text-Video Prediction, technical design choices, human evaluations, along with failure case analysis.", "audience_yn": "Yes", "audience_explanation": "The proposed method, in my view, showcases how effective engineering can extend the usefulness of existing Text-Video Prediction models.", "requested_changes": "- Please add Open-Sora and VideoCrafter2 as same-backbone baselines, each with LoRA-tuned vs FCA-tuned variants\n- It would be great to also compare against other adaptation-based methods (AdaLoRA, DoRA)", "broader_impact_concerns": "Yes, the authors mention the broader societal impacts in Appendix D.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)