FrameBridge: Improving Image-to-Video Generation with Bridge Models
Authors: Yuji Wang
Zehua Chen
Chen Xiaoyu
Yixiang Wei
Jun Zhu
Jianfei Chen
Venue: ICML 2025 poster (2025)
Keywords: Image-to-Video Generation
Diffusion Models
Diffusion Bridge Models
Prior Distribution
Data-to-Data Generation
Primary area: deep_learning->generative_models_and_autoencoders
TL;DR: We propose a bridge-based data-to-data generation process for image-to-video (I2V) synthesis, and present two improving techniques for bridge-based I2V models, achieving superior quality than previous ones based on diffusion models.

Abstract: Diffusion models have achieved remarkable progress on image-to-video (I2V) generation, while their noise-to-data generation process is inherently mismatched with this task, which may lead to suboptimal synthesis quality. In this work, we present FrameBridge. By modeling the frame-to-frames generation process with a bridge model based data-to-data generative process, we are able to fully exploit the information contained in the given image and improve the consistency between the generation process and I2V task.
Moreover, we propose two novel techniques toward the two popular settings of training I2V models, respectively. Firstly, we propose SNR-Aligned Fine-tuning (SAF), making the first attempt to fine-tune a diffusion model to a bridge model and, therefore, allowing us to utilize the pre-trained diffusion-based text-to-video (T2V) models. Secondly, we propose neural prior, further improving the synthesis quality of FrameBridge when training from scratch. Experiments conducted on WebVid-2M and UCF-101 demonstrate the superior quality of FrameBridge in comparison with the diffusion counterpart (zero-shot FVD 95 vs. 192 on MSR-VTT and non-zero-shot FVD 122 vs. 171 on UCF-101), and the advantages of our proposed SAF and neural prior for bridge-based I2V models. The project page: https://framebridge-icml.github.io/

FrameBridge: Improving Image-to- Videeo Generation with Bridge Models
Yuji Wang" Zehua Chen
12
Xiaoyu Chen'
Yixiang Wei! ! Jun Zhu
Jianfei Chen'
Abstract
lo-frames generation task where maintaining appearance consistency and ensuring temporal coherence of generated video frames are the key evaluation criteria (Xing et al., 2024; Zhang et al., 2024a). With the recent progress in video synthesis (Brooks et al., 2024; Yang et al., 2024b; Blattmann et al., 2023; Bao et al., 2024), several dnffusion-based I2V frameworks have been proposed. with novel designs on net- work architecture (Xing et al.. 2024; Zhang et al., 2024a; Chen et al., 2023b; Ren al.; Lu al.), cascaded frame- work (Jain et al., 2024; Zhang et al., 2023), and motion representation (Zhang et al., 2024b; Ni et al., 2023). How- ever, although these methods have demonstrated the strong capability of diffusion models (Ho et al., 2020; Song et al.) for I2V synthesis, their noise-to-data generation process is inherently mismatched with the trame-to-franee synthe- sis of 12V task, making them suffer from the difficulty of generating high-quality video samples from uninformative Gaussian noise rather than the given image.
Diffusion models have achieved remarkable progress on image-to-video (12V) generation. while their noise-to-data generation process is in- herently mismatched with this task, which may lead to suboptimal synthesis quality. In this work. we present FradeBridge. By modeling the frame- to-frames generation process with a bridge model based data-to-data generative process, we are able fully exploit the information contained in the given image and improve the consistency be- tween the generation process and 12V task. More- over, we propose two novel techniques toward the two popular settings of training 12V models, respectively. Firstly, we propose SNR-Aligned Fine-tuning (SAF), making the first attempt to fine-tune diffusion model to a bridge model and, therefore, allowing iilfusion-bbsed text-to-video (T2V) models. Sec- ondly, we propose neural prior, further improv- ing the synthesis quality of FrameBridge when training from scratch. Experiments conducted on WebVid-2M and UCF-101 demonstrate the superior quality of FrameBridge in comparison with the diffusion counterpart (zero-shot FVD 95 vs. 192 on MSR-VTT and non-zero-shot FVD 122 vs. 171 on UCF-101), and the advantages of our proposed SAF and neural prior for bridge- based 12V models. The project page: https: / 1/framebridge i icml. g.t.giitubbb... io/.
utilize the pre-trained
In this work, inspired by recently proposed bridge mod- els (Chen et al.; Liu et al., 2023; Chen et al., 2023c), we present FrameBridge, a novel 12V framework to model the frame-to-frames synthesis process with data-to-data gener- ative framework instead of the noise-to-dee to-data one in diffusion models. Specifically, given the input image and video target, we first leverage variational auto-encoder (VAE) based com- pression network to transform them into continuous latent representations, and then take their latent representations as boundary distributions, Le., prior and target, to establish our data-to-data generative framework. Considering the static image has already been an informative prior for each of the consecutive frames in video target, we naturally replicate it to obtain the prior of the whole video clip, constructing the frumes-to-frames training data pairs for the prior-to- target generative framework in FrameBridge. Standing on constructed pairs, we establish bridge models (Tong et al., 2024; Zhou et al.; Chen et al., 2023c) between them to learn the I2V synthesis with Stochastie Differential Equa- tion (SDE) based generation process. In comparison with previous diffusion-based 12V methods, our FrameBridge uti- lizes given static image as the prior of video target, which is ad advantageous on preserving the appearance details of input image than conditionally generating video samples from ran- dom noise. Moreover, our
Introduction
Image-to-video (12V) generation, commonly referred as im- age animation, aims at generating consecutive video frames from a static image (Xing et al., 2024; Ni et al., 2023; Zhang et al., 2024a; Guo et al., 2024; Hu et al., 2022), i.e., a frrame-
Eqqual contribution 'Dept. of Comp. Sei. and Tech., Insti- tute for AI, BNRist Center, THBI Lab, Bsinghua-Bosch Joint ML Center, Tsinghua University Shu, Beijing, China. Corre- spondence to: Jianfei Chen <jianfeic@ caannfii@@sinnguu..dd... tsinghua.cdu.cn>.
Proceedings of the nnnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
bridge model learns image animation in model training rather than learn

---

FramcBridge: Improving Image-to- Vidco Generation with Bridge Models
Image-to- :inge-to-Video:
frame-to-framee generation
"lawecoe buoe
Bridge process: data-to-data generation
Deplicate
Diffusion process:
2
noise-to-data generation
110.1)
Gaussian Noise
Figure 1. Overview of FrameBridge and diffusion-based 12V models. The sampling process of FrameBridge (upper) starts from given static image, while diffusion models (lower) synthesize videos from uninformative Gaussian noise.
ing image-conditioned noise-to-video generation, which hances the consistency between generative framework and 12V task, i.e., data-to-datt for frame-to-fin to-fmmess and tends to benefit temporal coherence for 12V synthesis.
neural prior reduces the distance between prior and video target to greater extent, and alleviates the burden of gen- eration process further. Although more advanced methods can be leveraged to extract more informative neural prior, we empirically find that a coarse estimation for video target at the cost of single sampling step has already been benefi- cial FrameBridge. This further verifies our motivation present FrameBridge and shows novel method to enhance bridge-based I2V models. In this work, our contributions can be summarized as follows:
In practice, 12V systems usually take advantage pre- trained iiffusion-based text-to-video (T2V) model (Xing al., 2024; Chen al., 2023b; Ma al., 2024a) with fine-tuning process, to reduce the requirements of image- video data pairs and the computational resources the training stage of I2V generation. Toward efficiently utiliz- ing previously pre-trained diffusion- -asied T2V models, we propose SNR-Aligned Fine-tuning (SAF), a novel technique for fine-tuning them to bridge-based 12V models. Specifi- cally, we first reparameterize the bridge process in Frame- Bridge, enabling the noisy intermediate representations of our frames-to-framee to-frames process to be aligned with the ones in the noise-to-frie toffmams process of pre-trained diffusion mod- els, improving finettuning efficiency. Then, we change the timestep to match the signal-to to-iolsto (SNR) ratio between the input of the bridge model and the pre-trained diffusion model, remaining the differences between the diffusion and bridge process. Our SAF aligns the noisy intermediate rep- resentations of two generative frameworks while preserving the difference between them (i.e,, diffusion and bridge pro- cess), and therefore improves the final synthesis quality of FrameBridge when adapting pre-trained T2V diffusion models.
We propose FrameBridge, making the first attempt to model the frame-to-framen generation task of I2V with a data-to-dal generative framework.
We present two novel techniques, SAF and neural prior, further improving the performance of FrameBridge when fine-tuning from pre-trained T2V diffusion mod- els and training from scratch respectively.
We conduct experiments on two I2V benchmarks by training FrameBridge on Vid-iM (Bain et al., 2021) and UCF-101 (Soomro, 2012). Compared with its diffusion counterpart, FrameBridge fine-tuned with SAF reduces the zero-shot FVD (Unterthiner et al. (2018); lower is better) from 192 to 95 on MSR-VTT (Xu et al., 2016), and FrameBridge with neural prior trained from scratch reduces the non-zero- from 171 to 122 on UCF-101, highlighting the supe- riority of FrameBridge to their diffusion counterparts and the effectiveness of SAF and neural prior.
FVD
Compared diffusion models using Gaussian prior, Frame- Bridge takes the given static image as the prior of video target to improve 12V performance. Toward further im- proving bridge-based 12V synthesis quality, we present a stronger prior for FrameBridge. Given a static image, we design one-step mapping-baid with the video target, extracting a neural prior from the image for the video target. Compared to input image, this
2 Related Works
network and optimize it
Diffusion-based I2V Generation Diffusion models have recently achieved remarkable progress in I2V synthe- sis (Blattmann al., 2023; Chen et al., 2023a; Li al.,
2

---

FramcBridge: Improving Image-to-
Inagc-to-Video Vidco Generation with Bridge Models
12V/Image Animation
FrameBridge
Diuer (wes mr))
Diu (Mn, kr))
Di (zg, mr))
Static Image
Pr.tridge (Ra (oo; xy)
- MPAanndgee (Zo, zr), of "Arridge /3
Synchesizod Video
Diffusion
((ar(zo))
((He(zo))
((Ho(z)))
((Hessss(Ea) dasc!!
Figure 2. Visualization for the value of marginal distributions. We visualize the decoded mean value of bridge process and diffusion process. The prior and target of FrameBridge are naturally suitable for 12V synthesis.
2024) and proposed multi-stage generation system (Jain et al., 2024; Zhang et al., 2023; Shi et al., 2024; Zhang et al., 2025f), fusion module ( (Wang et al., 2024; Ren et al.) and improved network architectures (Wang et al., 2024; Xing al., 2024; Ma al., 2024a; Chen et al., 2023b; Ren al.; Zhang et al., 2025a;b;c;d). However, their noise-to-data generation process may be inefficient for I2V synthesis. To improve the uninformative prior of diffusion models (Fis- cher et al., 2023; Albergo et al., 2024; Yang et al., 2024a), PYoCo (Ge et al., 2023) proposes to use correlated noise for each frame in both training and inference. Consist12V (Ren al.), Freelnit (Wu et al., 2024), and CIL (Zhao et al.) present training-free ffee strategies to better align the training and sampling distribution of diffusion prior. These strate- gies improve the noise distribution to enhance the quality of synthesized videos, while they still suffer the restriction of noise-to-data diffusion framework, which may limit their en- deavor to utilize the entire information (e.g., both large-scale features and fine-grate -reiirdi details) contained in the given im- age. In this work, we propose data-to-datt framework and utilize clean and deterministic prior rather than Gaus- sian noise, allowing us to leverage the given image as prior information.
bridge models to 12V synthesis and further propose two improving techniques for bridge models, enabling efficient fine-tuning from diffusion models and stronger prior for video target.
3 Motivation
Diffusion-based 12V Synthesis 12V synthesis aims at generating video clip v E R x3 with L frames conditioning on a static image, egg., the initial frame of E " video clip v. In diffusion- based I2V sys- tems (Xing et al., 2024; Blattmann et al., 2023), an VAE- based compression network usually leveraged to first transform the video into latent pramframme manner with pre-trained image encoder E(v), p
Rxxxxuxd in
where h = H w W p P > 1 and d are the spatial com- pression ratio and the number of output channels. A for- ward diffusion process gradually converts the video latent po po 2², c)) 2 Pdatt (2ol=*, c) to a known prior distribution Pr (zr) - Pprior dif (ZT) with a forward SDE (Song et al.):
dz, = f()) g())dw, zo ~ Pdata (zolz',c), (I)
Bridge Models Recently, bridge models (Chen et al.; Tong et al., 2024; Liu et al., 2023; Zhou et al.; Chen al., 2023c; Zheng et al., 2024; He al.; De Bortoli al., 2021: Peluchetti, 2023), which overcome the restriction of Gaus- sian prior in diffusion models, have gained increasing at- tention. They have demonstrated the advantages of data- to-data generation process over the noise-to-data one on amage-to-image translation (Liu et al., 2023; Zhou et al.) and speech synthesis (Chen et al., 2023c; Li et al., 2025) tasks. In this work, we make the first attempt to extend
where w is a Wiener process, f and g are known coeffi- cients, 2 E gxxwxd is the compressed latent of the initial
frame v², and denotes other guidance such as the text prompt (Ma et al., 2024a; Chen et al., 2023b) or the class condition (Ni et al., 2023; Zhang et al., 2024b). In sam- pling, we first synthesize the latent z ~ (o with the backward SDE which shares the same marginal distribu- tion Pt.di (z. 2 e) (Song et al.):
3

---

FramcBridge: Improving Image-to-Videe Generation with Bridge Models
temporal axis obtain z' R the prior of video latent z E Rxxxxxxxdd and establish the bridge pro- cess follows.
dz;- [f(t) g(t) Pr,di (zuke', cc] dt +g(() zr ~ Pf (ZT), (ZT))
(2)
Bridge Process In Figure 1, we present the overview of FrameBridge and compare it with diffusion- bbased 12V gen- cration. Different from biffue 12V models using uninformative Gaussian prior, our FrameBridge replaces the Gaussian prior with Dirac prior 6g., building a bridge process (Zhou et al.) connect the video target and the replicated image prior Pprior (2T|2* c) _ 6g (ZT). Specifically, the forward process is changed from Equa- tion (1) in diffusion models to:
from Gaussian prior Pprior,dif/ (Zz) (0. and then decode the video clip with pre-trained VAE decoder (z)). To estimate the score function Vz, log Pr,dif (2, |2", c). a U- Net (Ronneberger et al., 2015; Ho et al., 2020) or DiT (Pee- bles& & Xie, 2023; Bao et al., 2023) based neural network is optimized with a denoising objective:
[())= e( .0),0,z.
s(0) (0(2),))',,))
01:20
(3)
dz: = [f((tzz + ((t)) t, 2r, 20)c dt
Here A(t) is a time-dependent weight function, and (z(zt)) t) an alternative parameterization method of the score func- tion (Ho et al., 2020).
gt)ddw zo Pdata (addazzol2 2 ,),, 2ς =z,,
(4)
where h(ze, 2,,,,,,,,,2, 2t, A Vz, log (2)) and PT PT, (ZT|2:) is the marginal distribution of diffusion process shown in Equation (I). For bridge process, we denote the marginal distribution of Equation (4) as Pt, Pt.bridge (2el2" c)). Similar the forward SDE Equation (1) in diffusion process, the forward process of bridge mod- els Equation (4) also has reverse process, which shares the same marginal distribution pr,bridge (Ze |2|2 c) and can be represented by the backward SDE:
Limitations As shown, the forward process of diffusion models gradually injects noise into data samples, which results in a boundary distribution at & = T sharing the same distribution with the injected noise, e.g., the standard Gaussian noise e ~ (O, J). Therefore, in generation, their sampling process has to start from the uninformative prior distribution Pp (ZT) ~ (O, and then iteratively synthesize the video latent Zo with learned conditional score function Vz, log Pe |2|22,100 c).
dz; = [f(1)) -
27. - -h(z,, t, 2 c) + g(t)duo
However, for 12V generation, the two key requirements are preserving the appearance details of the given static image (Ren al.; Ma et al., 2024a) and ensuring temporal coherence between generated video frames (Guo et al., 2024; Zhang et al., 2024c). The noise prior of diffusion models and the mismatch between noise-to-data generation and frame- to-frames synthesis inevitably increase the burden of the generation process when meeting these two requirements. In this work, we propose FrameBridge. By modeling 12V with data-to-data process, we simultaneously improve the prior of generation process for preserving the appearance details and enhance the consistency between the generative framework and I2V task for ensuring temporal coherence, leading to improved V2V performance.
(5)
zt = z',
where s(z,, zr, 2², c) e Ve, log Pt.brige (z, lT, 2², c².
The change from the diffusion to the bridge process re- moves the restriction of noisy prior, allowing the genera- tion process to start from a static image rather than previ- ous Gaussian noise. Moreover, as the perturbation kemel Pt, Pt.bridge (ze (u Zy; 2f, .) in bridge process remains Gaus- sian (Appendix A), it facilitates us to find connections be- tween the marginal distribution, i.e., the intermediate repre- sentations of diffusion and bridge process, and then leverage the power of pre-trained diffusion models for bridge models.
4 FrameBridge
Training Objective Analogous to diffusion models, we use a SDE solver to solve Equation (5) when sampling videos. Since hr t, 2r, 2", c) can be calculated ana- lytically (see Appendix A), we only need to estimate the unknown term s(z,, f, zr, 2" c) with neural networks (Kingma et al., 2021). After parameterization Appendix A, we train our models eă (z,, t, zr, 22,,", c)) with the denoising objective (Chen et al., 2023c):
4.1 Bridge-based I2V Synthesis
Considering the given image, i.e., initial frame 24, has pro- vided the appearance details and the starting point of ani- mation for video target, we take it as the prior of following frames. To construct the boundary distributions for bridge models, we replicate the image latent 2 for I. times along
shown in
4

---

FramcBridge: Improving Image-to- Vidco Generation with Bridge Models
"HIme-lapse blooming with
Pre-trained Diffusion
Bridge Fine-tuning
t
Not Aligned
w/ SAF
Z
Ze
Bridge with SAF
w/o SAF
Aligned Marginal
zi
Z
(a) SAF technique
(b) Effectiveness of SAF
Figure SNR-Aligned Fine-tuning for FrameBridge. (a) SAF technique aligns the noisy latents bridge process and diffusion process with respective timesteps, enabling efficient fine-tuning from diffusion-based T2V model to bridge-based I2V models. (b) FrameBridge with SAF can better leverage capability pre-trained models.
process.
Coridge (f) -I = pan- (Bo.s' =)),
In the meanwhile, as seen from our intermediate representa- tions, the data information, e.g., appearance details, is well preserved during the bridge process. In comparison, the prior and intermediate representations of diffusion process contain rare or coarse information of the target, which is uninformative and requires diffusion models to generate entire video information from scratch.
22,, 24, 2t 0x
The training of FrameBridge resembles that of Gaussian diffusion-based I2V models: We first sample video latent An and the condition from training set, extracting the first frame of Zu to construct z². The primary difference lies in the Gaussian perturbation kernel pt,bridge [Ze] 20, 2r, =", of Equation (6). As we replace the Gaussian prior with a deterministic representation Zr. the mean value is an inter- polation between data and žr instead of the decaying data in diffusion models, naturally preserving more data infor- mation and facilitating generative models to learn image animation rather than regenerating the information provided in static image.
4.2 Efficient Fine-tuning
A common practice of training 12V models is to fine- tune from pre-trained T2V diffusion models (Chen et al., 2023b;a; Xing et al., 2024; Blattmann et al., 2023; Ma et al., 2024a). The essential difference between the dif- fusion and bridge process lies in the distribution of noisy latents Zt Pt.diff (Z:) and Zz.biddge Pt.bridge (zz). For certain / [0, T), the pre-trained diffusion models only have the capability to denoise Z, Zt.dif/ while our fine-tuning target is to denoise brideeee and the substantial discrepancy between noisy latents makes it difficult to utilizing knowl- edge of pre-trained models. To address this issue, we believe that aligning the latents will allow us fully leverage the de- noising capability and learned representations of tre-trained models, which is critical to more efficient and effective fine- tuning process (Yu et al., 2024a). Thus, we propose the innovative SNR-Alig Fine-tuning (SAF) technique to align the latent Zt, Bi,bridee with a diffusion noisy latent Zi Note that we use different timestep ī o and we
Bridge Process vs Diffusion Process demonstrate the advantages of bridge process in I2V synthesis, we visualize the data part, i.e., the mean function of bridge and diffu- sion process, in Figure 2. As shown, when replicating the initial frame, I2V synthesis can be formulated as frames- to-frames generation task. With the data-to-data bridge pro- cess, the boundary distributions of our FrameBridge have been an ideal fit for the I2V task, which is helpful for gen- erative models to focus on modeling the image animation
5

---

FramcBridge: Improving Image-to- Emage-to-Video Generation with Bridge Models
Input
Prompt
Static Image
vilusi dress, wnnoote entrokeered gloves, leathered Tet curred haidd, pala stor"
Neural Prior
Dynanicrottar
Reference
BEINE
Figure 4. Case of neural prior: Our neural prior provides more motion information than the given static image, and intuitively closer to the video target, further improving the prior of generation process.
Generated Frames
Figure 5. Qualitative comparisons between FrameBridge and other baselines. FrameBridge outperforms diffusion baseline methods appearance consisteney and video quality.
only change the noisy latent as tuputtrrpparameteriiaaion of bridge models. The forward and backward process is still a bridge process and different from diffusion 12V models.
!t c bridge (Ze, (, A di at (2,, c), and initialize l allignedd with the pre-trained T2V diffusion models. SAF enables bridge models to fully exploit the denoising capa- bility of pre-trained diffusion models as the marginal dis- tribution of ž aligned with azzo + oze. We provide more details in Appendix A.
Reparameterization of Bridge Process. In bridge pro- cess, the perturbed latent 2 at timestep t can be written as the linear combination of 2o. 2r and Gaussian noise e: 2: = Gg + b, + c&E (detailed expression of ae, be, Ct can be found in Equation (12)), which takes a different form from 0x + trE in diffusion models. Therefore, the pre-trained diffusion models have limited ability to directly denoise such a 2c, which impairs effective fine-tuning. To match the distributions of Z₁, reparameterize the bridge process by
4.3 Improved Prior
By establishing a data-to-data process for I2V synthesis, have been able reduce the distance between the prior and the target from noise-to-frames to frames-to-frames, and therefore facilitate the generation process and aim at improving the synthesis quality. To further demonstrate the function of improving prior information for I2V synthesis, we extend our design of FrameBridge from replicated initial frame z' to neural representations F,, (2, c). which serves a stronger prior for video frames.
2e biZT
at
Ct of
2:
Z0 o7
ag + +
+ c7
Then, 2, can be represented as the combination of clean data Zo and Gaussian noise, with the squre sum of coefficients equal to 1. Thus, the reparameterized bridge process Zt exactly aligns with a VP diffusion process.
As shown in Figure 4, although the static frame has provided indicative information such as the appearance details of the background and different objects, it may not be informative for the motion information in consecutive frames. When the distance between the prior frame and the target frame large, bridge models are faced with the challenge to generate the motion trajectory. Therefore, we present stronger prior than simply duplicating the initial frame, neural prior, which achieves a coarse estimation of the target at first, and then bridge models generate the high-quality target from this coarse estimation.
based Latent Alignment Although the marginal distribution of 2, resembles that of diffusion process, there is still a mismatch between the input of bridge models and pre-trained diffusion models (i.e., t) and (a.zo + o,c,f)), as it is not guaranteed that ar Vatto? a to a: (see Figure 3). To handle that, we change the timestep t to another i such that 0g- Vas+o? oi
and then 2, has the same SNR as açZo + oge
Vatto?
in diffusion process. According to the above deriva- tion, we reparameterize the input of bridge models as
Considering bridge models synthesize target data with itera-
6

---

FramcBridge: Improving Image-to-Videe Generation with Bridge Models
Table Zero-shot 12V generation UCF-101 and MSR-VTT (256 256, 16 frames). w/o SAF means FrameBridge without SAF
techniques when tuning For each metric, we mark the best one with i and the second one with f. Iterated videos is the number of videos iterated during training of model (batch size x iterations). results reported in Xing (2024). reprodueed with the open-sourced training code
UCF-101
MSR-VTT
Method
Iterated Videos
FVD
IS f
PIC
FVD 4
CLIPSIM 1
PIC
SVD (Blattmann et al., 2023)
236
114
SEINE (Chen et al., 2023b)
461
22.32
0.6665
245
0.2250
0.6848
Consist2V (Ren et al.)
32.64M
202
39.76
0.7638'
106
0.2249
0.7551
SparseCtrl (Guo et al., 2025)
722
19.45
0.4818
311
0.2245
0.4382
12VGen-XI. (Zhang et al., 2023)
571
0.5313
289
0,5352
DynamiCrafter" (Xing et al., 2024)
1.28M
485
29.46
0.6266
192
0.2245
0.6131
DynamiCrafter'
6.4M
429
0.6078
234
0,5803
(w/o eeee SAF)
VideoCrafter
1.28M
433
38.61
0.5989
229
0.2246
0.5559
(w/ Tranerrrge-VideoCrrffer SAF)
VideoCrafter
1.28M
312
39.89
0.6697
99
0.2250
0.6963
FrameBridge- VideoCrafter TraneerideeVideeCrrftee (w/ SAF)
6.4M
258
44.13!
0.7274
95'
0.2250
0.7142
FrameBridge-Cope Voggod (w/ SAF)
6.4M
235
39.83
0.75631
96*
0.2250
0.7566
Table VBench-I2V (Huang et al., 2024a:b) scores for different models. For each metric, we mark the best one with and the second one with (higher score means better performance). The abbreviations represents Camera Motion (CM), 12V-Subject Consistency (I2V-SC), 12V-Background Consistency (I2V-BC), Subject Consistency (SC), Background Consistency (BC), Motion Smoothness (MS), Dynamic Degree (DD), Aesthetic Quality (AQ), Imaging Quality (IQ). Total Score: weighted average of all dimensions which evaluates the overall quality. Scores are calculated with the official code VBench. results reported in Huang al. (2024b).
Total Score
Detailed Qulity Dimensions
Model
CM
I2V-SC
I2V-BC
SC
BC
MS
DD
AQ
IQ
DynamiCrafter:256
84.35
22.18
95.40
96.22
94.60
98.30
97.82
38.69
59.40
62.29
SEINE-256 x 256
82.12
15.91
93.45
94.21
93.94
97.01
96.20
24.55
56.55
70.52
SEINE-512 x 320'
83.49
23.36
94.85
94.02
94.20
97.26
96.68
34.31
58.42
70.97'
SparseCtrl
80.34
25.82
88.39
92.46
85.08
93.81
94.25
81.95t
49.88
69.35
Consist12V*
83.30
33.60*
94.69
94.57
95.27'
98.28
97.38
18.62
59.00
66.92
FrameBridgge CraneBridge-iideoCrateee
VideoCrafter
85.37*
30.72
96.241
97.251
94.63*
98.92
98.51
35.77
59.38*
63.28
FrameBridge- VideoX
85,931
92.06
95.42
97.13*
93.60
98.62
97.57
48.291
54.28
60.00
tive sampling steps, we develop a one-step mapping-buit based prior network taking both image latent z and text or label condition as input, and separately train the prior network with regression loss in latent space:
to the target than the provided initial frame, and bridge model synthesize the video target with a coarse-to-fin iter- ative sampling process. Although more advanced methods can be designed further improve neural prior, present design with simple training objective and one-step pling, demonstrating the performance of enhancing prior information on 12V synthesis.
(n) (e,,, (e [[( - (8)
=||]]_
With this objective, it can be proved that (2 .c learns to predict the mean value of subsequent frames, as shown in Appendix A. Given pre-trained F,(2',)) we build FrameBridge-ND from its output and target video latent by replacing the prior in Equation (6) with the neural prior (2 c). More details of the training and sampling algorithm can be found in Appendix B.
5 Experiments
We carry out experiments on UCF-101 (Soomro, 2012) and WebVid-2M (Bain et al., 2021) datasets to demonstrate the advantages of our data-to-data generation framework for 12V tasks. More details can be found in Appendix D.
In generation, neural prior model (2, provide coarse estimation with a single deterministic step, which closer

---

FramcBridge: Improving Image-to- Image-to-Video Generation with Bridge Models
Table Non-zero-shot 12V generation on UCF-101. The best and second results are marked with i and f.
Table 5. Ablation of neural prior. Condition means whether the
model conditions on F, (2 ())....
Method
FVD 4
IS f
1
Method
Prior
Condition
FVD 4
ExtDM
649
21.37
VDT-I2V
Gaussian
171
VDT-12V
171
62.61
0.7401
VDT-I2V
Gaussian
132
FrameBridge
154
64.01
0.7443*
FrameBridge
replicated
X
154
Fride-r
122
63.60*
0.7662
FrameBridge
replicated
129
FrameBridge-Ve
neural
122
Table 4. Ablation of SAF technique UCF-101 (non-zero-shot).
Method
Iterations
FVD 4
IS 1
PIC f
5.2 Neural Prior for Bridge Models
Diffusion
10k
176
53.60
0.7011
We train class-conditional FrameBridge model with neural prior (FrameBrridge-NP) on UCF-101 based the model of
Bridge (w/o SAF)
10k
176
53.93
0.7371
Bridge (w/o SAF)
5k
284
49.40
0.6557
Bridge (w/ SAF)
5k
141
55.98
0.8200
Latte-S/2 (Ma et al., 2024b) by replacing diffusion process with the Bridge-gmax bridge process (Chen et al., 2023c).
Comparison with Baselines We reproduce two diffusion models ExIDM (Zhang et al., 2024b) and VDT (Lu et al.) on UCF-101 dataset for the class-conditional 12V task as our baselines. Table 3 shows that FrameBridge-Ny has superior video quality and consistency with condition images. Here VDT-I2V direct counterpart of FrameBridge models as they share the same network architecture and training configurations. More qualitative results are shown in Ap- pendix F. The experiments reveal that bridge-base based I2V models outperform their diffusion counterparts with both replicated prior and neural prior, justifying the usage of the data-to tata-to-data generation process for I2V tasks. Additionally, FrameBridge can further benefit from neural prior (2,(2,, c) as it actually narrows the gap between the prior and data distribution of bridge process.
5.1 Fine-tuning from pre-trained diffusion models
Following Xing et al. (2024), we fine-tune text-conditional FrameBridge model with replicated prior z' from the open- sourced T2V diffusion model VideoCrafterl (Chen et al., 2023a) and CogVideos VideoX-2B (Yang et al., 2024b) on WebVid- 2M dataset.
Comparison with Baselines We choose DynamiCrafter (Xing al., 2024), SEINE (Chen et al., 2023b), 12VGen-XL (Zhang et al., 2023), SVD (Blattmann et al., 2023), Con- sistI2V (Ren et al.) and SparseCtrl (Guo et al., 2025) as text-conditional 12V baselines. Table shows zero-shot metrics on UCF-101 and MSR-VTT after fine-tuning on WebVid-2N9 MM. Note that DynamiCrafter trained with 6.4M videos a direct counterpart of FrameBridge- VideoCrafter, which uses the same model architecture, base T2V diffusion model and training budget, which shows that powerful I2V models can achieve better generation performance by re- placing diffusion process with a data-to- tott-tt--ata bridge process. We also evaluate FrameBridge and other baselines with a comprehensive benchemark for video quality, i.e., VBench- 12V (Huang et al., 2024a:b) (see Table 2, all the scores are calculated with the official " FrameBridge can effectively leverage the knowledge from pre-trained T2V diffusion models and generate videos with higher quality and consistency than the diffusion counterparts. We further discuss the trade-off between the dynamic degree and con- sistency in Appendix C. Qualitative results are shown in Figure 5. In the Figure, both FrameBridge and Dynami- Crafter model are fine-tuned from VideoCrafterl with 20k steps. To the best of our knowledge, our trial is the first time to fine-tune bridge models from pre-trained diffusion models.
5.3 Ablation Studies
SNR-Aligned Fine-tuning When fine-tuned with SAF, FrameBridge leverage the pre-trained T2V diffusion models efficiently and effectively. To ablate on the SAF technique, we fine-tune a pre-trained class-conditional video generation model Latte-XI/2 on UCF-101. Table 4 shows that SAF improves fine-tuning performance of Frame- Bridge. To conduct an ablation under the WebVid-2M train- ing setting, we also fine-tune FrameBridge models from VideoCrafterl and VigeoXX2222 with the same configu- ration except the usage of SAF technique, and compare the zero-shot metrics in Appendix C.5.
Neural Prior To showcase the effectiveness of neural prior, we compare five different models varying in priors and network conditions. More details of the configurations can be found in Appendix D. Results in Table reveal that Fy, c) is indeed more informative than a single frame z and can be fully utilized by FrameBridge through the
8

---

FramcBridge: Improving Image-to-Videe Generation with Bridge Models
change of prior.
In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 22669-22679, 2023,
Conclusions
Bao, F., Xiang, C., Yue, G., He. G., Zhu, H., Zheng, K., Zhao, M., Liu, S., Wang, Y., and Zhu, Vidu: highly consistent, dynamic and skilled text-to-video generator with diffusion models. arXiv preprini arXiv:2405.0422 2024.
In this work, we propose FrameBridge, building a data-to- data generation process, which matches the frame-to-frane nature of this task, and therefore further improving the 12V synthesis quality of strong diffusion baselines. Additionally, targeting at two typical scenarios of training I2V models, namely tine-tuning from pre- tre-trained diffusion models and training from scratch, we present SNR-Aligned Fine-tuning (SAF) and neural prior respectively to further improve the generation quality of FrameBridge. Extensive experiments show that FrameBridge generate videos with enhanced ap- pearance consistency with image condition and improved temporal coherence, demonstrating the advantages of Frame- Bridge and the effectiveness of two proposed techniques.
Blattmann, A., Dockhorn, T., Kulal, S., Mendelevitch, D., Kilian, M., Lorenz, D., Levi, Y., English, Z., Voleti, V., Letts, A., et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. CoRR, 2023.
Brooks, T., Peebles, B., Holmes, C., DePue, W., Guo, Y., Jing. L.., Schnurr, D., Taylor, J., Luhman, T., Luhman, E., Ng. C., Wang, R., and Ramesh, A. Video generation models world simulators. 2024.
Chen, H., Xia, M., He, Y., Zhang, Y., Cun, X., Yang, S., Xing, J., Liu, Y., Chen, Q.. Wang, X., et al. Videocrafterl: Open diffusion models for high-quality video generation. arXiv preprint arXiv:2310. 19512, 2023a.
Acknowledgement
The authors sincerely thank Min Zhao, Chendong Xiang, Kaiwen Zheng for the insightful discussions and sugges- tions. This work was supported by the NSFC Projects (Nos. 62376131, 92270001, U24A20342). J.Z. is also supported by the XPlorer Prize.
Chen, T., Liu, G.-H., and Theodorou, E. Likelihood train- ing of schrüdinger bridge using forward-backwar sdes theory. In International Conference on Learning Repre- sentations.
Impact Statement
Chen, X., Wang, Y., Zhang, L.., Zhuang, S., Ma, X., Yu, J., Wang, Y., Lin, D., Qiao, Y., and Liu, Seine: Short-to- long video diffusion model for generative transition and prediction. In The Twelfth International Conference on Learning Representations, 2023b.
Our method FrameBridge can improve the quality of image- to-video generation, and the proposed techniques, i.e. and neural prior, can furthur enhance FrameBridge. How- ever, our method is broadly applicable, and FrameBridge 12V models can be fine-tuned from various T2V diffusion models, which potentially leads to the misuse of I2V gener- ation models.
Chen, Z., He, G., Zheng, K., Tan, X., and Zhu, J. Schrodinger bridges beat diffusion models on text-to- speech synthesis. arXiv preprint trXiv:2312.03444 2023c.
References
De Bortoli, V., Thornton, J., Heng, J., and Doucet, A. Diffu- sion schrödinger bridge with applications to score-based generative modeling. Advances Neural Information Processing Systems, 34: 17695-17709, 2021.
Albergo, M. S., Boffi, N. M., and Vanden-Eijnden, E. Stochastic interpolants: A unifying framework for flows and diffusions. arXiv preprint 27Xiv22003.08797, 2023.
Albergo, M. S., Goldstein, M., Boffi, N. M., Ranganath, R., and Vanden- Eijnden, E. Stochastic interpolants with data- data-dependent couplings. In International Conference on Machine Learning, pp. 921-937. PMLR, 2024.
Fischer, J. S., Gui, M., Ma, P., Stracke, N., Baumann, S. A., and Ommer, B. Boosting latent diffiusion with flow match- ing. arXiv preprint arXiv 2023.
Fu, S., Tamir, N., Sundaram, S., Chai, L., Zhang, R., Dekel, T., and Isola, Dreamsim: Learning new dimensions of human visual similarity using synthetic data. Advances Neural Information Processing Systems, 36:50742- 50768, 2023.
Bain, M., Nagrani, A., Varol, G., and Zisserman, A. Frozen in time: A joint video and image encoder for end-to-end retrieval. Proceedings of the IEEE/CVF international conference on computer vision, pp. 1728-1738, 2021.
Ge, S., Nah, S., Liu, G., Poon, T., Tao, A., Catanzaro, B., Ja- cobs, D.., Huang, J.-B., Liu, M. Y., and Balaji, Y. Preserve your own correlation: A noise prior for video diffusion
Bao, F., Nie, S., Xue, K., Cao, Y., Li, C., Su, H., and Zhu, All are worth words: vit backbone for diffusion models.

---

FrameBridge: Improving Image-to-Videe Generation with Bridge Models
models. In Proceedings of the IEEE/CVE International Conference Computer Vision, pp. 22930-229444 2023.
Huang, Z., Zhang, F., Xu, X., He, Y., Yu, J., Dong, Z., Ma, Q. Chanpaisit, N., Si, C., Jiang, Y., et al. Vbench++: Comprehensive and versatile benchmark suite for video generative models. arXiv preprint arXiv:2411.13503, 2024b.
Ge, S., Mahapatra, A., Parmar, G., Zhu, J.-Y., and Huang, J.-B. On the content bias in fréchet video distance. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 7277-7288, 2024.
Jain, S., Watson, D., Tabellion, E., Poole, B., Kontkanen, J., et al. Video interpolation with diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7341-7351, 2024.
Guo, Y., Yang. C., Rao, A., Liang, Z., Wang, Y., Qiao, Y., Agrawala, M., Lin, D., and Dai, B. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. In ICLR, 2024.
Kingma, D., Salimans, T., Poole, B., and Ho, J. Varia- tional diffusion models. Advances in neural information processing systems, 34:21696-21707 2021.
Guo, Y., Yang, C., Rao, A., Agrawala, M., Lin, D., and Dai, B. Sparsectrl: Adding sparse controls to text-to-video diffusion models. European Conference on Computer Vision, pp. 330-348. Springer, 2025.
Li, C., Chen, Z., Bao, F., and Zhu, J. Bridge-sr: Schrödinger bridge for efficient sr. ICASSP. 2025.
He, G., Zheng, K., Chen, J., Bao, F., and Zhu, J. Consistency diffusion bridge models. In The eight Annual Conference on Neural Information Processing Systems.
Li, Z., Tucker, R., Snavely, N., and Holynski, A. Genera- tive image dynamics. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 44-1245,,, 2024.
He, Y.. Yang. I., Zhang, Y., Shan, Y., and Chen, Q. La- tent video diffusion models for high-fidelity long video generation. arXiv preprint
Lin, S., Liu, B., Li, J., and Yang, X. Common diffusion noise schedules and sample steps are flawed. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp. 5404-5411, 2024.
2022.
Ho, J., Jain, A., and Abbeel, P. Denoising diffusion proba- bilistic models. Advances in neural information process- ing systems, 33:6840-6851, 2020.
Liu, G.-H,, Vahdat, A., Huang, D.-A., Theodorou, E. A., Nie, W., and Anandkumar, A. 1 sb: Image-to-image schrödinger bridge. In International Conference on Ma- chine Learning, pp. 22042-22062. PMLR, 2023.
Ho, J., Chan, W., Saharia, C., Whang, J., Gao, R., Gritsenko, A., Kingma, D. P., Poole, B., Norouzi, M., Fleet, D. J., et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arX:v:2210002303, 2022a.
Lu, H., Yang, G., Fei, N., Huo, Y., Lu, Z., Luo, P., and Ding, M. Vdt: General purpose video diffusion trans- formers via mask modeling. In The Twelfth International Conference on Learning Representations.
General-purpose
Ho, J., Salimans, T., Gritsenko, A., Chan, W., Norouzi, M., and Fleet, J. Video diffusion models. Advances in Neural Information Processing Systems, 35:8633-866 2022b.
Ma, X., Wang, Y., Jia, G., Chen, X., Li, Y.-F., Chen, C., and Qiao, Y. Cinemo: Consistent and controllable image animation with motion diffusion models. CoRR, 2024a.
Hu, L. Animate anyone: Consistent and controllable image- to-video synthesis for character animation. In Proceed- ings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 8153-8163, 2024.
Ma, X., Wang, Y., Jia, G., Chen, X., Liu, Z., Li, Y.F., Chen, C., and Qiao, Y. Latte: Latent diffusion transformer for video generation. CoRR, 2024b.
Hu, Y., Luo, C., and Chen, Z. Make it move: control- lable mmage-to-vides generation with text descriptions. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 18219-18228, 2022.
Meng, C., He, Y., Song, Y., Song, J., Wu, J., Zhu, J.-Y., and Ermon, S. Sdedit: Guided image synthesis and editing with stochastic differential equations. In International Conference on Learning Representations.
Huang, 7., He, Y., J., Zhang, F., Si, C., Jiang, Y., Zhang, Y., Wu, T., Jin, Q., Chanpaisit, N., et al. Vbench: Compre- hensive benchmark suite for video generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 21807-21818. 2024a.
Ni, H., Shi, C., Li, K., Huang, S. X., and Min, M. R. Condi- tional image-to-vider generation with latent flow diffu- sion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 18444 18455, 2023.
10

---

FrameBridge: Improving Image-to-Videe Generation with Bridge Models
Nichol, A. Q.. Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., Megrew, B., Sutskever, I., and Chen, M. Glide: Towards photorealistic image generation and edit- ing with text- diffusion models. In International Conference on Machine Learning, pp. 16784-16804. PMLR, 2022.
generation with explicit motion modeling. In ACM SIG- GRAPH 2024 Conference Papers, pp. 1-11, 2024.
Singer, U., Polyak, A., Hayes, T., Yin, X., An, J., Zhang, S., Hu, Q., Yang, H., Ashual, O., Gafni, O., al. Make-a- video: Text-to-video generation without text-video data. In The Eleventh International Conference on Learning Reprexentations.
Peebles, W. and Xie, S. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 4195-4205, 2023.
Skorokhodov, I,, Tulyakov, S., and Elhoseiny, M. Stylegan- V: A continuous video generator with the price, image quality and perks of stylegan2. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3626-3636, 2022.
Peluchetti, S. Non-denoising forward time diffusions. arXiv preprint arXiv:2312. /4589, 2023.
Podell, D., English, 7.,, Lacey, K., Blattmann, A., Dockhorn, T., Müller, J., Penna, J., and Rombach, R. Sdxl: Im- proving latent diffusion models for high-resolution image synthesis. In The Twelfth International Conference on Learning Representations.
Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Er- mon, S., and Poole, B. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp.
Soomro, K. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv: 2012.
Tong, A., Malkin, N., Fatras, K., Atanackovic, L., Zhang, Y., Huguet, G., Wolf, G., and Bengio, Y. Simulation- free schrödinger bridges via score and flow matching. In The 27th International Conference on Artificial Intelli- gence and Statistics, pp. 1279-1287. Journal of Machine Learning Research-Proceedde Track, 2024.
PMLR, 2021.
Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., and Chen, M. Hierarchical text-conditional image generation with clip latents, 2022. URL https: //apii//a/iii//iaa org/ abs/ 2204. 06125.
Ren, W., Yang, H., Zhang, G., Wei, C., Du, X., Huang, W., and Chen, W. Consisti2v: Enhancing visual consistency for image-to-video generation. Transactions Machine Learning Research.
Unterthiner, T., Van Steenkiste, S., Kurach, K., Marinier, R., Michalski, M., and Gelly, S. Towards accurate generative models of video: A new metric & challenges. arXiv preprint 1rXiv:1812.0111
2018.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent difffusion models. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pp. 10684-10695, 2022.
Vahdat, A., Kreis, K., and Kautz, J. Score-based generative modeling in latent space. Advances in neural information processing systems, 34:1 1287-13213 2021.
Wang, X., Yuan, H., Zhang, S., Chen, D., Wang, J., Zhang, Y., Shen, Y., Zhao, D., and Zhou, J. Videocomposer: Compositional video synthesis with motion controllabil- ity. Advances Neural Information Processing Systems, 36, 2024.
Ronneberger, O., Fischer, P., and Brox, T. U-net: Con- volutional networks for biomedical image segmenta- tion. In Medical image computing and computer-assisted intervention-MICCAA 2015: 18th international confer- ence, Munich, Germany, October 5-9, 2015, proceedings, part 18, pp. 234-241. Springer, 2015.
Wang. Y., Chen, X., Ma. X., Zhou, S., Huang, Z., Wang, Y., Yang, C., He, Y., Yu, J., Yang, P., et al. Lavie: High- quality video generation with cascaded latent diffusion models. International Journal of Computer Vision, 133 (5)3009333778 2025.
Saito, M., Matsumoto, E., and Saito, S. Temporal generative adversarial nets with singular value clipping. In Proceed- ings of the IEEE international conference on computer vision, pp. 2830-2833 2017.
Wu, C., Huang, L., Zhang, Bi, Yang, F., Sapiro, G., and Duan, N. Godiva: Generating open- domain videos from natural descriptions. arXiv preprint arXiv:2 /4806, 2021.
Shi, X., Huang, Z., Wang, F.-Y,, Bian, W., Li, D., Zhang, Y., Zhang, M., Cheung, K. C., See. S., Qin, H., et al. Motion-i2y: Consistent and controllable image-to-video

---

FramcBridge: Improving Image-to- Anngcoto-Vidco Generation with Bridge Models
Wu, J. Z.,, Ge, Y., Wang, X., Lei, S. W. Gu, Y., Shi, Y. Hsu, W., Shan, Y., Qie, X., and Shou, M. Z. Tune-a- video: One-shot tuning of image diffusion models for text-to-video generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7623- 7633, 2023.
Zhang, J., Xiang, C.,, Huang, H., Wei, J., Xi, H., Zhu, J., and Chen, J. Spargeattn: Accurate sparse attention accelerat- ing any model inference. In International Conference on Machine Learning (ICML), 2025d.
Zhang, J., Xu, X., Wei, J., Huang, H., Zhang, P., Xiang, C., Zhu, J., and Chen, J. Sageattention2+4: A more efficient implementation of sageattention2. arXiv preprint arXiv:2505.21330 2025e.
Wu, T., Si, C., Jiang. Y. Huang, Z., and Liu, Z. Freeinit: Bridging initialization gap in video diffusion models. In European Conference on Computer Vision, pp. 378-394. Springer, 2024.
Zhang, S., Wang, J., Zhang, Y., Zhao, K., Yuan, H., Qin, Z., Wang. X., Zhao, D., and Zhou, 12vgen-xl: High-quality image-to-video synthesis via cascaded diffusion models. arXiv preprint arXiv:231 12.3311.0444 1404 2023.
Xing, J., Xia, M., Zhang, Y., Chen, H., Yu, W., Liu, H., Liu, G., Wang, X., Shan, Y., and Wong, T.-T. Dynamicrafter: Animating open-domain images with video diffusion pri- ors. In European Conference on Computer Vision, pp- 399-417. Springer. 2024.
Zhang, Y., Xing, Z., Zeng, Y., Fang, Y., and Chen, K. Pia: Your personalized image animator via plug-and-play modules in text-to- text-to-image models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7747-7756, 2024a.
Xu, J., Mei, T., Yao, T., and Rui, Y. Msr-vtt: large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.
2016.
Zhang, Y., Wei, Y., Lin, X., Hui, Z., Ren, P., Xie, X., and Zuo, W. Videoelevator: Elevating video generation qual- ity with versatile text-to-image diffusion models. In Pro- ceedings of the AAAI Conference on Artificial Intelli- gence, volume 39, pp. 10266-10277 2025f.
Yang. Q. Chen, H., Zhang, Y., Xia, M., Cun, X., Su, Z.,, and Shan, Y. Noise calibration: Plug-and-play content- preserving video enhancement using pre-trained video diffusion models. European Conference on Computer Vision, pp. 307-326. Springer, 2024a.
Zhang, Z., Hu, J., Cheng, W., Paudel, D., and Yang. J. Extdm: Distribution extrapolation diffusion model for video prediction. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 19300, 2024b.
Yang, Z. Teng. J., Zheng, W., Ding, M., Huang, S., Xu, J., Yang, Y., Hong, W., Zhang, X., Feng, G., et al. Cogvideox: Text-to-video diffusion models with an pert transformer. CoRR, 2024b.
Yu, S., Kwak, S., Jang, H., Jeong, J.. Huang, J., Shin, J., and Xie, S. Representation alignment for generation: Training diffusion transformers easier than you think. arXiv preprint arXiv:2410.066: (06400, 2024a.
Zhang, Z., Long, F., Pan, Y., Qiu, Z., Yao, T., Cao, Y., and Mci, T. Trip: Temporal residual learning with image noise prior for image-to-video diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8671-8681, 2024c.
Yu, S., Nie, W., Huang, D. A., Li, B., Shin, J., and Anand- kumar, A. Efficient video diffusion models via content- frame motion-latent decomposition. In 12th International Conference Learning Representations, ICLR 2024, 2024b.
Zhao, M., Zhu, H., Xiang, C., Zheng, K., Li, C., and Zhu, J. Identifying and solving conditional image leakage in image-to-video diffusion model. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.
Zhang. J.. Huang, H., Zhang, P., Wei, J., Zhu, J., and Chen, J. Sageattention2: Efficient attention with thorough outlier smoothing and per-thread int4 quantization. Interna- tional Conference on Machine Learning (ICML), 2025a.
Zheng, K., He, G., Chen, J., Bao, F., and Zhu, J. Diffusion bridge implicit models. arXiv preprint arXiv:2405. 158855 2024.
Zhang, J., Wei, J., Zhang, P., Xu, X., Huang, H., Wang, H., Jiang, K., 7hu, J., and Chen, Sageattention3: Microscal- ing fp4 attention for inference and an exploration of 8-bit training, arXiv preprint arXiv: 105555..11594.4 1594, 2025b.
Zhou, I., Lou, A., Khanna, S., and Ermon, S. Denoising diffusion bridge models. In The Twelfth International Conference on Learning Representations.
Zhang, J., Wei, J., Zhang, P., Zhu, J., and Chen, J. Sageatten- tion: Accurate 8-bit attention for plug-and-play inference acceleration. In International Conference on Learning Representations (ICLR), 2025c.
12

---

FrameBridge: Improving Image-to-Videe Vidco Generation with Bridge Models
A Proof and Derivation
A.1 Basics of Denoising Diffusion Bridge Model (DDBM)
We provide the derivations of Pt,bridge (z Zu. Zr,; 2", c) and h(z,t, t, 2 c) used in Section 4.1.
Similar to the proofs in (Zhou et al.), we calculate Pt, Pr,bridge (zi Zo, Zr. 2", c) by applying Bayes' rule:
Pt.bridge (Zi Zo, zr, 2",,2,00 Pt,difs (zi 2r, .)) Pr,dif (Zr (2, 2o, 2", dip (2 Z0, 2 2*,c)
Pt,diff (2r 10 zt,
pr.diff pr. (e (20)
1 uses the Markovian of the diffusion process 2, (Kingma et al., 2021).
The perturbation kernels pr,aif (Zrlz))))))))))) Pr.di, Pr,a (zT zo) is Gaussian and takes the form of:
(r.4:f7(27/2)) = N(zr; ạT
(ož
a
0t
(10)
Pr.dsfy(Zolzo) (Zz az 0)
pr Pr,dif (ZT (z) N(Zz ar o
Following (Zhou et al.), can be derived that Pt,bridge (2: YT; 20; z¹, e) is also Gaussian, and Pr.bridge(2, 2T, 20: 20; 2c,c) N(z (Zo, 2r), o² i.bridge I), where
SNRT
SNRT Oi SNR, OT
14 (zg: 2T) = =0:(1
'00 +
2TT
SNR,
(11)
SNRT
0f ibbridge of
SNR, ).
Specifically, 2, of bridge process can be reparameterized by 2, = 0,40 + b.2r + CFC, where
a₂ = are11 SNRT
SNR.
b, SNRT Ot SNR, ocr
(12)
SNRT
of
SNR,
Here, SNR, o.
(Kingma et al., 2021) is the signal-to-noise ratio of diffusion process.
Then we calculate h(z, 1, = Vz. logPr, (Zr (2:)), (
As (r.asf((z(zzi) = NN(zr or ziz (o off we have
at
2
aT. 2T
Pr.aos(Zz(zz)
exp
at
(13)
2(07
07 TI of
v 2T
of
2
ZT
Z
log Pr. (2r(22))
+ C,
(14)
2(0
a7
a7
13

---

FrameBridge: Improving Image-to-
Amagc-to-Vidco Vidco Generation with Bridge Models
where C' is a constant independent of Zr.
2r Z₁
2T
or Zt
Vn, v, logpT,ary(ZT(z)) Vx,
(15)
2(07
at 27 01)
(o}
of
So, hlz, t',) =
Note that for the diffusion process we commonly use, er Rs 0 and or RJ 1, and we
(o] have h(,,,, y.
A.2 Parameterization of FrameBridge
Proposition The score estimation so(z,, t, zr, 21, ,c) of bridge process pt.dridge (ze Zr, 2¹, c) can be reparamterized by
1 Eg (ze, zr, 21,22,c)
SNRT
2T
S6 80(ze, 2r, 2r,22, c))
(16)
oi
SNR, of
SNR( SNR,
where SNR, = ot and Eg (Ze, 1i,,,271 2T, ,e) trained with the objective
Cbridge (0)
ī(t)
E i (ze,t, 2T,
Zi agZ0
(17)
(Mo Por-Paaaa (2 o) =p=z Peroraag (z. wo zr 2t,
Ot
Here X(t) is the weight function of timestep and take X(t) 1 unless otherwise specified. When SNRT ze O(which is often the case for diffusion process), there exists such that
so 20, 24, c (z,, tz,,,, zr, 2), V Tc,
(18)
Proof. We denote the desnoising target 8:-0,20 by e (2,, 2o, t), and define a = ar(1
SNRT
SNR, ), b
SNRT ai SNR,
Ct.
V o0
SNRr SNR.
From Equation (11), we have
Zi
GgZp
b,z
z log Pt,bridge (z/zo, Zy 2mp
(19)
ci
which is the target of Denoising Bridge Score Matching (Zhou et al.). Our goal is to represent this target with z,, 2y, and (ze, Zo, t).
From the definition (ze, 2o, 1), we have
O3E & (Zz,
Zo,
Zg
(20)
14

---

FrameBridge: Improving Image-to-Videe Vidco Generation with Bridge Models
Plug it into Equation (19), it can be derived that
R: 3.0 2 (2.)20.1)
Vz log Pr.bridge (z) Zu, Zr 2 ar=
C2 2
Ct
0 0 + 0,0 L (Z,, 2o. 1) arbbuT
accy
0,0 2 (2,, n0, 1)
(ax+ 01)22 aebix
(21)
acci
SNRT ax SNR,
a2 SNRT zz' ap SNR,
21
(Zt, Zo,t) b)
0t
a,o7 [1
SNRT SNI,
1
$ (Ze, Zu.t)
SNRT 2,
at z
0t
SNR,
of
SNRr
SNR.
)
As the Denoising Bridge Score Matching takes the form of
Löridge (0) = Eq 21= (a(t) s0 Ze, t, 2T, 2', - Vz log Pt. bridge (z|2o: 2T
(z,,
1= )
[1]
(22)
when we parameterize s8(2e. t, ZT, c) written as
J oo Eo (ze, t, 2',,2',)
SNRT SNR,
the training objective can be SwiTT
of
A(t)
cg (z,, L WT, 20 x², - E < zo, 1) 11].
Coridge (0) = El .1,.
(23)
which proves the first part of the proposition if we take A(t)
X(0) of
For the second part, when SNRT ~ 0, there exists an 0, such that
has an upper bound M. Since
020
SNR.
of SNRT a = oT a.op SNR, or th L (Zr, L ZT, c).
= O when SNRT 2 0, it can be directly inferenced from Equation (16) that so so t, Zr, 21, 2)
Remark A.I. From the first part of the proposition, we parameterize bridge models to predict ما It is similar to that de used in Chen et al. (2023c) although their parameterization is derived from the forward- backdddd diffusion process of Schrödinger Bridge problems. The statement and proof of this proposition reveals that DDBM and Diffusion Schrödinger Bridges are closely related. Additionally, the second part shows that our parameterization resembles the Denoising Score Matching in diffusion models.
A.3 SNR-Aligned Fine-tuning
Existence and Uniqueness of i In Section 4.2, we need to find i such that og Vat+
0i
Vat+o
Cr
Since
SNR2) = SNR, - SNR7, it is a monotonically decreasing function of t. As SNR, is also a monotonically SNR.
decreasing function which ranges over (0, oc), we can take i = SNR- ( and the uniqueness of such i can also be
oe
guaranteed. Next, we provide more general form of SAF, where the schedule {ax, of beea7 of the pre-trained diffusion models and bridge models are not necessarily the same.
Proposition 2. Suppose we fine-tune a Gaussian diffusion model En,22,,,, with schedule (āe, ãz hejo, to a diffussio bridge model b.riidge (zr, L, ZT, 2², c))= A el, (Zr, I, zT, 2² =²,c) with schedule {au, of Jeeeo,, If we use the same dataset Pdata (Zo, 2¹, c) for training (2,, c) and fine-tuning e a (Ze, i, 2r, 2², c). Then, for each c, the imput (ze, () of én has the same marginal distribution the input (Zr, 8) of C align (2,, t, 2T, 2. c). Here
&
2 b,z%
2.
v Va + c
(24)
i = SNR
15

---

Anndcoto-Vidco FramcBridge: Improving Image-to- Vidco Generation with Bridge Models
&
(SNR
is the signal-to-noise ratio of pre-trained diffusion models.) oy
Proof. Since SN is also a monotonically decreasing function ranging over (0, oo), the uniqueness and existence of can also be guaranteed by the above analysis.
For fixed e, t, we denote the probability density function of ž by q(Z Then
J g(Zi|2"; t)pddtal (2 22)d2*
((
[. g(Zx) zo: 2²; t)paataa (20, 2' "dodd=
20
[ [ N(zi Vat +
at 20:
Ct
Pataaa (zo: z')doo
ai
20
(25)
[ N(zz; ā,zo, oe P'dora (zo, 2' d2zz
N(z;; ā,zo ogD Pdata (zo, =² )dzz)dd
N(2;; 0,z0, of Paaata (ao ddzo
which equals to the marginal distribution of the pre-trained diffusion process Pt,diff (ze)-
Output Parameterization Our previous descriptions show how to align the input of the network when fine-tuning from T2V diffusion models to 12V bridge models. When the output parameterization teacher diffusion models deviates significantly from the bridge parameterization e we can also reparameterize the network output to achieve better alignment. We take Cog -BdeoX222B as an example, where v-prediction is used for teacher diffusion models. The teacher diffusion models predict Og36 ugZo from (0,20 + oze, i). After the input alignment of bridge schedule, we have
at
Ct
2,
20
+
0j
+
ci
ai
+
ci
(26)
C1
90
a
+
o
+
c7
To align the network output with the teacher, we can set the target of prediction as
Vuz+ot
VaZ+o?
20.
A.4 Neural Prior with Regression Training Objective.
Proposition If we train Fz'2', c) with the regression training objective
Cp(1) = E(go.z" pe)- (Zo.s' .)) [l(((')))) - 20
(27)
and the neural network is optimized sufficiently, then we have
F,,(",,) P,, (, A pe [20]]
(28)
Proof. For each (e), (p( optimizes the following objective:
(, (ng [| -2ol))
= [F 2) (=
((z",,e .). Bez (20 () [2o]) + Be panlea [20i]]
(29)
= =1|F,(=*, (F, Peos [zol) + C.
16

---

FrameBridge: Improving Image-to-Vidde Vidco Generation with Bridge Models
Algorithm 1 Training algorithms for I2V diffusion models.
Output: Trained I2V diffusion model e E,
Set diffusion process {az, or)-=- if Fine-tuned from pre-trained diffuion model c (Zz, i, c) Initialize Eg with the weight of Ez( else
then
Randomly initialize Eg (Ze, t, c).
end if
repeat
Sample data (zo. c) ~ Pdato (Zo, c), timestep and 0g + osE, where ~ 1). Take the first frame of zo as the image condition 2².
((0) || [Z, 2,2, ,2,
ell?
Update ! with the optimizer and loss function ((0)
until Reach the training budget
Algorithm 2 Sampling algorithms for FrameBridge.
Output: Video latent Zp-
Prepare a trained FrameBridge model cg (ze. f, 27, 2", c) and timestep schedule to < ... <tN = Obtain the given input image 2 and additional conditions c. if Neural prior is used then
2r + F2 c). Here F, should the same neural prior model used the training process.) else
Construct Zr by replicating 2
end if
for k = N downto do
Calculate the score function of bridge process Vz log Pbridge, (z/zr. 2², c)la-e,, with ex (Zix tw. 27,22 Utilize SDE solver solve the backward bridge SDE dz, [f(()z- ((()) L, ET, x", h(z,, (, 21,,),,,) di+ g(t)dw from z(tz) Zea to obtain - end for Return Zu-
where C is a constant independent of When the network is optimized sufficiently, (, c) takes the minimum for each (z", c), so we have
Fy .) arg min [|x||² - (x, BB (ao (201)
(30)
It can be solved that F2 e) = Eng (ro [zo]-
B Pseudo Code for the Training and Sampling of FrameBridge
We provide the pseudo code for the training and sampling process of FrameBridge (See Algorithm and 2). Meanwhile, we also provide that of diffusion-based I2V models (See Algorithm I and 3) show the distinctions between FrameBridge and diffusion-based I2V models.
C Detailed Analysis of I2V Generation Performance
In this section, we provide further discussions and analysis of the results provided in Section
C.1 Dynamic Degree of Generated Videos
As shown by (Zhao et al.), there usually a trade-off between dynmaic motion and condition alignment for I2V models, and the high dynamic degree scores of some bascline models Table are at the cost of condition and temporal consistency. FrameBridge can reach balance demonstrated by the multi-dimensional evaluation on VBench-12y. Table shows
17

---

FramcBridge: Improving Image-to-Video Vidco Generation with Bridge Models
Algorithm 3 Sampling algorithms for 12V diftusion models.
Output: Video latent 2o-
Prepare a trained 12V diffusion model Eg E0 t, 2", c) and timestep schedule 0 to < < < IN =T Obtain the given input image 2 and additional conditions
Sample latent Zr ~ (0, o1)
for k = N downto do Calculate the score function of diffusion process Vz log Puss (z)2", c)l== with c(z tk. 2),
Utilize SDE solver solve the backward diffusion SDE dz, [f(1)z V log ( di g())))dd from zffx) Zte obtain -12-1" end for Return Zu-
Table VBench-I2V scores related the motion of videos for different 12V models. For the evaluation dimensions, higher score means better performance. For results marked by we directly use the data of VBench-12V Leaderboard.
Dynamic Degree
Temporal Flickering
Motion Smoothness
Model
FrameBridge-VieeoChhtttt
35.77
98.01
98.51
DynamiCrafter-256
38.69
97.03
97.82
SEINE-256 256
24.55
95.07
96.20
SEINE-512 320*
34.31
96.72
96.68
SEINE-511 x 512*
27.07
97.31
97.12
Consistl2V
18.62
97.56
97.38
VBench-I2V scores related dynamic degree and temporal consistency.
Mcanwhile, some techniques proposed for 12V diffusion models to improve the dynamic degree and we find they are also applicable to FrageB To be more specific, we fine-tune FrameBrigge-VdeeCCrffee VideoCrafter by adding noise to the
image condition (Blattmann et al., 2023; Zhao et al.) and use higher value of frame-stride conditioning (Xing et al., 2024) respectively, and conduct a user study to evaluate the dynamic degree and overall video quality. We randomly sample 50 prompts from VBench-122 I2V and generate one video with each prompt for each model. Participants are asked two questions for each group of videos:
Rank the videos according the dynamic degree. Higher rank (i.e. lower ranking number) corresponds to higher dynamic degree.
Rank the videos according the overall quality. Higher rank (i.e. lower ranking number) corresponds to higher quality.
We recruited 18 participants and use Average User Ranking (AUR) preference metric (lower for better performance). The results are shown in Table 7.
C.2 Content-Debiased FVD
Ge et al. (2024) points out that the FVD metric has content bias and may misjudge the qualify of videos. As supplementary, we also provide the evaluation results of the Content-Debiased FVD (CD-FVD) on MSR-VTY in Table 8.
C.3 Learning Curve of Video Quality
To illustrate the change of video quality during training, reproduce the training process of DynamiCrafter for 20k iterations and compare the zero-shot CD-FVD metric on MSR-VTT dataset with FrameBridge model trained during the training process. As we use the same training batch size and model structure for FradeBridge and DynamiCrafter in this experiment, the training budget for two models at the same training step is also the same. As demonstrated by Figure 6, the video quality of FrameBridge is superior to that DynamiCrafter during the training process and it also converges faster than its diffusion counterpart (i.e., DynamiCrafter).
18

---

FramcBridge: Improving Image-to-Videe Generation with Bridge Models
Table Results of user study. All the models fine-tuned from VideoCrafterl. For FrameBridge- Franedrree,,,,tridde, increase the value of conditioning frame stride (from 3 to 5) when sampling. For fine-tuning process.
add noise to the image condition in the
Model
AUR of dynamic degree
AUR of overall quality +
DynamiCrafter
2.85
3.04
FrameBridge
2.74
2.26
Frameeridge-FrameStrie
2.12
2.34
FrameBrigee-NoisyCoottiio
2.29
2.35
Table 8. Zero-shot CD-FVD metric MSR-VTT dataset. We also include the FVD metric reference
Model
CD-FVD 4
FVD $
DynamiCrafter
207
234
SEINE
420
245
Consist12V
192
106
SparseCtrl
454
311
TrameBridge-VideoCraaee
148
95
C.4 Sampling Efficiency of FrameBridge
Since sampling efficiency is also important for 12V models, also conduct experiments show the quality of videos sampled with different number of sampling timesteps and compare it with DynaniCrafter and SEINE. Figure C.4 shows that the quality of vidcos sampled by FrameBridge is better than that of DynamiCrafter and SEINE with different timesteps (i.e,, 250, 100, 50, 40, 20). Moreover, we also measure the actual execution time of the sampling algorithm and show the result Figure C.4. As illustrated by these two figures, FrameBridge can achieve good balance between sample efficiency and video quality, and there is no significant degradation in video quality when decreasing the sampling timestep from 250 50 or even smaller.
C.5 Sli-Ald Fine-tuning on WedVid-2M
To ablate SAF technique on We,, we fine-tune FrameBridge models from VideoCrafter1 with the same configuration except the usage of SAF for 1.6k steps. The zero-shot metries are reported in Table Similar ablation conducted with FrameBridge models fine-tuned from Cog VideoX-2B for 5k steps, and the zero-shot metrics are reported in Table 10.
D Experiment Details
We provide descriptions of the datasets and metrics used in our experiments, along with implementation details for different 12V models.
D.1 Datasets
UCF-101 an open-sourced video dataset consisting of 13320 videos clips, and each video clip are categorized into one of 101 action classes. There are three official train-test split, each of which divide the whole dataset into 9537 training video clips and 3783 test video clips. We use the whole dataset the training data for I2V models trained from scratch evaluate zero-shot metries for models fine-tuned on Vid- When we evaluate
on UCF-101, and use the test zero-shot metries on UCF-101 for text-conditional I2V models, use the class label as the input text prompt.
WebVid-2N is an open-sourced dataset consisting of about 2.5 million video-text pairs, which is a subset of Web Vid- We only use Web Vid-2M as the training data when fine-tuning 12V models from T2V diffusions in Section 5.1.
MSR-VTT is an opuncsourced dataset consisting 10000 video-text pairs, and we only use the test set to compute zero-shot metries for fine-tuned models.
19

---

FramcBridge: Improving Image-to- Vidco Generation with Bridge Models
FrameBridge DynamiCrafter
283
CD-FVD
244
211
182
157
116
100
3000
6000
9000
12000
15000
18000
21000
Training Steps
Figure
learning curve of FrameBridge and DynamiCrafter.
Table 9. Zero-shot metrics on UCF-101 and MSR-VTT for FrameBridge- Videdddreeee models.
UCF-101
MSR-VTT
Model
FVD 4
IS f
PIC
FVD 4
CLIPSIM
PIC t
FrameBridge- VideoCrafter (w/o SAF)
431
45.88
0.6765
151
0.2248
0.6493
FrameBrigg-ViddoCrafiee VideoCrafter (w/SAF)
354
46.09
0.7060
132
0.2248
0.6778
Preprocess of Training Data: For both UCF-101 and WebVid-2M dataset, we sample frames from each video clip with fixed frame stride of when training. Then we resize and center-crop the video clips 256 256 before input it to the models.
D.2 Metries
Fréchet Video Distance Unnterthinrr et (2018); FVD) evaluates the quality of synthesized videos by computing the perceptual distance between videos sampled from the dataset and the models. We follow the protocol used in StyleGAN-V (Skorokhodov et al., 2022) to caleulate FVD. First, sample 2048 video clips with 16 frames and frame stride of from the dataset. Then, generate 2048 videos from the 12V models. All videos are resized to 256 x 256 before calculating FVD except for ExtDM. (ExtDM generate videos with resolution 64 x 64, so we compute FVD on this resolution.) After that, we extract features of those videos with the same I3D model used the repository of StyleGAN-V and calculate the Fréchet Distance.
Inception Score (Saito al. (2017); IS) also evaluates the quality of the generated videos. However, computing IS need pre-trained classifier and we only apply this metric on UCF-101. When computing IS, we use the open-sourced evaluation code and pre- pre-trained classifier for videos from the repository of StyleGAN-V.
CLIPSIM (Wu et al., 2021) evaluates the consistency between video frames and the text prompt by computing the average CLIP similarity score between each frame and the prompt. We use the VIT-B/32 CLIP model (Radford et al., 2021) when evaluating zero-shot metrics on MSR-VI -TT.
PIC is metric used by Xing al. (2024) evaluate the consistency of video frames and the given image by the computing average Dreamsim (Fu et al., 2023) distance between generated frames and the image condition.
20

---

FramcBridge: Improving Inage-to-Video Vidco Generation with Bridge Models
FrameBridge DynamiCrafter
FrameBridge
DynamiCratter
1024
0.9
PVD
24
0.7
338
0.5
64
100
150
70
Sampling Steps
Sampling Steps
(a) Zero-shot FVD with different sampling timesteps
(b) Zero-shot PIC with different sampling timesteps
Figure 7. Video quality sampled with different number of timesteps.
Table 10. Zero-shot metrics on UCF-101 and MSR- VTT for FrameBridge- VideoX models.
UCF-101
MSR-VTT
Model
FVD 4
IS 1
PIC f
FVD 4
CLIPSIM f
PIC f
CrameBridge-Cog VideoX (w/o SAF)
359
36.84
0.5868
209
0.2250
0.6056
FrameBridge-Cog VideoX (w/SAF)
347
41.12
0.6563
185
0.2250
0.6587
D.3
Implementation FrameBridge and Other Baselines
We offer the implementation details I2V models which are fine-tuned on WebVid-2M or trained from scratch on UCF-101.
D.3.1 FRAMEBRIDGE
Fine-tuning on Vid2M For FrameBridge- VideoCrafter, we refer the codebase of Dynamicrafter' fine-tune FrameBridge, and initialize model from the pre-trained VideoCrafterl (Chen al., 2023a) checkpoint. For Frdde CogVideoX, refer to the official codebase and initialize our model from the pre-trained Cog CogVideoX22B (Yang et al., 2024b) checkpoint. For the schedule of bridge, adopt the Bridge-gmat gmax schedule of (Chen et al., 2023c), where f(t) 0.
g(t)² = 90 + t(31 Ba), 04 1, of [ - fo)t² + Pot with 3n 0.01, 81 = 50. We fine-tune the models e for 20k iterations 100k iterations with batch size 64. We use the AdamW optimizer with learning rate x 10-5 and mixed precision of BFloat16. We do not apply ema to the model weight during fine fine-tuning, The conditions and 2 are incorporated into the network in the same way as DynamiCrafter, and we concatenate with x' along the channel or temporal axis to condition the network on the prior (we find that the performance is quite similar whether we conduct the concatenation along channel or temporal axis). As the schedule (at,, hee is different from that of the pre-trained diffusion models, we use the generalized SAF (Proposition 2).
Training From Scratch on UCF-101 We reference the codebase of Latte? to train FrameBridge from scratch on UCF-101. We adopt Latte-S/2 as our bridge model with the same schedule as above and train FrameBridge for 400k iterations with batch size 40. For FrameBridge with neural prior, also implement FG, (2',, c) with Latte-S/2 except that the conditioning of timestep is removed from the model. To match 2 with the input shape of Latte, replicate 2 for L times and concatenate them along temporal axis. We train (2 c) for 400k iterations with batch size 32 before training bridge models if the
21

---

FramcBridge: Improving Inage-to-Video Vidco Generation with Bridge Models
FrameBridge DynemiCrafter
FrameBridge
DynamiCratter
1024
0.9
PVD
0.7
338
0.5
64
70
Sampling Timetsi
Sampling Timetsi
(a) Zero-shot FVD with different execution time
(b) Zero-shot PIC with different execution time
Figure 8. Video quality sampled with different exccution time.
neural prior is applied. For both the training of bridge models and F,,2*,,,, rate 1 x 10-6 and ema is not applied. The conditions are incorporated into the network in the same way Latte. Since Latte-S/2 is transformer-based diffusion network, we incorporate the condition z² by concatenate with video latent z, in the token sequence. To condition the network on prior z' or F2,(2,,, c), we concatenate them with z, along the channel axis.
the AdamW optimizer with learning
SNR-Aligned Fine-tuning When implementing the SAF technique, we need calculate the inverse function of SNR for the teacher diffusion schedule i SNR- ((=)) However, some T2V diffusion models use discrete timesteps and we need to approximate the aligned i. In our experiments, we choose to find discrete timestep t, such that SN R(tn)) _ > Rttn(()+1) and assume SN R(-) is linear function with respect to the input of diffusion schedule in the interval (tn, tn+1) to obtain the aligned f.
(
D.3.2 BASELINES FOR TEXT-CONDITIONAL 12V GENERATION
For SVD (Blattmann al., 2023), SEINE (Chen al., 2023b), Consist12V (Ren al.) and SparseCtrl (Guo al., 2025), use the official model checkpoints and sampling code to sample videos for evaluation. For DynamiCrafter (Xing et al., 2024), we sample videos with the official model checkpoints. We also use the official training code to train a DynamiCrafter for 20k iterations with batch size of 64 as a diffusion-based 12V fine-tuning baseline to compare it with FrameBridge fine-tuned under the same training budget.
D.3.3 BASELINES FOR CLASS-COUE CONDITIOAAL 12V GENERATION
ExtDM (Zhang et al., 2024b) is diffusion-basseeee based video prediction model, which is trained predict the following m frames with the given first frames of video clip. We train ExtDM with their official implementation" and set our 12V setting UCF-101.
1, m 15 for
VDT-I2VV is our implementation of the 12V method proposed by Lu et al.. They use transformer cunssfd diffusion network for I2V generation by directly concatenating the image condition with the token sequence of the noisy video latent 2, We also implement their I2V method on Latte-S/2 model considering the similarities among transformer-based diffusion models.
"htps://gttuuh.coo//// http::/hhhhhb.cmmo/cccccccchnhnn
22

---

FramcBridge: Improving Image-to-Videe Vidco Generation with Bridge Models
Algorithm 4 Training algorithms for FrameBridge.
Output: Trained FrameBridge model ex (zz, t, t,,,, Set bridge process {at, 12, dz, 0e==o- if Neural prior is used then
Train neural prior model (2 c) with Equation (8) before training FrameBridge. end if'
if Fine-tumed from pre-trained diffuion model cs t, c) then if SAF used then
Re-parameterize the input of eg (Zi, t, 2r, 2", c) by ex (Zz, t, 2r, 2r,"", 2²,c) Plalign (Ze, i, 2r, 2r, 20,c) with Equation (24).
c Initialize Egalign with the weight of (z,, t).
else
Initialize cx with the weight of end if
else
Randomly initialize cg (z,, t, ZT, 2T, e).
end if
repeat
Sample data (zo, c) ~ Paata (Zo, c), timestep and 2, ~ Phridge, (Ze to, 2T). Take the first frame of Zo as the image condition 2². if Neural prior is used then
Zr c))
else
Construct zr by replicating z².
end if
I(0) co (zz, t, zT, c) 2x
Update 9 with the optimizer and loss function I(0). until Reach the training budget
D.3.4 ABLATION STUDIES ON NEURAL PRIOR
In Section 5.3, we ablate on the neural prior technique by comparing the performance of four models:
VDT-I2V: The same model as our diffusion baseline on UCF-101.
VDT-I2V with neural prior as the network condition: The same model as VDT-I2V except that we additionally condition the network on (2', c).
FrameBridge without neural prior: A FrameBridge model implemented by utilizing the replicated image z' the prior.
FrameBridge with neural prior only as the network condition: FrameBridge model implemented by utilizing z' as the prior. However, we condition the bridge model on (2", e) by additionally feeding it into the network through concatenation with 2, along the channel axis.
FrameBridge-NP: FrameBridge model implemented by utilizing F2 c) as the prior.
E
Discussion On Related Works
Video Diffusion Models Inspired by the success of text-to-image (T2I) diffusion models (Ramesh et al., 2022; Nichol et al., 2022), numerous studies have investigated cistusiion-basee text-to-video (T2V) models (Blattmann et al., 2023; Yang et al., 2024b; Singer et al.) by designing 3D spatial-temporal U-Net (Ho et al., 2022b;a) and Diffusion Transformers (DiT) (Peebles & Xie, 2023; Bao et al., 2023; Zhang et al., 2025c). To improve memory and computation efficiency. Latent Diffusion Models (LDM) (Rombach al., 2022; Vahdat et al., 2021) are utilized where the diffusion process is applied in
23

---

FramcBridge: Improving Image-to-Videe Vidco Generation with Bridge Models
the compressed latent space of video samples (Bao et al., 2024; Brooks et al., 2024; He et al., 2022). Meanwhile, some other works designed cascaded diffusion models to generate motion representation (Yu et al., 2024b) videos with lower resolution (Ho et al., 2022a; Wang et al., 2025) first, which are utilized to synthesize the result videos in the subsequent stages. Another line of research (Zhang et al., 2025f; Guo et al., 2024; Wu et al., 2023) focuses on leveraging T2I diffusion models to enhance the performance of T2V generation, achieving high spatial quality and motion smoothness at the same time.
Diffusion- 12V Generation The main difference between 12V and T2V is the incorporation of image conditions into the sampling process. Xing et al. (2024) utilizes the features of a CLIP image encoder and lightweight transformer to inject image conditions into the backbone of a T2V model. Ma et al. (2024a) and Zhang et al. (2024c) propose to directly model the residual between the subsequent frames and the given initial frame with diffusion for 12V generation. Moreover, Ma et al. (2024a) also uses the DCTInit technique to enhance the consistency of video content with the given image. Chen al. (2023b) presents to train short-to-long video generation models with masked diffusion models. Guo al. (2024) and Zhang et al. (2024a) propose utilize pre-trained T2I models for image animation by training additional component to model the relationship between video frames. SparseCtrl (Guo et al., 2025) and Animate Anyone (Hu, 2024) design specific fusion modules for video diffusion models to adapt to various types of conditions including RGB images. Ren et al. propose improved network architecture and sampling strategy for image-to-video generation at the same time to enhance the controllability of image conditions. Jain et al. (2024), Zhang et al. (2023) and Shi et al. (2024) design cascaded diffusion systems for 12V generation. VIDIM (Jain et al., 2024) consists of one base diffusion model and another two diffusion models for spatial and temporal super-resolution respectively. Zhang et al. (2023) uses a base diffusion model generate videos with low resolutions, which serve the input of the following video super-resolution diffusion model. Shi et al. (2024) first generates the optical flow between the subsequent frames and given image with diffusion process, and use the optical flow as conditions of another model to generate videos. Ni et al. (2023) and Zhang et al. (2024b) train an autoencoder to represent the motions between frames in a latent space, and use diffusion models to generate motion latents. However, previous 12V diffusion models are built on the toise-to-data generation of conditional diffusion process and the sampling remains a denoising process conditioned on given images. In contrast, FrameBridge replaces the diffusion process with a bridge process and the sampling directly model the animation of static images.
Noise Manipulation for Video Diffusion Models Several works have explored to improve the uninformative prior distribution of diffusion models. PYoCo (Ge et al., 2023) recently proposes to use correlated noise for each frame in both training and inference. Consist12V (Ren al.), FreeInit (Wu et al., 2024), and CIL (Zhao et al.) present training-free strategies better align the training and inference distribution of diffusion prior, which is popular in diffusion models (Lin et al., 2024; Podell et al.; Blattmann et al., 2023). Noise Calibration (Yang al., 2024a) proposed to enhance the video quality of SDEdit (Meng et al.) with iterative calibration of initial noise These strategies focus on improving the noise distribution to enhance the quality of synthesized videos, while they still suffer the restriction of noise-to-data diffusion framework, which may limit their endeavor to utilize the entire information (e.g., both large-scale features and fine-gr fineigriiedd details) contained in the given image. In contrast, we propose a data-to to-data framework and utilize deterministic prior rather than Gaussian noise, allowing us to leverage the clean input image as prior information.
Comparison with Previous Works of Bridge Models and Coupling Flow Matching In Section 4, we leverage the forward SDE of bridge models (Zhou et al.) and the backward sampler proposed by Chen et al. (2023c) to build FrameBridge. We unify their theoretical frameworks to establish our formulation, and emphasize that bridge models are suitable for image-to-video generation, which is typical data-to-data generation task. Liu et al. (2023) and Chen et al. (2023c) apply bridge models to image-to-image translation and text-to-speech synthesis tasks respectively. Similar as bridge models, flow matching can also be used to construct the data-de data-dependent stochastic interpolants (Albergo et al., 2024; Fischer et al., 2023; Albergo et al., 2023) for paired-data generation and has been used in image-to-image generation. However, whether the coupling flow matching is suitable for image-to-vided generation has not been fully explored. Compared with their works, we focus on 12V tasks, building our bridge-basee framework by utilizing the frames-to-framee essence and presenting two innovative techniques for two scenarios of training 12V models, namely fine-tuning from pre-trained text-to- video diffusion models and training from scratch.
24

---

FramcBridge: Improving Image-to-
Mnndco Vidco Generation with Bridge Models
Input
Prompt
"A rabit, running fast"
Ours
ConamiCrafter SEINE
Generated Frames
Figure 9. Another case of qualitative comparison between FrameBridge and other baselines. FrameBridge outperforms diffusion baseline methods in appearance consistency and video quality. FrameBridge and DynamiCrafter models are fine-tuned from VideoCrafterl for 20k steps.
F More Qualitative Results of FrameBridge
We show several randomly selected samples of FrameBridge below, and more synthesized samples can be visited at: https: / https:://Errmbbrrdgge i cml wit.llgtthubiiii io.
25

---

FramcBridge: Improving Image-to- Vidco Generation with Bridge Models
Input
Prompt
a blue fishing boat is navigating in the ocean next to a cruise ship
FrameBridge
DynamiCrafter
SEINE
Consisti2V
Generated Frames
Figure 10. Qualitative comparison between FrameBridge and other baselines. Here FrameBridge is fine- tune- from CogVideoX-22 for 100k steps, and the samples of other baselines generated with their official checkpoints. DynamiCrafter, SEINE, ConsistIV fine-tuned from VideoCrafterl, inflated Stable Diffusion 2.1-Base and LaVie respectively.
26

---

FramcBridge: Improving Image-to- Vidco Generation with Bridge Models
Input
Prompt
bridge that the middle river, camera zooms out
FrameBridge
DynamiCrafter
SEINE
Consist12V
Generated Frames
Figure 11. Qualitative comparison between FrameBridge and other baselines. Here FrameBridge is fine- tune- from CogVideoX-22 for 100k steps, and the samples of other baselines generated with their official checkpoints. DynamiCrafter, SEINE, ConsistIV fine-tuned from VideoCrafterl, inflated Stable Diffusion 2.1-Base and LaVie respectively.
27

---

FramcBridge: Improving Image-to- Video Generation with Bridge Models
Figure 12. Zero-shot generation results of fine-tuned FrameBridge (with SAF) on UCF-101.
28

---

FramcBridge: Improving Image-to- Vidco Generation with Bridge Models
Phana la rrse plaping iille and voman play sland"
cabling par sotting
rido aperan
kabres
WERTE mary takes picc
talkingg
heeding a small poopy"
Figure 13. Zero-shot generation results of fine-tuned FrameBridge (with SAF) on MSR-VTT.
29

---

FramcBridge: Improving Image-to- Vidco Generation with Bridge Models
Figure 14, Non-zero-shot generation results FrameBridge-Ne NP on UCF-10L. We use two lines to present neural prior and the corresponding generated video.
30

---

FramcBridge: Improving Image-to- Vidco Generation with Bridge Models
Out
and quality
helpoerr
Light
Figure 15. Comparisons between finc-tuned FramcBridge and other diffusion-bassee 12V models.
31


Reviews:
[{"review_id": "mYDxkrzMYD", "reviewer": "Reviewer_fXNN", "summary": "This work focuses on the mismatching issue of diffusion models and I2V generation tasks, and propose FrameBridge, which build a data-to-data generation process with bridge model, making the generation procedure more in line with the frame-to-frames nature of I2V task. For fine-tuning scenario, a SNR-Aligned Fine-tuning is proposed  to take advantage of pretrained diffusion models. For training from scratch, a trained prior model provides initial prior of non-first frames.\n\n## update after rebuttal\n\nI have carefully read the response from authors and the comments of other reviewers. My major concerns about the adaption to multi-shot generation and limited dynamics were also raised by other reviewers. My concerns have been addressed. I will keep my initial rating.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer fXNN,\n\nWe sincerely appreciate your acknowledgement of our methods and proposed techniques as \"well-motivated\", \"practical and effective\". We are happy to engage in a thorough discussion and hope it will address your concerns and questions. Tables and demos are provided in https://framebridge-icml.github.io/rebuttal-demo-page/.\n\n## Reply to Q2\n\n**Firstly, we will offer some empirical solutions to improve the dynamic degree of FrameBridge.** The limited dynamic degree is a well-studied issue for diffusion I2V models [2,3,4]. We adapt two techniques used in diffusion I2V models to FrameBridge: (1) Add frame-stride condition; (2) Add noise to image condition. Demonstrated by our experments on FrameBridge-VideoCrafter, we find that these techniques are also effective for FrameBridge, and can **improve the DD score from 35.77 to 48.62**, which is much higher than the diffusion counterpart (38.69 for DynamiCrafter). (See **Table 2,7,8**. -MI for Motion-Improved).\n\n**Secondly, we would like to mention that the potential limited dynamic degree is an empirical issue instead of the inherent weakness of bridge-based methods.** Just as diffusion models, if we have the perfect bridge score function and a perfect sampler for the backward bridge SDE (Eq. 5 in the paper), we can sample from the true data distribution, which is a mathematically well-established conclusion in [6]. It may be counterintuitive. To understand it **more intuitively (but not that rigorously)**, we would like to provide discussions from three aspects:\n\n1. **Marginal:** Consider the marginal $z_t=a_t z_0 + b_t z_T + c_t \\epsilon$, where $a_0=b_1=1, a_1=b_0=c_0=c_1=0$. Different from the diffusion process, the coefficient of noise $c_t$ first increases then decreases when $t$ varies from $T$ to $0$ in the sampling process. So, the latent $z_t$ **will continuously become noisier in the first half part of sampling process**.\n\n2. **Forward-Backward SDE:** In [6] a forward diffusion process\n$$\n\\mathrm{d}z_t = f(t)z_t \\mathrm{d}t + g(t)\\mathrm{d}w,\n$$\nis used to construct the forward bridge process\n$$\n\\mathrm{d}z_t=[f(t)z_t + g(t)^2 \\nabla_{z_t} \\log p_{T,diff}(z_T|z_t)]\\mathrm{d}t+g(t)\\mathrm{d}w,\n$$\nCompared with diffusion process, the additional term $g(t)^2 \\nabla_{z_t} \\log p_{T,diff}(z_T|z_t)$ gradually \"pushes\" $z_t$ to the value $z_T$ and finally the prior $z_T$ will become a fixed point. Reversely, compared with diffusion backward process, the backward bridge process will additionally \"pull\" $z_t$ away from $z_T$.\n\n3. **Comparison between bridge model and inference-time noise manipulation:** In FrameInit, the prior $z_T$ is constructed by combining low-frequency information from a static video with Gaussian noise. (Similar to FrameBridge at first glance.) Then, $z_T$ is denoised with a **trained diffusion model**. Although $z_T$ has been influenced by the low-frequency components of a static video, the diffusion process will sample from it in the same way as a Gaussian noise, **since the training process of diffusion model is \"ignorant\" of the noise manipulation and still learns the score function of a standard diffusion process $z_t=\\alpha_tz_0+\\sigma_t\\epsilon$, which may cause train-test mismatch and limit the dynamic degree**. (This may also explain why the Dynamic Degree score of ConsistI2V is lower.) However, for bridge model, we change the prior $z_T$ **along with the diffusion process. The model is \"aware\" of that and will learn the score function of a bridge process $z_t=a_0z_0+b_tz_T+c_t\\epsilon$ with bridge score matching loss.**\n\n## Reply to Q1\n\nThis is a valuable question, and multi-shot/scene I2V generation is a highly demanding task for I2V generation without mature benchmarks or baselines currently. It will also be challenging for FrameBridge as repeating the first frame as prior may not be a proper design. Inspired by RPGDiffusion [8] and other previous works about long video generation [9,10, 11] , we propose a possible hierarchical solution as below:\n\n1. Use LLM planner to divide the whole video into several sub-video clips $v_1, v_2, ..., v_M$, and generate image prompts for the start of each clip.\n\n2. Generate the start of each clip $i_1, i_2,..., i_M$ with a T2I diffusion model.\n\n3. For $v_t, 1 \\leq t \\leq M - 1$, construct a neural prior by interpolating between $i_t, i_{t + 1}$ and train a FrameBridge for video interpolation (which is also a typical task for I2V generation [12]) to generate it. Generate $v_M$ with the static prior constructed by $i_M$ using FrameBridge.\n\nOur consideration is to generalize the neural prior with LLM planner and use FrameBridge to leverage the locally reliable prior.\n\n## Other Discussions\n\nThe line 168 is a typo and should be $(z_T | z^i, c)$. We thank the reviewer to point it out and will revise it later. \n\nDue to word limit, we can not elaborate on all the details and please feel free to tell us if you still have any concern.\n\nReferences: See response to Reviewer g6Xo.", "claims_and_evidence": "Yes. This work proposes to model the I2V generation task from frame-to-frames to a data-to-data framework based on the schr\\\"odinger bridge theory, which is implemented by a SNR aware finetuning or an advanced prior. The effectiveness of the proposed bridge based I2V method is validated on various benchmarks across different metrics.", "methods_and_evaluation_criteria": "Yes. The benchmarks used to validate are commonly used in this field, including the MSR-VTT and UCF-101, which are standard benchmarks for video generation task.", "theoretical_claims": "Yes, the theoretical claims in the appendix is reviewed, including the analysis of parameterization of the bridge optimization objectives and the analysis of SAF and the proposed prior for training from scratch.", "experimental_designs_or_analyses": "The proposed method is validated on MSR-VTT, webvid and UCF-101 compared with previous I2V works. The details of these datasets and the evaluation metrics are elaborated in the appendix.", "relation_to_broader_scientific_literature": "This work is mainly related to I2V video generation task, but implemented based on the schr¨odinger bridge theory. This theory has been explored these years for synthetic tasks like style transfer, I2I generation, etc. The bridge theory is more in line with these data-to-data task, compared with the commonly used diffusion method.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths:\n1. This paper identifies a clear mismatch issue between diffusion models and the frame-to-frames I2V task, which is well-motivated. A Schrödinger bridge-based method is proposed to deal with I2V task.\n2. The proposed SAF and neural prior are practical and effective to take advantage of pretrained diffusion models and enable efficient from scratch training.\n3. The paper is well-written and easy to follow. The details of experiments are elaborated.\n\nWeaknesses:\n\nplease refer to the question part.", "other_comments_or_suggestions": "- In line 168 right side, the $i$ in $(z_T|i, c)$ is not explained.", "questions_for_authors": "1. I certainly understand the underlying bridge theory and the proposed method for I2V, considering that some modern video generation methods can process multi-shot or multi-scene video generation, in which scenario the content of multiple shots may be much different with the given first frame, dose the hypothesis of starting from repeated first frame via bridge still make sense? Is there any more improvement or consideration of the issue?\n1. In the I2V task, repeating the first frames across the temporal dimension may influence the final motion of the generation result, limiting achieving high dynamic motions. Could the authors discuss about this issue?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YAG80rmFkd", "reviewer": "Reviewer_WAVy", "summary": "This paper proposes a bridge model-based image-to-video generation model. It first formulates image-to-video generation as data-to-data generation instead of noise-to-data generation. Under this formulation, the generation should be easier because it starts from a strong prior of the image instead of the Gaussian prior. Furthermore, two designs are proposed to help finetune a text-to-video model for image-to-video generation: an SNR-aligned parameterization and a neural prior that predicts the mean image using a neural network. The proposed method is evaluated on WebVid-2M, UCF-101 and VBench-I2V, showing advantageous performance over different base models.\n\n## update after rebuttal\nThe authors have addressed all of my concerns with clarifications and experimental results. Therefore, I am raising my score to Weak accept.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer WAVy,\n\nWe sincerely appreciate your recognition of the strengths of our work and providing valuable suggestions to help us improve it. We hope the following discussions can address your concerns. Tables and demos are provided in https://framebridge-icml.github.io/rebuttal-demo-page/.\n\n## Performance on VBench-I2V\n\n**1. Reply to W5**\n\nWe find there are discrepancies between the results of Table 2 in our paper and the latest version of VBench-I2V [1] and thank the reviewer for bringing this to our attention. We present the modified version in **Table 1**. Note that **two FrameBridge models still achieve higher total score than baselines**.\n\n**2. Reply to W4**\n\nIt is a valuable consideration and we will provide our empirical results and theoretical discussions. Before that, we would like to gently point out that the much higher DD of SparseCtrl may lead to inferior overall video quality as shown by **Table 1 and 6**. We kindly invite you to read the discussions in our response to Reviewer fXNN (**Reply to Q2** part) due to the character limit.\n\nWe would appreciate it if you could kindly take the time to read that, and it may also provide intuitions about why the samples can \"deviate\" from the prior, which is also appicable to the scenarios involved in W1 and W3.\n\n## Implementation Details\n**Reply to W1**\n\nFirstly, we would like to calrify that neural prior predicts the **mean video conditioning on the first frme** instead of \"mean image\". (i.e., Frames can be different.) We list the model input, condition, priors of diffusion and bridge models below.\n|Model|Input|Condition|Prior|\n|-|-|-|-|\n|Diffusion|$z_t,t,c,z^i$|$c, z^i$|$z_T\\sim\\mathcal{N}(0, I)$|\n|Bridge|$z_t,t,c,F_\\eta(z^i, c)$|$c, z^i,F_\\eta(z^i, c)$|$z_T=F_\\eta(z^i,c)$|\n\nHere $F_\\eta(z^i, c)\\in \\mathbb{R}^{F \\times h \\times w \\times d}$ has the same shape as video latents and we do not concatenate $F_\\eta(z^i, c)$ with the original image when it serves as a prior (as illustrated in our pseudo code, line 846), and this design is not a hack. However, both diffusion and bridge model should take original image as an input to ensure it is learning the conditional score function. Blurry prior will not degrade quality of final videos (as shown by our experiments) and bridge models here can be seen as a refiner. (e.g., Bridge can be applied to deblurring [5].)\n\n## Experimental Design\n\n**1. Reply to W2**\n\nTo address the concern, we conduct ablations of SAF in zero-shot setting and provide results in **Table 3, 4**. Neural prior is designed for training from scratch and the ablation setting is aligned with the main experiments in our paper.\nWe would like to gently point out that the ablation with VideoCrafter is already provided in the Table 1 of paper (line 342, 344, 345). We thank the reviewer's suggestion and will improve the clarity of ablation studies. For Figure 6, the first point is evaluated when the model is fine-tuned for 3k steps. When initialized, both models generate noise and thus we omitted it. To address the concern, we provide the CD-FVD curve before 3k in **Figure 1**.\n\n**2. Reply to W3**\n\nThis is a thoughtful suggestion to help us show the effectiveness of FrameBridge more thoroughly. Due to the constraints of time and computational resources, we temporarily provide the experiment results with 24 frames in **Table 5**.\nConducting experiments with long video requires significant computational resources: The base model should be large enough to have capability to deal with long video and training with more frames will also add to computational complexity. As far as we know, baselines and benchmarks under this setting (I2V for longer than 16 frames) is not mature enough. Taking all the above into account, we compare FrameBridge with a diffusion I2V model fine-tuned from the same CogVideoX-2B, and find FrameBridge outperforms the diffusion counterpart. Please feel free to tell us if you still have concerns and we are willing to provide more experimental evidence and discussions.\n\n## Other Discussions\n**1. Reply to Q1**: This is an insightful question and we think the input/output parameterization is not the most important difference. We kindly invite you to read our response to Reviewer g6Xo (**Comparing noise-to-data and data-to-data** part.)\n\n**2. Reply to Q2**: We kindly invite you to read our response to Reviewer fXNN (**Reply to Q1** part).\n\n**3. Additional references**: We thank the reviewer for pointing out the related works we unintentionally left out, and will add citations and discussions in our paper later.\n\n**4. More demos of FrameBridge**: We provide more demos generated by FrameBridge in **Table 9-11**, which we hope could address the concern of dynamic degree.\n\nDue to the word limit, we could only elaborate on some important points. Please feel free to tell us if you still have any concerns and we would be happy to have a kind and thorough discussion further.\n\nReferences: See response to Reviewer g6Xo.", "claims_and_evidence": "The paper makes three main claims: 1) image-to-video generation can be formulated as a data-to-data generation task, 2) SNR-aligned parameterization helps finetune a text-to-video model to image-to-video generation, 3) neural prior helps train the image-to-video model from scratch. While the first two claims are supported by extensive experimental evidence, I have concerns about the third claim:\n\n* **W1**: The predicted neural prior looks blurry (as shown in Figure 4), which could lead to inferior detail in the generated video. The authors seem to adopt an implementation of concatenating the neural prior with the original image to preserve the details (the second row of Figure 4), but this is more of a hack and weakens the contribution of this design.", "methods_and_evaluation_criteria": "The proposed method is suitable for image-to-video generation because it translates the noise-to-data generation problem into an easier data-to-data generation form. The evaluation criteria are also quite comprehensive, including FVD, IS, and PIC on UCF-101, FVD, CLIPSIM, and PIC on MSR-VTT, and various metrics in VBench-I2V.", "theoretical_claims": "The paper makes two theoretical claims: 1) SNR-aligned parameterization aligns with a VP diffusion process, 2) the neural prior actually predicts the mean value of subsequent frames. The proofs of these two claims are provided in Appendix A, and the proofs look solid.", "experimental_designs_or_analyses": "The experimental designs could be improved in the following directions:\n\n* **W2**: Inconsistent base model and setup across experiments. The current ablation studies use a different base model (VDT-I2V) and setup (non-zero-shot UCF-101) than the main experiments, which is confusing. It would be better if the authors could perform ablation studies using the same base model and setup (i.e., DynamicCrafter with zero-shot UCF-101). Even more, it is unclear to me whether the DynamicCrafter baseline is properly initialized in the main experiments, since the convergence curves in Figure 6 do not seem to share the same starting point.\n* **W3**: Evaluate on long video generation. The current method is evaluated only on 16-frame (2s) video generation, where the proposed strong image prior is clearly beneficial, as short videos don't deviate much from the starting frame. However, long video generation (e.g. $\\ge$ 5s) requires the generation of new content that is significantly different from the starting frame, where a strong image prior could become a limitation. It would be better if the authors could investigate its effectiveness across different generation lengths.", "relation_to_broader_scientific_literature": "The paper leverages the diffusion bridge model [Zhou’23] for image-to-video generation, extending its previous success in image-to-image translation [Liu’23, Zhou’23].", "essential_references_not_discussed": "There are a few seminal works on bridge models that are not cited [De Bortoli'21, Peluchetti'21].\n\n---\n\n[1] De Bortoli, et al. Diffusion Schrödinger Bridge with Applications to Score-based Generative Modeling. NeurIPS 2021.\n\n[2] Peluchetti. Non-Denoising Forward-Time Diffusions. 2021.", "other_strengths_and_weaknesses": "Strengths:\n\n* The idea of applying the bridge model to image-to-video generation is novel and elegant. It reduces the gap between prior distribution and target distribution in video generation.\n* The theoretical analysis on SNR-aligned parameterization and neural prior enhances the completeness of the paper.\n\nWeaknesses:\n\n* **W4**: The proposed method shows a much inferior dynamic degree on VBench-I2V (48.29 vs. 81.95). This is a fundamental limitation associated with the proposed strong static image prior. The authors should carefully discuss this limitation and propose possible solutions.\n* **W5**: The results on VBench-I2V seem to be different from the original paper [Huang'24] or the official website. I suggest the authors further check the evaluation protocols and make sure that the baseline results are consistent.\n\n---\n\n[1] Huang, et al. VBench++: Comprehensive and Versatile Benchmark Suite for Video Generative Models. 2024.", "other_comments_or_suggestions": "Judging from the demo samples provided on the website, the generated videos do not have a high dynamic degree or large camera motion. It would be better if the authors could include more demo samples with high motion.", "questions_for_authors": "* **Q1**: Intuitively, the difference between noise-to-data generation and data-to-data generation is not that great, since the model outputs differ only by a constant offset value $z^i$, which is given in the image condition. It should not be difficult for the model to copy the image value $z^i$ and add it to its output. What do you think contributes to the performance gap, is it due to the closer value range of source and target latents, which makes it more friendly for the neural network?\n* **Q2**: How does the model handle scene cuts, since it imposes such a strong static image prior?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RGpH4kFKAW", "reviewer": "Reviewer_g6Xo", "summary": "This paper introduces FrameBridge, reformulating the image-to-video task as data-to-data generation through a bridge model. Different from image-to-video as first frame condition (i.e., noise-to-data generation), data-to-data generation achieves better consistency and can well preserve information in the first frame. Besides, the authors further introduce SNR-Aligned fine-tuning and neural prior to improve the performance of FrameBridge.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer g6Xo,\n\nWe sincerely appreciate your acknowledgement of the strengths of our work and instructive suggestions to help us improve that. We hope the following discussions could address your concerns and questions. Tables are provided in https://framebridge-icml.github.io/rebuttal-demo-page/.\n\n## Ablations between noise-to-data and data-to-data\n\nWe thanks the reviewer's suggestion and provide the experimental results of this ablation.\n\n1. For the setting of fine-tuning, we provide the results in **Table 3, 4**. Here we use the same base model, network architecture, training data, training budget for noise-to-data and data-to-data models. As we need to fine-tune bridge models from pre-trained diffusion models, it is important to align the two process with our proposed SAF technique. A concurrent work [13] reaches similar conclusions when fine-tuning from flow-matching to diffusion process.\n\n2. For the setting of training-from-scratch, since VDT-I2V has the same network architecture (Latte-S/2), training dataset, training budget as FrameBridge, we would like to gently point out that Table 3, 5 in the paper can offer experimental evidence for the ablation between noise-to-data and data-to-data. We appreciate the reviewer's question which help us to identify the lack of clarity in the setup (details are now dispersed in Sec. 5.2, App. D.3.3, D.3.4) and will revise it for better clarity.\n\n## Does data-to-data learn a shotcut to more static motion?\n\nThis is a thoughtful question. Bridge models will not learn a shortcut to more static motion. Due to character limit, we kindly invite you to read our response to Reviewer fXNN for more discussions (**Reply to Q2** part).\n\n## Comparing noise-to-data and data-to-data\n\nWe wold like to offer more intuitive comparisons between the two frameworks. \n\n1. When considering the differences between noise-to-data and data-to-data frameworks, we suggest to view from the perspective of forward-backward process instead of pointwise denosing. Theoretically, denoising score matching guarantees perfect score function for each $(z_t, t)$. But practically the score function is not perfect for each point, and we should consider the complexity for modeling the whole process.\n\n2. Differences in process: For noise-to-data I2V generation, the image can only be injected into the model by network conditioning. For data-to-data I2V generation, the whole diffusion process is changed to a bridge process and now we can utilize the image condition from two aspects (network conditioning and process). As shown by Figure 2 in our paper, the marginal distribution $z_t$ in bridge process carries more information than in diffusion. In alignment with our experiments, **modeling the whole bridge process** with bridge score function could be easier than **modeling the whole diffusion process** with diffusion score function.\n\nWe hope the discussion could provide some intuitive insights for the readers and are willing to have a further discussion if there are other cocerns or questions.\n\n## Comparing bridge and flow-matching\n\nWe conduct experiments with 3 flow-matching models and compare them with diffusion and FrameBridge:\n\n1. Vanilla flow-matching: The noisy latent takes the form of $z_t = (1-t)z_0 + t \\epsilon, \\epsilon \\sim \\mathcal{N}(0, I)$\n2. Coupling flow-matching with $\\sigma=1.0$ [18]: $z_t = (1-t)z_0 + t (z^i + \\epsilon)$\n3. Coupling flow-matching with $\\sigma=0$ [19]: $z_t = (1-t)z_0 + t z^i$\n\nResults are shown in **Table 12** of our link. We compare bridge and flow-matching in both methods and experiment results as below:\n\n1. The algorithm and performance of Vanilla flow-matching is quite similar to that of diffusion.\n2. Coupling flow-matching with $\\sigma=0$ will generate almost static video. In [19], it is used to refine the results of a diffusion, and need condition augmentation, which is not suitable for single-stage I2V.\n3. Coupling flow-matching with $\\sigma=1$ has better performance than diffusion models. Different from bridge models, the prior is still a Gaussian (with non-zero mean), while the prior of bridge is a deterministic point. Empirically, as shown by the experiment results, brige models can utilize the condition more effectively.\n\nDue to character limit, we can not elaborate on all the details. Please feel free to tell us if you still have any concern and we are willing to have a further discussion.\n\nReferences (Due to character limit, we can only provide abbreviated forms of some works where there is no ambiguity):\n\n[1] VBench++\n\n[2] Stable Video Diffusion\n\n[3] CogVideoX\n\n[4] Conditional Image Leakage\n\n[5] I^2SB\n\n[6] DDBM\n\n[7] ConsistI2V\n\n[8] RPGDiffusion\n\n[9] MovieDreamer\n\n[10] VideoGen-of-Thought\n\n[11] VideoDirectorGPT\n\n[12] VIDIM\n\n[13] SANA-Sprint\n\n[14] VideoElevator\n\n[15] Noise Calibration\n\n[16] SDEdit\n\n[17] Flow Matching for Generative Modeling\n\n[18] Stochastic Interpolants with Data-Dependent Couplings\n\n[19] Boosting Latent Diffusion with Flow Matching", "claims_and_evidence": "The experiments in Table 4 and Table 5 demonstrate part of the claim. However, the authors do not directly compare noise-to-data and data-to-data in the ablation studies. Although the authors compare their approach to other noise-to-data paradigms, like ExtDM, the training settings and architectures are not matched.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Yes.", "experimental_designs_or_analyses": "I think most of the experimental designs are soundness. However, it is better to directly compare noise-to-data and data-to-data in ablation study.", "relation_to_broader_scientific_literature": "The contributions are broadly related to distribution transfer between arbitrary two data distributions. This paper resorts to a bridge model to translate data-to-data distribution, while this idea can also be implemented with flow-matching.", "essential_references_not_discussed": "N/A.", "other_strengths_and_weaknesses": "Strengths: 1) FrameBridge is well-motivated and the neural prior is a novel idea for initializing data distribution.\nWeaknesses: 1) The authors should carefully ablate noise-to-data and data-to-data paradigms with fair training settings. Does data-to-data learn a shortcut to more static motion? It would be better to analyze the differences between these two paradigms to provide more insights for readers.", "other_comments_or_suggestions": "1) The author should discuss/experiment with flow-matching, since flow-matching can also be adopted to data-to-data transfer. Is it any benefit to use bridge model than flow-matching?", "questions_for_authors": "N/A.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZUIBQ3bWC9", "reviewer": "Reviewer_2c9b", "summary": "This paper introduces FrameBridge, a novel approach to improve image-to-video (I2V) generation using diffusion models. The authors address the mismatch between the noise-to-data generation process of traditional diffusion models and the I2V task, which can lead to suboptimal results. FrameBridge proposes a bridge model that adopts a data-to-data generative process, enabling better utilization of the given image and enhancing consistency in video generation. The paper also introduces two innovative techniques: SNR-Aligned Fine-tuning (SAF), which adapts pre-trained text-to-video models for I2V tasks, and a neural prior to further improve the model when training from scratch. Experimental results on WebVid-2M and UCF-101 demonstrate that FrameBridge outperforms existing diffusion models in synthesis quality, with significant improvements in FVD scores. The SAF and neural prior techniques also contribute to the enhanced performance of bridge-based I2V models.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 2c9b,\n\nWe sincerely appreciate your recognition of the strengths and effectiveness of our work and the valuable suggestions to help us improve that. We are happy to have a discussion and hope it could address your concerns. Tables are provided in https://framebridge-icml.github.io/rebuttal-demo-page/.\n\n## Contributions of FrameBridge\n\n**1. Reply to W1**\n\nWe would like to gently point out that the contribution of our work is not limited and our FrameBridge is a novel bridge-based I2V framework:\n\n(1) **SAF technique: Applying bridge to I2V generation is not trivial.** As shown by Table 4 of our paper, directly fine-tuning **T2V diffusion** to **I2V bridge** may cause suboptimal video quality, and we propose SAF technique to align the two process. **As far as we know, it is the first trial to fine-tune bridge models from diffusion models.** A concurrent work [13] also shows the importance of aligning different processes in fine-tuning. **Table 3, 4 in our link** provide more evidence for the effectiveness of SAF.\n\n(2) **We propose neural prior to further improve the performance of bridge model.** As bridge-models benefit from informative priors, it is well-motivated and effective to further investigate the choice of prior.\n\n(3) We provide theoretically sound proof and derivations for all the techniques we proposed.\n\n**In conclusion, we are the first to apply bridge models to I2V generation, which may not achieve superior generation quality without the two novel proposed techniques (i.e., SAF and neural prior).**\n\n**2. Discussions about other related works**\n\nWe thank the reviewer for pointing out the related works we unintentionally left out, and will add citations and discussions in our revised paper. VideoElevator [14] proposed to improve the T2V generation quality with T2I diffusions, achieves high spatial quality and motion smoothness at the same time. Noise Calibration [15] proposed to enhance the video quality of SDEdit [16] with iterative calibration of initial noise. These are valuable works investigating into the improvement of T2V diffusion models, laying foundations for better I2V generation by building enhanced base models and providing viable techniques for improving diffusion-based I2V generation.\n\n## Qualitative Comparisons\n\n**Reply to S1 and W2:** We thank the reviewer for the instructive and valuable suggestions and add a later academic baseline ConsistI2V (TMLR 2024) [7] in our qualitative comparisons. Although we fine-tune a FrameBridge model from CogVideoX, we choose CogVideoX-2B as base model and fine-tune for several hundred GPU hours, which is $< 1\\%$ of that used to train CogVideoX-I2V (fine-tuned from CogVideoX-5B), HunyuanVideo and other industry-grade models. We provide more qualitative comparisons in https://framebridge-icml.github.io/rebuttal-qualitative/, and pre-trained models are specified. Please feel free to tell us if you still have any concern, and we are willing to provide more experimental evidence or have a further discussion.\n\n## Comparing FrameBridge with Inference-Time Noise Manipulation\n\n**Reply to W3:** If we understand correctly, the provided baseline is a inference-time noise manipulation which only changes the prior when sampling and do not modify the diffusion process. (We are not certain if we have understood correctly. Feel free to point it out if there is misunderstanding and we are willing to provide more experimental resutls.) This baseline will generate almost static videos, and thus the quality is much inferior to FrameBridge and diffusion I2V models. \n\nTable 1. Zero-shot metrics on MSR-VTT and VBench-I2V score.\n|Model|FVD|CLIPSIM|PIC|VBench-I2V Score|\n|-|-|-|-|-|\n|Baseline|644|0.2249|0.56|77.36|\n|FrameBridge|**99**|**0.2250**|**0.70**|**85.37**|\n\nDue to character limit, we kindly invite you to read our response to Revewer fXNN (**Comparison between bridge model and inference-time noise manipulation** part), where we provide more intuitive explanations. In case that you are interested in the comparison between bridge models and flow-matching (which shares some similarities with the mentioned the baseline), we kindly invite you to read our response to Reviewer g6Xo (**Comparing bridge and flow-matching** part).\n\n## Other Discussions\n**Reply to S2**: Thanks for the suggestion and we will revise that in out paper later. (In this response, we temporarily align the decimal places with the current version.)\n\nDue to the character limit, we can not elaborate on all the details. Please feel free to tell us if you still have any concern or question. We are willing to continue to have a kind and thorough discussion.\n\nReferences: See response to Reviewer g6Xo.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Yes. I checked the basics of DDBM, parameterization of FrameBridge, SNA-Aligned Fine-tuning and training objective in Appendix.", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "This paper mainly focuses on improving the appearance and temporal consistency of video diffusion models and has shown its effectiveness on various models.", "essential_references_not_discussed": "[1] VideoElevator: Elevating Video Generation Quality with Versatile Text-to-Image Diffusion Models. arXiv:2403.05438\n[2] Noise Calibration: Plug-and-play Content-Preserving Video Enhancement using Pre-trained Video Diffusion Models. arXiv 2407.10285\n\nThese two papers also investigate how to improve the performance of various video diffusion models, and the authors are encourgaed to discuss them in this paper.", "other_strengths_and_weaknesses": "1. The technical contributions are somewhat limited. The proposed FrameBridge seems like a simple combination of video diffusion models and bridge models.\n2. The paper only provides qualitative experimental comparisons with earlier video generation models (e.g., DynamiCrafter), but does not include comparisons with the latest models (e.g., CogVideoX, HunyuanVideo).\n3. In Fig.1, the key idea of FrameBridge is to take duplicated images as the initial video. The reviewer is curious that: what is the effect of directly adding noise to the initial video and using a video diffusion model to denoise it? Will FrameBridge perform better than this baseline?", "other_comments_or_suggestions": "1. The authors are encouraged to provide more video comparisons in supplementary materials and specify the name of pre-trained video diffusion models.\n2. The number of decimal places in the table should be kept consistent as much as possible, e.g., Table 1 and Table 2.", "questions_for_authors": "Please see weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents FrameBridge, a Schrödinger Bridge-based approach for image-to-video generation that reframes the task as data-to-data synthesis. While initial concerns regarding the neural prior's blurriness and inconsistent baselines were raised, the authors addressed these via clarifying design choices (e.g., concatenation preserves details) and supplemental experiments with unified settings. The method’s effectiveness is validated across benchmarks (MSR-VTT/UCF-101) with metrics (FVD, CLIPSIM), and new 32-frame results in the rebuttal alleviate concerns about long-video limitations . Though technical contributions appear incremental (bridges + diffusion), the SNR-aware finetuning (SAF) and theoretical grounding provide actionable insights. Thus, I recommendation to accept this paper.
Decision: Accept (poster)