FrameBridge: Improving Image-to-Video Generation with Bridge Models

## Abstract

Abstract Image-to-video (I2V) generation is gaining increasing attention with its wide application in video synthesis. Recently, diffusion-based I2V models have achieved remarkable progress given their novel design on network architecture, cascaded framework, and motion representation. However, restricted by their noise-to-data generation process, diffusion-based methods inevitably suffer the difficulty to generate video samples with both appearance consistency and temporal coherence from an uninformative Gaussian noise, which may limit their synthesis quality.
In this work, we present FrameBridge, taking the given static image as the prior of video target and establishing a tractable bridge model between them. By formulating I2V synthesis as a frames-to-frames generation task and modelling it with a data-to-data process, we fully exploit the information in input image and facilitate the generative model to learn the image animation process.
In two popular settings of training I2V models, namely fine-tuning a pre-trained text-to-video (T2V) model or training from scratch, we further propose two techniques, SNR-Aligned Fine-tuning (SAF) and neural prior, which improve the fine-tuning efficiency of diffusion-based T2V models to FrameBridge and the synthesis quality of bridge-based I2V models respectively.
Experiments conducted on WebVid-2M and UCF-101 demonstrate that: (1) our FrameBridge achieves superior I2V quality in comparison with the diffusion counterpart (zero-shot FVD 83 vs. 176 on MSR-VTT and non-zero-shot FVD 122 vs. 171 on UCF-101); (2) our proposed SAF and neural prior effectively enhance the ability of bridge-based I2V models in the scenarios of fine-tuning and training from scratch. Demo samples can be visited at: https://framebridge-demo.github.io/ .

## 1 Introduction

Image-to-video (I2V) generation, commonly referred as image animation, aims at generating consecutive video frames from a static image (Xing et al., ; Ni et al., ; Zhang et al., ; Guo et al., ; Hu et al., ), \ie, a frame-to-frames generation task where maintaining appearance consistency and ensuring temporal coherence of generated video frames are key evaluation criteria (Xing et al., ; Zhang et al., ).
With the recent progress in video synthesis (Brooks et al., ; Yang et al., ; Blattmann et al., ; Bao et al., ), several diffusion-based I2V frameworks have been proposed, with novel designs on network architecture (Xing et al., ; Zhang et al., ; Chen et al., ; Ren et al., ; Lu et al., ), cascaded framework (Jain et al., ; Zhang et al., ), and motion representation (Zhang et al., ; Ni et al., ).
However, although these methods have demonstrated the potential of diffusion models (Ho et al., ; Song et al., ) in I2V synthesis, restricted by their noise-to-data generation process, they inevitably suffer the difficulty to generate video samples required by both appearance consistency and temporal coherence from uninformative random noise.
With the noise-to-data sampling trajectory which is inherently mismatched with the frame-to-frames synthesis process of I2V task, previous diffusion-based methods increase the burden of generative models, which may result in limited synthesis quality.

Figure: Figure 1: Overview of FrameBridge and diffusion-based I2V models. The sampling process of FrameBridge (upper) starts from the informative given image, while diffusion models (lower) synthesize videos from uninformative noisy representation.
Refer to caption: x1.png

In this work, we present FrameBridge, a novel I2V framework to model the frame-to-frames synthesis process with recently proposed data-to-data generative framework (; Liu et al., ; Chen et al., ).
Specifically, given the input image and video target, we first leverage variational auto-encoder (VAE) based compression network to transform them into continuous latent representations, and then take their latent representations as boundary distributions, \ie, prior and target, to establish our data-to-data generative framework.
Considering the static image has already been an informative prior for each of the consecutive frames in video target, we naturally replicate it to obtain the prior of the whole video clip, constructing the frames-to-frames pairs for the prior-to-target generation process in FrameBridge.
Standing on constructed frames-to-frames pairs, we establish bridge models (Tong et al., ; Zhou et al., ; Chen et al., ) between them to learn the I2V synthesis with Stochastic Differential Equation (SDE) based generation process, implicitly modelling the image animation with SDE-based generative models.
In comparison with previous diffusion works, our FrameBridge utilizes given static image as the prior of video target, which is advantageous on preserving the appearance details of input image than conditionally generating video samples from random noise.
Moreover, our frames-to-frames bridge model naturally learns image animation in model training rather than learning the image-conditioned noise-to-video generation. The improved consistency between generative framework and I2V task, \ie, data-to-data for frame-to-frames, tends to benefit temporal coherence for I2V synthesis.

In practice, I2V systems usually leverage the potential of a pre-trained diffusion-based text-to-video (T2V) model (Xing et al., ; Chen et al., ; Ma et al., ) with a fine-tuning process, to reduce the requirements of image-video data pairs and the computational resources at training stage. Therefore, in the application of FrameBridge, we aim to efficiently fine-tune a pre-trained diffusion-based T2V model to our proposed bridge-based I2V model.
Toward this target, we propose SNR-Aligned Fine-tuning (SAF), which aligns the two frameworks, i.e., diffusion models and bridge models, by adjusting the bridge process. Specifically, we first reparameterize the bridge process in FrameBridge, enabling the noisy intermediate latents of our frames-to-frames process to be aligned with the ones in the noise-to-data process of pre-trained diffusion models.
Then, we change the timestep to match the signal-to-noise (SNR) ratio between the input of the bridge model and the pre-trained diffusion model.
SAF avoids the mismatch between the two different generative frameworks at pre-training and fine-tuning stage, and therefore improves the synthesis quality of FrameBridge when adapting pre-trained T2V diffusion models.

Compared to diffusion models using Gaussian prior, FrameBridge takes static image as prior to improve I2V performance. A natural question to our proposed method is if we could further improve the I2V quality with a stronger prior. We answer this question by further exploiting the prior of our method. Given a static image, we design a one-step mapping-based network and optimize it with the video target, extracting a neural prior from the image for the video target. Compared to input image, this neural prior reduces the distance between prior and video target to a greater extent, and alleviates the burden of generative models further.
Although more advanced methods can be leveraged to extract more informative neural prior, we empirically find that a coarse estimation for video target at the cost of a single sampling step has already been beneficial to FrameBridge.
This further verifies our motivation to present FrameBridge and shows a novel method to enhance bridge-based I2V models.
In this work, our contributions can be summarized as follows:

- •
We propose FrameBridge, making the first attempt to model the frame-to-frames generation task of I2V with a data-to-data generative framework.
- •
We present two novel techniques, SAF and neural prior, further improving the performance of FrameBridge when fine-tuning from pre-trained T2V diffusion models and training from scratch respectively.
- •
We conduct experiments on two I2V benchmarks by training FrameBridge on WebVid-2M (Bain et al., ) and UCF-101 (Soomro, ). FrameBridge fine-tuned with SAF reduces the zero-shot FVD (Unterthiner et al. (); lower is better) from 176 to 83 on MSR-VTT (Xu et al., ), and FrameBridge with neural prior trained from scratch reduces the non-zero-shot FVD from 171 to 122 on UCF-101, highlighting the superiority of FrameBridge to their diffusion counterparts and the effectiveness of SAF and neural prior.

## 2 Related Works

##### Diffusion-based I2V Generation

I2V generation (Blattmann et al., ; Chen et al., ; Li et al., ) aims to synthesize videos based on the content of a given image and additional conditions (\eg, textual descriptions). Recently, diffusion-based I2V methods have achieved remarkable results. Several approaches develop a multi-stage diffusion framework, where intermediate videos generated in earlier stages guide the subsequent generation process (Jain et al., ; Zhang et al., ; Shi et al., ). To enhance the temporal consistency and improve the controllability of video content (Wang et al., ; Ren et al., ), recent works have proposed carefully designed network architectures (Xing et al., ; Ma et al., ; Chen et al., ; Ren et al., ) or specific components (Guo et al., ; Zhang et al., ) to better integrate image conditions into the backbone of video diffusion models. However, the process and frameworks of diffusion-based I2V models has not been fully explored, which may exist a room for improvement on I2V synthesis quality.

##### Noise Manipulation for Video Diffusion Models

Several works have explored to improve the uninformative prior distribution of diffusion models. PYoCo (Ge et al., ) recently proposes to use correlated noise for each frame in both training and inference. ConsistI2V (Ren et al., ), FreeInit (Wu et al., ), and CIL (Zhao et al., ) present training-free strategies to better align the training and inference distribution of diffusion prior. These strategies focus on improving the noise distribution to enhance the quality of synthesized videos, while they still suffer the restriction of noise-to-data diffusion framework, which may limit their endeavor to utilize the entire information (\eg, both large-scale features and fine-grained details) contained in the given image. In contrast, we propose a data-to-data framework and utilize deterministic prior rather than Gaussian noise, allowing us to leverage the clean input image as prior information.

##### Bridge Models

Recently, bridge models (; Tong et al., ; Liu et al., ; Zhou et al., ; Chen et al., ), which overcome the restriction of Gaussian prior in diffusion models, have gained increasing attention. They have demonstrated the advantages of data-to-data generation process over the noise-to-data one on image-to-image translation (Liu et al., ; Zhou et al., ) and text-to-speech synthesis (Chen et al., ) tasks. In this work, we make the first attempt to extend bridge models to I2V synthesis and further propose two improving techniques for bridge models, enabling efficient fine-tuning from diffusion models and stronger prior for video target.

## 3 Background

##### Problem Formulation

I2V aims at generating an video clip ${\mathbf{v}}\in\mathbb{R}^{L\times H\times W\times 3}$ with a number of $L$ frames from a static image, \eg, the initial frame $v^{i}\in\mathbb{R}^{H\times W\times 3}$ of video clip ${\mathbf{v}}$.
In I2V systems (Xing et al., ; Blattmann et al., ), an VAE-based compression network is usually leveraged to first transform the video ${\mathbf{v}}$ into a latent ${\mathbf{z}}\in\mathbb{R}^{L\times h\times w\times d}$ in a per-frame manner with a pre-trained image encoder $\mathcal{E}({\mathbf{v}})$, where $h=\frac{H}{p}$, $w=\frac{W}{p}$, $p>1$ and $d$ are the spatial compression ratio and the number of output channels.
Then, generative models are designed in this compressed space to learn the conditional distribution $p_{{\mathbf{z}}}({\mathbf{z}}|z^{i},c)$, where $z^{i}\in\mathbb{R}^{h\times w\times d}$ is the compressed latent of the initial frame $v^{i}$ and $c$ denotes other guidance such as the text prompt (Ma et al., ; Chen et al., ) or the image class condition (Ni et al., ; Zhang et al., ).
In sampling, we first synthesize the latent ${\mathbf{z}}$ conditioned on the latent of given static frame $z^{i}$, and then decode the video clip with pre-trained VAE decoder $\mathcal{D}({\mathbf{z}})$. As formulated, I2V synthesis actually seeks to generate consecutive frames in ${\mathbf{v}}$ from a single frame $v^{i}$, \ie, a frame-to-frames generation process where the appearance details of $v^{i}$ should be preserved in generated ${\mathbf{v}}$ and the animation in ${\mathbf{v}}$ should start from $v^{i}$ and remain temporal coherent.

##### Diffusion-based I2V Synthesis

Diffusion models (Ho et al., ; Sohl-Dickstein et al., ) have been popularly leveraged to learn the conditional distribution $p_{{\mathbf{z}}}({\mathbf{z}}|z^{i},c)$. These models are composed of two processes. A forward process gradually converts the video latent $p_{0}({\mathbf{z}}_{0}|z^{i},c)\triangleq p_{data}({\mathbf{z}}_{0}|z^{i},c)$ to a known prior distribution $p_{T,diff}({\mathbf{z}}_{T})\triangleq p_{prior,diff}({\mathbf{z}}_{T})$ with a forward SDE (Song et al., ):

$$ $\mathrm{d}{\mathbf{z}}_{t}={\bm{f}}(t){\mathbf{z}}_{t}\mathrm{d}t+g(t)\mathrm{ d}\mathbf{w},\quad{\mathbf{z}}_{0}\sim p_{data}({\mathbf{z}}_{0}|z^{i},c),$ (1) $$

where $\mathbf{w}$ is a Wiener process, ${\bm{f}}:\mathbb{R}^{D}\times[0,T]\xrightarrow{}\mathbb{R}^{D}$ is the drift coefficient, and $g:[0,T]\xrightarrow{}\mathbb{R}$ is the diffusion coefficient. $D$ represents the dimension of data ${\mathbf{z}}_{0}$.
The marginal distribution of ${\mathbf{z}}_{t}$ satisfies $p_{t,diff}({\mathbf{z}}_{t}|z^{i},c)=\int_{{\mathbf{z}}_{0}\sim p_{0}({\mathbf
{z}}_{0}|z^{i},c)}p_{t,diff}({\mathbf{z}}_{t}|{\mathbf{z}}_{0})p_{0}({\mathbf{
z}}_{0}|z^{i},c)\mathrm{d}{\mathbf{z}}_{0}$. Here $p_{t,diff}({\mathbf{z}}_{t}|{\mathbf{z}}_{0})=\mathcal{N}(\alpha_{t}{\mathbf{z
}}_{0},\sigma_{t}^{2}I),\alpha_{t}=e^{\int_{0}^{t}f(\tau)d\tau}$, $\sigma_{t}^{2}=\alpha_{t}^{2}\int_{0}^{t}\frac{g(\tau)^{2}}{\alpha_{\tau}^{2}}
\mathrm{d}\tau$ (Kingma et al., ).

Given the forward process defined by eq. 1, there exists a reverse process with a backward SDE which shares the same marginal distribution $p_{t,diff}({\mathbf{z}}_{t}|z^{i},c)$ (Song et al., ):

$$ $\mathrm{d}{\mathbf{z}}_{t}=\left[{\bm{f}}(t){\mathbf{z}}_{t}-g(t)^{2}\nabla_{{ \mathbf{z}}_{t}}\log p_{t,diff}({\mathbf{z}}_{t}|z^{i},c)\right]\mathrm{d}t+g( t)\mathrm{d}\bar{\mathbf{w}},\quad{\mathbf{z}}_{T}\sim p_{prior,diff}({\mathbf {z}}_{T}),$ (2) $$

where $\bar{\mathbf{w}}$ is a backward Wiener process.

To learn the unknown term in eq. 2, \ie, the score function $\nabla_{{\mathbf{z}}_{t}}\log p_{t,diff}({\mathbf{z}}_{t}|z^{i},c)$, usually a U-Net (Ronneberger et al., ; Ho et al., ) or DiT (Peebles & Xie, ; Bao et al., ) based neural network is optimized with a denoising objective to predict the noise:

$$ $\mathcal{L}_{diff}(\theta)=\mathbb{E}_{({\mathbf{z}}_{0},z^{i},c)\sim p_{data} ({\mathbf{z}}_{0},z^{i},c),t\sim q(t),{\mathbf{z}}_{t}\sim p_{t,diff}({\mathbf {z}}_{t}|{\mathbf{z}}_{0})}\left[\lambda(t)\left\lVert\bm{\epsilon}_{\theta}({ \mathbf{z}}_{t},t,z^{i},c)-\frac{{\mathbf{z}}_{t}-\alpha_{t}{\mathbf{z}}_{0}}{ \sigma_{t}}\right\rVert^{2}\right],$ (3) $$

where $q(t)$ is a distribution of $t$ supporting on $[0,T]$, $\lambda(t)$ is a time-dependent weight function, and $\bm{\epsilon}_{\theta}({\mathbf{z}}_{t},t)$ is an alternative parameterization method of the score function (Ho et al., ).

##### Limitations

As shown, the forward process of diffusion models gradually injects noise into data samples, which results in a boundary distribution at $t=T$ sharing the same distribution with the injected noise, \eg, the standard Gaussian noise $\bm{\epsilon}\sim\mathcal{N}(\bm{0},\bm{I})$.
Therefore, in generation, their sampling process has to start from the uninformative prior distribution $p_{prior,diff}({\mathbf{z}}_{T})\sim\mathcal{N}(\bm{0},\bm{I})$ and then iteratively synthesize the video latent ${\mathbf{z}}_{0}$ with learned conditional score function $\nabla_{{\mathbf{z}}_{t}}\log p_{t}({\mathbf{z}}_{t}|z^{i},c)$.
Formulating a frame-to-frames generation task with a conditional noise-to-data sampling process, diffusion-based I2V systems suffer the difficulties to generate high-quality samples from uninformative Gaussian noise while preserving the appearance details with condition and keeping the temporal coherence, which may result in limited I2V performance.

## 4 Method

Figure: Figure 2: Visualization for the mean value of marginal distributions. We visualize the decoded mean value $D(\mu_{t}(\cdot))$ of bridge process and diffusion process, where $D$ is the pre-trained VAE decoder. As shown, the prior and target of FrameBridge are naturally fitted with I2V synthesis. The data information is preserved along bridge process, while it gradually vanishes in the forward diffusion process.
Refer to caption: x2.png

In this section, we analyze our motivation to build bridge-based I2V system, present our FrameBridge, and elaborate our proposed SNR-Aligned Fine-tuning (SAF) and neural prior to further improve bridge-based I2V synthesis quality.

### 4.1 Motivation

As discussed above, on one hand, the static image in I2V generation has already provided strong condition information, \eg, the appearance details which should be preserved in generated video. However, the prior of diffusion models ignore this information, rendering diffusion-based I2V systems regenerate entire video information from Gaussian noise.
On the other hand, the image animation process should be temporal coherent, ensuring the consistency between video frames. However, the prior of diffusion models makes previous works generate each frame from random noise. Although the generation is conditioned on static image, it would be difficult for a noise-to-data generation process to guarantee the temporal coherence between generated frames.

Toward reducing the mismatch between the generation process and the I2V task, we make the first attempt to establish data-to-data process, \ie, bridge models (Chen et al., ; Zhou et al., ), for the frame-to-frames synthesis of I2V. Taking the static image as the prior of consecutive frames in video, we build a bridge between the initial frame and the following ones. In generation, each video frame is generated from the initial frame, which is naturally helpful to preserve appearance details in I2V synthesis. Moreover, as our model learns the frame-to-frames synthesis instead of conditional noise-to-frames generation, we facilitate our model to focus on learning image animation, which benefits temporal coherence of generated video.

### 4.2 FrameBridge

Considering the given image, \ie, initial frame $z^{i}$, has provided the appearance details and the starting point of animation for video target, we take it as the prior of following frames. To construct the boundary distributions for bridge models, we replicate the image latent $z^{i}$ for $L$ times along temporal axis to obtain ${\mathbf{z}}^{i}\in\mathbb{R}^{L\times h\times w\times d}$ as the prior of video latent ${\mathbf{z}}\in\mathbb{R}^{L\times h\times w\times d}$, and establish the bridge process as follows.

Figure: Figure 3: SNR-Aligned Fine-tuning for FrameBridge. (a) SAF technique aligns the marginal distributions of FrameBridge with that of pre-trained Gaussian diffusion models, to improve fine-tuning efficiency. (b) A case study showing FrameBridge with SAF can better leverage the capability of pre-trained models.
Refer to caption: x3.png

##### Bridge Process

In Figure 1, we present the overview of FrameBridge and compare it with diffusion-based I2V generation. Different from diffusion-based I2V models using uninformative Gaussian prior, our FrameBridge replaces the Gaussian prior with a Dirac prior $\delta_{{\mathbf{z}}^{i}}$, building a bridge process (Zhou et al., ) to connect the video target and the replicated image prior $p_{prior,bridge}({\mathbf{z}}_{T}|i,c)\triangleq\delta_{{\mathbf{z}}^{i}}({
\mathbf{z}}_{T})$. Specifically, the forward process is changed from eq. 1 in diffusion models to:

$$ $\mathrm{d}{\mathbf{z}}_{t}=\left[{\bm{f}}(t){\mathbf{z}}_{t}+g(t)^{2}{\bm{h}}( {\mathbf{z}}_{t},t,{\mathbf{z}}_{T},z^{i},c)\right]\mathrm{d}t+g(t)\mathrm{d}{ \bm{w}},\quad{\mathbf{z}}_{0}\sim p_{data}({\mathbf{z}}_{0}|z^{i},c),\quad{ \mathbf{z}}_{T}={\mathbf{z}}^{i},$ (4) $$

where ${\bm{h}}({\mathbf{z}}_{t},t,{\mathbf{z}}_{T},z^{i},c)\triangleq\nabla_{{
\mathbf{z}}_{t}}\log p_{T,diff}({\mathbf{z}}_{T}|{\mathbf{z}}_{t})$ and $p_{T,diff}({\mathbf{z}}_{T}|{\mathbf{z}}_{t})$ is the marginal distribution of diffusion process shown in eq. 1. For bridge process, we denote the marginal distribution of eq. 4 as $p_{t,bridge}({\mathbf{z}}_{t}|z^{i},c)$. Similar to the forward SDE eq. 1 in diffusion process, the forward process of bridge models eq. 4 also has a reverse process, which shares the same marginal distribution $p_{t,bridge}({\mathbf{z}}_{t}|z^{i},c)$ and can be represented by the backward SDE:

$$ $\displaystyle\mathrm{d}{\mathbf{z}}_{t}=\left[{\bm{f}}(t){\mathbf{z}}_{t}-g(t) ^{2}({\bm{s}}({\mathbf{z}}_{t},t,{\mathbf{z}}_{T},z^{i},c)-{\bm{h}}({\mathbf{z }}_{t},t,{\mathbf{z}}_{T},z^{i},c))\right]\mathrm{d}t+g(t)\mathrm{d}\bar{{\bm{ w}}},\quad{\mathbf{z}}_{T}={\mathbf{z}}^{i},$ (5) $$

where ${\bm{s}}({\mathbf{z}}_{t},t,{\mathbf{z}}_{T},z^{i},c)\triangleq\nabla_{{
\mathbf{z}}_{t}}\log p_{t,bridge}({\mathbf{z}}_{t}|{\mathbf{z}}_{T},z^{i},c)$.
The change from diffusion process to bridge process remove the restriction of noisy prior in diffusion models, allowing the generation process to start from static image rather than previous Gaussian noise. Moreover, as the perturbation kernel $p_{t,bridge}({\mathbf{z}}_{t}|{\mathbf{z}}_{0},{\mathbf{z}}_{T},z^{i},c)$ in bridge process remains Gaussian (Appendix A), it facilitates us to find connections between the marginal distribution, \ie, the intermediate representations of diffusion and bridge process, and then leverage the power of pre-trained diffusion models for bridge models.

##### Training Objective

Analogous to diffusion models, we use a SDE solver to solve eq. 5 when sampling videos. Since
${\bm{h}}({\mathbf{z}}_{t},t,{\mathbf{z}}_{T},z^{i},c)$ can be calculated analytically (see Appendix A), we only need to estimate the unknown term ${\bm{s}}({\mathbf{z}}_{t},t,{\mathbf{z}}_{T},z^{i},c)$ with neural networks (Kingma et al., ). After parameterization (more details can be found in Appendix A), we train FrameBridge models $\bm{{\epsilon}}_{\theta}^{\hat{\Psi}}({\mathbf{z}}_{t},t,{\mathbf{z}}_{T},z^{i
},c)$ with the following denoising objective (Chen et al., ):

$$ $\mathcal{L}_{bridge}(\theta)=\mathbb{E}_{\begin{subarray}{c}({\mathbf{z}}_{0}, z^{i},c)\sim p_{data}({\mathbf{z}}_{0},z^{i},c),\\ {\mathbf{z}}_{T}={\mathbf{z}}^{i},t,{\mathbf{z}}_{t}\sim p_{t,bridge}({\mathbf {z}}_{t}|{\mathbf{z}}_{0},{\mathbf{z}}_{T},z^{i},c)\end{subarray}}\left[\left \lVert\bm{{\epsilon}}_{\theta}^{\hat{\Psi}}({\mathbf{z}}_{t},t,{\mathbf{z}}_{T },z^{i},c)-\frac{{\mathbf{z}}_{t}-\alpha_{t}{\mathbf{z}}_{0}}{\sigma_{t}} \right\rVert^{2}\right].$ (6) $$

The training of FrameBridge resembles that of Gaussian diffusion-based I2V models: We first sample a video latent ${\mathbf{z}}_{0}$ and the condition $c$ from training set, extracting the first frame of ${\mathbf{z}}_{0}$ to construct ${\mathbf{z}}^{i}$. The primary difference lies in the Gaussian perturbation kernel $p_{t,bridge}({\mathbf{z}}_{t}|{\mathbf{z}}_{0},{\mathbf{z}}_{T},z^{i},c)$ of eq. 6. As we replace the Gaussian prior with a deterministic data point ${\mathbf{z}}_{T}$, the mean value is an interpolation between data and ${\mathbf{z}}_{T}$ instead of the decaying data in diffusion models, naturally preserving more data information and facilitating generative models to learn image animation rather than regenerating the information provided in static image.

##### Bridge Process vs Diffusion Process

To demonstrate the advantages of bridge process in I2V synthesis, we visualize the data part, \ie, the mean function of bridge and diffusion process, in Figure 2.
As shown, when replicating the initial frame, I2V synthesis can be formulated as a frames-to-frames generation task.
With the data-to-data bridge process, the boundary distributions of our FrameBridge, \ie, the prior and the target, have been an ideal fit for the I2V task, which is helpful for generative models to focus on modelling the image animation process.
In the meanwhile, as seen from our intermediate representations, the data information, \eg, appearance details, is well preserved during the bridge process.
In comparison, the prior and intermediate representations of diffusion process contain rare or coarse information of the target, which is uninformative and requires diffusion models to generate entire video information from scratch conditioned on static image.

### 4.3 SNR-Aligned Fine-tuning

To implement FrameBridge, one straightforward strategy is to train bridge models from scratch using eq. 6. Meanwhile, another common practice of training I2V models is to fine-tune from pre-trained T2V diffusion models (Chen et al., ; ; Xing et al., ; Blattmann et al., ; Ma et al., ). A key challenge arises as the marginal distribution of bridge models $p_{t,bridge}({\mathbf{z}}_{t})$ differs from that of diffusion models $p_{t,diff}({\mathbf{z}}_{t})$, limiting the generation performance of fine-tuned FrameBridge as illustrated in Figure 3. To address this issue, we propose the innovative SNR-Aligned Fine-tuning (SAF) technique. By bridging the gap between the two distributions during fine-tuning, SAF enables more seamless knowledge transfer between the two model families. To the best of our knowledge, this is the first attempt to explore how to effectively fine-tune bridge models from pre-trained diffusion models. Our SAF technique consists of the following two steps:

##### Reparameterization of Bridge Process.

In bridge process, the perturbed latent ${\mathbf{z}}_{t}$ at timestep $t$ can be written as the linear combination of ${\mathbf{z}}_{0}$, ${\mathbf{z}}_{T}$ and a Gaussian noise $\bm{{\epsilon}}$: ${\mathbf{z}}_{t}=a_{t}{\mathbf{z}}_{0}+b_{t}{\mathbf{z}}_{T}+c_{t}\bm{{
\epsilon}}$ (detailed expression of $a_{t},b_{t},c_{t}$ can be found in eq. 13), which is different from $\alpha_{t}{\mathbf{z}}_{0}+\sigma_{t}\bm{{\epsilon}}$(^1^11Without loss of generality, we consider Variance-Preserving (VP) diffusion where $\alpha_{t}^{2}+\sigma_{t}^{2}=1$) in diffusion models due to the change of prior and forward process. Therefore, the pre-trained diffusion models have limited ability to directly denoise such a ${\mathbf{z}}_{t}$, which impairs effective fine-tuning. To match the distributions of ${\mathbf{z}}_{t}$, we reparameterize the bridge process by

$$ $\tilde{{\mathbf{z}}}_{t}=\frac{{\mathbf{z}}_{t}-b_{t}{\mathbf{z}}_{T}}{\sqrt{a _{t}^{2}+c_{t}^{2}}}=\frac{a_{t}}{\sqrt{a_{t}^{2}+c_{t}^{2}}}{\mathbf{z}}_{0}+ \frac{c_{t}}{\sqrt{a_{t}^{2}+c_{t}^{2}}}\bm{{\epsilon}}.$ (7) $$

Then, $\tilde{{\mathbf{z}}}_{t}$ can be represented as the combination of clean data ${\mathbf{z}}_{0}$ and a Gaussian noise, with the squre sum of coefficients equal to 1. Thus, the reparameterized bridge process $\tilde{{\mathbf{z}}}_{t}$ exactly align with a VP diffusion process,

##### SNR-based Latent Alignment

Although $\tilde{{\mathbf{z}}}_{t}$ has the same perturbation kernel as a diffusion process, the marginal distribution of $\tilde{{\mathbf{z}}}_{t}$ does not match that of the pre-trained diffusion at timestep $t$, \ie, $\alpha_{t}{\mathbf{z}}_{0}+\sigma_{t}{\epsilon}$, hindering bridge models to seamlessly leverage the knowledge of pre-trained diffusion models (see Figure 3). Therefore, we leverage signal-to-noise ratio (SNR) as an indicator to unify the two marginal distributions (Kingma et al., ). Specifically, we change the timestep $t$ to another $\tilde{t}$ such that $\alpha_{\tilde{t}}=\frac{a_{t}}{\sqrt{a_{t}^{2}+c_{t}^{2}}}$, $\sigma_{\tilde{t}}=\frac{c_{t}}{\sqrt{a_{t}^{2}+c_{t}^{2}}}$, and then $\tilde{{\mathbf{z}}}_{t}$ has the same SNR as $\alpha_{\tilde{t}}{\mathbf{z}}_{0}+\sigma_{\tilde{t}}\bm{{\epsilon}}$ in diffusion process. According to the above derivation, we reparameterize the input of bridge models as $\bm{{\epsilon}}_{\theta,bridge}^{\hat{\Psi}}({\mathbf{z}}_{t},t,i,c)\triangleq
\bm{{\epsilon}}_{\theta,aligned}^{\hat{\Psi}}(\tilde{{\mathbf{z}}}_{t},\tilde{
t},i,c)$, and initialize $\bm{{\epsilon}}_{\theta,aligned}^{\hat{\Psi}}$ with the pre-trained T2V diffusion models. Since the marginal distribution of $\tilde{{\mathbf{z}}}_{t}$ is complemetely aligned with the marginal of diffusion process at timestep $\tilde{t}$, SAF enables bridge models to fully exploit the denoising capability of pre-trained diffusion models. We provide a general statement of SAF technique and more details in Appendix A.

### 4.4 Neural Prior

Figure: Figure 4: Case of neural prior. The neural prior provides more motion information than a static image, which has been closer to the reference video in dataset.
Refer to caption: x5.png

By establishing a data-to-data process for I2V synthesis, we have been able to reduce the distance between the prior and the target from noise-to-frames to frames-to-frames, and therefore reduce the burden of generative models and aim at improving synthesis quality. To further demonstrate the function of improving prior information for I2V synthesis, we extend our design of FrameBridge from replicated initial frame ${\mathbf{z}}^{i}$ to neural representations $F_{\eta}(z^{i},c)$, which serves as a stronger prior for video frames.

As shown in Figure 4, although the static frame has provided indicative information such as the appearance details of the background and different objects, it may not be informative for the motion information in consecutive frames. When the distance between the prior frame and the target frame is large, bridge models are faced with the challenge to generate the motion trajectory.
Therefore, we present a stronger prior than simply duplicating the initial frame, neural prior, which achieves a coarse estimation of the target at first, and then bridge models generate the high-quality target from this coarse estimation.

**Table 1: Zero-shot I2V generation on UCF-101 and MSR-VTT (256 $\times$ 256, 16 frames). w/o SAF means FrameBridge without SAF techniques when fine-tuning. Results marked by ^∗ are reported in Xing et al. ().**
| Method | UCF-101 | MSR-VTT |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| FVD $\downarrow$ | IS $\uparrow$ | PIC $\uparrow$ | FVD $\downarrow$ | CLIPSIM $\uparrow$ | PIC $\uparrow$ |  |
| DynamiCrafter (Reproduce) | 503 | 28.85 | 0.5506 | 265 | 0.2240 | 0.5093 |
| DynamiCrafter^∗ | 429 | – | 0.6078 | 234 | – | 0.5803 |
| SEINE | 461 | 22.32 | 0.6665 | 176 | 0.2250 | 0.4217 |
| FrameBridge (Ours, w/o SAF) | 433 | 28.61 | 0.5989 | 228 | 0.2246 | 0.5559 |
| FrameBridge (Ours, w/ SAF) | 283 | 42.85 | 0.7101 | 83 | 0.2249 | 0.7185 |

Figure: Figure 5: Qualitative comparisons between FrameBridge and other baselines. FrameBridge outperforms other diffusion-based methods in appearance consistency and video quality.
Refer to caption: x6.png

Considering bridge models synthesize target data with iterative sampling steps, we develop a one-step mapping-based prior network taking both image latent $z^{i}$ and text or label condition $c$ as input, and separately train the prior network with a regression loss in latent space:

$$ $\mathcal{L}_{prior}(\eta)=\mathbb{E}_{({\mathbf{z}},z^{i},c)\sim p_{data}({ \mathbf{z}},z^{i},c)}\left[\left\lVert F_{\eta}(z^{i},c)-{\mathbf{z}}\right \rVert^{2}\right].$ (8) $$

With this objective, it can be proved that $F_{\eta}(z^{i},c)$ learns to predict the mean value of subsequent frames, as shown in Appendix A). Given pre-trained $F_{\eta}(z^{i},c)$, we build FrameBridge-NP from its output and target video latent ${\mathbf{z}}$:

$$ $\mathcal{L}_{bridge-NP}(\theta)=\mathbb{E}_{\begin{subarray}{c}({\mathbf{z}}_{ 0},z^{i},c)\sim p_{data}({\mathbf{z}}_{0},z^{i},c),t,\\ {\mathbf{z}}_{T}=F_{\eta}(z^{i},c),{\mathbf{z}}_{t}\sim p_{t,bridge}({\mathbf{ z}}_{t}|{\mathbf{z}}_{0},{\mathbf{z}}_{T},z^{i},c)\end{subarray}}\left[\left \lVert\bm{{\epsilon}}_{\theta}^{\hat{\Psi}}({\mathbf{z}}_{t},t,{\mathbf{z}}_{T },z^{i},c)-\frac{{\mathbf{z}}_{t}-\alpha_{t}{\mathbf{z}}_{0}}{\sigma_{t}} \right\rVert^{2}\right].$ (9) $$

In generation, neural prior model $F_{\eta}(z^{i},c)$ provide a coarse estimation with a single deterministic step, which is closer to the target than the provided initial frame, and bridge model synthesize the video target with a coarse-to-fine iterative sampling process. Although more advanced methods can be designed to further improve neural prior, we present a design with simple training objective and one-step sampling, demonstrating the performance of enhancing prior information on I2V synthesis.

## 5 Experiments

We carry out extensive experiments on UCF-101 (Soomro, ) and WebVid-2M (Bain et al., ) datasets to compare FrameBridge with other baselines, demonstrating the advantages of our data-to-data generation framework for I2V tasks.

### 5.1 Setups

We conduct experiments on class-conditional I2V generation and text-conditional I2V generation tasks, where the condition $c$ is a class label or textual description of the video content.

##### Class-conditional I2V Generation

For class-conditional I2V generation, we train FrameBridge and other baselines on UCF-101 dataset. Fréchet Video Distance (Unterthiner et al. (); FVD) and Inception Score (Saito et al. (); IS) are used to evaluate the quality of generated videos. Meanwhile, we use Perceptual Input Conformity (PIC) (Xing et al., ) to evaluate the consistency of the synthesized frames with given image conditions.

##### Text-conditional I2V Generation

For text-conditional I2V generation, both FrameBridge and baselines are fine-tuned from pre-trained T2V diffusion models on WebVid-2M dataset. We use zero-shot FVD, IS, PIC on UCF-101 and zero-shot FVD, CLIPSIM (Wu et al., ), PIC on MSR-VTT (Xu et al., ) to evalaute the quality and consistency with image conditions for generated videos. More details about datasets and metrics can be found in Appendix C.

### 5.2 Fine-tuning from pre-trained diffusion models

**Table 2: Non-zero-shot I2V generation on UCF-101. Different from zero-shot metrics, here all models are trained on UCF-101 dataset.**
| Method | FVD $\downarrow$ | IS $\uparrow$ | PIC $\uparrow$ |
| --- | --- | --- | --- |
| ExtDM | 649 | 21.37 | 0.9651 |
| VDT-I2V | 171 | 62.61 | 0.7401 |
| FrameBridge | 154 | 64.01 | 0.7443 |
| FrameBridge-NP | 122 | 63.60 | 0.7662 |

##### Implementation Details

Following (Xing et al., ), we fine-tune FrameBridge with the replicated prior ${\mathbf{z}}^{i}$ from the open-sourced T2V diffusion model VideoCrafter1 (Chen et al., ). When fine-tuning, we apply SAF technique proposed in Section 4.3 and initialize the network after alignment $\bm{{\epsilon}}_{\theta,aligned}^{\hat{\Psi}}({\mathbf{z}}_{t},t,z^{i},c)$ with the weights of VideoCrafter1 at 256 $\times$ 256 resolutions. We fine-tune FrameBridge and baselines for 20k iterations with batch size 64. All synthesized videos are sampled through the first-order SDE solvers with 250 steps (Chen et al., ). More details can be found in Appendix C.

##### Comparison with Baselines

We reproduce DynamiCrafter (Xing et al., ) for text-conditional I2V as baselines. Table 1 shows zero-shot metrics on UCF-101 and MSR-VTT after fine-tuning on WebVid-2M. As demonstrated by quantitative results in the table, bridge-based I2V models can effectively leverage the knowledge from pre-trained T2V diffusion models and generate videos with higher quality and consistency than their diffusion counterparts. Moreover, it is indicated that SAF can further improve the performance of fine-tuned FrameBridge, showcasing the advantages of matching marginal distributions of the bridge process and diffusion process. Qualitative results are shown in Figure 5.

Both the quantitative and qualitative comparisons demonstrate the advantages of FrameBridge and the data-to-data generation farmework for I2V tasks. To the best of our knowledge, our trail is the first time to fine-tune diffusion bridge models from pre-trained diffusion models, and SAF is crucial to boosting the performance of fine-tuned bridge models. Moreover, we note that the zero-shot metrics on UCF-101 is better than those of DynamiCrafter fine-tuned with 100k iterations (FVD 429.23 and PIC 0.6078 as reported in Xing et al. ()), highlighting the efficiency of SAF.

### 5.3 Neural Prior for Bridge Models

##### Implementation Details

We implement FrameBridge-NP based on the class-conditional video diffusion model Latte-S/2 (Ma et al., ) by replacing diffusion process with the Bridge-gmax bridge process. We freeze the parameters $\eta$ of neural prior when training bridge models $\bm{{\epsilon}}_{\theta}^{\hat{\Psi}}({\mathbf{z}}_{t},t,{\mathbf{z}}_{T},z^{i
},c)$. All synthesized videos are sampled through the first-order SDE solvers with 250 steps. More details can be found in Appendix C.

##### Comparison with Baselines

We reproduce two diffusion models ExtDM (Zhang et al., ) and VDT (Lu et al., ) on UCF-101 dataset for the class-conditional I2V task as our baselines. Table 2 shows that FrameBridge-NP has superior video quality and consistency with condition images. (The PIC value of ExtDM is not comparable to other methods as its resolution 64 $\times$ 64 is much lower than others and it generate videos with slower motion.) More qualitative results are shown in Appendix D. The experiments reveal that bridge-based I2V models outperform their diffusion counterparts with both replicated prior and neural prior, justifying the usage of the data-to-data generation process for I2V tasks. Additionally, FrameBridge can further benefit from neural prior $F_{\eta}(z^{i},c)$ as it actually narrows the gap between the prior and data distribution of bridge process.

### 5.4 Ablation Studies

##### SNR-Aligned Fine-tuning

Table 2 has already shown the advantage of SAF technique, we use another ablation experiment to further elucidate that. We fine-tune a pre-trained class-conditional video generation model Latte-XL/2(^2^22https://huggingface.co/maxin-cn/Latte/tree/main) on UCF-101 with 20k iterations. The quantitative results presented in Table 4 shows that SAF improves generation performance of FrameBridge.

##### Neural Prior

To showcase the effectiveness of neural prior, we compare four different models: VDT-I2V (Gaussian priors, condition on $\{z^{i},c\}$), FrameBridge without neural prior (replicated prior, condition on $\{z^{i},c\}$), FrameBridge with neural prior only as the network condition (replicated prior, condition on $\{z^{i},c,F_{\eta}(z^{i},c)\}$, FrameBridge-NP (neural prior, condition on $\{z^{i},c,F_{\eta}(z^{i},c)\}$). More details of the configurations can be found in Appendix C. Results in Table 4 reveal that $F_{\eta}(z^{i},c)$ is indeed more informative than a single frame $z^{i}$ and can be fully utilized by FrameBridge through the change of prior.

**Table 3: Ablation of SAF technique on UCF-101. All models are fine-tuned from Latte-XL/2 class-conditional video diffusion model for 20k iterations. We use repeating regression for FrameBridge, and w/o SAF means fine-tuning without SAF technique.**
| Method | FVD $\downarrow$ | IS $\uparrow$ | PIC $\uparrow$ |
| --- | --- | --- | --- |
| VDT-I2V | 177 | 55.22 | 0.7700 |
| FrameBridge(w/o SAF) | 157 | 37.65 | 0.7498 |
| FrameBridge(w/ SAF) | 148 | 55.60 | 0.8198 |

## 6 Conclusions

In this work, we propose FrameBridge, building a data-to-data generation process for I2V synthesis, which matches the frame-to-frames nature of this task. Additionally, targeting at two typical scenarios of training I2V models, namely fine-tuning from pre-trained diffusion models and training from scratch, we present SNR-Aligned Fine-tuning (SAF) and neural prior respectively to further improve the generation quality of FrameBridge. Extensive experiments show that FrameBridge generate videos with enhanced appearance consistency with image condition and improved temporal coherence between frames, demonstrating the advantages of FrameBridge over diffusion-based I2V methods and the effectiveness of two proposed techniques.

## Appendix A Proof and Derivation

### A.1 Basics of Denoising Diffusion Bridge Model (DDBM)

We provide the derivations of $p_{t,bridge}({\mathbf{z}}_{t}|{\mathbf{z}}_{0},{\mathbf{z}}_{T},z^{i},c)$ and $h({\mathbf{z}},t,{\mathbf{y}},z^{i},c)$ used in Section 4.2.

Similar to the proofs in (Zhou et al., ), we calculate $p_{t,bridge}({\mathbf{z}}_{t}|{\mathbf{z}}_{0},{\mathbf{z}}_{T},z^{i},c)$ by applying Bayes’ rule:

$$ $\displaystyle p_{t,bridge}({\mathbf{z}}_{t}|{\mathbf{z}}_{0},{\mathbf{z}}_{T}, z^{i},c)=p_{t,diff}({\mathbf{z}}_{t}|{\mathbf{z}}_{0},{\mathbf{z}}_{T},z^{i},c)$ $\displaystyle=\frac{p_{T,diff}({\mathbf{z}}_{T}|{\mathbf{z}}_{t},{\mathbf{z}}_ {0},z^{i},c)p_{t,diff}({\mathbf{z}}_{t}|{\mathbf{z}}_{0},z^{i},c)}{p_{t,diff}( {\mathbf{z}}_{T}|{\mathbf{z}}_{0},z^{i},c)}$ (10) $\displaystyle\overset{\leavevmode\hbox to11.37pt{\vbox to11.37pt{\pgfpicture \makeatletter\hbox{\hskip 5.68657pt\lower-5.68657pt\hbox to0.0pt{ \pgfsys@beginscope\pgfsys@invoke{ }\definecolor{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0} {0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox to 0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{ {{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{{{}}}{{}}{}{}{}{}{}{}{ }{}{}{{}\pgfsys@moveto{5.48657pt}{0.0pt}\pgfsys@curveto{5.48657pt}{3.03018pt}{ 3.03018pt}{5.48657pt}{0.0pt}{5.48657pt}\pgfsys@curveto{-3.03018pt}{5.48657pt}{ -5.48657pt}{3.03018pt}{-5.48657pt}{0.0pt}\pgfsys@curveto{-5.48657pt}{-3.03018 pt}{-3.03018pt}{-5.48657pt}{0.0pt}{-5.48657pt}\pgfsys@curveto{3.03018pt}{-5.48 657pt}{5.48657pt}{-3.03018pt}{5.48657pt}{0.0pt}\pgfsys@closepath\pgfsys@moveto {0.0pt}{0.0pt}\pgfsys@stroke\pgfsys@invoke{ } }{{{{}}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@transformcm{1.0}{0.0}{0.0}{1 .0}{-2.5pt}{-3.22221pt}\pgfsys@invoke{ }\hbox{{\definecolor{pgfstrokecolor}{ rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ } \pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{1}} }}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{{{}}}{}{}\hss} \pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hss}} \lxSVG@closescope\endpgfpicture}}}{=}\frac{p_{T,diff}({\mathbf{z}}_{T}|{ \mathbf{z}}_{t})p_{t,diff}({\mathbf{z}}_{t}|{\mathbf{z}}_{0})}{p_{T,diff}({ \mathbf{z}}_{T}|{\mathbf{z}}_{0})}.$ $$

1 uses the Markovian of the diffusion process ${\mathbf{z}}_{t}$ (Kingma et al., ).

The perturbation kernels $p_{T,diff}({\mathbf{z}}_{T}|{\mathbf{z}}_{t}),p_{t,diff}({\mathbf{z}}_{t}|{
\mathbf{z}}_{0}),p_{T,diff}({\mathbf{z}}_{T}|{\mathbf{z}}_{0})$ is Gaussian and takes the form of:

$$ $\displaystyle p_{T,diff}({\mathbf{z}}_{T}|{\mathbf{z}}_{t})=\mathcal{N}({ \mathbf{z}}_{T};\frac{\alpha_{T}}{\alpha_{t}}{\mathbf{z}}_{t},(\sigma_{T}^{2}- \frac{\alpha_{T}^{2}}{\alpha_{t}^{2}}\sigma_{t}^{2})I),$ (11) $\displaystyle p_{t,diff}({\mathbf{z}}_{t}|{\mathbf{z}}_{0})=\mathcal{N}({ \mathbf{z}}_{t};\alpha_{t}{\mathbf{z}}_{0},\sigma_{t}^{2}I),$ $\displaystyle p_{T,diff}({\mathbf{z}}_{T}|{\mathbf{z}}_{0})=\mathcal{N}({ \mathbf{z}}_{T};\alpha_{T}{\mathbf{z}}_{0},\sigma_{T}^{2}I).$ $$

Following (Zhou et al., ), it can be derived that $p_{t,bridge}({\mathbf{z}}_{t}|{\mathbf{z}}_{T},{\mathbf{z}}_{0},z^{i},c)$ is also Gaussian, and $p_{t,bridge}({\mathbf{z}}_{t}|{\mathbf{z}}_{T},{\mathbf{z}}_{0},z^{i},c)=
\mathcal{N}({\mathbf{z}}_{t};\mu_{t}({\mathbf{z}}_{0},{\mathbf{z}}_{T}),\sigma
_{t,bridge}^{2}I)$, where

$$ $\displaystyle\mu_{t}({\mathbf{z}}_{0},{\mathbf{z}}_{T})=\alpha_{t}(1-\frac{ \mathrm{SNR}_{T}}{\mathrm{SNR}_{t}}){\mathbf{z}}_{0}+\frac{\mathrm{SNR}_{T}}{ \mathrm{SNR}_{t}}\frac{\alpha_{t}}{\alpha_{T}}{\mathbf{z}}_{T},$ (12) $\displaystyle\sigma_{t,bridge}^{2}=\sigma_{t}^{2}(1-\frac{\mathrm{SNR}_{T}}{ \mathrm{SNR}_{t}}).$ $$

Specifically, ${\mathbf{z}}_{t}$ of bridge process can be reparameterized by ${\mathbf{z}}_{t}=a_{t}{\mathbf{z}}_{0}+b_{t}{\mathbf{z}}_{T}+c_{t}\bm{{
\epsilon}}$, where

$$ $\displaystyle a_{t}$ $\displaystyle=\alpha_{t}(1-\frac{\mathrm{SNR}_{T}}{\mathrm{SNR}_{t}}),$ (13) $\displaystyle b_{t}$ $\displaystyle=\frac{\mathrm{SNR}_{T}}{\mathrm{SNR}_{t}}\frac{\alpha_{t}}{ \alpha_{T}},$ $\displaystyle c_{t}$ $\displaystyle=\sqrt{\sigma_{t}^{2}(1-\frac{\mathrm{SNR}_{T}}{\mathrm{SNR}_{t}} )}.$ $$

Here, $\mathrm{SNR_{t}}=\frac{\alpha_{t}^{2}}{\sigma_{t}^{2}}$ (Kingma et al., ) is the signal-to-noise ratio of diffusion process.

Then we calculate ${\bm{h}}({\mathbf{z}},t,{\mathbf{y}},z^{i},c)=\nabla_{{\mathbf{z}}_{t}}\log p_
{T,diff}({\mathbf{z}}_{T}|{\mathbf{z}}_{t})|_{{\mathbf{z}}_{t}={\mathbf{z}},{
\mathbf{z}}_{T}={\mathbf{y}}}$.

As $p_{T,diff}({\mathbf{z}}_{T}|{\mathbf{z}}_{t})=\mathcal{N}({\mathbf{z}}_{T};
\frac{\alpha_{T}}{\alpha_{t}}{\mathbf{z}}_{t},(\sigma_{T}^{2}-\frac{\alpha_{T}
^{2}}{\alpha_{t}^{2}}\sigma_{t}^{2})I)$, we have

$$ $p_{T,diff}({\mathbf{z}}_{T}|{\mathbf{z}}_{t})=\frac{1}{\sqrt{2\pi(\sigma_{T}^{ 2}-\frac{\alpha_{T}^{2}}{\alpha_{t}^{2}}\sigma_{t}^{2})}^{D}}\exp\left(-\frac{ \left\lVert{\mathbf{z}}_{T}-\frac{\alpha_{T}}{\alpha_{t}}{\mathbf{z}}_{t} \right\rVert^{2}}{2(\sigma_{T}^{2}-\frac{\alpha_{T}^{2}}{\alpha_{t}^{2}}\sigma _{t}^{2})}\right),$ (14) $$

$$ $\log p_{T,diff}({\mathbf{z}}_{T}|{\mathbf{z}}_{t})=-\frac{\left\lVert{\mathbf{ z}}_{T}-\frac{\alpha_{T}}{\alpha_{t}}{\mathbf{z}}_{t}\right\rVert^{2}}{2( \sigma_{T}^{2}-\frac{\alpha_{T}^{2}}{\alpha_{t}^{2}}\sigma_{t}^{2})}+C,$ (15) $$

where $C$ is a constant independent of ${\mathbf{z}}_{T}$.

$$ $\nabla_{{\mathbf{z}}_{t}}\log p_{T,diff}({\mathbf{z}}_{T}|{\mathbf{z}}_{t})= \nabla_{{\mathbf{z}}_{t}}\left(-\frac{\left\lVert{\mathbf{z}}_{T}-\frac{\alpha _{T}}{\alpha_{t}}{\mathbf{z}}_{t}\right\rVert^{2}}{2(\sigma_{T}^{2}-\frac{ \alpha_{T}^{2}}{\alpha_{t}^{2}}\sigma_{t}^{2})}\right)=-\frac{{\mathbf{z}}_{T} -\frac{\alpha_{T}}{\alpha_{t}}{\mathbf{z}}_{t}}{(\sigma_{T}^{2}-\frac{\alpha_{ T}^{2}}{\alpha_{t}^{2}}\sigma_{t}^{2})}.$ (16) $$

So, ${\bm{h}}({\mathbf{z}},t,{\mathbf{y}},z^{i},c)=-\frac{{\mathbf{y}}-\frac{\alpha
_{T}}{\alpha_{t}}{\mathbf{z}}}{(\sigma_{T}^{2}-\frac{\alpha_{T}^{2}}{\alpha_{t
}^{2}}\sigma_{t}^{2})}$.
Note that for the diffusion process we commonly use, $\frac{\alpha_{T}}{\alpha_{t}}\approx 0$ and $\sigma_{T}\approx 1$, and we have ${\bm{h}}({\mathbf{z}},t,{\mathbf{y}},z^{i},c)\approx-{\mathbf{y}}$.

### A.2 Parameterization of FrameBridge

###### Proposition 1 .

The score estimation ${\bm{s}}_{\theta}({\mathbf{z}}_{t},t,{\mathbf{z}}_{T},z^{i},c)$ of bridge process $p_{t,bridge}({\mathbf{z}}_{t}|{\mathbf{z}}_{T},z^{i},c)$ can be reparamterized by

$$ ${\bm{s}}_{\theta}({\mathbf{z}}_{t},t,{\mathbf{z}}_{T},z^{i},c)=-\frac{1}{ \sigma_{t}}\bm{\epsilon}_{\theta}^{\hat{\Psi}}({\mathbf{z}}_{t},t,{\mathbf{z}} _{T},z^{i},c)-\frac{\mathrm{SNR}_{T}}{\mathrm{SNR}_{t}}\frac{{\mathbf{z}}_{t}- \frac{\alpha_{t}}{\alpha_{T}}{\mathbf{z}}_{T}}{\sigma_{t}^{2}(1-\frac{\mathrm{ SNR}_{T}}{\mathrm{SNR}_{t}})},$ (17) $$

where $\mathrm{SNR}_{t}=\frac{\alpha_{t}^{2}}{\sigma_{t}^{2}}$, and $\bm{\epsilon}_{\theta}^{\hat{\Psi}}({\mathbf{z}}_{t},t,{\mathbf{z}}_{T},z^{i},c)$ is trained with the objective

$$ $\mathcal{L}_{bridge}(\theta)=\mathbb{E}_{\begin{subarray}{c}({\mathbf{z}}_{0}, z^{i},c)\sim p_{data}({\mathbf{z}}_{0},z^{i},c),\\ {\mathbf{z}}_{T}={\mathbf{z}}^{i},t,{\mathbf{z}}_{t}\sim p_{t,bridge}({\mathbf {z}}_{t}|{\mathbf{z}}_{0},{\mathbf{z}}_{T},z^{i},c)\end{subarray}}\left[\tilde {\lambda}(t)\left\lVert\bm{{\epsilon}}_{\theta}^{\hat{\Psi}}({\mathbf{z}}_{t}, t,{\mathbf{z}}_{T},z^{i},c)-\frac{{\mathbf{z}}_{t}-\alpha_{t}{\mathbf{z}}_{0}} {\sigma_{t}}\right\rVert^{2}\right].$ (18) $$

Here $\tilde{\lambda}(t)$ is the weight function of timestep $t$ and we take $\tilde{\lambda}(t)=1$ unless otherwise specified.

When $\mathrm{SNR_{T}}\approx 0$(which is often the case for diffusion process), there exists $\epsilon$ such that

$$ ${\bm{s}}_{\theta}({\mathbf{z}}_{t},t,{\mathbf{z}}_{T},z^{i},c)\approx-\frac{1} {\sigma_{t}}\bm{\epsilon}_{\theta}^{\hat{\Psi}}({\mathbf{z}}_{t},t,{\mathbf{z} }_{T},z^{i},c),\quad\forall t\in[\epsilon,T-\epsilon].$ (19) $$

###### Proof.

We denote the desnoising target $\frac{{\mathbf{z}}_{t}-\alpha_{t}{\mathbf{z}}_{0}}{\sigma_{t}}$ by $\bm{{\epsilon}}^{\hat{\Psi}}({\mathbf{z}}_{t},{\mathbf{z}}_{0},t)$, and define $a_{t}=\alpha_{t}(1-\frac{\mathrm{SNR}_{T}}{\mathrm{SNR}_{t}})$, $b_{t}=\frac{\mathrm{SNR}_{T}}{\mathrm{SNR}_{t}}\frac{\alpha_{t}}{\alpha_{T}}$, $c_{t}=\sqrt{\sigma_{t}^{2}(1-\frac{\mathrm{SNR}_{T}}{\mathrm{SNR}_{t}})}$.

From LABEL:eq:bridge_marginal, we have

$$ $\nabla_{\mathbf{z}}\log p_{t,bridge}({\mathbf{z}}|{\mathbf{z}}_{0},{\mathbf{z} }_{T})|_{{\mathbf{z}}={\mathbf{z}}_{t},{\mathbf{z}}_{T}={\mathbf{z}}^{i}}=- \frac{{\mathbf{z}}_{t}-a_{t}{\mathbf{z}}_{0}-b_{t}{\mathbf{z}}^{i}}{c_{t}^{2}},$ (20) $$

which is the target of Denoising Bridge Score Matching (Zhou et al., ). Our goal is to represent this target with ${\mathbf{z}}_{t}$, ${\mathbf{z}}_{T}$, and $\bm{{\epsilon}}^{\hat{\Psi}}({\mathbf{z}}_{t},{\mathbf{z}}_{0},t)$.

From the definition of $\bm{{\epsilon}}^{\hat{\Psi}}({\mathbf{z}}_{t},{\mathbf{z}}_{0},t)$, we have

$$ ${\mathbf{z}}_{0}=\frac{{\mathbf{z}}_{t}-\sigma_{t}\bm{{\epsilon}}^{\hat{\Psi}} ({\mathbf{z}}_{t},{\mathbf{z}}_{0},t)}{\alpha_{t}}.$ (21) $$

Plug it into eq. 20, it can be derived that

$$ $\displaystyle\nabla_{\mathbf{z}}\log p_{t,bridge}({\mathbf{z}}|{\mathbf{z}}_{0 },{\mathbf{z}}_{T})|_{{\mathbf{z}}={\mathbf{z}}_{t},{\mathbf{z}}_{T}={\mathbf{ z}}^{i}}$ $\displaystyle=-\frac{{\mathbf{z}}_{t}-a_{t}\frac{{\mathbf{z}}_{t}-\sigma_{t} \bm{{\epsilon}}^{\hat{\Psi}}({\mathbf{z}}_{t},{\mathbf{z}}_{0},t)}{\alpha_{t}} -b_{t}{\mathbf{z}}^{i}}{c_{t}^{2}}$ (22) $\displaystyle=-\frac{\alpha_{t}{\mathbf{z}}_{t}-a_{t}{\mathbf{z}}_{t}+a_{t} \sigma_{t}\bm{{\epsilon}}^{\hat{\Psi}}({\mathbf{z}}_{t},{\mathbf{z}}_{0},t)- \alpha_{t}b_{t}{\mathbf{z}}_{T}}{\alpha_{t}c_{t}^{2}}$ $\displaystyle=-\frac{a_{t}\sigma_{t}\bm{{\epsilon}}^{\hat{\Psi}}({\mathbf{z}}_ {t},{\mathbf{z}}_{0},t)}{\alpha_{t}c_{t}^{2}}-\frac{(\alpha_{t}-a_{t}){\mathbf {z}}_{t}-\alpha_{t}b_{t}{\mathbf{z}}^{i}}{\alpha_{t}c_{t}^{2}}$ $\displaystyle=-\frac{1}{\sigma_{t}}\bm{{\epsilon}}^{\hat{\Psi}}({\mathbf{z}}_{ t},{\mathbf{z}}_{0},t)-\frac{\alpha_{t}\frac{\mathrm{SNR}_{T}}{\mathrm{SNR}_{t }}{\mathbf{z}}_{t}-\frac{\alpha_{t}^{2}}{\alpha_{T}}\frac{\mathrm{SNR}_{T}}{ \mathrm{SNR}_{t}}{\mathbf{z}}^{i}}{\alpha_{t}\sigma_{t}^{2}(1-\frac{\mathrm{ SNR}_{T}}{\mathrm{SNR}_{t}})}$ $\displaystyle=-\frac{1}{\sigma_{t}}\bm{{\epsilon}}^{\hat{\Psi}}({\mathbf{z}}_{ t},{\mathbf{z}}_{0},t)-\frac{\mathrm{SNR}_{T}}{\mathrm{SNR}_{t}}\frac{{\mathbf {z}}_{t}-\frac{\alpha_{t}}{\alpha_{T}}{\mathbf{z}}^{i}}{\sigma_{t}^{2}(1-\frac {\mathrm{SNR}_{T}}{\mathrm{SNR}_{t}})},$ $$

As the Denoising Bridge Score Matching takes the form of

$$ $\mathcal{L}_{bridge}(\theta)=\mathbb{E}_{({\mathbf{z}}_{0},z^{i},c).{\mathbf{z }}_{T}={\mathbf{z}}^{i},t,{\mathbf{z}}_{t}}\left[\lambda(t)\left\lVert{\bm{s}} _{\theta}({\mathbf{z}}_{t},t,{\mathbf{z}}_{T},z^{i},c)-\nabla_{\mathbf{z}}\log p _{t,bridge}({\mathbf{z}}|{\mathbf{z}}_{0},{\mathbf{z}}_{T})|_{{\mathbf{z}}={ \mathbf{z}}_{t},{\mathbf{z}}_{T}={\mathbf{z}}^{i}}\right\rVert^{2}\right],$ (23) $$

when we parameterize ${\bm{s}}_{\theta}({\mathbf{z}}_{t},t,{\mathbf{z}}_{T},z^{i},c)=-\frac{1}{
\sigma_{t}}\bm{\epsilon}_{\theta}^{\hat{\Psi}}({\mathbf{z}}_{t},t,{\mathbf{z}}
_{T},z^{i},c)-\frac{\mathrm{SNR}_{T}}{\mathrm{SNR}_{t}}\frac{{\mathbf{z}}_{t}-
\frac{\alpha_{t}}{\alpha_{T}}{\mathbf{z}}_{T}}{\sigma_{t}^{2}(1-\frac{\mathrm{
SNR}_{T}}{\mathrm{SNR}_{t}})}$, the training objective can be written as

$$ $\mathcal{L}_{bridge}(\theta)=\mathbb{E}_{({\mathbf{z}}_{0},z^{i},c).{\mathbf{z }}_{T}={\mathbf{z}}^{i},t,{\mathbf{z}}_{t}}\left[\frac{\lambda(t)}{\sigma_{t}^ {2}}\left\lVert\bm{{\epsilon}}_{\theta}^{\hat{\Psi}}({\mathbf{z}}_{t},t,{ \mathbf{z}}_{T},z^{i},c)-\bm{{\epsilon}}^{\hat{\Psi}}({\mathbf{z}}_{t},{ \mathbf{z}}_{0},t)\right\rVert^{2}\right],$ (24) $$

which proves the first part of the proposition if we take $\tilde{\lambda}(t)=\frac{\lambda(t)}{\sigma_{t}^{2}}$.

For the second part, when $\mathrm{SNR}_{T}\approx 0$, there exists an ${\epsilon}>0$, such that $\frac{1}{\sigma_{t}^{2}(1-\frac{\mathrm{SNR}_{T}}{\mathrm{SNR_{t}}})}$ has an upper bound $M$. Since $\frac{\mathrm{SNR}_{T}}{\mathrm{SNR_{t}}}\frac{\alpha_{t}}{\alpha_{T}}=\alpha_
{T}\frac{\sigma_{t}^{2}}{\alpha_{t}\sigma_{T}^{2}}\approx 0$ when $\mathrm{SNR}_{T}\approx 0$, it can be directly inferenced from eq. 17 that ${\bm{s}}_{\theta}({\mathbf{z}}_{t},t,{\mathbf{z}}_{T},z^{i},c)\approx-\frac{1}
{\sigma_{t}}\bm{\epsilon}_{\theta}^{\hat{\Psi}}({\mathbf{z}}_{t},t,{\mathbf{z}
}_{T},z^{i},c)$.
∎

###### Remark .

From the first part of the proposition, we parameterize bridge models to predict $\frac{{\mathbf{z}}_{t}-\alpha_{t}{\mathbf{z}}_{0}}{\sigma_{t}}$. It is similar to that used in Chen et al. () although their parameterization is derived from the forward-backward diffusion process of Schrödinger Bridge problems. The statement and proof of this proposition reveals that DDBM and Diffusion Schrödinger Bridges are closely related. Additionally, the second part shows that our parameterization resembles the Denoising Score Matching in diffusion models.

###### Proposition 1 .

###### Proof.

###### Remark .

### A.3 SNR-Aligned Fine-tuning

##### Existence and Uniqueness of t ~ ~ 𝑡 \tilde{t} over~ start_ARG italic_t end_ARG

In Section 4.3, we need to find a $\tilde{t}$ such that $\alpha_{\tilde{t}}=\frac{a_{t}}{\sqrt{a_{t}^{2}+c_{t}^{2}}}$, $\sigma_{\tilde{t}}=\frac{c_{t}}{\sqrt{a_{t}^{2}+c_{t}^{2}}}$. Since $\frac{a_{t}^{2}}{c_{t}^{2}}=\frac{\alpha_{t}^{2}}{\sigma_{t}^{2}}(1-\frac{
\mathrm{SNR}_{T}}{\mathrm{SNR}_{t}})=\mathrm{SNR}_{t}-\mathrm{SNR}_{T}$, it is a monotonically decreasing function of $t$. As $\mathrm{SNR}_{t}$ is also a monotonically decreasing function which ranges over $(0,\infty)$, we can take $\tilde{t}=\mathrm{SNR}^{-1}(\frac{a_{t}^{2}}{c_{t}^{2}})$ and the uniqueness of such $\tilde{t}$ can also be guaranteed. Next, we provide a more general form of SAF, where the schedule $\{\alpha_{t},\sigma_{t}\}_{t\in[0,T]}$ of the pre-trained diffusion models and bridge models are not necessarily the same.

###### Proposition 2 .

Suppose we fine-tune a Gaussian diffusion model $\tilde{\bm{\epsilon}}_{\eta}({\mathbf{z}}_{t},t,c)$ with schedule $\{\tilde{\alpha}_{t},\tilde{\sigma}_{t}\}_{t\in[0,T]}$ to a diffusion bridge model $\bm{\epsilon}_{\theta,bridge}^{\hat{\Psi}}({\mathbf{z}}_{t},t,{\mathbf{z}}_{T}
,z^{i},c)\triangleq\bm{\epsilon}_{\theta,align}^{\hat{\Psi}}(\tilde{{\mathbf{z
}}}_{t},\tilde{t},{\mathbf{z}}_{T},z^{i},c)$ with schedule $\{\alpha_{t},\sigma_{t}\}_{t\in[0,T]}$.
If we use the same dataset $p_{data}({\mathbf{z}}_{0},z^{i},c)$ for training $\tilde{\bm{\epsilon}}_{\eta}({\mathbf{z}}_{t},t,c)$ and fine-tuning $\bm{\epsilon}_{\theta,align}^{\hat{\Psi}}(\tilde{{\mathbf{z}}}_{t},\tilde{t},{
\mathbf{z}}_{T},z^{i},c)$. Then, for each $c$, the input $({\mathbf{z}}_{t},t)$ of $\tilde{\bm{\epsilon}}_{\eta}$ has the same marginal distribution as the input $(\tilde{{\mathbf{z}}}_{t},\tilde{t})$ of $\bm{\epsilon}_{\theta,align}^{\hat{\Psi}}(\tilde{{\mathbf{z}}}_{t},\tilde{t},{
\mathbf{z}}_{T},z^{i},c)$. Here

$$ $\displaystyle\tilde{{\mathbf{z}}}_{t}=\frac{{\mathbf{z}}_{t}-b_{t}{\mathbf{z}} ^{i}}{\sqrt{a_{t}^{2}+c_{t}^{2}}},$ (25) $\displaystyle\tilde{t}=\widetilde{SNR}^{-1}(\frac{a_{t}^{2}}{c_{t}^{2}}).$ $$

($\widetilde{SNR}=\frac{\tilde{\alpha}_{t}^{2}}{\tilde{\sigma}_{t}^{2}}$ is the signal-to-noise ratio of pre-trained diffusion models.)

###### Proof.

Since $\widetilde{SNR}$ is also a monotonically decreasing function ranging over $(0,\infty)$, the uniqueness and existence of $\tilde{t}$ can also be guaranteed by the above analysis.

For a fixed $c,t$, we denote the probability density function of $\tilde{{\mathbf{z}}}_{t}$ by $q(\tilde{{\mathbf{z}}}_{t};t)$.
Then

$$ $\displaystyle q(\tilde{{\mathbf{z}}}_{t};t)$ $\displaystyle=\int_{z^{i}}q(\tilde{{\mathbf{z}}}_{t}|z^{i};t)p_{data}(z^{i}) \mathrm{d}z^{i}$ (26) $\displaystyle=\int_{z^{i}}\int_{{\mathbf{z}}_{0}}q(\tilde{{\mathbf{z}}}_{t}|{ \mathbf{z}}_{0},z^{i};t)p_{data}({\mathbf{z}}_{0},z^{i})\mathrm{d}{\mathbf{z}} _{0}\mathrm{d}z^{i}$ $\displaystyle=\int_{z^{i}}\int_{{\mathbf{z}}_{0}}\mathcal{N}(\tilde{{\mathbf{z }}}_{t};\frac{a_{t}}{\sqrt{a_{t}^{2}+c_{t}^{2}}}{\mathbf{z}}_{0},\frac{c_{t}}{ \sqrt{a_{t}^{2}+c_{t}^{2}}}I)p_{data}({\mathbf{z}}_{0},z^{i})\mathrm{d}{ \mathbf{z}}_{0}\mathrm{d}z^{i}$ $\displaystyle=\int_{z^{i}}\int_{{\mathbf{z}}_{0}}\mathcal{N}(\tilde{{\mathbf{z }}}_{t};\tilde{\alpha}_{\tilde{t}}{\mathbf{z}}_{0},\tilde{\sigma}_{\tilde{t}}^ {2}I)p_{data}({\mathbf{z}}_{0},z^{i})\mathrm{d}{\mathbf{z}}_{0}\mathrm{d}z^{i}$ $\displaystyle=\int_{{\mathbf{z}}_{0}}\mathcal{N}(\tilde{{\mathbf{z}}}_{t}; \tilde{\alpha}_{\tilde{t}}{\mathbf{z}}_{0},\tilde{\sigma}_{\tilde{t}}^{2}I)( \int_{z^{i}}p_{data}({\mathbf{z}}_{0},z^{i})\mathrm{d}z^{i})\mathrm{d}{\mathbf {z}}_{0}$ $\displaystyle=\int_{{\mathbf{z}}_{0}}\mathcal{N}(\tilde{{\mathbf{z}}}_{t}; \tilde{\alpha}_{\tilde{t}}{\mathbf{z}}_{0},\tilde{\sigma}_{\tilde{t}}^{2}I)p_{ data}({\mathbf{z}}_{0})\mathrm{d}{\mathbf{z}}_{0},$ $$

which equals to the marginal distribution of the pre-trained diffusion process $p_{t,diff}({\mathbf{z}}_{t})$.

∎

###### Proposition 2 .

###### Proof.

### A.4 Neural Prior with Regression Training Objective.

###### Proposition 3 .

If we train $F_{\eta}(z^{i},c)$ with the regression training objective

$$ $\mathcal{L}_{prior}(\eta)=\mathbb{E}_{({\mathbf{z}}_{0},z^{i},c)\sim p_{data}( {\mathbf{z}}_{0},z^{i},c)}\left[\left\lVert F_{\eta}(z^{i},c)-{\mathbf{z}}_{0} \right\rVert^{2}\right],$ (27) $$

and the neural network is optimized sufficiently, then we have

$$ $F_{\eta}(z^{i},c)=F_{\eta}^{*}(z^{i},c)\triangleq\mathbb{E}_{{\mathbf{z}}_{0} \sim p_{data}({\mathbf{z}}_{0}|z^{i},c)}\left[{\mathbf{z}}_{0}\right].$ (28) $$

###### Proof.

For each $(z^{i},c)$, $\mathcal{L}_{prior}(\eta)$ optimizes the following objective:

$$ $\displaystyle l_{\eta}(z^{i},c)$ $\displaystyle=\mathbb{E}_{{\mathbf{z}}_{0}\sim p_{data}({\mathbf{z}}_{0}|z^{i} ,c)}\left[\left\lVert F_{\eta}(z^{i},c)-{\mathbf{z}}_{0}\right\rVert^{2}\right]$ (29) $\displaystyle=\left\lVert F_{\eta}(z^{i},c)\right\rVert^{2}-\langle F_{\eta}(z ^{i},c),\mathbb{E}_{{\mathbf{z}}_{0}\sim p_{data}({\mathbf{z}}_{0}|z^{i},c)} \left[{\mathbf{z}}_{0}\right]\rangle+\left\lVert\mathbb{E}_{{\mathbf{z}}_{0} \sim p_{data}({\mathbf{z}}_{0}|z^{i},c)}\left[{\mathbf{z}}_{0}\right]\right \rVert^{2}$ $\displaystyle=\left\lVert F_{\eta}(z^{i},c)\right\rVert^{2}-\langle F_{\eta}(z ^{i},c),\mathbb{E}_{{\mathbf{z}}_{0}\sim p_{data}({\mathbf{z}}_{0}|z^{i},c)} \left[{\mathbf{z}}_{0}\right]\rangle+C.$ $$

where $C$ is a constant independent of $\eta$.
When the network is optimized sufficiently, $l_{\eta}(z^{i},c)$ takes the minimum for each $(z^{i},c)$, so we have

$$ $F_{\eta}(z^{i},c)=\arg\min_{\mathbf{x}}\left(\left\lVert{\mathbf{x}}\right \rVert^{2}-\langle{\mathbf{x}},\mathbb{E}_{{\mathbf{z}}_{0}\sim p_{data}({ \mathbf{z}}_{0}|z^{i},c)}\left[{\mathbf{z}}_{0}\right]\rangle\right)$ (30) $$

It can be solved that $F_{\eta}(z^{i},c)=\mathbb{E}_{{\mathbf{z}}_{0}\sim p_{data}({\mathbf{z}}_{0}|z
^{i},c)}\left[{\mathbf{z}}_{0}\right]$.
∎

###### Proposition 3 .

###### Proof.

## Appendix B Detailed Discussion On Related Works

##### Video Diffusion Models

Inspired by the success of text-to-image (T2I) diffusion models (Ramesh et al., ; Nichol et al., ), numerous studies have investigated diffusion-based text-to-video (T2V) models (Blattmann et al., ; Yang et al., ; Singer et al., ) by designing 3D spatial-temporal U-Net (Ho et al., ; ) and Diffusion Transformers (DiT) (Peebles & Xie, ; Bao et al., ). To improve memory and computation efficiency, Latent Diffusion Models (LDM) (Rombach et al., ; Vahdat et al., ) are utilized where the diffusion process is applied in the compressed latent space of video samples  (Bao et al., ; Brooks et al., ; He et al., ). Meanwhile, some other works designed cascaded diffusion models to generate motion representation (Yu et al., ) or videos with lower resolution (Ho et al., ; Wang et al., ) first, which are utilized to synthesize the result videos in the subsequent stages.

##### Diffusion-based I2V Generation

The main difference between I2V and T2V is the incorporation of image conditions into the sampling process. Xing et al. () utilizes the features of a CLIP image encoder and a lightweight transformer to inject image conditions into the backbone of a T2V model. Ma et al. () and Zhang et al. () propose to directly model the residual between the subsequent frames and the given initial frame with diffusion for I2V generation. Moreover, Ma et al. () also uses the DCTInit technique to enhance the consistency of video content with the given image. Chen et al. () presents to train short-to-long video generation models with masked diffusion models. Guo et al. () and Zhang et al. () propose to utilize pre-trained T2I models for image animation by training an additional component to model the relationship between video frames. Ren et al. () propose improved network architecture and sampling strategy for image-to-video generation at the same time to enhance the controllability of image conditions. Jain et al. (), Zhang et al. () and Shi et al. () design cascaded diffusion systems for I2V generation. VIDIM (Jain et al., ) consists of one base diffusion model and another two diffusion models for spatial and temporal super-resolution respectively. Zhang et al. () uses a base diffusion model to generate videos with low resolutions, which serve as the input of the following video super-resolution diffusion model. Shi et al. () first generates the optical flow between the subsequent frames and given image with a diffusion process, and use the optical flow as conditions of another model to generate videos. Ni et al. () and Zhang et al. () train an autoencoder to represent the motions between frames in a latent space, and use diffusion models to generate motion latents. However, previous I2V diffusion models are built on the noise-to-data generation of conditional diffusion process and the sampling remains a denoising process conditioned on given images. In contrast, FrameBridge replaces the diffusion process with a bridge process and the sampling directly model the animation of static images.

##### Comparison with Previous Works of Bridge Models

In Section 2, we leverage the forward SDE of bridge models (Zhou et al., ) and the backward sampler proposed by Chen et al. () to build FrameBridge. We unify their theoretical frameworks to establish our formulation, and emphasize that bridge models are suitable for image-to-video generation, which is a typical data-to-data generation task. Liu et al. () and Chen et al. () apply bridge models to image-to-image translation and text-to-speech synthesis tasks respectively. Compared with their works, we focus on I2V tasks, building our bridge-based framework by utilizing the frames-to-frames essence and presenting two innovative techniques for two scenarios of training I2V models, namely fine-tuning from pre-trained text-to-video diffusion models and training from scratch.

## Appendix C Experiment Details

We provide descriptions of the datasets and metrics used in our experiments, along with implementation details for different I2V models.

### C.1 Datasets

UCF-101 is an open-sourced video dataset consisting of 13320 videos clips, and each video clip are categorized into one of the 101 action classes. There are three official train-test split, each of which divide the whole dataset into 9537 training video clips and 3783 test video clips. We use the whole dataset as the training data for I2V models trained from scratch on UCF-101, and use the test set to evaluate zero-shot metrics for models fine-tuned on WebVid-2M. When we evaluate zero-shot metrics on UCF-101 for text-conditional I2V models, we use the class label as the input text prompt.

WebVid-2M is an open-sourced dataset consisting of about 2.5 million video-text pairs, which is a subset of WebVid-10M. We only use WebVid-2M as the training data when fine-tuning I2V models from T2V diffusions in Section 2.

MSR-VTT is an open-sourced dataset consisting of 10000 video-text pairs, and we only use the test set to compute zero-shot metrics for fine-tuned models.

##### Preprocess of Training Data:

For both UCF-101 and WebVid-2M dataset, we sample 16 frames from each video clip with a fixed frame stride of 3 when training. Then we resize and center-crop the video clips to 256 $\times$ 256 before input it to the models.

### C.2 Metrics

Fréchet Video Distance ( Unterthiner et al. (); FVD) evaluates the quality of synthesized videos by computing the perceptual distance between videos sampled from the dataset and the models. We follow the protocol used in StyleGAN-V (Skorokhodov et al., ) to calculate FVD. First, we sample 2048 video clips with 16 frames and frame stride of 3 from the dataset. Then, we generate 2048 videos from the I2V models. All videos are resized to 256 $\times$ 256 before calculating FVD except for ExtDM. (ExtDM generate videos with resolution 64 $\times$ 64, so we compute FVD on this resolution.) After that, we extract features of those videos with the same I3D model used in the repository of StyleGAN-V (^3^33https://github.com/universome/stylegan-v) and calculate the Fréchet Distance.

Inception Score (Saito et al. (); IS) also evaluates the quality of the generated videos. However, computing IS need a pre-trained classifier and we only apply this metric on UCF-101. When computing IS, we use the open-sourced evaluation code and pre-trained classifier for videos from the repository of StyleGAN-V.

CLIPSIM (Wu et al., ) evaluates the consistency between video frames and the text prompt by computing the average CLIP similarity score between each frame and the prompt. We use the VIT-B/32 CLIP model (Radford et al., ) when evaluating zero-shot metrics on MSR-VTT.

PIC is a metric used by Xing et al. () to evaluate the consistency of video frames and the given image by the computing average Dreamsim (Fu et al., ) distance between generated frames and the image condition.

### C.3 Implementation of FrameBridge and Other Baselines

We offer the implementation details of I2V models which are fine-tuned on WebVid-2M or trained from scratch on UCF-101.

#### C.3.1 FrameBridge

##### Fine-tuning on WebVid2M

We reference the codebase of Dynamicrafter(^4^44https://github.com/Doubiiu/DynamiCrafter) to fine-tune FrameBridge, and initialize our model from the pre-trained VideoCrafter1 (Chen et al., ) checkpoint. For the schedule of bridge, we adopt the Bridge-gmax schedule of (Chen et al., ), where $f(t)=0$, $g(t)^{2}=\beta_{0}+t(\beta_{1}-\beta_{0})$, $\alpha_{t}=1$, $\sigma_{t}^{2}=\frac{1}{2}(\beta_{1}-\beta_{0})t^{2}+\beta_{0}t$ with $\beta_{0}=0.01$, $\beta_{1}=50$. For both FrameBridge with and without SAF technique, we fine-tune the models $\bm{{\epsilon}}^{\hat{\Psi}}$ for 20k iterations with batch size 64. We use the AdamW optimizer with learning rate $1\times 10^{-5}$ and mixed precision of BFloat16. We do not apply ema to the model weight during fine-tuning. The conditions $c$ and $z^{i}$ are incorporated into the network in the same way as DynamiCrafter, and we concatenate ${\mathbf{z}}_{t}$ with ${\mathbf{z}}^{i}$ along the channel axis to condition the network on the prior. As the schedule $\{\alpha_{t},\sigma_{t}\}_{t\in[0,T]}$ is different from that of the pre-trained diffusion models, we use the generalized SAF (Proposition 2).

##### Training From Scratch on UCF-101

We reference the codebase of Latte(^5^55https://github.com/Vchitect/Latte) to train FrameBridge from scratch on UCF-101. We adopt Latte-S/2 as our bridge model with the same schedule as above and train FrameBridge for 400k iterations with batch size 40. For FrameBridge with neural prior, we also implement $F_{\eta}(z^{i},c)$ with Latte-S/2 except that the conditioning of timestep $t$ is removed from the model. To match $z^{i}$ with the input shape of Latte, we replicate $z^{i}$ for $L$ times and concatenate them along temporal axis. We train $F_{\eta}(z^{i},c)$ for 400k iterations with batch size 32 before training bridge models if the neural prior is applied. For both the training of bridge models and $F_{\eta}(z^{i},c)$, we use the AdamW optimizer with learning rate $1\times 10^{-5}$ and ema is not applied. The conditions $c$ are incorporated into the network in the same way as Latte. Since Latte-S/2 is a transformer-based diffusion network, we incorporate the condition $z^{i}$ by concatenate it with video latent ${\mathbf{z}}_{t}$ in the token sequence. To condition the network on prior ${\mathbf{z}}^{i}$ or $F_{\eta}(z^{i},c)$, we concatenate them with ${\mathbf{z}}_{t}$ along the channel axis.

#### C.3.2 Baselines on UCF-101.

ExtDM (Zhang et al., ) is a diffusion-based video prediction model, which is trained to predict the following $m$ frames with the given first $n$ frames of a video clip. We train ExtDM with their official implementation(^6^66https://github.com/nku-zhichengzhang/ExtDM) and set $n=1,m=15$ for our I2V setting on UCF-101.

VDT-I2V is our implementation of the I2V method proposed by Lu et al. (). They use a transformer-based diffusion network for I2V generation by directly concatenating the image condition with the token sequence of the noisy video latent ${\mathbf{z}}_{t}$. We also implement their I2V method on a Latte-S/2 model considering the similarities among transformer-based diffusion models.

#### C.3.3 Baselines on WebVid-2M.

DynamiCrafter (Xing et al., ) fine-tunes an I2V diffusion model based on the T2V diffusion model VideoCrafter1. We reproduce the fine-tuning process on WebVid-2M dataset with the same training budget of FrameBridge for fair comparison (\ie, 20k iterations with batch size of 64).

SEINE (Chen et al., ) is an I2V diffusion model fine-tuned from Lavie (Wang et al., ). As the fine-tuning code of SEINE has not been released, we directly use their pre-trained model to compute zero-shot metrics on UCF-101 and MSR-VTT.

#### C.3.4 Ablation Studies on Neural Prior

In Section 5.4, we ablate on the neural prior technique by comparing the performance of four models:

- •
VDT-I2V: The same model as our diffusion baseline on UCF-101.
- •
FrameBridge without neural prior: A FrameBridge model implemented by utilizing the replicated image ${\mathbf{z}}^{i}$ as the prior.
- •
FrameBridge with neural prior only as the network condition: A FrameBridge model implemented by utilizing ${\mathbf{z}}^{i}$ as the prior. However, we condition the bridge model on $F_{\eta}(z^{i},c)$ by additionally feeding it into the network through concatenation with ${\mathbf{z}}_{t}$ along the channel axis.
- •
FrameBridge-NP: A FrameBridge model implemented by utilizing $F_{\eta}(z^{i},c)$ as the prior.

## Appendix D More Qualitative Results

We show several randomly selected samples of FrameBridge below, and more synthesized samples can be visited at: [https://framebridge-demo.github.io/](https://framebridge-demo.github.io/)

Figure: Figure 6: Zero-shot generation results of fine-tuned FrameBridge (with SAF) on UCF-101.
Refer to caption: x7.png

Figure: Figure 7: Zero-shot generation results of fine-tuned FrameBridge (with SAF) on MSR-VTT.
Refer to caption: x8.png

Figure: Figure 8: Non-zero-shot generation results of FrameBridge-NP on UCF-101. We use two lines to present a neural prior and the corresponding generated video.
Refer to caption: x9.png

Figure: Figure 9: Comparisons between fine-tuned FrameBridge and other diffusion-based I2V models.
Refer to caption: x10.png

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: I hope the authors can address the concerns raised in the "Weaknesses" section.
soundness: 2
strengths: S1: Unlike previous diffusion-based methods, FrameBridge formulates I2V synthesis as an interesting frames-to-frames generation task rather than a conditional noise-to-frames generation.

S2: The paper presents good qualitative and quantitative results.
summary: The paper introduces FrameBridge, which formulates I2V synthesis as a frames-to-frames generation task rather than a conditional noise-to-frames generation. This approach allows the model to concentrate more effectively on learning image animation, enhancing both the temporal and appearance coherence of the generated video. Furthermore, to better utilize the priors from pre-trained T2V models, SNR Aligned Fine-tuning (SAF) and a neural prior are proposed to promote fine-tuning efficiency and improve the quality of generated videos. Extensive experiments demonstrate the effectiveness of FrameBridge in image-to-video generation tasks.
weaknesses: W1: In Table 1, some important baselines are missing, such as SVD [r1] and I2VGen-XL [r2], both of which are strong baselines and would be valuable for comparison. Additionally, I noticed in line 473 that DynamiCrafter was reproduced in this paper. I’m curious why the original weights couldn’t be used for testing, as the videos generated by DynamiCrafter in Figure 5 and on the project page seem to exhibit significant inconsistencies. This seems unusual, given that DynamiCrafter claims to rank first on the I2V benchmark of VBench [r3]. Clarifying the reproduction process and discussing potential discrepancies would be helpful. 

[r1] Blattmann, Andreas, et al. "Stable video diffusion: Scaling latent video diffusion models to large datasets." arXiv preprint arXiv:2311.15127 (2023).
[r2] Zhang, Shiwei, et al. "I2vgen-xl: High-quality image-to-video synthesis via cascaded diffusion models." arXiv preprint arXiv:2311.04145 (2023).
[r3] Huang, Ziqi, et al. "Vbench: Comprehensive benchmark suite for video generative models." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

W2: In Figure 5 and on the project page, most of the generated videos seem to exhibit only minor movements, which raises a concern that the proposed method may tend to produce videos with limited motion. Could the authors provide examples of the method generating videos with larger-scale actions, such as camera movements or vehicles in motion? Alternatively, discussing any limitations in generating more dynamic scenes would be valuable.

W3: In line 470 of the paper, it is noted that all video synthesis methods sample 250 steps, which is quite time-consuming. Since most models achieve good results with only 50 steps, could the authors clarify the justification for using 250 steps? Additionally, it would be helpful if the authors could provide a comparison of quality vs. number of steps, along with the inference time for different step counts.

W4: In Table 2, FrameBridge-NP is mentioned. Could the authors clarify whether the default FrameBridge includes a neural prior? Additionally, it would be helpful to specify if NP is used only in the context of training from scratch or if it is a default setting. If NP is only applicable or beneficial in specific training scenarios, clarifying these conditions would be useful.

W5: Table 2 should include experimental results of classic I2V network architectures, such as Dynamicrafter and I2VGEN-XL. Furthermore, I think it may be better to conduct ablation studies based on the setting in Table 1, as suggested in the Dynamicrafter paper, since generalizability is crucial for image-to-video generation models.

---

confidence: 5
contribution: 3
initial_rating: 5
presentation: 3
questions: Based on the weaknesses mentioned, this paper requires further revisions, therefore, I currently give it a score of 'marginally below the acceptance threshold'.
soundness: 3
strengths: 1.	FrameBridge introduces a new data-to-data generation framework for I2V synthesis, which is different from traditional noise-to-data diffusion-based methods.
2.	By taking the given static image as the prior of the video target, FrameBridge reduces the burden of generative models and improves synthesis quality.
3.	The proposed SAF technique enables seamless knowledge transfer between pre-trained diffusion models and FrameBridge, improving fine-tuning efficiency.
summary: This paper proposes a novel image-to-video (I2V) generation framework called FrameBridge, which models the frame-to-frames synthesis process with a data-to-data generative framework. Unlike traditional diffusion-based I2V methods, FrameBridge takes the given static image as the prior of the video target, establishing a bridge model between them. This approach reduces the burden of generative models and improves synthesis quality. The authors also propose two techniques, SNR-Aligned Fine-tuning (SAF) and neural prior, to further enhance the performance of FrameBridge.
weaknesses: 1.	My primary concern about this paper is the experiment. I acknowledge that the bridge model may be a promising method for image-to-video diffusion models, and this paper has made an effort to fully utilize input image information. However, this paper only provides baselines for SEINE and DynamiCrafter, other works such as Animate Anyone [1] and Sparsectrl [2] have attempted to introduce an additional appearance encoding and fusion module to support more refined conditional image information utilization, which is an advantage over these methods. Therefore, this paper needs to provide a more detailed discussion. Moreover, ConsistI2V [3] designed FrameInit, which incorporates image priors into the initial noise, breaking the noise-to-data pattern and leading to better results. This paper does not provide a comprehensive comparison with other state-of-the-art I2V generation methods.
2.	Regarding evaluation metrics, FVD and IS are no longer reliable metrics. In fact, there are many benchmarks focused on video generation, such as T2V-CompBench [4] and VBench [5], which provide reliable evaluation metrics. The authors should use the most advanced benchmarks. Furthermore, this paper does not provide user studies.

[1] Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation, Hu et al., CVPR 2024
[2] Sparsectrl: Adding sparse controls to text-to-video diffusion models, Guo et al., ECCV 2024
[3] ConsistI2V: Enhancing Visual Consistency for Image-to-Video Generation, Re et al., TMLR 2024
[4] T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generationrk, Sun et al., arXiv 2024 
[5] VBench: Comprehensive Benchmark Suite for Video Generative Models, Huang et al., CVPR 2024

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 3
questions: Can you show the best ten cases from FrameBridge? In the field of video generation, the upper bound of the model is also a very important reference indicator.
soundness: 2
strengths: 1. Solving the distribution gap of training and inference is a reasonable motivation.
2. The video generated by FrameBridge is smoother with less mutation.
3. Compared to previous methods, FrameBridge achieves better metric scores on WebVid-2M and UCF-101.
summary: In this paper, the authors propose FrameBridge to achieve the image-to-video generation. Unlike previous diffusion-based methods, FrameBridge starts from a deterministic data point, the static image, and then maps it to the video latent through a bridge model. To achieve that, they further propose two techniques, SNR-
Aligned Fine-tuning (SAF) and neural prior.
weaknesses: 1. Insufficient reference to related work. For example, “Common Diffusion Noise Schedules and Sample Steps are Flawed” proposes to align the SNR between the training stage and the inference stage. 
2. Some videos on the website can not be played. It seems that authors still modify the website when the review period starts. 
3. The video quality is not satisfactory. First, Maybe due to the size of the training set, the visual quality is not competitive (the worst one in my assigned papers). 
4. It seems that neural prior will limit the dynamics of the generated results. The transformation starting point of all frames comes from the transformation of the same frame. In this case, most generated results only contain small motions. The selected indicators are not helpful in judging the motion of the video. Alternatively, evaluating on a comprehensive benchmark like VBench will make the comparison more convincing.
5. Abstract: “modelling” -> “modeling”.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: I want the authors to show the training/inference pseudo code of their proposed method and standard video diffusion methods. It is the best way to provide a clear difference of these two methods.
soundness: 3
strengths: - The overall story is smooth and sounds interesting.

- The discussion on related works is sufficient.

- The experimental results seems to support the claim of contributions in the paper.
summary: The paper introduces an interesting image-to-video diffusion-based methods by leveraging the Denoising Diffusion Bridge Models (DDBM). It sets the initial state of denoising as temporally replicated first frame (i.e., static videos) and uses the DDBM to connect the distributions of the static videos and real videos sharing the same first frame. The reported results is better than naive diffusion models.
weaknesses: 1. **I have a fundamental concern that I think the proposed method has no inherent difference between naive video diffusion models, especially considering the re-parameterization trick proposed in the paper.**

The paper says "Formulating a frame-to-frames generation task with a conditional noise-to-data sampling process, diffusion-based I2V systems suffer the difficulties to generate high-quality samples from uninformative Gaussian noise". This is to say that the authors believe the model can hardly learn to predict good videos $x_0$ from the mixture state of noise and data, that is $x_t = \alpha_t x_0 + \sigma_t \epsilon$.  Instead, providing the model the information from the statics images can help the model learn. The input for bridge models is formulated as $x_t = a_t x_0 + b_t \epsilon + c_t x_T$, where $x_T$ is the replicated initial frames.  This is also the reasonability of the bridge models.  

However, as proposed in Equation 7 of the paper, the authors remove all the information $x_T$ (The information of the initial frame is all removed), and the noisy state $x_t$ returns to the normal forms of standard diffusion models, which is the weighted sum of the target $x_0$ and the noise $\epsilon$.  Even though the authors claim that this operation helps them to adapt the standard diffusion models into the diffusion bridge models, I think this operation breaks the basic property of diffusion bridge models.

To this end, the only difference between the proposed method and the standard diffusion model is the prediction types. For standard DDPM, the model learns to predict the $\epsilon$. For the proposed method, the model learns to predict the $\epsilon + m_t x_T$. $m_t$ can be some time-dependent factors (not strict symbol form).   Since the model's input is the same now and the predictions just have an additional term $m_t x_T$ which is known at either training and inference stages, the influence of $x_T$ will be eliminated due to the sampling starting point difference between diffusion models and bridge models. I don't think the prediction type difference can make any difference. Therefore, I am in deep concern that the proposed method might have no intrinsic difference from naive video diffusion models. 

I am not very sure about my judgement though. I hope to propose the concern and inspire other experienced reviewers do further check on the  proposed technique.


2. **Connecting similar distributions is not uncommon.** Many works proposed similar ideas on image fields [1] [2] [3] for applications that they build a connection between the target image fields and the source image fields instead of simply using the source images as conditions. 

3.  **Detailed FVD learning curve for different baselines.** I would like the authors provide the detailed staged FVD learning curves (or other metrics) for different baselines. They can clearly show the learning speed of different methods and show performance gap of different methods trained with the same training budget.

4. In table 4, I would like to see the FVD for gaussian prior and Fn conditions. Since we can see from the table that, when both using Fn as conditions, the model have very similar performance with replicate prior and neural prior. This might indicate that the condition is more important than prior.

5. It is known that FVD has a very strong content bias and tends to misjudge the quality of video motion. Therefore I want the authors report the new-FVD proposed in the previous work. [4]
 


[1]  FMBoost: Boosting Latent Diffusion with Flow Matching. ECCV 2024 Oral

[2]  ResShift: Efficient Diffusion Model for Image Super-resolution by Residual Shifting. NeurIPS 2023, Spotlight, TPAMI@2024.

[3] Residual Denoising Diffusion Models. CVPR 2024.

[4] Content-Debiased FVD for Evaluating Video Generation Models.  CVPR 2024.

Meta-review: All reviewers agree to reject the paper. Reviewers mainly complain about the result quality and unfair comparisons to SOTA methods, which could not be addressed in the rebuttal. The authors are encouraged to address the concerns and submit elsewhere.

Decision: Reject