CamI2V: Camera-Controlled Image-to-Video Diffusion Model

## Abstract

Abstract Recently, camera pose, as a user-friendly and physics-related condition, has been introduced into text-to-video diffusion model for camera control.
However, existing methods simply inject camera conditions through a side input. These approaches neglect the inherent physical knowledge of camera pose, resulting in imprecise camera control, inconsistencies, and also poor interpretability.
In this paper, we emphasize the necessity of integrating explicit physical constraints into model design.
Epipolar attention is proposed for modeling all cross-frame relationships from a novel perspective of noised condition.
This ensures that features are aggregated from corresponding epipolar lines in all noised frames, overcoming the limitations of current attention mechanisms in tracking displaced features across frames, especially when features move significantly with the camera and become obscured by noise.
Additionally, we introduce register tokens to handle cases without intersections between frames, commonly caused by rapid camera movements, dynamic objects, or occlusions.
To support image-to-video, we propose the multiple guidance scale to allow for precise control for image, text, and camera, respectively.
Furthermore, we establish a more robust and reproducible evaluation pipeline to solve the inaccuracy and instability of existing camera control measurement.
We achieve a 25.5% improvement in camera controllability on RealEstate10K while maintaining strong generalization to out-of-domain images. Only 24GB and 12GB are required for training and inference, respectively. We plan to release checkpoints, along with training and evaluation codes. Dynamic videos are best viewed at https://zgctroy.github.io/CamI2V .

## 1 Introduction

The remarkable 3D consistency demonstrated in videos generated by Sora (Brooks et al., ) has highlighted the powerful capabilities of diffusion models (Ho et al., ; Rombach et al., ), showcasing their potential as a world simulator.
Many researchers have attempted to enable the model to understand real-world knowledge (Chen et al., ; Liu et al., ).

Condition or guidance (Ho & Salimans, ; Dhariwal & Nichol, ) is widely recognized as a crucial factor in enhancing generation quality.
This is attributed to the fundamental principles that diffusion models denoise along the gradient of the log probability density function (score function) (Song et al., ), moving towards a high density region.
However, this characteristic has varying effects at different noise levels (Tang et al., ).
As shown in Fig. 1(a), the high density region under high noise level becomes the overlap of numerous noisy samples, resulting in visual blurriness.
By providing the model with conditions such as $c_{\text{dog}}$ and $c_{\text{cat}}$, it can rapidly eliminate incorrect generations.
This illustrates that effective conditions can guide the model towards desired outcomes while reducing uncertainty.

Consequently, incorporating physics-related or more detailed conditions into the diffusion model is an effective way of improving its world understanding.
Considering that video generation requires providing condition for each frame, it is essential to identify a condition that is physics-related and also user-friendly.
Recently, some camera-conditioned text-to-video diffusion models such as MotionCtrl (He et al., ) and CameraCtrl (Wang et al., ) have proposed using camera poses of each frame as a new type of condition.
However, these methods simply inject camera conditions through a side input (like T2I-Adapter (Mou et al., )) and neglect the inherent physical knowledge of camera pose, resulting in imprecise camera control, inconsistencies, and also poor interpretability.
This raises a critical question: How can we effectively inject camera poses into the diffusion model to enhance its ability to physical world knowledge?

In this paper, we integrate geometric priors into the model and propose a novel framework for injecting camera poses.
We first identify shortcomings in existing attention mechanisms regarding the ability of tracking noised features across frames. As illustrated in Fig. 2, separated spatial and temporal attention serves as an indirect form of 3D attention. The cross-frame interaction in temporal attention is confined to features at the same location in the image, rendering it ineffective for tracking significant movements resulting from large camera shifts.
3D full attention is widely applied in advanced video diffusion models such as OpenSora (Zheng et al., ) and CogVideoX (Yang et al., ), due to its extensive receptive field.
From the novel perspective of the noised conditions mentioned in Fig. 1, the broad receptive field of 3D full attention allows it to access more noised conditions. However, noised conditions act as a double-edged sword; at high noise levels, deterministic information is often obscured by random noise, resulting in minimal reduction of uncertainty despite increased access.
As previously highlighted in Fig. 1, the quality of a condition is determined solely by its ability to reduce the model’s uncertainty, rather than by the number of conditions it can access.

To address these issues, we propose to apply plücker coordinates (Plücker, ) as absolute 3d ray embedding for implicit learning of 3d space and propose a novel epipolar attention mechanism that introduces an explicit constraint.
This ensures that even when feature tracking is compromised under high noise conditions, only features along the epipolar line are aggregated.
By doing so, our approach minimizes the search space and reduces potential errors, ultimately enhancing 3D consistency across frames and improving overall controllability.
Our explicit epipolar attention can also be regarded as a form of clean condition, as it reliably filters out features that do not conform to 3D geometry across varying noise levels, significantly enhancing controllability and stability.
Additionally, inspired by Timothée et al. (), we incorporate register tokens into epipolar attention to address scenarios where there are no intersections between frames, often caused by rapid camera movements, dynamic objects, or occlusions.

For inference, we propose a multiple classifier-free guidance scale to control images, text, and camera respectively. If needed, several forward passes can be combined into a single pass by absorbing the scales of image, text, and camera into the model input, similar to timestep conditioning according to (Meng et al., ). For evaluation, we identify inaccuracies and instability in the current measurements of camera controllability due to the intrinsic limitations of SfM-based methods such as COLMAP (Schonberger & Frahm, ), which rely on identifying keypoint pairs and is quite challenging on generated videos with low resolution, high frame stride, and 3D inconsistencies. Considering the importance of accurate evaluation in this field, we establish a more robust, precise, and reproducible evaluation pipeline by implementing several enhancements. More details are provided in Section 5.

We conduct experiments on the RealEstate10k dataset and evaluate video generation quality using FVD (Unterthiner et al., ), as well as camera controllability metrics including RotError, TranError (Wang et al., ), and CamMC (He et al., ). The results demonstrate that the proposed epipolar attention mechanism across all noised frames significantly enhances geometric consistency and improves camera controllability.
To facilitate further research, we will release all models trained on open-source frameworks such as DynamiCrafter, along with high-resolution checkpoints and training/evaluation codes, as soon as possible. To summarize, our key contributions are as follows:

- •
We expand the camera-conditioned text-to-video diffusion model to support reference image at random frame and propose multiple guidance scale to allow for precise control for text, image, and camera, respectively.
- •
From a novel perspective of noised condition, we propose epipolar attention added with register token as a explicit constraint for modeling all cross-frame relationships. This addresses the issue of noised features tracking due to the movement of camera, leading to precise camera control and better 3d consistency. It also reduces the controllability instability due to randomness introduced by noise during generation.
- •
We point out and analyze the reasons for inaccurate measurement of camera controllability caused by the inherent limitations of SfM evaluator and re-establish a more robust, accurate and reproducible evaluation pipeline. Furthermore, we achieve the sota performance on RealEstate10K dataset and out-of-domain dataset.

## 2 Related Work

Diffusion-based Video Generation.
With the advancement of diffusion models (Rombach et al., ; Ramesh et al., ; Zheng et al., ), video generation technology has progressed significantly. Given the scarcity of high-quality video-text datasets (Blattmann et al., ; ), many researchers have sought to adapt existing text-to-image (T2I) models for text-to-video (T2V) generation. Some efforts involve integrating temporal blocks into original T2I models, training these additions to facilitate the conversion to T2V models.
Examples include AnimateDiff (Guo et al., ), Align your Latents (Blattmann et al., ), PYoCo (Ge et al., ), and Emu video (Girdhar et al., ). Additionally, methods such as LVDM (He et al., ), VideoCrafter (Chen et al., ; ), ModelScope (Wang et al., ), LAVIE (Wang et al., ), and VideoFactory Wang et al. () have adopted a similar structure, using T2I models as initialization weights and fine-tuning both spatial and temporal blocks to achieve better visual effects.
Building on this foundation, Sora (Brooks et al., ) and CogVideoX (Yang et al., ) have significantly enhanced video generation capabilities by introducing Transformer-based diffusion backbones (Peebles & Xie, ; Ma et al., ; Yu et al., ) and leveraging 3D-VAE technology, thereby opening up the possibility of world simulators.
Furthermore, works such as Dynamicrafter (Xing et al., ), SVD (Blattmann et al., ), Seine (Chen et al., ), I2vgen-XL (Zhang et al., ), and PIA (Zhang et al., ) have extensively explored image-to-video generation, achieving substantial progress.

Controllable Generation.
With the development of image controllable generation technology (Zhang et al., ; Jiang et al., ; Mou et al., ; Zheng et al., ; Peng et al., ; Ye et al., ; Wu et al., ; Song et al., ; Wu et al., ), video controllable generation has gradually become a highly focused direction. Significant progress has been made in areas such as pose (Ma et al., ; Wang et al., ; Hu, ; Xu et al., ), trajectory (Yin et al., ; Chen et al., ; Li et al., ; Wu et al., ), subject (Chefer et al., ; Wang et al., ; Wu et al., ), and audio (Tang et al., ; Tian et al., ; He et al., ), greatly facilitating users to generate desired videos according to their needs.

Camera-controlled Video Generation.
AnimateDiff (Guo et al., ) utilizes LoRA (Hu et al., ) fine-tuning to achieve specific camera movements. MotionMaster (Hu et al., ) and Peekaboo (Jain et al., ) explore a training-free method for coarse-grained camera movement generation, but they lack precise control.
VideoComposer (Wang et al., ) offers global motion guidance by adjusting pixel-level motion vectors. In contrast, MotionCtrl (Wang et al., ), CameraCtrl (He et al., ), and Direct-a-Video (Yang et al., ) incorporate camera pose information as side input; however, these methods primarily focus on text-to-video generation and do not effectively leverage 3D geometric priors in camera pose.
Concurrent work, CamCo (Xu et al., ), also facilitates controllable camera generation in the image-to-video task by using epipolar lines to ensure consistency between generated frames and a single reference frame only. However, it does not account for scenarios where frames lack overlapping regions with the reference frame and can thus be regarded as a degenerate version of our approach.

## 3 Method

### 3.1 Preliminaries

Figure: Figure 3: Parameterizations for cameras. Left: Camera representation and trajectory visualization in the world coordinate system. Right: The transformation from camera representations to 3D ray representations as Plücker coordinates given pixel coordinates.
Refer to caption: x3.png

3D Ray Embedding.
Considering camera intrinsics $K\in{\mathbb{R}}^{3\times 3}$ and extrinsics (rotation $R\in\text{SO}(3)$, translation $T\in{\mathbb{R}}^{3}$), it parameterizes the transform from world coordinates to pixel coordinates by projection $u=K\begin{bmatrix}R~{}|~{}T\end{bmatrix}x$.
This low-dimensional representation may hinder neural networks from direct regression.
Instead, we follow (Tseng et al., ) to represent cameras as ray bundles:

$$ $\mathcal{R}=\{r_{1},\ldots,r_{n}\},$ (1) $$

where each ray $r_{i}\in{\mathbb{R}}^{6}$ is associated with a known pixel coordinate $u_{i}$.
Each ray ${\bm{r}}$ can be parameterized by ray direction $d\in{\mathbb{R}}^{3}$ from camera center $P\in{\mathbb{R}}^{3}$ as Plücker coordinates:

$$ $r=\left<m,d\right>\in{\mathbb{R}}^{6},$ (2) $$

where $m=p\times d\in{\mathbb{R}}^{3}$ is the moment vector.
When normalize $d$ to unit length, the norm of the moment $m$ represents the distance from the ray to the world origin.
Given a set of 2D pixel coordinates $\{{(u,v)}_{i}\}^{n}$, ray directions $d$ can be computed by the unprojection transform:

$$ $d=R^{-1}K^{-1}\cdot(u,v,1)^{\rm T},\ m=(-R^{-1}T)\times d$ (3) $$

Text-guided Image to Video Diffusion Model.
Text-guided Image to Video Diffusion Model (Zhang et al., ; ; Xing et al., ) learn a video data distribution by the gradual denoising of a variable sampled from a Gaussian distribution.
For image to video generation, first, a learnable auto-encoder (consisting of an encoder $\cal E$ and a decoder $\cal D$) is trained to compress the video into latent space.
Then, a latent representation $z={\cal E}(x)$ is trained instead of a video $x$.
Specifically, the diffusion model $\epsilon_{\theta}$ aims to predict the added noise $\epsilon$ at each timestep $t$ based on the text condition $c_{\text{txt}}$ and the reference image condition $c_{\text{img}}$, where $t\in\mathcal{U}(0,1)$. The training objective can be simplified as a reconstruction loss:

$$ $\mathcal{L}=\mathbb{E}_{z,c_{\text{txt}},c_{\text{img}},\epsilon\sim\mathcal{N }(0,\mathbf{\mathrm{I}}),t}\left[\left\|\epsilon-\epsilon_{\theta}\left(z_{t}, c_{\text{txt}},c_{\text{img}},t\right)\right\|_{2}^{2}\right],$ (4) $$

where $z\in\mathbb{R}^{F\times H\times W\times C}$ is the latent code of video data with $F,H,W,C$ being frame, height, width, and channel.
Besides, $c_{\text{text}}$ is the text prompt for input video, and $c_{\text{img}}$ is the reference frame of video.
A noise-corrupted latent code $z_{t}$ from the ground-truth $z_{0}$ is formulated as $z_{t}=\alpha_{t}z_{0}+\sigma_{t}\epsilon$, where $\sigma_{t}=\sqrt{1-\alpha_{t}^{2}}$, $\alpha_{t}$ and $\sigma_{t}$ are hyperparameters to control the diffusion process.

Figure: Figure 4: Pipeline of camera-conditioned image-to-video diffusion model.
Refer to caption: x4.png

### 3.2 Overall Pipeline

In this section, we present our novel camera-conditioned method for geometry-consistent image-to-video generation, as shown in Fig. 4.
We first describe cross-frame epipolar line and discreterized epipolar mask, grounded in the principle of camera projection.
Next, we propose epipolar-constrained attention module for the base model in a plug-and-play manner, which effectively make use of feature correlations along epipolar lines.
Further, we discuss the situation when epipolar lines of all frames are outside the image plane and introduce register tokens as a simple yet effective fix.
Finally, we leverage multiple CFG to balance visual quality and camera pose consistency.

Figure: Figure 5: Epipolar line and mask. Left: Epipolar constraint of the $j$-th frame from one pixel at $(u,v)$ on the $i$-th frame. Middle: Epipolar mask discretized by the distance threshold $\delta$, so that only neighboring pixels in green are allowed to attend while those red lined are not. Right: Multi-resolution epipolar mask adaptive to the feature size in U-Net layers.
Refer to caption: x5.png

### 3.3 Epipolar Attention for Noised Features Tracking

Epipolar line and mask.
The proposed epipolar attention mechanism seeks to establish a connection between frames, as shown on the left-hand side of Fig. 5.
Its primary concept involves utilizing the epipolar line as a constraint, which effectively narrows down the potential matching pixels from one target frame to any other frames.
For a single pixel at coordinate $(u,v)$ on the $i$-th frame, the corresponding epipolar line $l_{ij}\in{\mathbb{R}}^{3}$ on the $j$-th frame can be formulated as:

$$ $l_{ij}{(u,v)}=F_{ij}\cdot\begin{pmatrix}u,v,1\end{pmatrix}^{\rm T},$ (5) $$

where $F_{ij}$ is the camera fundamental matrix of two frames, which can be derived as $F_{ij}=K_{j}^{\rm-T}\cdot E_{ij}\cdot K_{i}^{-1}$ given the camera intrinsics $K_{i},K_{j}\in{\mathbb{R}}^{3\times 3}$ and the camera essential matrix $E_{ij}\in{\mathbb{R}}^{3\times 3}$.
We transform the camera pose of the $j$-th frame to be relative to the $i$-th frame for simplicity, thus it holds that $E_{ij}=T_{i\rightarrow j}\times R_{i\rightarrow j}$, where $R_{i\rightarrow j}\in{\mathbb{R}}^{3\times 3}$ and $T_{i\rightarrow j}\in{\mathbb{R}}^{3}$ are the relative rotation matrix and translation vector, respectively.
Due to the contiguous representation of the epipolar line $l_{ij}=Ax+By+C$, we convert it to attention mask by calculating per-pixel distance at coordinate $(u^{\prime},v^{\prime})$ on the $j$-th frame to the epipolar line as

$$ $d_{ij}{(u^{\prime},v^{\prime})}=\frac{(A,B,C)\cdot(u^{\prime},v^{\prime},1)}{ \sqrt{A^{2}+B^{2}}},$ (6) $$

and filtering out those values that are larger than a threshold $\delta$.
We empirically choose half of the diagonal of the feature grid size as the threshold.
This approach optimizes the correspondence search space by significantly reducing the number of candidates from $hw$ to $l$, with $l\ll hw$, thereby enhancing efficiency and accuracy.

Figure: Figure 6: Epipolar attention mask with register tokens. We specify query pixel by red point in the $i$-th frame for clarity. Epipolar attention mask is constructed by concatenating epipolar masks along all frames. We insert register tokens to key/value sequence to deal with zero epipolar scenarios.
Refer to caption: x6.png

Epipolar attention.
We extend current temporal attention with epipolar constraint to leverage cross-frame relationship and inject geometry consistency for video generation.
We denote the noised latent of the $i$-th frame at timestamp $t$ as $z_{t}^{i}\in\mathbb{R}^{hw\times c}$.
And the query, key and value are defined as $q=W_{q}z_{t}^{i}\in\mathbb{R}^{hw\times c}$, $k=W_{k}\{z_{t}^{j}\}_{j=1}^{N}\in\mathbb{R}^{Nhw\times c}$ and $v=W_{v}\{z_{t}^{j}\}_{j=1}^{N}\in\mathbb{R}^{Nhw\times c}$, where $W_{q}$, $W_{k}$ and $W_{v}$ are the projection matrices respectively.
Given the epipolar attention mask $m_{i}\in\mathbb{R}^{hw\times Nhw}$ introduced in Section 3.3, our epipolar attention that captures relevant contextual information between the $i$-th frame and all $N$ frames is then computed as

$$ ${\rm EpipolarAttn}(z_{t}^{i},\{z_{t}^{j}\}_{j=1}^{N},m_{i})=\mathop{\rm softmax }\left(\frac{qk^{\rm T}}{\sqrt{d}}\odot m_{i}\right)v,$ (7) $$

where $\odot$ denotes Hadamard product and $d$ is the dimension of attention heads for attention score normalization. For detailed computation procedures, please refer to Appendix A.

Register tokens for special cases.
For videos with significant camera movements, dynamic objects, or occlusions, there may be cases where some pixels from the $i$-th frame have no corresponding epipolar lines within the image planes of all $N$ frames. This situation can lead to a zero epipolar mask, affecting the computational stability of the epipolar attention mechanism.

To address this issue, we draw inspiration from Timothée et al. () and introduce additional register tokens to the input sequence as a straightforward solution, as illustrated in Fig. 6. Additionally, register tokens are learnable, enabling adaptive learning to address various special cases.

### 3.4 Multiple Classifier-free Guidance

Control for multiple condition.
Similar to DynamicCrafter (Xing et al., ; Esser et al., ), we introduce two guidance scales $s_{\text{img\&txt}}$ and $s_{\text{camera}}$ to text-conditioned image animation, which can be adjusted to trade off the impact of two control signals:

$$ $\displaystyle\mathbf{\hat{\epsilon}}_{\theta}\left(\mathbf{z}_{t},\mathbf{c}_{ \text{camera}},\mathbf{c}_{\text{img\&txt}}\right)$ $\displaystyle=\mathbf{\epsilon}_{\theta}\left(\mathbf{z}_{t},\mathbf{c}_{\text {camera}},\varnothing\right)$ $\displaystyle+s_{\text{img\&txt}}(\mathbf{\epsilon}_{\theta}\left(\mathbf{z}_{ t},\mathbf{c}_{\text{camera}},\mathbf{c}_{\text{img\&txt}}\right)-\mathbf{ \epsilon}_{\theta}\left(\mathbf{z}_{t},\mathbf{c}_{\text{camera}},\varnothing \right))$ (8) $\displaystyle+s_{\text{camera}}(\mathbf{\epsilon}_{\theta}\left(\mathbf{z}_{t} ,\mathbf{c}_{\text{camera}},\mathbf{c}_{\text{img\&txt}}\right)-\mathbf{ \epsilon}_{\theta}\left(\mathbf{z}_{t},\varnothing,\mathbf{c}_{\text{img\&txt} }\right)).$ $$

Multiple scale distillation for acceleration.
If needed, we can distill (Xing et al., ) the
two guidance scales $s_{\text{img\&txt}}$ and $s_{\text{camera}}$ into the model to further avoid the extra inference time brought by three times of forward:

$$ $\mathbf{{\epsilon}}_{\theta}\left(\mathbf{z}_{t},\mathbf{c}_{\text{camera}}, \mathbf{c}_{\text{img\&txt}},s_{\text{camera}},s_{\text{img\&txt}}\right)= \mathbf{\hat{\epsilon}}_{\theta}\left(\mathbf{z}_{t},\mathbf{c}_{\text{camera} },\mathbf{c}_{\text{img\&txt}}\right)$ (9) $$

## 4 Metrics and Evaluation

In this section, we present our reproducible evaluation pipeline. Previous studies have employed various evaluation protocols, resulting in inconsistent metrics due to the lack of a common benchmark. The structure-from-motion (SfM) method such as COLMAP (Schonberger & Frahm, ), struggles to produce stable and accurate predictions when applied to generated videos. This challenge arises because SfM relies on SIFT operators for keypoint identification, which can lead to erroneous matches when assessing generated content. Such inaccuracies may result in unsolvable equations or significantly flawed estimates of camera extrinsics. Contributing factors include the low resolution of these videos (256x256), the presence of dynamic scenes, the absence of true 3D consistency, and issues related to lighting variations and object distortion.

To address these limitations, we adapt the global structure-from-motion method GLOMAP (Pan et al., ) to validate camera pose consistency. Our evaluation pipeline comprises three steps: feature extraction, exhaustive matching, and global mapping. To enhance robustness, we share GT priors for camera intrinsics ($f_{x}$, $f_{y}$, $c_{x}$, $c_{y}$) and allow the structure-from-motion process to focus primarily on optimizing camera extrinsics. Detailed CLI parameters can be found in Appendix B.

Before calculating metrics, we canonicalize the estimated camera-to-world matrices by converting each frame relative to the first frame and normalizing the scene scale using the ${\cal L}_{2}$ distance from the first camera to the furthest cameras. To account for randomness introduced by GLOMAP, we conduct five individual trials for each of the 1,000 sampled videos, averaging only those trials that are successful per sample. The final metrics, including $\rm RotErr$, $\rm TransErr$, and $\rm CamMC$, are averaged on a sample-wise basis.

RotError. We evaluate per-frame camera-to-world rotation accuracy by the relative angles between ground truth rotations $R_{i}$ and estimated rotations $\tilde{R}_{i}$ of generated frames. We report accumulated rotation error along 16 frames in radians.

$$ ${\rm RotErr}=\sum_{i=1}^{n}\cos^{-1}{\frac{\mathop{\rm tr}(\tilde{R}_{i}R_{i}^ {\rm T})-1}{2}}$ (10) $$

TransError. We evaluate per-frame camera trajectory accuracy by the camera location in the world coordinate system, i.e. the translation component of camera-to-world matrices. We report the sum of ${\cal L}_{2}$ distance between ground truth translations $T_{i}$ and generated translations $\tilde{T}_{i}$ for all 16 frames.

$$ $\vspace{-1mm}{\rm TransErr}=\sum_{i=1}^{n}{\left\|\tilde{T}_{i}-T_{i}\right\|_ {2}}\vspace{-1mm}$ (11) $$

CamMC. We also evaluate camera pose accuracy by directly calculating ${\cal L}_{2}$ similarity of per-frame rotations and translations as a whole. We sum up the results of 16 frames.

$$ $\vspace{-1mm}{\rm CamMC}=\sum_{i=1}^{n}{\left\|\begin{bmatrix}\tilde{R}_{i}| \tilde{T}_{i}\end{bmatrix}-\begin{bmatrix}R_{i}|T_{i}\end{bmatrix}\right\|_{2} }\vspace{-1mm}$ (12) $$

FVD. Additionally, to ensure that proposed method coherently improve generative capability and visual quality of base I2V model, we evaluate the distance of generated frames from training distribution by Fréchet Video Distance (FVD) (Unterthiner et al., )(^1^11We use “fvdcal” to calculate FVD metric.).

## 5 Experiments

### 5.1 Setup

Dataset. We train our model on RealEstate10K (Zhou et al., ) dataset, which contains approximately 70K video clips at the resolution of around 720P with camera poses annotated by SLAM-based methods.
We resize video clips from dataset to 256 while keeping the original aspect ratio and perform center cropping to fit in our training scheme.
We sample 16 frames from single video clip when training with a random frame stride ranging from 1 to 10.
We set fixed frame stride of 8 for inference.
We take random condition frame for generation as data augmentation.

Implementation Details. We choose DynamiCrafter (Xing et al., ) as our base image-to-video (I2V) model and implement proposed method on the top of it.
For fair comparision, we also make reproduction work of MotionCtrl (Wang et al., ) and CameraCtrl (He et al., ), since their public accessible versions are either T2V or SVD-based.
We project Plücker embedding into base model by a pose encoder similar to the architecture in CameraCtrl.
We freeze all parameters from base model and train proposed method at the resolution of 256$\times$256.
We set 2 register tokens for the epipolar module to attend when no relevant pixels are on the epipolar line.
We apply the Adam optimizer with a constant learning rate of $1\times 10^{-4}$.
We follow DynamiCrafter to choose Lightning as our training framework with mixed-precision fp16 and DeepSpeed ZeRO-1.
We train proposed method and variants on 8 NVIDIA 3090 GPUs with effective batch size of 64 for 50K steps.

### 5.2 Quantitative Comparison

We compare our CAMI2V with the latest methods in camera controlled image-to-video generation, including DynamiCrafter (Xing et al., ), MotionCtrl (Wang et al., ) and CameraCtrl (He et al., ). As reported in Table 1, our CAMI2V significantly improves the camera controllability, with substantial reductions in RotErr, TransErr and CamMC. Compared to MotionCtrl, our method reduces RotErr by 0.512, translating to a 3.67^∘ decrease in rotational error, which marks a significant improvement. However, enhancing camera controllability inevitably leads to a decline in FVD. Nevertheless, our method surpasses the state-of-the-art method CameraCtrl in both camera controllability and FVD metrics.

### 5.3 Ablation Study

**Table 1: Quantitative comparison with state-of-the-art methods. Our method achieves a significant improvement in camera controllability (25.5% in rotation and 7.77% in translation compared to previous sota), where a decrease of 0.1 in RotErr corresponds to a reduction of 0.72 degrees in rotational error. * denotes the results we reproduced using DynamiCrafter as base I2V model.**
| Method | Publication | TransErr$\downarrow$ | RotErr$\downarrow$ | CamMC$\downarrow$ | FVD$\downarrow$ |  |
| --- | --- | --- | --- | --- | --- | --- |
| VideoGPT | StyleGAN |  |  |  |  |  |
| DynamiCrafter | ECCV 2024 | $9.674$ | $3.086$ | $11.28$ | $113.25$ | $97.181$ |
| DynamiCrafter+MotionCtrl* | SIGGRAPH 2024 | $2.632$ | $0.955$ | $3.197$ | $\bf 72.073$ | $\bf 62.069$ |
| DynamiCrafter+CameraCtrl* | arXiv 2024 | $1.636$ | $0.595$ | $1.958$ | $78.795$ | $66.916$ |
| DynamiCrafter+CamI2V (ours) | - | $\bf 1.509$ | $\bf 0.443$ | $\bf 1.726$ | $77.265$ | $65.806$ |

**Table 2: Ablation study on model variants.**
| Method | Plücker | Epipolar | 3D Full | TransErr$\downarrow$ | RotErr$\downarrow$ | CamMC$\downarrow$ | FVD$\downarrow$ |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| VideoGPT | StyleGAN |  |  |  |  |  |  |  |
| DynamiCrafter+CamI2V (ours) | ✓ | ✓ |  | $\bf 1.509$ | $\bf 0.443$ | $\bf 1.726$ | $77.265$ | $65.806$ |
| ✓ |  | ✓ | $1.605$ | $0.497$ | $1.851$ | $\bf 70.068$ | $\bf 60.830$ |  |
| ✓ |  |  | $1.516$ | $0.482$ | $1.757$ | $72.069$ | $61.999$ |  |
|  | ✓ |  | $4.613$ | $1.844$ | $5.799$ | $107.97$ | $91.318$ |  |
|  |  | ✓ | $6.627$ | $3.677$ | $9.287$ | $115.36$ | $100.58$ |  |
| DynamiCrafter |  |  |  | $10.10$ | $3.268$ | $11.79$ | $127.41$ | $107.12$ |

We conduct ablation study of proposed method on Plücker embedding, 3D full attention and our proposed epipolar attention.
All model variants are trained under the same protocol mentioned in Section 5.1 for 50K steps.
Evaluation results under CFG $s_{\text{img\&txt}}=4$ and $s_{\text{camera}}=1.875$ are provided in Table 2.
In this section, we will discuss the validity of our Plücker pose encoder and epipolar-constrained attention module.

Plücker embedding. The experimental results illustrate that Plücker embedding significantly enhance the capability of base model for camera control.
It also coherently improve visual quality of generated frames, witnessed by a large drop of FVD.
We conclude that Plücker embedding serves as a more expressive position encoding and is more effective for the model to leverage geometric correspondence, thus better cross-frame consistency.

3D full attention alone without Plücker embedding.
The use of 3D Full Attention alone, without employing Plücker embeddings to provide global positional encoding (a form of clean condition), fails to achieve camera controllability. This indicates that even with a 3D global receptive field, the presence of noise limits the model’s ability to generate videos that follow specific camera trajectories, resulting instead in random video generation that conforms to the dataset distribution.

Epipolar attention alone without Plücker embedding.
The use of Epipolar Attention alone, without Plücker embeddings for global positional encoding (a form of clean condition), still retains a degree of camera controllability. This notable phenomenon suggests that explicit constraints can enable the model to grasp knowledge about the physical world.

3D full attention along with Plücker embedding.
The use of 3D full attention with employing Plücker embeddings to provide global positional encoding (a form of clean condition), achieves the best FVD but results in the poorest camera controllability metrics. This supports our earlier discussion in the introduction that 3D full attention loses its ability to track features under high noise conditions. However, in small noise scenarios, it can access a greater number of features, allowing for the generation of videos that conform to the dataset distribution, though these may not align with the camera trajectories provided by users.

Epipolar attention along with Plücker embedding.
When the explicit constraint of Epipolar Attention is combined with the implicit prior provided by Plücker embeddings for global positional encoding, the best camera controllability is achieved. This underscores the necessity of explicit constraints like Epipolar Attention. Although the FVD declines, this is understandable; when FVD reaches a saturation point, enhancing controllability inevitably leads to deviations from the current distribution. Achieving the best FVD with 3D Full Attention alone, without controllability, lacks practical value in the field of controllable generation.

### 5.4 Qualitative Comparison

Visualization on RealEstate10K.
As shown in Fig. 7, we present the visualization results of DynamiCrafter, MotionCtrl, CameraCtrl and our CamI2V. It can be observed that the camera trajectory of our method aligns more closely with GT compared with other methods, and the rendering of certain details appears more realistic in the video generated by our CamI2V.

Figure: Figure 7: Qualitative Comparison on RealEstate10K.
Refer to caption: extracted/5945018/figs/exp_SOTA.png

Out-of-domain visualization. Our CamI2V demonstrates strong generalization capabilities, enabling direct application to camera controlled video generation across out-of-domain content, such as oil paintings, photography, and animation, as shown in Fig. 8.

Figure: Figure 8: Out-of-Domain Visualization.
Refer to caption: extracted/5945018/figs/Exp_out_dom.png

## 6 Conclusion

In this paper, we address the integration of camera poses into diffusion models to enhance their understanding of the physical world in text-guided image-to-video generation. We propose a novel framework utilizing Plücker coordinates as 3D ray embeddings and introduce an epipolar attention mechanism that aggregates features along epipolar lines, ensuring robust tracking even under high noise conditions. Additionally, we incorporate register tokens to manage scenarios where frames lack intersections due to rapid camera movements or occlusions. Our methods significantly improve controllability and stability, achieving state-of-the-art performance on RealEstate10K and out-of-domain datasets. However, challenges remain in high-resolution generation, handling complex camera trajectories, and maintaining generation quality in long videos. Future work will focus on these aspects, alongside releasing checkpoints and training/evaluation codes to support further research.

## Appendix A Core Codes

Figure: Algorithm 1 Spatial Attention Block

Figure: Algorithm 2 Temporal Attention Block with Camera Control

Figure: Algorithm 3 Epipolar Attention Mask

## Appendix B Colmap & Glomap Configuration

We assume SIMPLE_PINHOLE as the common camera model for all video clips and all 16 frames from the same video clip share the same camera intrinsics.
For the feature extractor, we enable estimate_affine_shape and domain_size_pooling in SiftExtraction, while fix camera intrinsics by passing ($f_{x}$, $f_{y}$, $c_{x}$, $c_{y}$) into ImageReader.camera_params.
For the exhaustive matcher, we enable guided_matching and set max_num_matches to 65536 in SiftMatching to make possible more underlying matches.
For the global mapper, we disable BundleAdjustment.optimize_intrinsics and relax the geometric constraint by extending RelPoseEstimation.max_epipolar_error to 4.

## Appendix C GPU Memory and Speed

**Table 3: Comparison on GPU memory usage and speed under DeepSpeed ZeRO-1. * denotes our reproduction on DynamiCrafter. We report full parameter fine-tuning results of DynamiCrafter. Our model can be trained on 24GB custom-level GPUs despite the additional epipolar attention.**
| Method | # Trainable Parameters | GPU Memory (GiB) $\downarrow$ | Time (s) $\downarrow$ |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Inference | Training | Forward | Backward | Optimizer |  |  |
| DynamiCrafter | $1.4$ B | $11.14$ | $23.72$ | $0.413$ | $0.856$ | $1.959$ |
| DynamiCrafter+MotionCtrl* | $63.4$ M | $11.18$ | $16.75$ | $0.387$ | $0.198$ | $0.636$ |
| DynamiCrafter+CameraCtrl* | $211$ M | $11.56$ | $18.44$ | $0.398$ | $0.247$ | $0.723$ |
| DynamiCrafter+CamI2V (ours) | $261$ M | $11.67$ | $21.71$ | $0.403$ | $0.458$ | $0.974$ |

## Appendix D Extra Out-of-Domain Visualizations

Figure: Figure 9: Orbit right on unseen images.
Refer to caption: x7.png

Figure: Figure 10: Zoom in on unseen images.
Refer to caption: x8.png

Figure: Figure 11: Orbit left on unseen images.
Refer to caption: x9.png

Figure: Figure 12: Zoom out on unseen images.
Refer to caption: x10.png

## Appendix E Extra RealEstate10K Visualizations

Figure: Figure 13: Extra RealEstate10K Visualizations.
Refer to caption: x11.png

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. The concept of clean vs. noisy conditions is also neat. The authors even make a figure to illustrate it (Fig. 1), but the knowledge of clean vs. noisy conditions seems not fully exploited in the method? For example, Fig. 1 points out that text and RT are clean conditions, but I don't see this new insight leads to new architectural designs? Adding camera information through Plücker embedding and epipolar attention is a natural choice due to the nature of multi-view geometry, not because it is a clean condition. I wonder if making such separation is really necessary in the exposition.  (This is more of a question for presentation, not technical details.) 

2. More generated video results. All video examples in the supplementary material show only zooming-out camera motion, which, in my experience, is the easiest one to be learnt by the network. Please provide other examples, such as panning left/right, moving up/down, etc.
soundness: 3
strengths: The paper is well written with a few nicely created figures, e.g., Fig. 1 and Fig. 2. The ideas of (1) clean vs. noisy condition & (2) register tokens are neat (but they are also related to the weakness and questions below). Despite tuned with static/rigid scene dataset -- RealEstate10k, from the few generated videos in the supplementary, the motion dynamic of foreground is not lost too much. The discussion in L457-479 is good and clearly supports the design choices. For fair comparison, the authors spend effort reproducing the recipes of baselines (MotionCtrl and CameraCtrl) in their framework (instead of applying the released ckpt out of the box) such that they all stand on a common ground.
summary: This paper extends an existing I2V method (DynamiCrafter) to accept additional camera viewpoint control. The input cameras are parameterized as Plücker rays, serving as a position encoding. The authors additionally inset learnable Epipolar attention layers before temporal attn, which explicitly model cross-view geometry to further enhance the adherence to the input camera.
weaknesses: * Contribution is clearly articulated in L.135-139 but not verified in the experiment: using Plücker embedding and/or epipolar attention has become a norm in the 3D generation literature, e.g. [1]. Applying one of them, if not both, in the video generation field has also been done, e.g., CameraCtrl, CamCo etc. Therefore, the biggest technical contribution I see in this work, to my knowledge, is applying the idea of register tokens to account for occlusion, zero epipolar scenarios, etc. Despite simple, I find this idea neat, but I don't see any experiments ablating this key idea to analyze the effect. A contribution/novelty has be verified by the experiments. If this concern can be addressed, I am happy to raise the rating.

* Method description is a bit insufficient. I need to read Sec. 5.3 to realize Plücker rays are also used as global positional encoding similar to CameraCtrl. This is also part of the final method so it has to be described in Section 3. 



[1] Kant et al., SPAD : Spatially Aware Multiview Diffusers, CVPR24. https://yashkant.github.io/spad/

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 2
questions: Please refer to the weakness section above.
soundness: 3
strengths: + The epi-polar mask attention layer proposed in the paper helps to enhance the camera control ability for video diffusion models. Also, it is plug-and-play – we don't need to retrain other modules of the original pretrained VDM.

+ It is an interesting idea to include the register token to handle cases where the epipolar constraint on correspondences fails, although more discussion and evaluation on it are needed (see the weakness section).

+ The paper addresses the inaccuracy in the SfM for robust evaluation metrics. Specifically, GLOMAP, a more state-of-the-art dense SfM pipeline, is used to validate the camera pose consistency.
summary: The paper proposes a new method to inject camera pose conditions for controlled video generation using a diffusion model. The method uses Plücker coordinates to represent the camera rays and introduces epi-polar mask attention for cross-view attention, which is more efficient than the full 3D attention counterpart. The proposed method is evaluated on the RealEstate10K dataset and achieves state-of-the-art performance.
weaknesses: + Discussion and experiment missing for a key statement in the paper: While the paper states (in L112) that register tokens are included to handle rapid camera movements, occlusions, and dynamic objects, this contribution (also the key difference from CamCo) is not discussed in more detail. For example, how does this additional token help to deal with the non-epipolar constrained correspondences? Can the image-level register token handle pixel-level dense correspondences across frames (like moving arms of people)? In addition, the register token is not ablated, so it’s unclear if it actually helps for dynamic scene generation.

Furthermore, since this is the key difference from CamCo, more discussion and evaluation could help to distinguish this submission from CamCo.

+ Some parts of the presentation are confusing: While the discussion in Fig. 1 and Sec. 1 on different types of conditions based on how much uncertainty the condition can reduce is interesting, how is that related to the epipolar condition? Does the epipolar condition reduce more uncertainty, hence making it a good condition? How does this view of condition serve as a principle to introduce the epipolar condition in the paper? More clarification is needed.

+ For the quantitative results, cross-view consistency is missing, although it has been highlighted in the qualitative subsection (Fig. 7). Is there any reason why it is ignored in the tables as a metric? Cross-frame consistency can be another useful metric to show how the epipolar attention helps, in addition to camera controllability.

+ In Fig. 5, the multi-resolution epipolar mask is shown but not discussed in the text. What is the motivation for using a multi-resolution epipolar mask?

---

confidence: 5
contribution: 1
initial_rating: 3
presentation: 3
questions: See weakness
soundness: 3
strengths: The paper is easy to follow, figures are intuitive and look good. 

Experiments shows the model outperforms all available state-of-the-art works.
summary: This paper introduces CAMI2V, a camera controlled I2V model that integrates several camera-based components, such as epipolar attention and plucker embeddings, image conditions and text prompts.
weaknesses: Novelty of the paper: The contribution of this paper highly resembles CamCo, which is released in June (3.5 months prior to the submission deadline). Both methods use plucker embedding, epipolar lines, and are aimed for image-2-video model. The paper mentioned that CamCo does not supports video trajectories with non-overlapping frames, and the introduction of register token alleviate this problem. Yet this can be  considered as a relatively minor improvement, and is not well supported by experiments. While CamCo does not release its code for reproduction, an ablation study on the token (comparing to other trivial techniques, such as averaging all K/V together), might be helpful to show its effectiveness. Aside from CamCo, CVD (Kuang et.al. 2024) also applies the epipolar attention in its cross-view modules, and CameraCtrl/CVD both use plucker embedding. All these methods challenge the novelty of this work. 

While many of the components proposed by the paper are originated from other prior works, the paper spent a huge portion of space to explain these components, overcomplicating the model itself. For example, in Figure 1/2 the authors try to analyze the differences between image/text conditions and noisy latents and show the importance of the epipolar attention, and later provides very detailed calculation on how to compute the attention maps. These contents are somehow redundant since they are already been proposed in prior works. 

The author also claims the improvement of the evaluation protocol as one of the major contributions, yet the changes are rather trivial. It only replaces COLMAP with GLOMAP, and fixing the GT camera parameters to the registration. 

All of the examples shown in the paper are in relatively small camera changes, which does not support the claims that CAMI2V can handle large camera movements with non-overlapped area. One of the example in Ours_I2V_on_unseen_image_3.gif also shows strong inconsistency across frames (5’th one from the left)

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. Elaboration on the insight of the _clean/noised conditions_. The Pl&uuml;cker embeddings and the proposed epiplolar attention are regarded as clean conditions, but no further clarification is given. It would be preferable to include some mathematical analysis on how clean conditions can reduce uncertainty more effectively than noisy conditions, as the current perspective seems just intuitive and empirical rather than theoretically supported.
2. I appreciate the idea of reserving register tokens for special cases, but the authors fail to demonstrate its effectiveness through essential ablation studies. 
3. In the epipolar attention part, a hard mask strategy is employed, which filters out pixels far from epipolar lines, while [1] proposes the epipolar weight matrix (a form of soft masks). It would be interesting to draw a comparison between them. The hard mask strategy might be computationally more efficient, so for example, more specific details about the computational complexity or running time would be beneficial.
4. The epipolar geometry is based on the assumption that observations are all static. Wouldn't the proposed epipolar attention incur degradation in cases involving dynamic objects?
5. The multiple guidance scale is not only similar to [2] but also lacks explanation and experimental support. The authors should reconsider whether to count it as one contribution (Line 133-134).
6. Limited qualitative results. Camera trajectories used in supplementary materials are similar and relatively simple. Diverse camera trajectories for evaluating are in great need. 

- The authors may have to check the citation in Line 208-209: we follow _(Tseng et al., 2023)_ to represent cameras as ray bundles. 
- In the top middle of Fig. 4, what do the "Project" module and its learnable weights refer to? 
- Please check the format of notations. Make sure they are consistent throughout the paper. For example, the latent code should be in bold: $\mathbf{z}_t$.

[1] Hung-Yu Tseng, et al. "Consistent View Synthesis with Pose-Guided Diffusion Models." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2023.

[2] Jinbo Xing, et al. "DynamiCrafter: Animating Open-domain Images with Video Diffusion Priors." Proceedings of the 18th European Conference on Computer Vision (ECCV). 2024
soundness: 3
strengths: - The idea of introducing epipolar geometry to the temporal attention is straightforward and well-motivated, which allows the model to explicitly reason about the 3D world.
- It is interesting and technically sound to configure register tokens for degenerate cases where no corresponding epipolar lines are detected.
- Experiments on RealEstate10K and in the wild have shown that the proposed method improves the camera motion controllability and stability.
summary: This paper retrofits the text-guided video diffusion model with a camera control module, enhancing the camera trajectory controllability and 3D consistency for image animation. 

By representing a pinhole camera model as a bundle of ray-level 3D embeddings through Pl&uuml;cker coordinates, the denoising process is conditioned on camera poses in a _clean_ and expressive manner. To facilitate multi-view 3D reasoning and efficiency, the epipolar attention mechanism is developed, which only aggregates cross-view information within a limited region along the epipolar line. 

The authors test the method on RealEstate10K and out-of-domain datasets using an improved evaluation pipeline, and demonstrate its superiority over other alternatives.
weaknesses: - One major concern is that the proposed method is somewhat lack of novelty. The epipolar attention mechanism is not new [1], while the Pl&uuml;cker embedding is adopted from [2].  There are some impressive innovations, e.g. the concept of _clean/noised conditions_ and the register tokens, but related exposition and analysis could be more thorough. Please see Q1-5 in the question section for details.
- In the conclusion, the authors announce that their method _significantly_ improves the controllability and stability (Line 534-535), which may be an overstatement. In Tab.1, the advantage of CamI2V (the proposed method) over CameraCtrl is kind of minor, and in Tab.2, the introduction of the epipolar attention quantitatively appears to bring in only limited improvement. It would be nice to include additional (qualitative) results that demonstrate the effectiveness of each contribution. Also see Q6.
- One contribution that the authors adapt GLOMAP for more robust evaluation pipeline raises another concern. It is not convincing that GLOMAP can effectively address limitations such as low resolution and dynamic scenarios. More details and experimental comparisons would be helpful to demonstrate the merits of the proposed evaluation protocol.

[1] Hung-Yu Tseng, et al. "Consistent View Synthesis with Pose-Guided Diffusion Models." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2023.

[2] Hao He, et al. "CameraCtrl: Enabling Camera Control for Text-to-Video Generation." arXiv preprint arXiv:2404.02101. 2024.

Meta-review: This paper presents an image-to-video diffusion model that enables camera control. The input cameras are represented as Plücker ray embeddings. Epipolar attention is used to aggregate cross-frame features and enhance cross-view geometry. The paper introduces register tokens as placeholders to avoid NaN in cases without intersections between frames.

While the proposed method has some acknowledged merits, reviewers have expressed concerns about its novelty and view its contributions as incremental. The area chair has also reviewed the paper and shares similar reservations. Despite the authors' efforts to clarify key contributions in their rebuttal and revision, the concerns remain. Consequently, the paper received final ratings of 5, 5, 6, and 6, which indicates that it does not have enough support for acceptance.

Decision: Reject