CamCo: Camera-Controllable 3D-Consistent Image-to-Video Generation

## Abstract

Abstract Recently video diffusion models have emerged as expressive generative tools for high-quality video content creation readily available to general users. However, these models often do not offer precise control over camera poses for video generation, limiting the expression of cinematic language and user control.
To address this issue, we introduce CamCo , which allows fine-grained Cam era pose Co ntrol for image-to-video generation.
We equip a pre-trained image-to-video generator with accurately parameterized camera pose input using Plücker coordinates. To enhance 3D consistency in the videos produced, we integrate an epipolar attention module in each attention block that enforces epipolar constraints to the feature maps. Additionally, we fine-tune CamCo on real-world videos with camera poses estimated through structure-from-motion algorithms to better synthesize object motion.
Our experiments show that CamCo significantly improves 3D consistency and camera control capabilities compared to previous models while effectively generating plausible object motion.
Project page: https://ir1d.github.io/CamCo/ .

## 1 Introduction

The rapid evolution of diffusion-based generative models ; ; ; ; has empowered general users to generate video content from textual or visual inputs easily.
Recent advancements in video generative models ; ; ; have underscored the importance of user control ; ; ; over the generated content. Rather than relying on the cumbersome and error-prone process of prompt engineering, controllable generation techniques ensure that the output adheres to more specified and fine-grained control signals, thereby enhancing user satisfaction by producing more desirable outcomes.
Recent studies ; ; have incorporated additional training layers to integrate these control signals, further refining content generation capabilities.

Despite the diverse control signals available (e.g., depth, edge map, human pose) ; ; ; ; ; ; , controlling camera viewpoints in generated content has received little attention.
Camera motion, a crucial filmmaking technique , enables content creators to shift the audience’s perspective without cutting the scene , thereby conveying emotional states effectively . This technique is vital for ensuring that videos are practically usable in downstream applications such as augmented reality, filmmaking, and game development . It allows creators to communicate more dynamically with their audience and adhere to a pre-designed script or storyboard.

To enable camera control in content generation, early methods ; utilize a fixed number of categories to represent camera trajectories, providing a coarse control over the camera motion. They use LoRA  to fuse the camera information and conditionally generate videos that belong to certain categories of camera motion. To improve the camera motion granularity, has proposed to use adapter layers to accept normalized camera extrinsic matrices.
However, this approach uses one-dimensional numeric values to represent the camera pose, which struggles to precisely control the video content when provided with complex camera motion.

To overcome the above issues, we introduce CamCo, a camera-controllable image-to-video generation framework that produces 3D-consistent videos.
Our framework is built upon a pre-trained image-to-video diffusion model, preserving the majority of the original model parameters to maintain its generative capabilities.
To enhance the accuracy of camera motion control, we represent camera pose with Plücker coordinates that encode both camera intrinsics and extrinsics into a pixel-wise embedding, which is a dense conditioning signal for video generation.
Moreover, to improve the geometric consistency of synthesized videos, we introduce a new epipolar constraint attention module in each attention block to enforce epipolar constraints across frames.
Finally, we implement a data curation pipeline that annotates in-the-wild video frames with estimated camera poses, enhancing our capability of generating videos with large object motions.
Our experiments on various domains demonstrate that CamCo outperforms previous state-of-the-art methods in terms of visual quality, camera controllability, and geometry consistency.

Our contributions can be summarized as follows,

- •
We propose CamCo, a novel camera-controllable image-to-video generation framework that can generate high-quality, 3D-consistent videos.
- •
We build the first 3D-consistent video diffusion model by adapting a pre-trained image-to-video diffusion model. We parameterize camera information via Plücker coordinates and incorporate epipolar constraints via new epipolar constraint attention modules.
- •
We introduce a data curation pipeline to handle in-the-wild videos with dynamic subjects and fine-tune CamCo on the curated dataset to enhance its ability to generate videos with both camera ego-motion and dynamic subjects.
- •
Our method exhibits superior 3D consistency, visual quality, and camera controllability when compared with previous works.

## 2 Related Work

### 2.1 Diffusion-based Video Generation

The recent advances in diffusion models ; ; ; ; have provided users with enhanced flexibility in visualizing their imaginations. Leveraging large-scale image-text paired datasets ; , diffusion models have become the state-of-the-art generators for text-to-image (T2I) synthesis. However, due to the lack of high-quality video-text datasets ; , researchers have extensively explored adapting text-to-image generators into text-to-video (T2V) generators.
Text2Video-Zero  builds the first training-free video generation pipeline. AnimateDiff  constructs a motion module applicable to various base T2I models. VideoLDM  and SVD  inflate the T2I models with additional temporal layers and train on curated datasets to improve the visual quality.
The recently released Sora model  has demonstrated impressive video generation results, drawing attention to transformer-based diffusion backbones ; ; ; .
Our work builds upon the open-source SVD  model. We introduce novel camera conditioning and geometry-aware blocks to achieve camera controllability and geometry consistency for the challenging image-to-video generation task.

### 2.2 Controllable Content Creation

Along with the rapid development of generative models that produce content from various input modalities, improving user control over generation has also attracted significant attention.
ControlNet , T2I-adapter , and GLIGEN  pioneered the introduction of control signals, including depth, edge maps, semantic maps, and object bounding boxes, to text-to-image generation. These approaches ensure the synthesis backbone  remains intact while adding trainable modules to maintain both control and generative capabilities.
Similar approaches have been applied to video generation, supporting controls like depth, bounding boxes, and semantic maps ; ; ; . However, controlling camera motion has received limited attention.
AnimateDiff  and SVD  explore class-conditioned video generation, clustering camera movements, and using LoRA  modules to generate specific camera motions. MotionCtrl  enhances control by employing camera extrinsic matrices as conditioning signals. While effective for simple trajectories, their reliance on 1D numeric values results in imprecise control in complex real-world scenarios.
Our work significantly improves granularity and accuracy in camera control by introducing Plücker coordinates and geometry-aware layers to image-to-video generation. Concurrently, CameraCtrl  also employs Plücker coordinates but focuses on the text-to-video generation while paying less attention to 3D consistency.

### 2.3 3D Scene Synthesis

Due to the lack of high-quality 3D scene datasets ; , it is challenging to adapt the success of image and video generation ; ; ; ; ; into 3D generation tasks ; . DreamFusion  addresses this issue by introducing score distillation sampling, which distills knowledge from 2D diffusion models.
This technique has been successfully applied to 3D object synthesis from text ; ; and image ; ; input.
However, due to the limited ability of 2D diffusion models to generate complex scenes, score distillation is not directly applicable to scene generation. Consequently, compositional generation ; ; ; has emerged as a popular approach to tackle these challenges.
Video generators ; ;
have also been adapted to generate simple objects by conditioning the three degrees of freedom (DoF) camera information on the time embeddings of diffusion models.
However, these approaches are not directly applicable to real-world scenarios because their simplifications of camera information do not suffice for real videos that contain diverse extrinsic and intrinsic parameters.
Our work focuses on generating 3D-consistent videos from in-the-wild images, providing insights into the challenging direction of 3D scene synthesis.

## 3 Method

Figure: Figure 2: Overview of our proposed CamCo framework. (a) shows the architecture, where we introduce Plücker coordinates as an effective camera parameterization and an epipolar constraint attention block to enforce geometry consistency. (b) illustrates our epipolar constraint attention block. For each queried pixel via grid sampling from the $i$-th frame $z_{t}^{i}$, we gather information from the corresponding epipolar line in the source frame $z_{t}^{1}$ using a cross-attention layer. The features $E^{i}_{t}$ along the latent space epipolar line encode the local regions around it in the image space.
Refer to caption: x2.png

In this section, we present our novel method for generating camera-controllable, geometry-consistent videos, as shown in Fig. 2. First, we outline the preliminaries of the pre-trained image-to-video diffusion model. Then, we introduce our camera parameterization and control modules, which integrate camera control into the pre-trained video diffusion model. Further, we describe our epipolar constraint attention mechanism, ensuring the geometric consistency of generated videos. Finally, we discuss how to curate data annotations to generate object motion effectively.

### 3.1 Image-to-video Generation

#### Problem Setting

The task of image-to-video generation involves using a single image $I_{0}$ as input of a video generator to produce a sequence of output frames $O_{1},\cdots,O_{n}$ with temporal consistency. In our experiments, $n$ is typically set to 14. To add camera control, we retain the original input and output format while additionally introducing a set of camera information $C_{1},\cdots,C_{n}$. These camera parameters ${C_{i}}$ are used as fine-grained conditioning of the video generator, ensuring that the generated frames follow the viewpoint changes specified by the camera sequence.

#### Base Model Architecture

We train our CamCo based on a publicly available pre-trained image-to-video diffusion model, Stable Video Diffusion (SVD) . SVD is built upon Stable Diffusion 2.1 model  that is originally trained through an EDM  framework.
SVD inserts temporal convolution and attention layers after spatial layers following VideoLDM .
Unlike previous works that only train the temporal layers ; or are training-free , SVD fine-tunes all parameters.

#### Training

SVD incorporates a continuous-time noise scheduler , which supports a continuous range of noise levels. The diffusion model learns to gradually denoise a high-variance Gaussian noise towards the (video) data distribution $\mathbf{x}_{0}\sim p_{0}$. Let $p(\mathbf{x};\sigma(t))$ denote the marginal probability of noisy data $\mathbf{x}_{t}=\mathbf{x}_{0}+\mathbf{n}(t)$ where the added Gaussian noise $\mathbf{n}(t)\sim\mathcal{N}(0,\sigma^{2}(t)\mathbf{I})$, the iterative denoising process corresponds to the probability flow ordinary differential equation (ODE):

$$ $d\mathbf{x}=-\dot{\sigma}(t)\sigma(t)\nabla_{\mathbf{x}}\log p(\mathbf{x}; \sigma(t))dt,$ (1) $$

where $\nabla_{\mathbf{x}}\log p(\mathbf{x};\sigma(t))$ is the score function parameterized by a neural network $D_{\boldsymbol{\theta}}$ through
$\nabla_{\mathbf{x}}\log p(\mathbf{x};\sigma)\approx\left(D_{\boldsymbol{\theta
}}(\mathbf{x};\sigma)-\mathbf{x}\right)/\sigma^{2}$. The training objective is denoising score matching:

$$ $\mathbb{E}\left[\left\|D_{\boldsymbol{\theta}}\left(\mathbf{x}_{0}+\mathbf{n}; \sigma,\mathbf{c}\right)-\mathbf{x}_{0}\right\|_{2}^{2}\right].$ (2) $$

where $\mathbf{c}$ denotes the conditioning information (e.g., the input image $I_{0}$).

Classifier-free guidance  (CFG) is also adopted for better conditioning following.
During training, the condition signal $\mathbf{c}$ is randomly set to a zero tensor $\emptyset$ to properly learn an unconditional model $D(\mathbf{x};\emptyset)$ with a 10% probability.
At inference time, CFG is formulated as follows,

$$ $D_{\omega}(\mathbf{x};\mathbf{c})=\omega(D(\mathbf{x};\mathbf{c})-D(\mathbf{x} ;\emptyset))+D(\mathbf{x};\emptyset),$ (3) $$

where $\omega$ is the weighting coefficient. Following SVD , $\omega$ is empirically set to linearly increase from 1 for the first frame and 3 for the last frame.

### 3.2 Injecting Camera Control to Video Diffusion Model

#### Camera Parameterization

Unlike generating canonicalized 3D objects that operate in a 3-DoF camera space, video diffusion models dealing with real-world dynamic scenes must manage 6-DoF cameras with diverse intrinsic parameters. Consequently, representing camera information using elevation  or camera extrinsics  is suboptimal. Moreover, using one-dimensional numeric values for camera extrinsic as conditioning leads to imprecise camera control  in challenging camera trajectories.
To comprehensively represent both camera extrinsic and intrinsic information, we draw inspiration from light field networks  and adopt Plücker coordinate  . Specifically, let $o$ be the ray origin and $d$ be the ray direction of a pixel. The Plücker coordinate is formulated as $P=(o\times d^{\prime},d^{\prime})$, where $\times$ represents the cross product and $d^{\prime}$ is the normalized ray direction $d^{\prime}=\frac{d}{||d||}$. Given camera extrinsics $E=[\bf{R}|\bf{T}]$ and intrinsics $\bf{K}$,
the ray direction $d_{u,v}$ for 2D pixel located at $(u,v)$ is defined as $d=\bf{R}\bf{K}^{-1}(\begin{smallmatrix}u\\
v\\
1\end{smallmatrix})+\bf{T}$.
All camera poses are defined relatively to the first frame.
Plücker coordinates uniformly represent rays, making them a suitable positional embedding for the network, and they have proven successful in parameterizing 360-degree unbounded light fields .

#### Camera Control Module

To incorporate the camera embeddings, we add a simple adapter layer to each temporal attention block in the pre-trained video generator.
This design best preserves the generation abilities the base model acquired during pre-training ; ; .
Specifically, in each temporal attention block, we first
concatenate the Plücker coordinates with the network features along the channel dimension, and then pass them into the $1\times 1$ convolution layer to project back to the original feature space. The projected features are passed to the remaining temporal attention layers.
Plücker embeddings are downsampled via interpolation if their spatial resolution mismatches with the network features at bottleneck layers.
Inspired by ControlNet , we zero-initialize partial weights of the convolution layer so that at the start of training, the whole network remains unaffected by these additional layers, and we gradually add the camera control through the gradient descent.

### 3.3 Ensuring 3D-Consistent Generation

#### Drawback of vanilla architecture

While using the Plücker coordinates results in more fine-grained camera control, it does not directly ensure the geometric consistency of the generated videos.
This inconsistency issue is primarily rooted in the inefficiency of the vanilla architecture of video diffusion models (e.g., SVD) in modeling geometric relationships across frames.
Recall that the vanilla architecture introduces dense self-attention in spatial and temporal dimensions, where any pixel at any frame is free to attend to any other pixels, potentially leading to pixel-copying behaviors  that appear correct but do not adhere to geometric constraints.
To address this issue, we need to design an attention masking mechanism to ensure each pixel attends to features that are geometrically correlated.

#### Epipolar Constraint Attention (ECA)

Due to the missing depth information from the in-the-wild input images, we do not have the exact pointwise correspondences across frames. Nevertheless, for each pixel in the target view, we can derive an epipolar line in the source view. Epipolar constraint describes that the corresponding point of a feature in one image must lie on the epipolar line associated with that feature’s projection in another image.
We then propose epipolar constraint attention (ECA), which conducts cross-attention between the features on the epipolar line and features at the target location, ensuring that the current frames adhere to projective geometry. We visualize our ECA block in Fig. 2(b) and provide more details in Sec. A.

Our epipolar constraint attention associates the source view $O_{1}$ with the target views $O_{i},i\in[2,n]$ through the epipolar lines.
Due to the diverse camera pose sequences, the epipolar lines can vary in direction and length, posing a challenge for constructing epipolar lines that efficiently support batch attention calculation. To ensure efficiency, we propose constructing the epipolar lines by resampling the visible pixel locations, thereby avoiding variable-length epipolar line features.

At timestep $t$, we represent $z_{t}^{i}\in\mathrm{R}^{hw\times d}$ as the latent feature of $i$-th frame and $E_{t}^{i}\in\mathrm{R}^{hw\times l\times d}$ as the point features along the corresponding epipolar lines, sampled from the feature map of the first frame, where $l$ is the number of points on the epipolar line.
Denote the query, key, and value by $q:=z_{t}^{1}W_{q}\in\mathbb{R}^{hw\times 1\times d}$, $k:=E_{t}^{i}W_{k}\in\mathbb{R}^{hw\times l\times d}$, and $v:=E_{t}^{i}W_{v}\in\mathbb{R}^{hw\times l\times d}$, respectively, where $W_{q}$, $W_{k}$ and $W_{v}$ are the projection matrices. Then, ECA between frame $i$ and the first frame is given by
$\text{ECA}(z_{t}^{i},E_{t}^{i})=\sigma(\frac{qk^{T}}{\sqrt{d}})v\in\mathbb{R}^
{hw\times d}$.
Unlike previous works that apply spatial binary masks  or weight maps  to the original cross-attention in $O((hw)^{2})$, our attention mechanism implements the cross-attention matrix of size $O(hwl)$.
This efficient design avoids unnecessary computation and improves the efficiency of imposing the epipolar constraint by one order of magnitude (assuming $l\approx O(h)\approx O(w)$), which is crucial for video generation where multiple frames are generated simultaneously.
For training, we only update the weights in the ECA layers while freezing the base video model. This preserves the base model’s generation and generalization abilities.

### 3.4 Improving Object Motion

While the above design enforces good camera control and geometric consistency, the model can easily overfit to generate only static scenes with little object motion because it lacks the training data with dynamic scenes.

#### Augmented Training Dataset with better Motion

To overcome this issue, we need to augment our training dataset with dynamic videos that contain rich object motion. However, the number of available dynamic videos with camera poses is greatly limited due to the difficulties in collecting annotations. Consequently, we first randomly sample videos from the WebVid  dataset, which only contains video frames and lacks camera annotations. Then, we annotate the camera poses in each video clip using Particle-SfM . Particle-SfM is a pre-trained framework capable of estimating camera trajectories for monocular videos captured in dynamic scenes. Due to the time-consuming nature of this process, we randomly sample 32 frames from each original video to feed into the Particle-SfM  pipeline.

#### Curating High-quality Samples

Note that the WebVid dataset contains videos of various kinds, including many with static cameras. To ensure the model learns to handle more challenging motions, we need to
filter out sequences where the estimated camera trajectories show minimal displacement.
Due to the inevitable ambiguity between object motion and camera motion, in-the-wild estimation of camera poses can be inaccurate.
We thus use the number of points in the reconstructed sparse point clouds from Particle-SfM  as an indicator of camera annotation quality.
The intuition is that an accurate camera pose estimation comes from well-registered frames. If the frames have more 3D-consistent pixels, more point correspondences will contribute to solving the camera parameters.
With this indicator, we only use videos with accurate camera pose estimation and high object motion,
and construct a training set of high-quality 12,000 video sequences annotated with camera poses.

Figure: Figure 3: Static scene video generation results. The last column provides reference videos that visualize the camera trajectories. The images and trajectories are unseen during training. Regions are highlighted to reveal camera motion. Please check the video results for better visualizations on the project page: [https://ir1d.github.io/CamCo/](https://ir1d.github.io/CamCo/).
Refer to caption: x3.png

Figure: Figure 4: Dynamic scene video generation results where the first frame is generated by SDXL from the prompt in the left. The last column provides reference videos that visualize the camera trajectories. The trajectories are unseen during training. Regions are highlighted to reveal camera motion and object motion better. Please check the video results for better visualizations on the project page: [https://ir1d.github.io/CamCo/](https://ir1d.github.io/CamCo/).
Refer to caption: x4.png

## 4 Experiments

### 4.1 Baselines

We mainly compare our method against Stable Video Diffusion , VideoCrafter , and MotionCtrl .
For MotionCtrl , we take the image-to-video generation checkpoint released by the authors for a fair comparison since their released checkpoint is also trained from Stable Video Diffusion .
For each sequence in RealEstate10k , we first randomly select a starting frame and then take every 8th frame to ensure reasonable camera change.
Note that in our evaluations and visualizations, none of the input images or camera trajectories was seen during training.

### 4.2 Metrics

Since our major goal is to generate videos that follow the camera motion input, we need to measure the camera pose from the output videos. Thanks to COLMAP ; , we are able to annotate the camera poses of videos we obtain. Because structure-from-motion algorithms can only estimate the scene structure up to a certain scale, we further canonicalize the estimated camera poses before calculating the differences. Specifically, we follow our training protocol where we first convert the camera systems to be relative to the first frame and then normalize the scale of the scene by finding the furthest camera against the first frame.

#### COLMAP error rate and matched points

Due to the randomness introduced in COLMAP’s pipeline, we report the average rate with five trials for each video.
We consider a reconstruction to be successful only when a sparse model is created with all 14 frames matched together.
We further report the number of points available in the reconstructed sparse point clouds as a reflection of the 3D consistency of the video frames.

**Table 1: Quantitative comparison against baseline methods on static videos. * denotes that the results of these metrics are averaged for sequences that are successfully processed by COLMAP.**
| Method | FID $\downarrow$ | FVD $\downarrow$ | COLMAP error$\downarrow$ | Points*$\uparrow$ | Translation* $\downarrow$ | Rotation* $\downarrow$ |
| --- | --- | --- | --- | --- | --- | --- |
| VideoCrafter | 30.58 | 342.66 | 93.9% | 263.12 | 4.2882 | 9.2891 |
| Stable Video Diffusion | 18.59 | 281.45 | 64.3% | 434.18 | 2.8476 | 9.5735 |
| MotionCtrl | 14.74 | 229.33 | 14.6% | 447.93 | 3.0445 | 8.9289 |
| Ours | 14.66 | 138.01 | 3.8% | 461.07 | 2.6655 | 7.0218 |

**Table 2: Ablation studies on model variants. * denotes that the results of these metrics are averaged for sequences that are successfully processed by COLMAP.**
| Method | FID $\downarrow$ | FVD $\downarrow$ | COLMAP error$\downarrow$ | Points*$\uparrow$ | Translation* $\downarrow$ | Rotation* $\downarrow$ |
| --- | --- | --- | --- | --- | --- | --- |
| Time Embedding | 16.08 | 152.81 | 12.9% | 453.46 | 3.0327 | 7.7283 |
| 1-Dim Camera | 16.12 | 150.57 | 12.7% | 452.98 | 3.1272 | 7.8846 |
| w/o Plücker | 15.95 | 144.16 | 14.7% | 458.42 | 2.7335 | 7.2427 |
| w/o Epipolar Attention | 15.51 | 144.30 | 10.2% | 458.71 | 3.0502 | 7.2215 |
| Full Model | 14.66 | 138.01 | 3.8% | 461.07 | 2.6655 | 7.0218 |

#### Pose Accuracy

We extract the camera-to-world matrices of the COLMAP predictions and obtain the estimated rotation and translation vectors, similar to CameraCtrl .
The relative rotation distances are then converted to radians, and we sum the total error of 14 frames,

$$ $R_{\text{err}}=\sum_{i=1}^{n}\text{arcos}(\frac{\text{tr}(R_{\text{out}_{i}}^{ T}R_{\text{gt}_{i}})-1}{2}).$ (4) $$

The norm of the relative translation vector for each frame is also summed together to form the translation error of the whole video,

$$ $T_{\text{err}}=\sum_{i=1}^{n}\left\|T_{\text{out}_{i}}-T_{\text{gt}_{i}}\right \|_{2}.$ (5) $$

#### FID and FVD

Additionally, we evaluate the visual quality of the generated video frames, regardless of the camera following ability or geometry quality. FID  and FVD  are calculated based on deep features extracted from video frames. We, therefore, use FID  and FVD  to measure the distance between the generated frames and the corresponding reference videos.

### 4.3 Main Results

**Table 3: Quantitative comparison on generated dynamic videos.**
| Method | Stable Video Diffusion | MotionCtrl | Static Only | Uncurated Dynamic | Full Model |
| --- | --- | --- | --- | --- | --- |
| FID $\downarrow$ | 27.43 | 36.21 | 30.78 | 23.22 | 22.19 |
| FVD $\downarrow$ | 121.05 | 188.86 | 169.48 | 148.08 | 137.59 |

#### Static Generation

Due to the lack of large-scale camera pose annotation of dynamic videos in the wild, we quantitatively evaluate our method against previous works on static videos. Specifically, we randomly sampled 1,000 videos from the Realestate-10k  test set. The camera poses were originally annotated using COLMAP ; . We provide qualitative and quantitative comparisons in Fig. 3 and Tab. 1, respectively. In Fig. 3, the ground truth cameras are visualized through the camera reference videos. Compared to the baselines, our method produces results with the best camera-following ability. In contrast, the results produced by Stable Video Diffusion do not follow the camera control, and MotionCtrl demonstrates inaccurate camera-following results. Our results outperform the baselines by a large margin in all metrics, indicating our superiority in visual quality, camera controllability, and geometric consistency. Note that for all experiments, neither the input image nor the camera trajectory was seen during training.

#### Dynamic Generation

Similar to the static scenario, we evaluate our method against previous works on randomly sampled 1,000 videos from the test split of our annotated WebVid  dataset. We provide qualitative and quantitative comparisons in Fig. 4 and Tab. 3, respectively. In Fig. 4, we provide the generated videos when inferencing text-to-image results produced by SDXL . The last column provides the camera trajectories. Compared to the baselines, our method excels at producing perspective changes as well as object motion. Notably, in the eagle case, the baselines fail to produce reasonable object motion or follow the camera change configurations, whereas our method successfully generates both camera changes and object movements vividly. In Tab. 4, we provide FID and FVD calculated against ground truth videos sampled from the WebVid dataset. Our full model produces the best FID and FVD values, indicating the model produces the best visual quality. In comparison, the model trained on the uncurated dynamic dataset and the static version model show inferior results, indicating poorer object motion.
Note that for all experiments, neither the input image nor the camera trajectory was seen during training.

### 4.4 Ablation Studies

In this section, we conduct several ablation studies on the model variants, including different versions of camera conditioning and the importance of our epipolar attention block. “w/o epipolar attention” denotes the baseline where we remove the epipolar constraint attention from our model and only use the temporal convolution layers. “w/o Plücker” refers to the model variant where we remove the Plücker coordinate input in the temporal blocks and rely on the epipolar constraint attention to modulate the camera control. “1-dim camera” means using 1-dimensional camera matrices as the conditioning signal for modulating the features in temporal blocks instead of using 2-dimensional Plücker coordinates. “Time embedding” refers to modulating the time embedding feature with camera matrices instead of using Plücker coordinates. As can be seen in Tab. 2, the full model delievers best results in both 2D visual quality (FID and FVD) and camera pose accuracy (COLMAP error, points, translation and rotation error). In contrast, the model variants lack the ability of accurately controlling the camera poses and can lead to deteriorated visual appearance.

## 5 Conclusion

In this paper, we introduced CamCo, a camera-controllable framework that produces 3D-consistent videos. Our method is built upon a pre-trained image-to-video diffusion model and introduces novel camera conditioning and geometry constraint blocks to ensure camera control accuracy and geometry consistency. Further, we proposed a data curation pipeline to improve the generation of object motion. Our experiments on images of various domains demonstrate the superiority of CamCo against previous works in terms of camera controllability, geometry consistency, and visual quality.

## Appendix A Additional Details on Epipolar Constraint Attention

An epipolar line refers to the projection on one camera’s image plane of the line connecting a point in 3D space with the center of projection of another camera. Specifically, let $\mathbf{K},\bf{R},\bf{T}$ be the relative camera pose between frame $i$ and the first frame,
the projections of point $p$ and camera origin $o=(\begin{smallmatrix}0\\
0\\
0\end{smallmatrix})$ on source view the first frame are denoted as $\text{Proj}\left(\mathbf{R}\left(\mathbf{K}^{-1}\mathbf{p}^{i}\right)+\mathbf{
T}\right)$ and $\text{Proj}\left(\mathbf{R}\left(\mathbf{K}^{-1}(\begin{smallmatrix}0\\
0\\
0\end{smallmatrix})\right)+\mathbf{t}\right)$, respectively, where Proj is the projection function. The epipolar line is, therefore, given by

$$ $\mathbf{L}=\mathbf{o}+c\left(\mathbf{p}-\mathbf{o}\right),$ (6) $$

where $c\in\{0,\infty\}\in\mathbb{R}$.

## Appendix B Experiment Details

### B.1 COLMAP configuration

We use the widely adopted COLMAP configuration for few-view 3D reconstruction [] and assume all frames in each video share the same camera intrinsics. For the feature extractor, we set
--SiftExtraction.estimate_affine_shape 1 --SiftExtraction.domain_size_pooling 1 --ImageReader.single_camera 1 . For the feature matching process, we set --SiftMatching.guided_matching 1 --SiftMatching.max_num_matches 65536. For each video, we retry the process five times, at most.

### B.2 Particle-sfm workflow

We use the full default configuration of Particle-sfm for the best performance. Videos are processed at their original resolution without any resizing or cropping operation. Given a long video sequence, we randomly sample a frame stride and then uniformly obtain a set of 32 frames according to the stride. The 32 frames are then sent into Particle-sfm for camera annotation.

### B.3 Additional Settings

We use the same train-test split of RealEstate-10k [] as in PixelSplat [].
Baseline methods are ran at their originally designed resolution. Thanks to the image-to-video nature, when the input image is at the resolution of $256\times 256$, despite what the output image resolution is, the output becomes undistorted when being resized to $256\times 256$. For baselines, we use the open-source checkpoints released by the authors. We use “clean-fid”(^1^11[https://github.com/GaParmar/clean-fid](https://github.com/GaParmar/clean-fid)) and “common-metrics-on-video-quality”(^2^22[https://github.com/JunyaoHu/common_metrics_on_video_quality](https://github.com/JunyaoHu/common_metrics_on_video_quality)) for calculating FID and FVD, respectively. For FVD, we report the results in VideoGPT [] format.

## Appendix C Implementation Details

Our CamCo model builds upon SVD [], which is a UNet-based open-source image-to-video diffusion model with architecture similar to VideoLDM [].
In our experiments, we use a relative camera system where all camera poses are converted to become relative to the first frame. The camera of the first frame is placed to the world origin and rotated to the face towards the x-axis. During training, we randomly sample the strides of the image sequences to augment the speed of the video. The sampled camera extrinsics are then normalized to have a unit maximum distance against the world origin to stabilize the training.
For simplicity and efficiency, the training frames are first center-cropped to square images and then downsampled to $256\times 256$ resolution. During inference, we use 25 sampling steps to obtain the 14 frames. The decoding chunk size of the latent decoder is set to 14 frames for the best visual quality. Our model is trained with batch-size 64 in total on 16 A100 GPUs for around two days. The learning rate is set to 2e-5 with a warmup of 1,000 steps using the Adam optimizer. We use fp16 training implemented with “accelerate”(^3^33[https://github.com/huggingface/accelerate](https://github.com/huggingface/accelerate)) library.

## Appendix D Limitations and Future Work

Because our training data are video frames with the same camera intrinsics, our model generates images that have the same camera intrinsic as the input image. Therefore, the model can not generate complex camera intrinsic changes such as dolly zoom. Additionally, CamCo is able to generate 14 frames at the resolution of $256\times 256$, which only covers limited viewpoints and can be insufficient for large scenes. We will explore generating longer, and larger resolution videos in future work.

## Appendix E Broad Impacts

The paper focuses on generating videos from image input. Like all other large models trained from large-scale datasets, our model can contain certain social biases. Such biases could perpetuate stereotypes or misrepresentations in generated videos, thus influencing public perceptions and potentially leading to societal harm.
Our T2I base model is trained with NSFW content removed and is also released with a safety checker. The publicly available safety checker maintained by the diffusers library can also protect our video generator from harmful usage.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: **MotionCtrl**

MotionCtrl offers user-specified object motions, and they show good object motions with custom trajectory. Why in the comparison section of the paper, the MotionCtrl always output very limited object motions.
soundness: 3
strengths: **Good presentation**

The paper is easy to read and have good figures to show the method. Overall the pipeline is reasonable and each module is clear introduced.

**Good experimental results**

The model shows superior results over existing controllable camera image-to-video generation results, with much lower COLMAP error and FVD.  

**Good generalization performance.**

I appreciate the author conduct rich experiments on multiple source unseen data to show the generalization ability. The author also shows the good 3D consistency in the generated videos.
summary: The paper proposes to address the image-to-video generation with precise camera control.  The author parameterizes the camera pose using Plücker coordinates and adopts a epipolar attention module to improve 3D consistency. The author further augments the training set with a curated dataset to better capture object movements. Experiments on different source datas demonstrate the effectiveness and generalization of the model.
weaknesses: **Very Limited Contribution**

The 2 fundamental modules introduced in this paper: Plücker embedding and epipolar attention are all commonly used techniques. For Plücker embedding, previous work in CameraCtrl (He et al., 2024)  also used the same technique. Even their used for text to video generation other than image-to-video, but the key techniques are the same, that is how to better incorporate the camera pose into the video generation model other than using R and t. Also works in 3D generation[1, 2] also uses the same one. Similar for epipolar attention used in (Tseng et al., 2023). Compare to Tseng, one improvement is the efficiency of the attention. However, I could not find experimental support to show the improvement of the speed or training time. Besides, the author doesn't compare to Tseng's attention, and I could not tell is there more significant difference with other implementations. Also for the dataset augmentation, it seems the pipeline directly follows MotionCtrl. 


**Training set**

Given the author augments the training set, but in order to demonstrate the effectiveness of each proposed module, the comparison to MotionCtrl should be conducted on the same training set. The author should further mention this in the paper to ensure the improvement is coming from the better camera encoding and epipolar attention other than high-quality data sources.  



**Object and camera movement decomposition**

MotionCtrl offers the flexibility to decompose the camera and object motions in generation. However, it seems the proposed method could not achieve this. Also the object motion shows in the paper and video seems very limited. I am expect to see more objects with higher dynamic motion for comparison.




[1] Chen et al. Ray Conditioning: Trading Photo-consistency for Photo-realism in Multi-view Image Generation. ICCV 2023
[2] kant et al. SPAD : Spatially Aware Multiview Diffusers. CVPR 204

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: N/A
soundness: 3
strengths: 1. The authors implement a data curation pipeline that annotates in-the-wild videos with estimated camera poses using structure-from-motion algorithms. This enhances the model's ability to generate plausible object motion in addition to camera movements, addressing the challenge of synthesizing dynamic scenes.
2. The paper provides thorough quantitative and qualitative evaluations, demonstrating that CamCo outperforms baseline methods in terms of visual quality, camera controllability, and geometric consistency. Metrics like FID, FVD, and COLMAP error rates support these claims.
3. The inclusion of ablation studies validates the effectiveness of the proposed components, such as the Plücker coordinate parameterization and the ECA module. This strengthens the paper's contributions by showing the impact of each component.
summary: The paper introduces CamCo, a novel framework for camera-controllable, 3D-consistent image-to-video generation, enabling fine-grained control over camera viewpoints while ensuring geometric consistency in the generated videos. This is achieved by: 1. camera pose parameterization with Plücker Coordinates; 2) epipolar attention to improve 3D consistency; 3) performing data curation and fine-tuning for dynamic Scenes using SFM. Results show a step forward comparing with prior camera controllable video generation in terms of accuracy of cameras and 3D consistency.
weaknesses: 1. The biggest weakness of the paper is its technical contribution, its main designs are Plücker coordinates and epipolar attention, but none of these is exactly novel, even on the constrained domain of camera controllable video generation --- the former was used in CameraControl [1] and the later was used in Collaborative Video Diffusion [2]. The authors should discuss what is novel about their approach while using these techniques.
2. Without sufficient dynamic training data, the model tends to overfit to static scenes with minimal object motion. Although the authors address this by curating additional dynamic data, it indicates a reliance on data quality and diversity --- this is especially concerning given that the authors choose do develop their model on SVD, a model that is known to generate very limited motions.
3. The quantitative evaluation primarily uses FID, FVD, and COLMAP error rates. While these are standard metrics, they may not fully capture perceptual quality, temporal coherence, or user satisfaction. For instance, FVD is known to be biased towards good qualitative single frames without taking the overall motion coherence into account. Incorporating additional metrics such as de-biased FVD [3] or user studies could provide a more comprehensive assessment of the model's performance.


[1] He et al. CameraCtrl: Enabling Camera Control for Text-to-Video Generation, in arXiv, 2024.

[2] Kuang et al. Collaborative Video Diffusion: Consistent Multi-video Generation with Camera Control, in NeurIPS, 2024.

[3] Ge et al., On the Content Bias in Fréchet Video Distance, in CVPR 2024.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. Why is epipolar attn. inserted before temporal attn., not after? Any intuition or empirical evidence?
2. Why training with WebVid doesn't result in a ShutterStock watermark? As far as I know, a big portion of WebVid videos contain a "ShutterStock" watermark. In fact, WebVid is also not publicly available anymore. Any method trained with the pre-downloaded copy (like the authors clarify in L546) is hard to reproduce. If any preprocessing is performed to prevent the watermark from emerging in the generated videos, the authors should disclose and explain the details.  
3. Table 3 doesn't have a corresponding discussion. L.505-506 says Table 4, but I don't see Table 4 anywhere, so I assume it's a typo?
4. Will the curated Particle-SfM annotations for dynamic videos be released?
soundness: 3
strengths: The proposed method is explained clearly. The authors compare with a few baselines and show the best adherence to the input camera according to Table 1 (despite the most important one is missing). The curated dynamic video dataset can be a good contribution if released.
summary: This paper extends the I2V model SVD by adding camera control. The proposed method, CamCo, parameterizes input camera poses as Plücker coordinates and feeds the condition into temporal attention and the newly added Epipolar attention layers. These two layers are tuned to teach the network how to react to the provided 3D camera condition, while the remaining layers, e.g., self-attn. layers, are frozen to retain the quality of the generated videos.
weaknesses: 1. Despite the effort of annotating the dynamic dataset WebVid, from the results in the suppl. page, the foreground dynamic is still largely lost. Even the eagle example doesn't show prominent motion; in another example where a bird flying above a lake, the proposed method does produce more object translation than baselines, but the object size is fairly small. Arguably, the proposed method still suffers from the common problem shared with the state-of-the-art camera-conditioned methods, i.e., lost of foreground dynamics. Any idea how to improve here?
2. Limited novelty: Epipolar attention and Plücker coordinate are very standard in 3D generation field, e.g., [1]. Existing video generation methods have applied one of them, if not both, to facilitate camera conditioning, e.g., CameraCtrl also adopts Plücker coordinate to parameterize cameras. If the ideas are similar, I expect to see in-depth discussion/analysis why one is superior than the other. The submission nonetheless compares with CameraCtrl only "qualitatively" not quantitatively. CameraCtrl has released the code a while back and this submission follows its evaluation metrics, so quantitative comparison should not be too difficult. 


[1] Kant et al., SPAD : Spatially Aware Multiview Diffusers, CVPR24. https://yashkant.github.io/spad/

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: - As mentioned above, how do the epipolar constraints affect the dynamic video cases? (also related to the last comment above)
- When comparing to CameraCtrl (which seems closely related), it seems the text 2 video version of CameraCtrl is used, this paper also shows results on SVD. Are the comparisons done for both models using SVD?
soundness: 3
strengths: - The presentation is clear.
- The ablation studies are good. The metrics used to evaluate the method are reasonable. A good set of visual results are provided in the supplementary.
- The design choices of using plucker embeddings and epipolar attention are reasonable.
summary: This paper presents a method to condition a pre-trained image to video model with explicit 3D camera control. The camera information is represented as plucker embeddings and features extracted from these are provided as input to the temporal layers. In addition, an epipolar attention layer is introduced to better preserve the geometry/rigidity of the reconstructed scenes. The method is trained on SVD using camera parameters obtained from both static and dynamic scenes.
weaknesses: - Many pieces of the work have been also discussed in other previous/concurrent work. Use of plucker embeddings for cameras is becoming a standard. The use of epipolar attention has been introduced in previous work that tackle multi-view generation. Hence, although reasonable, the paper does not introduce very specific novel contributions.
- While the use of epipolar attention improves scene rigidity, it is not discussed at all how this affects the dynamic videos. The epipolar constraints would not hold for objects that are moving.
- The dynamic video examples shown in the supplementary -seem to have relatively low motion.

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 4
questions: How does the proposed method compare with the CameraCtrl approach?
soundness: 3
strengths: The problem this method tackles is an important and impactful one. The manuscript is very clear and well written. The evaluation is adequate and the proposed method outperforms the shown baselines quantitatively. The qualitative results are certainly not as impressive as recent video generation models, like sora, but adequate given the limited capabilities of the SVD base model.
summary: The paper introduces a mechanism that adds camera control to a text-to-video diffusion model, i.e., SVD. For this purpose, the authors propose to finetune the model on the WebVid dataset using a conditioning mechanism that uses Plucker coordinates and epipolar attention to improve consistency. The proposed method is shown to outperform the MotionCtrl baseline.
weaknesses: In my assessment, the main weakness of this submission is a lack of novel contributions. This paper is not the first to propose camera conditioning for text-to-video generation (e.g., line 93), it does not introduce epipolar attention, and it does not introduce Pucker coordinates. Similar to other works in this general area this is a "systems paper, which combines several known components into an adapter network. Systems papers are important and valuable, but only if they enable a new capability or solve a problem in some novel and better way. This is not the case here, because very similar systems have been described before, like MotionCtrl, and the quantitative improvements of this work are marginal (see e.g. Table 1). Without a clear technical contribution on the methods side and only marginal improvements for the targeted application, it seems difficult to get excited about this submission.

What I also find concerning is that the CameraCtrl paper is only tangentially mentioned. This paper has been on arxiv for more than 7 months and basically claims the same system's level contributions, including Plucker conditioning. CameraCtrl is not included in the baseline comparisons. The authors claim that it's concurrent work, but in an area that is so fast paced, I find it difficult to convince myself that a paper that has been online for such a long time should be treated as being concurrent. Regardless, this is not my primary concern.

Meta-review: Summary:
The paper tackles the important problem of camera-controlled image-to-video generation. It achieves this by using Plücker coordinates to parameterize camera pose input and an epipolar attention module to enforce epipolar constraints. With fine-tuning on real videos and estimated camera poses, the experiments show improved 3D consistency and camera trajectory control compared to prior methods.  

Strength: 
- The exposition is good. The paper is easy to read and the figures are well-prepared.
- Improved results over existing controllable camera image-to-video generation results

Weakness:
- Limited technical contributions. 
- Foreground dynamics are limited (due to the use of epipolar constraints)

Justification: 
- Four reviewers are leaning negative about the paper, primarily due to the limited contributions from the paper. Unfortunately, the authors did not engage with the reviewers in the rebuttal period. As the concerns from the reviewers are not resolved, the AC finds no ground to accept.

Decision: Reject