Ctrl-V: Higher Fidelity Video Generation with Bounding-Box Controlled Object Motion

Authors: Ge Ya Luo, ZhiHao Luo, Anthony Gosselin, Alexia Jolicoeur-Martineau, Christopher Pal
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: video generation, video synthesis, computer vision, diffusion models, autonomous driving, controllable video generation

Abstract:
Controllable video generation has attracted significant attention, largely due to advances in video diffusion models. In domains like autonomous driving in particular it can be critical to develop highly accurate predictions for object motions. This paper tackles a crucial challenge of how to exert precise control over object motion for realistic video synthesis in a safety critical setting. To achieve this, we 1) use a separate, specialized model to predict object bounding-box trajectories given the past and optionally future locations of bounding boxes, and 2) generate video conditioned on these high quality trajectory predictions. This formulation allows us to test the quality of different model components separately and together. To address the challenges of conditioning video generation on object trajectories in settings where objects may disappear and appear within a scene, we propose an approach based on rendering 2D or 3D boxes as videos. Our method, Ctrl-V, leverages modified and fine-tuned Stable Video Diffusion (SVD) models to solve both trajectory and video generation. Extensive experiments conducted on the KITTI, Virtual-KITTI 2, BDD 100k, and nuScenes datasets validate the effectiveness of our approach in producing realistic and controllable video generation.

Decision: reject
Reviewer scores: 5, 5, 3, 3

Reviews and discussion:
<thread>
creation date:
1729562275660

thread_type:
official_review

<thread_object>
creator: reviewer_ora5

<content>
summary:
This paper proposes Ctrl-V, a method for bounding box controlled video generation. The proposed method is split into two parts. The first part is a generator that synthesizes images of bounding box sequences. The second part takes the bounding boxes images as an input to condition a pre-trained video model. The results demonstrate that the bounding boxes images can be used as a ControlNet style conditioning signal for video generation.

strengths:
- Writing: The paper is well written and easy to follow.
- Simple approach: The proposed two-stage method seems simple and easy to implement. Moreover, the method is generally sound.

weaknesses:
-  No video results: There is no supplementary website with video results. This makes the submission incomplete, as the most important part of a video generation work are the video generations. It’s impossible to judge the results at all, especially regarding temporal consistency.
- Generalizability: The method is trained on driving data and evaluated on that, even though the method is using a pre-trained video model trained on diverse data. It’s not clear how well this method generalizes, especially because the denoising U-Net is fine-tuned. All previous works demonstrate generalizability to any input, which is a clear advantage over the proposed method.
- Missing comparisons: Why does Tab. 2 with the bounding box control metrics not include other trajectory-conditioned video generation methods? It’s not clear why there is only this one made up baseline instead of adapting previous trajectory-conditioned video generation approaches to the same setup.
- Unclear metric scores: The average precision scores in Tab. 3 are not clear. The caption says that prior works don’t evaluate driving datasets. So are these methods adapted to these datasets or do the test datasets differ here? Because it sounds like different evaluation sets were used for different methods.
- No insights provided: The approach is very simple. Create bounding box images with one network, then do ControlNet to condition the generation with these bounding box images. If an approach is that simple without some interesting twist or super convincing results, the paper needs insights. There is no ablation in the paper. For example: How important is it to use bounding boxes images? What happens if just the coordinates and sizes are used as conditioning? There is no clear motivation why the proposed method is the best way to solve this issue.

questions:
I currently rate this paper below the acceptance threshold. Video results are missing, hence it’s impossible to judge the results. Furthermore, only driving datasets are shown while all previous works show generalizability to any kind of scene.

I would like authors to address following questions:
- Why are there no video results?
- Does the method transfer bounding box conditioning to OOD prompts?

I am open to adjusting my rating based on the rebuttal.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730536005404

thread_type:
official_review

<thread_object>
creator: reviewer_y7ge

<content>
summary:
This paper presents a bounding-box (bbox) controllable video generation method called "Ctrl-V." Ctrl-V employs a two-stage pipeline: it first generates a bbox sequence based on both the first and last frames, and then the diffusion model Box2Video generates a video conditioned on the sequence of generated bboxes. To evaluate the performance of this new problem formulation, a new benchmark has been introduced.

strengths:
This paper proposes an innovative video generation method that involves generating a sequence of bounding boxes. Compared to other approaches, it demonstrates superior generation quality.

weaknesses:
1. The figures in the paper require significant improvement. The references to the figures are unclear. For Figure 1, the information presented is sparse. Figure 2 is difficult for readers to interpret due to unclear connections. Additionally, Figure 3's purpose is not well-defined within the context of the paper.
2. The experimental section lacks validity. For Table 1, there is no information regarding the last column for Multi-view on the nuScenes dataset. The meaning of "Ctrl-V + BBox Generator + Box2Video" is ambiguous, as it appears that Ctrl-V is already integrated with the BBox Generator and Box2Video.

questions:
Where is the corresponding description for the figures in this paper?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730624206966

thread_type:
official_review

<thread_object>
creator: reviewer_aa8j

<content>
summary:
This paper presents an approach to generating driving scene videos by utilizing existing pre-trained image-to-video diffusion models, Stable Video Diffusion. Notably, all the bounding box conditionings are encoded into RGB (in contrast to feature representations), and the off-the-shelf VAE is used to render the conditionings to latent representation. The proposed method first interpolates bounding boxes placed on the first frame and the last frame (and optionally other frames?) to generate a sequence of bounding conditionings, and then realistic driving videos are generated with a control-net-based conditioning mechanism. The authors demonstrate reasonable generated quality.

strengths:
- The authors demonstrate that bounding box conditioning can be encoded with the off-the-shelf VAE, which is interesting.
- Investigating the ability to generate driving scene videos with existing image-to-video diffusion models (i.e., SVD) is interesting. 
- The quality of generated videos seems to be reasonable although the authors did not provide the sample of generated videos (in a video format) in the supplemental materials.

weaknesses:
Overall, paper clarity is lacking, and significant revision might be needed to meet the standard conference paper quality. Since I am not specifically an expert in driving video generation, it was difficult for me to precisely understand the technical contribution and soundness of the paper. (Hence, I set the confidence score as 3.)
Here are some suggestions to improve clarity:
- Generated videos (in a video format) are not provided in supplementary materials. Since this paper works on video generation, it is important to provide samples of generated videos in a video format.
- The figure of the overall framework (Figure 2) is not referred to in the paper. Since there is no explanation related to this figure (for instance what is c', k?), it is difficult for me to judge the correctness.
- Some equations are missing. For instance, the authors mentioned Equation 3.3, but the equation does not exist in the paper.
-  I would suggest defining all the notations in the authors' papers instead of directing the readers to other papers for the definition of notations (for instance, in section 3.1).
- In section 4.2, the authors set up baselines but it is not clear to me exactly what these baselines are. For instance, what is "Teacher-forced Box2Video generation"? Exactly how it is different from the proposed approaches. I would appreciate it if the authors could explain all the baselines in detail. 
- The motivation of input conditioning seems unclear. Could the authors provide reasons for why the authors experiment with first/last frame conditioning, 2D/3D conditioning, and first/three frame conditioning, trajectory conditioning in the last frame?
- Are there indeed no baselines from previous works that can be compared with the authors' works? How is the authors' method superior to relevant works? Since I am not familiar with driving scene generation, I will rely on other reviewers' judgments for this. But in general, I would like to see how relevant baselines (even if they are not directly applicable to your problem setting) are inferior to your approach.

questions:
I would appreciate it if the authors could revise the paper to improve clarity. Please see above for some suggestions.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730696647714

thread_type:
official_review

<thread_object>
creator: reviewer_esmh

<content>
summary:
This paper presents a controllable video generation pipeline for autonomous driving, where bounding boxes are first generated and then used as conditions for video generation. Experiments on various driving datasets are conducted to evaluate the pipeline's capabilities.

strengths:
* The paper introduces a novel diffusion-based approach for generating 2D/3D bounding-box trajectories at the pixel level.
* This paper proposes a two-part controllable video generation method that first generates trajectories and then uses these trajectories to condition video generation, achieving solid performance in autonomous driving scenarios.
* The experiments cover multiple autonomous driving datasets and provide sufficient quantitative and qualitative results, effectively validating the model’s applicability in autonomous driving contexts.

weaknesses:
* The primary distinction of the proposed two-part method from one-stage generation is the additional step of bounding-box trajectory generation. However, the paper does not discuss the specific advantages of this intermediate trajectory generation step. Including a comparison would help clarify the benefit of this two-part method.
* While AP is used to assess bounding-box generation location accuracy, this paper lacks evaluations of trajectory smoothness, rationality, and temporal consistency. These aspects are important in assessing the realism of generated motion trajectories.
* Methods compared in the paper, such as Boximator, are designed for general scenarios, while this paper only concentrates on driving scenes, making it difficult to ensure fair comparisons. The paper does not clarify whether the proposed method could work in broader contexts beyond autonomous driving.
* In Table 1, the Teacher-forced method performs worse than the BBox-Generator-and-Box2Video-combination method on KITTI and BDD datasets. Does this imply that the Box2Generator model is not optimal? Discussion on this phenomenon would be valuable.

questions:
* It would be helpful to see more ablation studies, such as whether using the bounding box of the last frame is necessary and the role of the adapter layer.
* More specific visualizations or numerical results would be valuable for cases involving occluded objects or objects that appear midway, as mentioned in the model.
* Discussion on failure cases is insufficient.
* The two-stage generation method involves two diffusion forward processes. However, there is a lack of discussion regarding the computational overhead and parameter cost associated with this approach.
* It would be insightful to explore whether the model can handle more control information.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732176159231

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Rebuttal Summary

comment:
Dear Reviewers,

We appreciate your feedback and have revised our manuscript accordingly:
### Writing Revision
- **Revised figures and captions** Figures 2 with added inline references (*Pg 5 & 9*). [`aa8J`, `Y7ge`]
- **Manuscript revision: Enhancing Writing Clarity** We have rewritten the entire Section 3 and revised portions of Section 4 (*Pg 4-6*). [`aa8J`, `Y7ge`]

### New Experiments
- Added new experiment results evaluating **performance without last-frame bounding boxes conditioning** (*Appendix C.8, Pg 28-30*).  [`EsMH`]

### New Discussions
- Provided **model capacity** information (*Section A.5 (Pg 16)*). [`EsMH`]
- Included discussion on **failure cases** (*Section E.1 (Pg 32)*). [`EsMH`]

### New Supplementary Materials
- Supplemented with **video demos** (https://anongreenelephant.github.io/ctrlv.github.io/). [`oRa5`, `aa8J`, `Y7ge`]
- Our supplementary materials now include simulated NuScene intersection scenarios featuring manually added agents on the bird's eye view map. These scenarios display the new agents' bounding boxes projected in the ego view and include generated videos by our model highlighting interactions with these agents. You can view these videos via the provided [link](https://anongreenelephant.github.io/ctrlv.github.io/).
- We also offer a car crash simulation in our demos (accessible via [link](https://anongreenelephant.github.io/ctrlv.github.io/)) to showcase our model's ability to generate outputs in **out-of-distribution scenarios**. [`oRa5`]

We hope the newly provided results alleviate any confusion. We welcome any further feedback you may have and look forward to the opportunity to continue refining our work.

Authors
</content>
</thread_object>

</thread>
<thread>
creation date:
1732598586416

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Manuscript Revision: Enhancing Writing Clarity

comment:
In response to Reviewer `aa8J` and `Y7ge` suggestions, we have made substantial revisions to **improve the clarity of our manuscript**. Specifically, we have completely reworked the presentation of our method in *Section 3 (Pg 4-6)* and updated *Figure 2 (Pg 5)*, which details our architecture. The revised method explanation utilizes *Figure 2* to guide the reader through the details of our approach. Additionally, we have included a table defining our notation in *Appendix E (Pg 32)*, as well as explanations for the notations used in prior work, which are detailed in the Preliminaries of *Section 3.1*. **All changes in response to reviewer feedback are marked in red.**

In addition, we firmly believe our research makes significant contributions to:
- **Motion Prediction:** Predicting object motions using diffusion network
- **Pixel Space Conditioning:** Exploring novel bounding box conditioning techniques in pixel space.
- **Multimodal Information Encoding:** Encoding object information through color and shapes.

We kindly invite you to review the video demonstrations on the anonymous webpage. These demonstrations highlight the exceptional results achieved by our method, emphasizing its significant contribution to advancing controllable video generation. We would like to emphasize that the autonomous vehicle (AV) research community invests considerable resources in developing highly accurate motion models for vehicles. By integrating our architecture, video generation models can accurately manipulate pedestrian and vehicle movements, addressing a critical need in AV research. This key application motivated our work and informed our architectural design.

We greatly value this opportunity to work with you in enhancing our manuscript. We welcome any further feedback you may have and look forward to continuing to refine our work.

Authors.
</content>
</thread_object>

</thread>


Metareview: