Ctrl-V: Higher Fidelity Autonomous Vehicle Video Generation with Bounding-Box Controlled Object Motion
Authors: Ge Ya Luo
ZhiHao Luo
Anthony Gosselin
Alexia Jolicoeur-Martineau
Christopher Pal
Venue: Accepted by TMLR ()

Abstract: Controllable video generation has attracted significant attention, largely due to advances in video diffusion models. In domains such as autonomous driving, developing highly accurate predictions for object motions is essential. This paper addresses the key challenge of enabling fine-grained control over object motion in the context of driving video synthesis. To accomplish this, we 1) employ a distinct, specialized model to forecast the trajectories of object bounding boxes, 2) adapt and enhance a separate video diffusion network to create video content conditioned on these high-quality trajectory forecasts, and 3) we are able to exert precise control over object position/movements using bounding boxes in both 2D and 3D spaces. Our method, Ctrl-V, leverages modified and fine-tuned Stable Video Diffusion (SVD) models to solve both trajectory and video generation. Extensive experiments conducted on the KITTI, Virtual-KITTI 2, BDD100k, and nuScenes datasets validate the effectiveness of our approach in producing realistic and controllable video generation. Project page: \url{https://oooolga.github.io/ctrl-v.github.io/}

Published in Transactions on Machine Learning Research (05/2025)
Ctrl-V: Higher Fidelity Autonomous Vehicle Video Genera- tion with Bounding-Box Box Controlled Object Motion
Ge Ya Luo Mila, Université de Montréal
olg..zu@kmontrral.cc
Zhi Hao Luo Mila, Polytechnique Montréal
siozzhiha@mmilln quebec
Anthony Gosselin
saathony-gossllinneee
Mila, Polytechnique Montréol
Alexia
Samsung- SAIT AI Lab, Montreal
Christopher Pal
christophhermml@@pllymt.oom
Mila, Polytecknique Montréal
Canada CIFAR Al Chair
Reviewed on OpenReview: https: /ttp://ppppe openreview. net/ forum? id= BMCskHB
Abstract
Controllable video generation has attracted significant attention, largely due to advances in vidco diffusion models. In domains such as autonomous driving, developing highly accurate predictions for object motions is essential. This paper addresses the key challenge of en- abling fine-grained control over object motion in the context of driving video synthesis. To accomplish this, 1) employ distinet, specialized model to forecast the trajectories of object bounding boxes, 2) adapt and enhance a separate video diffusion network to create video content conditioned on these high-quality trajectory forecasts, and 3) we are able to exert precise control over object position /oovements using bounding boxes in both 2D and 3D spaces. Our method, Ctrl-V, leverages modified and fine-tuned Stable Video Diffu- sion (SVD) models to solve both trajectory and video generation. Extensive experiments conducted the KITTI, Virtual-KITTI 2, BDD100k, and nuScenes datasets validate the ef- fectiveness of our approach in producing realistic and controllable video generation. Project page: https:// githubbbitttbbb..... cto/ctllv.... www.ivvb..t........... ioo
1 Introduction
Recent advances controllable image generation have enabled the creation of highly realistic images from various conditioning inputs, including points, bounding boxes, scribbles, segmentation maps, and skeleton poses. Yet, translating this control to video generation is markedly more challenging due to the added temporal dimension. Incorporating time dynamics into diffusion models significantly complicates controllable video generation, as it requires accounting for object interactions, physical consistency, and coherent motion across frames.
Numcrous recent studics have examined different forms of controllability for vidco generation. Rescarchers have used variety of methods for control, including conditioning on information such as canny edge and depth maps (Zhang et al. 2023b)), similar visual information (Chen et. al. (023)) ) optical flow (Hu & Xu

---

Published in Transactions on Machine Learning Research (05/2025)
BoundIng-box Prediction Samples
Videos Sampled from Box2Video
Inputs
First Frame
(bounding boxes)
Last Frame (bouncing boxes)
Ctrl
Figure Overview of Ctrl-V's generation pipeline: (Left) inputs: Our inputs include an initial frame, its corresponding bounding box image and the final frame's bounding box image. (Middle) generated bounding box trajectories: We demonstrate three distinct possible trajectory sequences produced by our diffusion-based bounding box motion generation model - BBox Generator. (Right) generated video clips: Our Box2Video model conditions on the generated bounding box trajectory frames to produce the final video clips.
2023 and pose sequences Karras et al. (2023) These control inputs are often expensive to produce, especially when sequences of them are required to condition video. Models that use accessible conditioning such as bounding boxes require additional input such as text to help with the generation process (Wang let al. (2024)). There a strong demand for a video generation model that allows for easy and effective control, particularly for practical uses such as navigation and safety in autonomous vehicles.
It is widely recognized that most of the state-of-the-art, real-world autonomous vehicle systems trained and/or tested in some manner in simulation. Current AV simulators generally rely traditional hand- engineered computer graphics environments, exemplified by the well-known (open source) CARLA sim- ulator (Dosovitskiy et al.) 2017). However, multiple experts in the field have envisioned future where it is possible to construct simulators from genorative models, potentially revolutionizing the way we train and test autonomous vehicles. Commercial grade neural sensor simulation approaches, like the one described by Yang et al., are starting to emerge publicly.
In this rescarch, objective is to create video generation model capable of controlling the movements of objects, particularly for videos related to autonomous driving. Specifically, we aim to generate higher fidelity videos controlled by the beginning and ending positions of 2D and 3D bounding boxes without the help of other modes of control. Our two-part method includes a diffusion-bbsset model that generates the motions and dynamics of objects in the form of bounding box videos (2D images of the bounding boxes evolving over time), and generative model of videos according to those bounding box videos. To this end, we choose to train and test our model on driving datascts as they contain challenging scenes rich with different types of bounding boxes as well as complex movement and irregular appearing and disappearing objects. In our experiments, we show that our model generates videos that adhere tightly to the desired bounding box motion conditioning, accurately depicting desired object movements. Additionally, through our novel pixel-level bounding box generator and conditioning, our method robustly handles the appearance and disappearance of different objects a scene, including cars, pedestrians, bikers, and others.
In this paper, we present Ctrl-V, a diffusion-based bounding box conditional vidco generation method that addresses multiple challenges and makes the following contributions to generate higher- higher-fidelity fidelity videos using diffusion techniques. Our contributions be enumerated as follows:
2

---

Published in Transactions on Machine Learning Research (05/2025)
1. A Novel Diffusion Approach for Predicting Bounding Box Trajectories: Our approach generates video frames with 2D or 3D bounding box Injeelories at the pixel-level based on their initial and final states, and the first frame of RGB video. Our results show that our proposed approach yields higher quality bounding box trajectory predictions than a recently proposed state of the art method (sce in Table D.
2. 2D and 3D Bounding Box Conditioned Vidco Generation with Diffusion: We present a mothod that allows video generation by conditioning on 2D or 3D bounding box trajectories which allows fine- grained control over the generated videos. Our results (in Table 2 indicate that this approach can generate video that is dramatically better than a state-of-the-art Stable Video Diffusion baseline fine- tuned without this conditioning mechanism across four commonly used quality metries. Our approach further improves upon prior work by enabling the following capabilities: a. Multi-subject Generation: Synthesizing multiple subjects in videos poses significant challenges, requiring coherent object placement across frames, particularly during interactions. Existing models typically demonstrate capabilities with up to three subjects in online demo clips. Recent advances include: Boximator (Wang et al., 2024h (which can synthesize up to subjects) and FACTOR (Li et al.] 2024) (up to 12 subjects). Our method enables synthesis of scenes with any number of objects, limited only by the number of clearly renderable bounding boxes per frame; b. Uninitialized Object Generation: Most bounding box-based generation met focus solely on objects that either remain present throughout the entire clip or appear in the first frame and persist until at least the middle of the clip. They typically overlook bounding boxes that appear only after the first frame (Wang et al. 2024 In this work, we train our model to be sensitive to all bounding boxes, whether they are present from the first frame or appear from the middle of the clip.
3. A New Benchmark for a New Problem Formulation: Given the novelty of our problem formulation,
there is no existing standard way to evaluate models that seek to predict vehicle video with high fidelity.
We therefore present a new benchmark consisting of a particular way of evaluating video generation
models using the KITTI (Geiger et al. 2013). Virtual KITTI 2 (vKITTI) (Cabon et al. 2020 the
Berkeley Driving Dataset (BDD 100k) (Yu al. 2020) and nuScenes (Caesar el al. 2019
2 Related Work
Video latent diffusion models (VLDMs) extend latent image diffusion techniques (Rombach et al. 2022
to video generation. Early VLDMs (Blttmaann et al. 2023b a He et al. 2023 Zeng et al., 2023 Wu et al.
2023 shows temporally consistent frame generation and are tailored for text-prompted video generation. However, these models often struggle with complex scenes and lack the capability for precise local control.
image-prompin
Conditional Video Diffusion techniques providing a cortain degrce of control. Methods like VideoCom- pose (Wang et al. 2023a Dreamix Molad el al. 2023 ; Pix2Video (Ceylan el al. 2023). ; and Dream
KKrrras et al.] 2023) propose various designs of novel adapters on top of VLDMs order to incorporate different conditioning to achieve frame-level control. ControlNet Adapted Video Diffusion, on the other
hand, achieve precise regional pixel-level control in video generation by utilizing ControlNet (Zhang et al.
2023al adapters within VL.DM frameworks. Models such as Control-A-Videw (Chen et al. 2023) Video
ControlNet (Hu & Xu, 2023 Chu et al.] 2023), ControlVideo (Zhang et al. 2023b), and ReVideo Mou ct al. 2024) show that these adapters are highly adaptable to various types of conditioning, easy to train, and allow for more precise manipulation and enhanced accuracy in editing and creating video content.
Motion Control with Bounding Box Conditioning There are many strategies of control that have been explored in controllable video generation research. Notably, ControlVideo (Zhang et al. 2023b) utilizes training-free strategy that employs pre-trained image LDMs and ControlNets to generate videos based on canmy edge and depth maps. Control-A-Vin Aideoo (Chen et al. 2023) leverages a controllable video LDM that combines a pre-trained text-to-video model with Net to manipulate videos using similar visual information. Video ControlNets (Hu & Xul 2023 Chu et al. 2023) uses optical flow information to enhance video generation, while ReVideo (Mou et al. 2024) depends on extracted video trajectories. DreamPose Karras et al. 2023) injects pose sequence information into the initial noise. VideoComposer (Wang et al. 2023al uses an array of shetch, depth, mask, and motion vectors as conditioning.
3

---

Published in Transactions on Machine Learning Research (05/2025)
Many of these conditions, such edge, depth, and optical flow maps, are costly to produce and lack the flexibility needed for customization. Bounding boxes emerge as a conditioning that are easily customizable and can be edited into different shape, size, locations and classes efficiently. To the best of our knowledge, six other research projects are currently exploring the use of bounding boxcs for motion control in video generation. However, it is important to note that our work is distinct from these in several critical respects.
Direct- A-ettAA-iideo TrailBlazer (Ma et al. 2024 and Peekaboo (Jain et al. 2024 are different training- free approaches that employ attention map adjustments to direct the model in generating a particular object within a defined region. Direct-A- Video, in particular, is text-to-video model that learns to control camera motion during training and then adopts a training-free approach to manipulate object movements using bounding boxes. FACTOR (Huang et al.) 2023) augmented the transformer-based generation model, Phenaki (Villegas et al, 2022 by integrating a box control module. TrailBlazer, Peekaboo and FACTOR. necessitate textual descriptions for individual boxes, thus lacking direct visual grounding.
Our lask setup shares mild similarities with Boximator Wang al. 2024 and TrackDiffusion (inifuusioooFschheerr Fischer et al., 2023p because we also utilize bounding box conditioning during training without relying on text descriptions for individual boxes. However, our approach diverges from these text-to-video models, our primary focus is on generating realistic videos conditioned only on a couple frames of bounding boxes, whereas Boximator and Difcssifn are designed to be conditioned on text information as they both are text-to- video models. Boximator and TrackDiffusion enhance their models by introducing new self-attention layers to 3D U-Net blocks. These layers incorporate additional conditional information, such as box coordinates and object IDs, into the pretrained VLDM model. Their bounding box information is processed using a Fourier embedder (Mildenhall et al., 2020 ; which is then passed through multi-layer perceptron layers to encode. In contrast, our approach uses ControlNet and does not involve training additional encoding layers or utilizing Fourier embedder to handle the bounding box information. Moreover, Boximator introduces self- trucking technique to ensure adherence to the bounding boxes in generated outputs, technique also adopted by TrackDiffusion. This enables the network to learn the object tracking task alongside video generation, but requires a two-stage training process: one with larget bounding boxes in frames, and another with the boxes removed. They demonstrate that without this technique, the model's performance markedly declines. Conversely, our model achieves alignment with the bounding box conditions without additional training.
Vehicle Oriented Generative Models DriveDreamer [aang el al. 2023b) presents noteworthy contri- bution from autonomous driving domain. It takes an oetion-based approach to video simulation. It also makes use of bounding boxes and generate actions along with a video rendering. Within the DriveDreamer framework, Fourier embeddings (Mildenhall et al. 2020 are also employed to encode bounding box informa- tion, and CLIP embeddings (Radford et al.] 2021 are used for box categorization. They focus on generating multiple camera views and do not condition on bounding box sequences, so cannot be directly compared with our problem setting. In contrast, the DriveGAN work of Kim et al. (2021) aims to learn a GAN based driving enviromment in pixel-space, complete with actions and an implicit model of dynamics encoded using the latent space of a VAE. While driving oriented, the approach does not focus on controlling the generation of vehicle video that respects well-defined object trajectories with high fidelity.
3 Our Method: Ctrl-V
3.1 Preliminaries
We begin here with an overview of the Stable Video Diffusion (SVD) model (Battmann et al. 2023a due to its importance in our approach. SVD is diffusion based image-to-video (I2V) model performed in latent embedding space Blattmann et al. 2023b). Using image f()) initial condition, SVD is able to extend that single frame into a video f = [f)) ..... [(N)] where N is the length of the sequence (i.e the total number of frames). Notably, SVD operates in latent space, where the diffusion and denoising process act upon the
latents z of the video f. Here, SVD employs an image encoder (E) and an image decoder (D) to translate each frame into and out of latent space (Kingma & Welling 2022 :: ((E(f())) = D(2()) = f(i). At each diffusion step, SVD progressively introduces noise into the latent representations. In this work, the amount of noise is dictated by Fuler discrete noise scheduling method (EDM) introduced in Karras et. al. 2022). A
4

---

Published in Transactions on Machine Learning Research (05/2025)
UNet based denoiser network within the SVD is used to predict this noise in order to recover the original latent representations. The UNet, Ug is parameterized as:
Wo (2t, pad ( t).
(1)
E B xX'Xx"" latent representation of N frames corrupted by noise at noise level t.
20)) E BRIx'xH''X latent representation of the initial frame.
(0)
pad E R xW" Padded =0() by repeating itself along the first dimension N times.
(.); CLIP encoding RRaford et al. 2021 of the initial frame.
The full denoiser network, Dø, with an EDM noise scheduler, is formulated as
ce .0)) = Askip (o,)= + (o = padd e0)) Anoise (o.))
(2)
Here Askips Aout, Am and Anoise denote scaling functions, while o, represents the computed noise at level
The precise mathematical definitions of these terms are detailed in Appendix [ Note that 3D UNet
Ug in Equation | is a re- re-parameterized version of the one in Equation Z (Ronneberger et al.) 2015). The
scaling terms absorbed and the inputs are simplified for clarity. In the following sections, follow the re-parameterize version in Equation D when refering to the UNets in our model.
3.2 Overview of our Method: Ctrl-V
Our controllable video generation method is illustrated in Figure 2 Tt consists of two components:
1. BBox Generator: Predicts bounding box trajectories based on initial and final states It is shown the left side of Figure 2 The generated frames contain only bounding baxes. They make up a video of moving (or stationary) bounding boxes and it serve as the 'skeleton" for the generated video.
2. Box2Video: Synthesizes high-fidelity videos conditioned on the predicted bounding box sequences. It is shown on the right side of Figure 2 The bounding boxes frames act as the control signal they determine the objects generated in the corresponding frames of the video.
BBox Generator and Box2Video each utilizes a modified SVD backbone illustrated by the SVD backbone in Figure g These backbones are adapted to their respective generation tasks. Details of each model are presented in their individual sections: BBox Generator Section 3.3 and Bix2Video Section 3.4
3.3 Ctrl-V: BBox Generator
The BBox Generator shown on the left in Figure aims to predict object bounding boxes across all video frames using SVD backbone. The four inputs to the model are b., ((0) ((N-1), 2(0), where: b, is the encoded "video" of bounding boxes with levels of noise added; ((0) is the encoded initial bounding box frame(s); ((N-1) is the encoded final bounding box frame; z0)) is the encoded initial video frame. During training, the model learns to predict the noise added in 6, according to the EDM noise scheduler. The model recovers the original b from its noisy version b, by calculating the noise with UNet outputs and climinating the noise through scaling functions. We opt to abstract this detail in the model diagram for readability.
practice, the four inputs are transformed and concatenated into a vector format accepted by the UNet adapter within the SVD backbone. Specifically, as shown in Figure 2 z0) E BIxX''x''' xw" is replicated along the first dimension, and its front and end (in the first dimension) are replaced by b(9), b(N-1) respectively. X'''C'X'''W'' The noise-added encoding of bounding
This forms = cb)cat(b(o) 20)), ...., (0)) ((N-1)
box video b, is then concatenated with xpad to form the final input to the UNet adapter. The network incorporates additional conditioning inputs, including a CLIP-encoded embedding of the initial frame (()) and a noiso-level embedding These embeddings are individually integrated into every sub-block of the U-Net through self-attention mechanism.
5

---

Published in Transactions on Machine Learning Research (05/2025)
BBox Generator
Box2Video
ControiNet Adaepter
Generated BBox Frames Floox
VAE Encoder
Controlltet
fosow
VAE Encoder
Bilox
Ulet Adupterr
SVD U/
Latents
C
VAE Image
Initial Fieee
PAG
Conditioning Frames
Latents [if
UNet Adapter
ZeTO ooeN
Combined
SVD uNet
Frume Latents
Initial Frame Fifl
SS Noise
embeddinn
CLIP
Noise Level Embedding
UNet Adapter
Image Decoder
CLIP Encoder
Other Frames (r)) (19-1)
10 1 Hoise lavel Embedding
(Above) Our Diffuslon-Basdd Bounding Box Generator.
embedding
(Right)
Diffusion- Based Boundling Box Video Frame Conditioned Video Generator
CLIP Encoder
Figure 2:
The diagram illustrates two components of Ctrl-V: (left) the BBox Generator and
(right) BixeVideo. For both models, we use a frozen, off-the-shelf VAE to encode images into latent space (E) and decode them back into pixel space (D). During training, (1) the BBox Generator (Sec. 3.3
learns to denoise the noisy bounding box frame latents 6,, conditioned on the first (b()) and last (b
bounding box frame latents and the padded initial frame latent = and (2) the Box2 (Sec. 3.4
denoises the target frame latents & by conditioning the initial frame's latent = (input to the SVD
UNet) and the bounding box frame latents b (input to the CoutrolNet).
Representing Bounding Boxes in Pixel Space
An important element of Ctrl-V is design choice of rendering bounding boxes in pixel space. The manner in which bounding box information is provided For example, prior work such as Boximator (Wang et al.] 2024h represents bounding boxes as a Fourier transformed concatenated vector of their raw coordinates, ID and other information. In contrast, in our work we choose to render bounding boxes into frames while maintaining minimal loss of meta information. Importantly, we also encode information such as track ID, object type, and orientation for each bounding box using combination of visual attributes, including border color, fill color, and markings. Specifically, the track ID represents unique identifier for each tracked object across frames, the object type specifies the category of the object (e.g,, car, pedestrian), and the orientation indicates the direction the object is facing, Further details about how these bounding box frames are rendered can be found in Appendix C.1 Crucially, our approach allows us to leverage the highly effective ControlNet approach to provide pixel-level guidance to influence diffusion generated imagery.
control signal to the video generator is important.
3.4 Ctrl-V: Box2Video
Box2Video shown on the right. in Figure 2 and aims generate high-fidelity videos controlled by bounding box frames, such as those generated by the BBox Generator network. Box2Vidco consists of SVD backbone for video generation, and an adapted ControlNet module to process the bounding box control signal. ControlNet is widely used network for controlling image generation. In this work, we modify ControlNet and adapt it to the video diffusion framework (as shown on the right in Figure 2 This architecture allows us to train Box2Video in a single stage without the need for additional optimization criteria, in contrast. to previous work such as Boximator and TrackDiffusion (Wang et. al. 2024 Li el al. 2024h, which require multi-stage learning with extra criteria to train their models.
The SVD component takes two inputs: 20) and d.. Here, ()) is the encoded initial video frame and is the encoded full video with | levels of noise added to it As shown in Figure 2 process these inputs

---

Published in Transactions on Machine Learning Research (05/2025)
by padding 2 (0) by repeating it along the first dimension before concatenating it with &, to create the final input to the UNet adapter of the SVD. The same input is also sent. to the ControlNet module through its own UNet adapter layers. Additionally, ControlNet also receives the encoded bounding box frames, b, as input, through ControlNet adapter layers. Both of these transformed input is then added together before processed by the ControlNet module. The output signal of the ControlNet module then goes through a zero-convolution before being sent to the SVD UNet decoder layers through residual paths as control signal. During training, the weights of the SVD model (0) are frozen, while only the weights in the ControlNet (E) are updated.
4 Experimental Analysis and Ablation Studies
Frame 1
Frame 7
Frame 13
Frame 19
Frame 25
BBox Generator's s
Output
Box2Video Generated Frames
Ground Truth Frames
SVD Baseline Generated
Frames
The first two rows illustrate video samples generated using the Ctrl-V pipeline, with one initial frame, three initial bounding box frames, and final bounding box frame input. The first row shows bounding box trajectories from the BBox-generator in pixel space (solid rectangles for predictions, wireframe rectangles for ground truth). The second row presents frames generated by the Box2Video model, conditioned on the genertornsssssrss output. The third row displays ground-truth frames, while the fourth row shows frames generated by the Stable Video Diffusion (SVD) baseline. In the Ctrl- V video, the car with the bright- green bounding box, which initially pokes oul ils mose the lane the left of the ego car, stays beside the ego car in the final frame. Meanwhile, the silver car with the olive bounding boz, which starts in the lane to the right of the ego cor, speeds off and is replaced by a new car (purple bounding box) entering the frame. These generated frames closely match the car positions seen in the conditioned inputs. In controst, the SVD-generated video shows the black car (marked by the green arrows on the final frame) on the left accelerating and moving ahead of the ego car, while the silver car (marked by the olive arrow on the final frome) remains in the same relative position to the ego car throughout.
Figure 3:
For quantitative evaluation, we assess the model's performance across four driving datasets on three key aspccts:
1. The overall visual qualily of the generaled results (Section
2. The alignment of the predicted bounding bor trajectories with the ground truth (Section 4.2
3. The fidelity of the generated objects in the video to the bounding box control signal (Section
4.4
For visual asscssment, Figure and Appendix F showcase sample demonstrations generated by our model. To assess video quality, we randomly select 200 initial frames from each dataset's testing set and generate videos. The results in this section are based analyses of these 200 generated videos per dataset. Furthermore, we

---

Published in Transactions on Machine Learning Research (05/2025)
explored different bounding box conditioning options: one three initial bounding box frames, followed by single final bounding box. Additional variations are discussed in Appendix E.8
4.1 Datasets
We evaluate the performance of our models across four autonomous--ehicle datasets: KITTI (Geiger et al. 2013), Virtual KITTI 2 (vKIттI) (Cabon et al. 2020 Berkeley Driving Dataset (BDD) Yu et al. 2020 with Multi-object Tracking labels (MOT2020), and the nuScenes Dataset (Caesar et al. 2019 KITTI comprises real-world driving clips with 3D object labelling. vKITTI consists of virtual simulated driving scenes, each offering weather variants, all including 3D object labelling. BDD is a large-scale real-world driving dataset, featuring 1603 2D-labeled sequences of driving videos. The muScenes dataset is large-scale driving dataset that includes 1000 scenes 20-second scenes annotated with 3D bounding boxes, multiple sensor data (lidar, radar and cameras) and map information. Further details on dataset configurations are provided in Appendix C.3
4.2
BBox Generator: Quantitative Evaluation
Frame
Frame
Frame
Frame 19
Frame 25
Trajectory Generation #1:
Trajectory Generation #2:
Figure 4: This figure visualizes two samples of bounding box trajectories generated by the BBex Generator, conditioned on the same set of three initial bounding box frames and one final bounding box frame (solid reclangles represent predictions, and wireframe reclangles represent ground truth). Although the intermedi- ate frames show notable differences, the initial and final frames align closely with the ground mauntttruhh bounding boxes.
Figure showcases two bounding box trajectory samples generated by the BBox Generator, conditioned on the same initial and final bounding box frames. To evaluate the quality of our bounding box generalions, we create mask images for both the ground-truth and generated bounding box sequences. The mask images are generated by converting the bounding box frames into binary masks (dctails be found in Appendix D.2 We then caleulate the generated averaged mask Intersection over Union (maskIoU) scores, averaged mask Precision (maskP) scores, and averaged mask Recall (maskR) scores against the ground-truth bounding box masks. To assess our bounding box trajectories, we applied the "best-out-of-K" method, selecting the model with the highest maskloU score for evaluation. In this instance, K equals 5. We compare our results with a baseline referred to as the "Trajeglish-Style model, an autoregressive GPT-like encoder-decoder that models the bounding box trajectories as a sequence of discrete motion tokens. This bascline is inspired by the work of Philion al. (2023) with implementation details provided in Appendix D We present our findings in Table and demonstrate examples of our bounding box generations each dataset in Appendix D
In the bounding box generation figures, our generator model achieves the closest alignment with the ground- truth in the first. and last. frames. This near-perfect alignment is primarily altributed to conditioning the model on the bounding boxes of these key frames. When considering all generated frames, the alignment scores decrease, as shown by the plotted demonstrations and metric results in Table | This because objects in frames do not move deterministically. The role of the bounding bor generator is to generate plausible trajectory for moving objects from the initial bounding box frame to the last.
8

---

Published in Transactions on Machine Learning Research (05/2025)
# Cond. BBox
maskIoU? (first+last)
maskPt
mask
Method
maskloUt
maskP
maskR
Hastt)
(firtt last)
BBox Geuerator (ours) Trajeglish-Style
.629 +: .212
.758 1. .176
.763 .188
.986 +: .012
.994 = .008
.992 1 000
1-to-1
.447 + .154
.568 + .172
.679 + .177
.561 + .151
.663 + .10
.789 + 166
BBox Generator (ours) Trajeglish-Style
.795 + .12
.881 + .082
.884 = .078
.986
.010
.992 +: .007
.994 + .005
3-to-1
.491 + .164
.622 + .773
.691 + .175
.576 + .154
.684 .19
.784 + .63
BBxx Generator (ours)
.710 = .205
.828 + .178
.809 .171
.943 + .048
.946 = .046
.997 = .006
vKIT
Trajeglish-Style
1-to-1
.471 + .171
.578 = 2000
.700 .187
.557 = .171
.628 .194
.835 .135
BBox Geucrator (ours)
.767 .131
.881 i: .16
.853 = .078
.944
.039
.948 1 .036
.996 1 .006
3-to-1
Trajellihh-Style
.500 i .162
630 1 .86
.741 1: .176
.575 1 .154
.657 1 .182
.836 1 .143
BBox Generator (ours)
.587 .214
.747 1 .187
.712 .194
.954
.047
.955
.047
.999 1 .002
1-to-1
BDD
Trajeglish-Style
.355 + .183
.372 .213
.658 2007
.432 + .171
.483 .192
.840 .166
BBox Generator (ours)
.647 + .176
.784 + .150
.783 = -116
.955 + .043
.955 + .042
.997 + .001
3-1o-1
Trajellih--Style
.373 + .185
.54 = 2066
.686 + .193
.492 = .190
.553 +: .208
.842 = .154
muSoenes
BBox Generator (ours)
1-to-1
.364 +: .242
.433 +: .278
.740
.186
.983
.013
.985 = .0112
.997 + .003
Trajeglish-Style
.405 -202
.506 + .20
.661 .216
.511 + 168
.603 .17
.789 .195
BBox Generator (ours)
.827 + .150
.892 + .120
.906
.099
.983 + .013
.985 +: .012
.998 + .003
3-to-1
Trajeglish-Slyle
.48 + .144
.554 + 213
.695
196
.529 + .172
.623 1: .177
.791 + 192
Comparing real and generated bounding boxes. We condition on or 3 initial bounding box frame and final bounding box or trajectory frame. The first three columns show evaluations on the entire generated bounding box sequence, measuring the alignment scores between our generated bounding box generations and ground-truth labels. The last three columns focus on testing the auto-encoding capability of the network, evaluating only the first and last frames of the generated sequence. "BBox Generator' is our method and "Trajeglish-Style is a baseline inspired from Philion et al. 2023) (see Appendix for implementation details on this baseline).
Table 1:
Despite the disparity between the ground-truth trajectory and the generated trajectory, our Box2Video con- sistently generates high-fidelity videos based on either trajectory provided. Further analysis of this aspect is provided the subsequent sections.
4.3
Ctrl-V: Generation Quality
To assess the quality video generation, we compare videos generated through 4 distinct pipelines:
1. Pre-trained Stable Video Diffusion (SVD) basclines] without fine-tuning (initial frame video)
2. Finc-tuned Stable Vidco Diffusion (SVD) basclines on the provided datasct (initial frame video)
3. Teacher-forced Box2Video generation (initial frame and all bounding box frames -> video)
4. bounding box generation with BBox Generator and Box2Video (initial frame, one or three initial and one last. bounding box frames -> in-between bounding box frames and video).
We evaluate our generation across four metrics: Fréchct Vidco Distance (FVD) (Unterthiner et al., 2019) Learned Perceptual Image Patch Similarity (LPIPS) (Zang al. 2018). Structural Similarity Index Measure (SSIM) (Wang et al., 2004b) and Peak Signal-to-Noise Ratio (PSNR). These metrics either measure the consistency of frame pixels with the ground truth or the consistency of the frame latents extracted by another network. FVI is an exception; it evaluates the generation distribution against the ground truth's 's
Stable Video Diffusion (SVD) baseline inage-to-video (I2V) model that generates video sequence conditioned single vidco frame.
2FVD is highly sensitive to vidco configuration parameters such frame rate, clip duration, and spatial resolution cooouutio-mmakingg direct comparisons FVD values across studies challenging. Additionally, the metric's as sensitivity sample raises concerns, as some datasets may lack sufficient. samples for convergence, leading unreliable estimates.
9

---

Published in Transactions on Machine Learning Research (05/2025)
Pipeline
# Cond. BBox
FVD1
LPIPS!
SSIMI
PSNR1
Stable Video Diffusion Bascline Blattnann et al. 2023a
Noue
1118.4
0.4375
0.2919
10.63
Stable Video Diffusion Fine-tuned (Blatcmann et al. 2023a
Noue
552.7
0.3504
0.4030
13.01
Ctrl-V: BBox Generator + Box2Videa((unny
1-to-1
467.7
0.3416
0.3241
13:21
Ctrl-V: BBox Generator + (oo2VVido((Oun)))
3-to-1
422.2
0.2981
0.4277
13.85
Ctrl-V: Teacher-foreed Box2Video(Ours)
All
435.6
0.2963
0.4394
14.10
Stable Video Diffusion Baseline Blattmann al.
2023n
None
922.7
0.3636
0.4740
14.61
BDD <kI111
Stable Video Diffusion Fine-tuned [Blsttmann
2023a
None
331.0
0.2852
0.5540
16.60
Ctrl-V: BBxx Generator
1-to-1
400.2
0.3179
0.4714
16.78
Ctrl-V: BBx Generator Box2Video(Ouss)
3-to-1
341.4
0.26415
0.5841
17.60
Ctrl-V: Teacher-foreod (Ons))d))OOurs)
All
313.3
0.2372
0.6203
18.41
Stable Video Diffusion Bascline Blattmann et al.
2023a
Noue
933.6
0,4880
0.3349
12.70
Stable Video Diffusion Fine-tuned Blattmann et al.
2023a
None
409.0
0.3454
0.5379
16.99
Ctrl-V: BBox Generator + Box2Videor
1-to-1
412.8
0.2967
0.5470
17.52
Ctrl-V; BBox Generator + Box2Videu(Quui
3-to-1
373.1 1
0.3071
0.5407
17.37
Ctrl-V: Teacher-foreed (ox2Video(Ours)
All
348.9
0.2926
0.5836
18.39
Stable Video Diffusion Baseline Blattmann al.
2023a
None
1179.4
0.5004
0.2877
13:31
Single- View
Stable Video Diffusion Fine-tuned [Blattmann et al.
2023a
None
316.6 6
0.2730
0.4787
18.58
Ctrl-V: BBox Generator + Box2Videa
1-to-1
285.3
0.2647
0.5050
18.93
Ctrl-V; BBox Generator + Box2Video(Qurs)
3-to-1
235.0
0.2235
0.5500
20.33
Ctrl-V: Teacher-foreod Box2 (uucoo(u))
All
235.5
0.2104
0.5705
23.36
moSecnes
DriveGAN (Kim ct al.
2021
Noue
390.8
DriveDreamer Wang
al.
2023h
All
340.8
WoVoGen Lu al.
2023
All
417.7
Drivingdiffusion
2023
All
332.0
Multi-view
Drive WM Lu
2023
None
212.5
BEVWorld Zhang
2024
None
154.0
Panacea (Wnn
2024
All
139.0
Drive-Wa WM
Lu et 20233
All
122.7
Drimeere Zhao et sl.
2024
None
105.1
Comparing the quality and diversity of the generated video models. The generated videos consist of 25 frames (except for our nuScenes models which consist of 11 frames videos at 4 Hz) at resolution of 312 x 520, while the reported metrics from this table are evaluated at a resolution of 256 x 410. The "# Cond. BBox' column reports the number of ground-truth imput bounding box frames used by the generation pipclines. "None' indicates that no ground-truth frames are used, while "All" indicates that all ground-truth bounding box frames are utilized. If "# Cond. BBox" is n-lo-m, then represents the number of initial bounding box frames used by the pipeline is and the number of final bounding box frames used by the pipcline is m.
Table 2:
distribution. It is important to note that while many papers report their best-out-of-K results on these metries, due to computational constraints, we evaluate our model on a single sample for each input.
The evaluated results are reported in Table and visualizations are available in Appendix E.I These results indicate that the generation quality improves as we condition on more ground-truth bounding box frames. Details regarding the metrics and their limitations are discussed in Appendix D.I
4.4 Box2Video: Motion Control Evaluation
Our Box2Vidoo is trained to control object motions through bounding boxes using a tcacher-foreing approach, where only ground-truth bounding box frames are provided during the training phase. In this section, we analyze the fidelity of our Box2Video generations to the ground-truth bounding box conditions. To access the consistency of objects' locations between our generated content and ground-truth, we compute the average precision of the bounding boxes in the generated frames and the ground-truth frames.
Average precision (AP) scores gauge the alignment of predicted/gentt generated bounding boxes with the ground- truth labeling. In all related prior studics, average precision (AP) scores have been consistently reported. However, it is important to acknowledge that AP scores can vary across studies, depending on the specifics of the task setup. Boximator (Wang et al. 2024h's motion control model predicts object locations in the
10

---

Published in Transactions on Machine Learning Research (05/2025)
Frame
Frame
Frame 13
Frame 19
Frame 25
KITTI
vKITTI
Ti
BDD
Figure 5: Illustrations of the Box2Vidco generations conditioned on ground truth 3D bounding box trajecto- ries (2D for BDD) across various dalasets. The 2D outlines of the ground-truth bounding boxes are overlaid on top.
Method
Dataset
Dataset Type
# Frames
mAP
A.soo
AP7t
A.
KITTI
Driving
25
0.547
0.712
0.601
0.327
vKITTI
Driving-sim
25
0.599
0.776
0.667
Ctrl-V
BDD
Driving
25
0.685
0.855
0.781
0.401
muScenes
Driving
25
0.661
0.833
0.734
0.381
Boximato
MSR-VTT Xu et al. NetivityNet Heilbron UCF-101 Soomro et
2016
Web videos
16
0.365
0.521
0.384
Wang
et.
2015
Human-action
0.394
0,607
0,409
2024
2012
Human-action
0.212
0.343
U:205
TrackDiffusion
YTVIS Yang et al.
2019
YouTube videos
16
0.467
0.656
Li et al. 2024
UCF-101 Soomro et
2012
Human-action
16
0.205
0.326
Table Average Precision scores obtained by comparing the YOLOvv bounding box estimations of real and generated samples. Prior works [Wang al. 2024 Li 2024 do not report results on driving datasets; thus, draw upon their reported performances
alternative datasets to provide comparative context. The backbone model of Ctrl-V produces videos with 25 frames, while Boximator and TrackDiffusion create videos with 16 frames. Longer videos tend to have reduced quality and lower detection rates, which presents an extra challenge for our model (as generates 56.25% more frames); yet. it achieves greater precision compared to the other baseline models.
scene, focusing solely objects with consistent appearances across all frames. Their AP implementation disregards the object locations in the intermediate frames, comparing the objects' locations only in the final frame. In contrast, TrackDiffusion (Li et al., 2024) uses TrackAP for evaluation, employing a QDTrack model (Fischer et al. 2023) to track instances in generated videos and comparing them to ground-true -tuth labels. As of now, existing AP metrics are designed for static object detection and do not fully capture the nuances of bounding- box-connitioned video generation, where objects may dynamically appear, disappear, or shift positions across frames. To address this, we propose revised AP metric that:
1. Evaluates bounding box consistency across all frames rather than just final frame positions.
2. Accounts for new object entries and occlusions rather than assuming a fixed object set.
3. Uses intersottion-ove--uiio (IoU) matching to compare gencrated and ground-truth objects across time, ensuring a frame-by-frame accuracy assessment.
We apply this metric measure how well Ctrl-V aligns object positions with ground-truth bounding box conditions. As shown in Table 3, our model outperforms prior methods in motion fidelity, particularly under lenient AP thresholds, which better capture realistic object tracking in video synthesis.
11

---

Published in Transactions on Machine Learning Research (05/2025)
Autonomous driving datasets often contain numerous object instances within scene, with objects contin- nously entering, exiting, and interacting with each other. In line with this complexity, we have introduced our own version of the AP metric in this work.
First, we utilize the state-of-the-an ahe- object detection tool, YOLOv8 (Ris et al. 2024h. to obtain the objects' trackings from the generated and ground-truth scenes. Detailed information about the tool and our config- urations is reported in Appendix D33 Next, we match objects in each generated- ve--eaated---grouu---ee ----groud-uuttt frame pair based spatial similarity calculating the intersection over union (IoU) score to determine the similarity in location between objects' bounding boxes. Our metric disregards object type and tracking IDs equivalence assuming that objects close in location should naturally have the same type and IDs. Finally, we compute the average precision score following MS COCO protocol (Lin et al. 2015). Details are provided in Ap- pendix D.4 and results are listed in Table g These results indicate that our Box2Video model is particularly
adept at adhering to the specified conditions, especially when evaluated with a more lenient metric (i.c., a lower IoU threshold for the AP computation).
5 Limitations
Our model has several key limitations. First, our experiments were conducted exclusively autonomous driving dataset, limiting the model's generalizability to other scenarios. Second, accurately generating street signs remains challenging. Third, the quality of generated videos deteriorates as the generation length increases. Fourth, the model struggles to maintain quality when the ego car is making a turn. Lastly, handling duttof-distributiin cases, such as car accidents, remains difficult, affecting the model's robustnoss in unseen environments. A detailed discussion of our model's limitations and failure cases, along with demos,
can be found in Appendix G
6 Conclusions
We have presented Ctrl-V, novel model capable of generating controllable aulonomous vehicle videos via bounding box trajectory conditioning, Our approach demonstrates that our BBox Generator technique can closcly follow generation requirements for the first and last frames and produce coherent bounding box track for intermediate frames. Moreover, our Box2Video network generates high-fidelity videos that strictly conform to the provided bounding boxes. Furthermore, our model accommodates both 2D and 3D bounding boxes and handles uninitialized objects appearing in the middle of the videos. Ctrl-V provides future rescarchers with an efficient way to simulate driving video data with flexible controllability in the form of bounding boxes. In addition, we further define improved metric to evaluate bounding box conditioned video generation to account for objects that not present in the first frame, and those that do not remain until the last frame. In Appendix G discuss potential future work for this project. With Ctrl-V and an improved metric for more accurate evaluation, we aim to establish solid foundation for future research in controllable video generation.
Acknowledgements
We thank CIFAR for their support under the AI chairs program, NSERC for their support under the Discovery Grants program, and Samsung for supporting this work.
References
Nir Aharon, Roy Orfaig, and Ben-Zion Bobrovsky. Bot-sort: Robust associations multi-pedestrian tracking. arXiv preprint Xrxi::2206. 14651, 2022.
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, Varun Jampani, and Robin Rombach. Stable video diffuuson: Scaling latent video diffusion models to large datasets, 2023a.
12

---

Published in Transactions on Machine Learning Research (05/2025)
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Scung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resoltion video synthesis with latent diffusion models, 2023b.
Yohann Cabon, Naila Murray, and Martin Humenberger. Virtual kitti 2, 2020.
Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. CoRR, abs/ 1999.993..1,27, 2019. URL http://arriv..orrr/a///000.....0 11027
Duygu Ceylan, Chun-Hao Paul Huang, and Niloy J. Mitra. Pix2video: Vidco oditing using image diffusion, 2023.
Weifeng Chen, Jie Wu, Pan Xie, Hefeng Wu, Jiashi Li, Xin Xia, Xuefeng Xiao, and Liang Lin. Control-a- video: Controllable text-to-video generation with diffusion models, 2023.
Ernie Chu, Shuo- Yen Lin, and Jun-Cheng Chen. Video controlnet: Towards temporally consistent synthetic- to-real video translation using conditional image diffusion models, 2023.
Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. Carla: An open urban driving simulator, 2017.
Tobias Fischer, Thomas E. Huang, Jiangmiao Pang, Linlu Qiu, Haofeng Chen, Trevor Darrell, and Fisher Yu. Qdtrack: Quasi-dense similarity learning for appearance-only multiple object tracking, 2023.
Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti dataset. International Journal of Robolics Research (IJRR), 2013.
Yingqing He, Tianyu Yang, Yong Zhang, Ying Shan, and Qifeng Chen. Latent video diffusion models for high-fidelity long video generation, 2023.
Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large- scale vidoo benchmark for human activity understanding. In 2015 IEEE Conference on Computer Vision and Pallern Recognition (CVPR), pp. 961-999 961-970, 2015. doi: 10. D0.119/CVVR.2015.7777
Zhihao Hu and Dong Xu. Videocontrolnet: A motion-guided video-to-video translation framework by using diffusion model with controlnet, 2023.
Hsin-Ping Huang, Yu-Chuan Su, Deqing Sun, Lu Jiang, Xuhui Jia, Yukun Zhu, and Ming-Hsuan Yang. Fine-grained controllable video generation via object appearance and context, 2023.
Yash Jain, Anshul Nasery, Vibhav Vineet, and Harkirat Behl. Poekaboo: Interactive video generation via masked-diffusion, 2024.
Glenn Jocher, Ayush Chaurasia, and Jing Qiu. Ultralytics YOLO, January 2023. URL https://githi com//albzaallytcccs/lllaalyyyiil
Johanna Karras, Aleksander Holynski, Ting- Chun Wang, and Ira Kenehnacher--Shizermn.. Dreampose: Fashion inage-to-video synthesis via stable diffusion, 2023.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models, 2022.
Seung Wook Kim, Jonah Philion, Antonio Torralba, and Sanja Fidler. Drivegan: Towards controllable high-quality neural simulation, 2021.
Diederik P Kingma and Max Welling. Auto-cncoding variational bayes, 2022. URL https://arxiv.org abs/ 13312.6114
Zoran Kotevski and Pece Mitrevski. Experimental comparison of psnr and ssim metrics for video quality estimation, 01 2010.
13

---

Published in Transactions on Machine Learning Research (05/2025)
Pengxiang Li, Kai Chen, Zhili Liu, Ruiyuan Gao, Lanqing Hong, Guo Zhou, Hua Yao, Yan Yeung, Huchuan Lu, and Xu Jia. Trackdiffusson: Tracklet-conditioned video generation via diffusion models, 2024.
Xiaofan Li, Yifu Zhang, and Xiaoqing Ye. Drivingdiffusio:: Layout-guided multi-view driving scene video generation with latent diffusion model. arXiv preprint (XXi:2310000777:: 2023.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár. Microsoft coco: Common objects in context, 2015.
Πya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019.
Jiachen Lu, Ze Huang, Jiahui Zhang, Zeyu Yang, and Li Zhang. Wovogen: World volume-aware diffusion for controllable multi-camera driving scene generation. arXiv preprint arXiv:231.00293, 2023.
Wan-Duo Kurt Ma, J. P. Lewis, and W. Bastiaan Kleijn. Trailblazer: Trajectory control for diffusion-basel video generation, 2024.
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing seenes as neural radiance fields for view synthesis, 2020.
Eyal Molad, Fliahu Horwitz, Dani Valevski, Alex Rav Acha, Yossi Matias, Yael Pritch, Yaniv Leviathan, and Yedid Hoshen. Dreamix: Video diffusion models general video editors, 2023.
Chong Mou, Mingdeng Cao, Xintao Wang, Zhaoyang Zhang, Ying Shan, and Jian Zhang. Revideo: Remake video with motion and content control, 2024.
Jonah Philion, Xue Bin Peng, and Sanja Fidler. Trajeglish: Learning the language of driving scenarios. arXiv preprint 2rX1..2312.04 04535, 2023.
Alcc Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramosh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. CoRR, abs/2103.00020, 2021. URL https:
//arxiv. www.ooggg/bbbbggeeee abs. 2ttp//ab.//1000222222 00020
Dillon Reis, Jordan Kupec, Jacqueline Hong, and Ahmad Daoudi. Real-time flying object detection with yolov8, 2024.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models, 2022.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation, 2015.
Mahmood Sharif, Lujo Bauer, and Michael K. Reiter. On the suitability of Ip-norms for creating and preventing adversarial examples. CoRR,
2018. URL
09653
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. UCF101: A dataset of 101 human actions classes from videos in the wild. CoRR, abs/1212.0402, 2012. URL
Thomas Unterthiner, Sjoerd Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Golly. Towards accurate generative models of video: A new metric & challenges, 2019.
Ruben Villegas, Mohammad Babaeizadeh, Pieter-Jan Kindermans, Hernan Moraldo, Han Zhang, Moham- mad Taghi Saffar, Santiago Castro, Julius Kunze, and Dumitru Erhan. Phenaki: Variable length video generation from open domain textual description, 2022.
Patrick von Platen, Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj, Dhruv Nair, Sayak Paul, William Berman, Yiyi Xu, Steven Liu, and Thomas Wolf. Diffusers: 2022.
State-of-the-art diffusion models. http://hhhh..ccoooooocc/ccccaaaa
14

---

Published in Transactions on Machine Learning Research (05/2025)
Jiawei Wang, Yuchen Zhang, Jiaxin Zou, Yan Zeng, Guoqiang Wei, Liping Yuan, and Hang Li. Boximator: Generating rich and controllable motions for video synthesis, 2024.
Xiang Wang, Hangjie Yuan, Shiwei Zhang, Dayou Chen, Jiuniu Wang, Yingya Zhang, Yujun Shen, Deli Zhao, and Jingren Zhou. Videocomposer: Compositional video synthesis with motion controllability, 2023a.
Xiaoleng Wang, Zheng Zhu, Guan Huang, Xinze Chen, Jiagang Zhu, and Jiwen Lu. Drivedreamer: Towards real-world-driven world models for autonomous driving, 2023b.
Zhou Wang and Alan C. Bovik. Mean squared error: Love it or leave it? a new look at signal fidelity measures. IEFE Signal Processing Magnzine, 26(1):98-117, 2009. doi: N..1099/MSP20008990
Zhou Wang, A.C. Bovik, H.R. Sheikh, and E.P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEFE Transactions on Image Processing, 13(4):600-61) -12, 2004a. doi: 10.1109/TIP.
2003.819861.
Zhou Wang, A.C. Bovik, H.R. Sheikh, and E.P. Simoncelli. Image quality assessment: from error visibility structural similarity. IEEE Transactions on Image Processing, 13(4):600 -12, 2004b. doi: 10.1109/ TIP.2003.819861
Yuqing Wen, Yucheng Zhao, Yingfei Liu, Fan Jia, Yanhui Wang, Chong Luo, Chi Zhang, Tiancai Wang, Xiaoyan Sun, and Xiangyu Zhang. Panacea: Panoramic and controllable video generation for autonomous driving. In Proceedings of the IEFE/CVE Conference on Computer Vision and Pottern Recognition, pp. 6902-6912, 2024.
Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tunc-a-video: One-shot tuning of image diffusion models for text-to- video generation, 2023.
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5288 5288-5296, 2016. doi:
Linjie Yang, Yuchen Fan, and Ning Xu. Video instance segmentation. CoRR, abs/1905.04804, 2019. URL https://arxiv. http:////waiii.ooo/////////e//// oww./bs/////oooo// 1905, 19011048040000444
Ze Yang, Yun Chen, Jingkang Wang, Sivabalan Manivasagam, Wei- Chiu Ma, Anqi Joyce Yang, and Raquel Urtasun. Unisim: A neural closed-loop sensor simulator, 2023.
Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Darrell. Bdd100k: A diverse driving dataset for heterogeneous multitask learning, 2020.
Yan Zeng, Guoqiang Wei, Jiani Zheng, Jiaxin Zou, Yang Wei, Yuchen Zhang, and Hang Li. Make pixels dance: High-dynamic video generation, 2023.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala, Adding conditional control to eoxt-to-image diffusion models, 2023a.
Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unreasonable effective- ness of deep features a perceptual metric, 2018.
Yabo Zhang, Yuxiang Wei, Dongsheng Jiang, Xiaopeng Zhang, Wangmeng Zuo, and Qi Tian. Controlvideo: Training-free controllable text-to-video generation, 2023b.
Yumeng Zhang, Shi Gong, Kaixin Xiong, Xiaoqing Ye, Xiao Tan, Fan Wang, Jizhou Huang, Hua Wu, and Haifeng Wang. Bevworld: A multimodal world model for autonomous driving via unified bev latent space. arXiv preprint 007:0:2407.05679, 2024.
Guosheng Zhao, Xiaofeng Wang, Zheng Zhu, Xinze Chen, Guan Huang, Xiaoyi Bao, and Xingang Wang. 2: Lim-enhanced world models for diverse driving video generation. arXiv preprint arXiv:2403.06845, 2024.
15


Reviews:
[{"review_id": "UlSdFEVx2l", "reviewer": "Reviewer_wMAd", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "1. **The proposed video generation method, which uses bounding boxes as control signals, allows for the presence of a large number of objects, distinguishing itself from previous approaches that could only generate a limited number of objects.**\n\n2. **Comprehensive evaluations have been conducted on this method, with extensive and detailed experiments supporting its effectiveness.**", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a video generation method that uses bounding boxes as control signals, leveraging the maximum possible number of generated bounding boxes to guide video synthesis. Additionally, the approach exhibits high sensitivity to bounding boxes in intermediate frames and introduces a new benchmark.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. **The logical flow of this paper is somewhat unclear, and the quality of English writing needs improvement.**  \n\n2. **The introduction mentions the introduction of a new benchmark as a major contribution, yet it is not adequately discussed in the main text.**  \n\n3. **This paper refers to \\( t \\) in the diffusion model as 'level \\( t \\),' whereas most researchers recognize it as 'time step \\( t \\).' Please provide an explanation for this choice.**  \n\n4. **Many aspects of the paper lack proper explanation and citations, such as VAE.**  \n\n5. **This paper employs a generative approach to produce bounding boxes, which serve as critical control signals for video generation. Why was a bounding box detection method not used instead? Does the generative approach outperform detection-based methods? Please provide a detailed experimental comparison.**", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "GymsGAo2qT", "reviewer": "Reviewer_GgyN", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Advantages:\n1. A controllable generation method suitable for autonomous driving video generation scenarios is proposed, and the visual generation quality is relatively high.\n2. The article is well-structured and easy to read.\n\nDisadvantages:\n1. The chosen base model is SVD, which is a very early model. Given the rapid development in the video generation field, selecting a more recent base model would have been a more reasonable choice.\n2. The model was evaluated only in driving scenarios, raising doubts about the generalizability of the method. The title of the paper is \"Ctrl-V: High Fidelity Video Generation with Bounding-Box Controlled Object Motion,\" which is not limited to driving scenarios. It should be evaluated on more types of datasets.\n3. There are missing training details, such as the dataset and hyperparameters. It’s unclear whether the training and evaluation were done with the same four datasets.\n4. The length of generated videos is limited.\n5. The homepage cannot be accessed.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a pipeline for controllable video generation based on bounding boxes. First, a model based on SVD is designed to generate bbox sequences. Then, using the bbox sequence as a condition, another method based on ControlNet + SVD is employed to generate controllable videos. Additionally, for this task, the paper introduces an evaluation benchmark.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Conduct experiments on the method using an new base model.\n2. Evaluate the method on a broader range of datasets.\n3. Add details about the training process in the main text.\n4. Fix the homepage.", "broader_impact_concerns": "No Impact Concern", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "YduLxZoDrv", "reviewer": "Reviewer_wj6p", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:\n* Clear and pleasant illustrations and visualizations.\n* The method pipeline is clear in terms of inputs and outputs, and conditioning mechanism (but might be complex to implement).\n* Great performance.\n* Detailed appendix!\n\n**Weaknesses**:\n* If the method is mainly aimed for autonomous driving applications, maybe make this information more explicit to the reader in the abstract/title, as the method itself is not evaluated on non-autonomous-driving datasets. If the authors want to claim more generality, it should be demonstrated.\n* Project site link in the main text doesn’t work for me, leads to a GitHub repository with template information (?). The link in the appendix works. Also, the caption of Figure 1 doesn’t seem to be the right location to place a project site link.\nWhile it has become rather easy to extract bounding boxes of common objects (such as vehicles) with prompt-based object-detection/segmentation models, it seems that the model still requires users to provide the bounding-boxes themselves (by hand or by running another model to get them) as input to the model, rather than just a single frame. I find this a limitation of the model to a certain degree.\n* Question: have the authors tried training a single model (instead of 2) where frames and BBs are generated together? I.e., the diffusion model denoises both BBs and frames (making the conditioning of frames on BBs implicit in the neural network, rather than explicit via the 2-stage approach). I understand that it has multiple limitations such as memory required, but I wonder if this is possible and how it would compare to the proposed approach (not asking the authors to implement and run it if they haven’t).\n* It seems that generating the BBox data is a complex manual pipeline that includes tracking and extraction of other domain-specific information (like labels). It seems like a weakness that significantly limits the applicability of this approach to other domains and requires domain-specific adaptations (like trackers and annotators). Is tracking really necessary? Are you providing BBs as images with additional annotations, like class or confidence? Is it really helpful or can one just use an image of BBs without any additional labels? Wouldn't a simple image object (not video) detection model get the job done?\n* Discussion of limitations: aside from Appendix G.1., there is not much discussion of limitations. I highly encourage the authors to include a brief discussion of limitations and failure cases in the main text.\n\n**Minor**:\n* Section 3.1: it is unclear what $N$ represents. At first, I thought it was the batch-size, but then you referred to it as the number of frames. Maybe make this more explicit, e.g., “... where $N$ is the total number of frames in the trajectory.”\n* Figure 2: used both “BBox” and “Bbox”.\n* Figure 3: it is very hard to understand the differences between the generations of the proposed approach and SVD. I appreciate the effort done in the caption to explain the differences, but it is still quite hard to notice the differences. I would recommend adding visual arrows on the frames to direct the reader where to look at.\n* Figure 4: “Frame 1” indicates the first generated frame or the first frame of the video?\n* Motivation: the introduction is mostly focused on why the problem of generation is challenging, but not on why it is interesting and important for autonomous driving applications. What are the use cases of such a model, why “In domains such as autonomous driving, it is essential to develop highly accurate predictions for object motions.” (abstract)?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a diffusion-based approach for trajectory and video generation (mainly in autonomous driving domains), by modifying Stable Video Diffusion to support bounding-boxes as part of the generation boxes. The main contribution is an extension of controlling the diffusion process with bounding-boxes to the temporal dimension for controllable video generation. The method is evaluated on several autonomous driving datasets, shows favorable performance and presents more accurate longer-horizon generations than previous diffusion methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* Fix project site link and move the link location from Figure 1’s caption.\n* Open-source code: do the authors plan to provide an open-source implementation for further research?\n* See “Weaknesses” and “Minor” above.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a controllable video generation algorithm, with a clear focus on the autonomous driving domain. The method extends diffusion-based models by incorporating bounding-box guidance across the temporal dimension, and the experimental results across several driving datasets support the claims. Reviewers generally acknowledged the technical soundness and relevance of the work, while pointing out some limitations, such as the narrow application scope and the reliance on external bounding-box input. The authors have responded constructively, clarifying the domain-specific focus, justifying design choices, and expanding the discussion of limitations in the main text.
Decision: Accept (poster)