Making Video Models Adhere to User Intent with Minor Adjustments
Authors: Daniel Ajisafe
Eric Hedlin
Helge Rhodin
Kwang Moo Yi
Venue: Accepted by TMLR ()

Abstract: With the recent drastic advancements in text-to-video diffusion models, controlling their generations has drawn interest. A popular way for control is through bounding boxes or layouts. However, enforcing adherence to these control inputs is still an open problem. In this work, we show that by slightly adjusting user-provided bounding boxes we can improve both the quality of generations and the adherence to the control inputs. This is achieved by simply optimizing the bounding boxes to better align with the internal attention maps of the video diffusion model while carefully balancing the focus on foreground and background. In a sense, we are modifying the bounding boxes to be at places where the model is familiar with. Surprisingly, we find that even with small modifications, the quality of generations can vary significantly. To do so, we propose a smooth mask to make the bounding box position differentiable and an attention-maximization objective that we use to alter the bounding boxes. We conduct thorough experiments, including a user study to validate the effectiveness of our method.

Published in Transactions on Machine Learning Research (03/2026)
Making Video Models Adhere to User Intent with Minor Adjustments
Daniel Ajisafe
dajisafe@cs.ube.. ubc.ca
Deportment of Computer Science The University of British Columbia
Eric Hedlin
iamerich@cs.ubee
Department Computer Science The University of British Columbia
Helge Rhodin
helge. Salgerrrhooiin@uuiibeeelleleld de
Facully of Technology
Bielefeld University
Department of Computer Science The University of British Columbia
Kwang Moo Yi
krregi Ccssi.es.com www.essw..b........... ubc. ca
Department of Computer Science The University of British Cohumbia
Reviewed on OpenReview: https: // openreview. net/ forum? id= 2R
Bascline
Our method
The jellyfish suimming
turtle swimming
wolf exploring
The singetailld exploring
Figure 1: Teaser We show example bounding box controlled video generations with (left) Trailblazer Ma et al. 2024b) in T2V-Turbo Li et al. 2025) and (right) the original Trailblazer Ma et al. (024b) respectively. On top the original control signal and below is our adjusted bounding boxes. show the original bounding boxes blue and the adjusted boxes as orange. While the modification is subtle, the difference in the quality of generation is large. We modify bounding boxes to adhere better to the cross-attention maps within the video models.
Abstract

---

Published in Transactions on Machine Learning Research (03/2026)
With the recent drastic advancements in text-to-video diffusion models, controlling their generations has drawn interest. A popular way for control is through bounding boxes or layouts. However, emforcing adherence to these control inputs is still an open problem. In this work, we show that by slightly adjusting user-provided bounding boxes we can improve both the quality of generalions and the adherence to the control inputs. This is achieved by simply optimizing the bounding boxes better align with the internal attention maps of the video difffusion model while carefully balancing the focus on foreground and background. In a sense, we are modifying the bounding boxes to be at places where the model is familiar with. Surprisingly, we find that even with small modifications, the quality of generations can vary significantly. To do so, we propose a smooth mask to make the bounding box position disferentiable and an atlention-mmaxiniaalle bounding boxes. We conduct thorough experiments, including a user study to validate the offectiveness of our method. Our code made available on the project webpage to foster future research from the communily.
objective that we use to alter the
1 Introduction
Text-to-video diffusion models have made groundbreaking advances in producing high-quality prompt- directed generations Ho et. al. (2022b); Singer et. al. (2023); Weissenborn et al. (2020); Arnab et al. (2021) 2022a); ; Chen et al. 2024a). Among the various directions, methods based on diffusion Ho et al.
Ho et al.
(2022b a): Chen et al. (2024a) and transformers Weissenborn et al. (2020): [Arab al. (2021) have become
popular. While these models can be controlled through proper prompling, this is not always straight.or-
ward and requires careful prompt engineering Liu et al. (2023). In particular, the spatial control of object placement and object trajectories remains difficult.
Naturally, researchers have sought to improve the controllability of text-to-video diffusion models. These include methods that specifically train a model in addition to the main model Zhang et al. (2023 which was shown to be especially effective for text-to-image generation [Wang et al. (2023); Zhang et al. (2023)
Patashnik et al. (2021). For video generation, however, training such additional models is computationally
expensive. For spatial control, using bounding boxes or layouts as control inputs Ma et al. (2024b); Zheng
ct al. 2023) has gained attention. These approaches work without additional training by simply modifying
the internal attention maps within the video diffusion model Ma et al.] (2024b); Herlz el al.] (2023); Chefer
et al. (003): Tumanyan et al. (2023) or through guidance Patashhik et al. (2021 ; Nichol et al. (2022 with a pro-trained classifier. While these methods are effective, they are still limited in terms of adherence to the control inputs. Generated videos contain artificial outcomes because of the mismatch between how the control signals affect the video generation process and how was trained without such injection; see seeFie 1.
In this work, we show that by slightly adjusting the control inputs, we can improve both the quality of generations and the control adherence. However, finding how to adjust bounding boxes to adhere to user control without hurting the final generation outcome is challenging. For instance, modifying and balancing attention within the difffusion model is effective for control but can lead to over-saturation. In general, changes that are outside of the model's internal understanding casily cause video generation to degrade. To address these, we optimize bounding boxes to better align with the internal attention maps of the video diffusion model, while still being close to the uscr-provided bounding boxcs. More specifically, we introduce an optinization framework that ensures that attention maps edits remain differentiable with respect to the bounding box parameters. With the differentiable pipeline, for improved alignment of the bounding box control and the video model, we then propose to maximize the attention within the bounding box of the next layer after the edit, which is representative of where the neural network is focusing on once the edits are applied. Further, while we enhance focus, we also balance attention between the foreground and background areas, so that generation process does not completely ignore the background. While the resulting adjustments to the bounding boxes are small, their impact on the vidco generation is significant; see Figure
To summarize, we make the following contributions:
2

---

Published in Transactions on Machine Learning Research (03/2026)
Differentiable
mask
Maximize
Ours
Baseline line
Attention Edit
(q.(6)
Frame
Frame 4
Frame 16
Frame 24
Figure 2: Overview - We inject bounding box control for video diffusion models by editing their cross attention maps within the network. However, not all such edits are friendly to video diffusion models as they are not trained with such edits. Thus, when applying these edits, we make sure that this editing process is differentiable (Section 3.2) and adjust the edit parameters in a way such that the network behaves as intended attention antenddd-ateentoon being focused on desired regions (Section 3.3). We show the original bounding boxes in blue and the adjusted bounding boxes in orange. Though the adjustments original user imput, they create drastic difference in terms of vidco generation quality and adherence to the bounding boxes.
minimal and close to the
demonstrate that small adjustments to user intent can lead to significant improvements in con- trolled video generation when adjusting with our method;
propose a novel method to optimize the control inputs to align well with the internal attention maps of the video diffusion model, while still being close to the uscr-provided bounding boxes; to do so, present editing pipeline that alters a non-differentiable method to be differentiable with respect to the bounding box parameters;
to find better bounding-bon parameters, considering the cross-attention maps of the next layer after the edits; and conduct experiments including user study to validate the effectiveness of our method, and show that it outperforms existing methods in terms of video generation quality and adherence to user intent.
propose balanced attention maximization objective
2 Related Work
Text-to-image generative models. Generative lexi-to-image models have shown ground-breaking results in terms of high-quality images Rombach et al. 2022 Nichol et al. (2022) synthesizing different objects, persons, and places. However, these models have been shown fail adhere faithfully to spatial user
intent Rombach el. al. (2022) Ramesh et al. 2022 Nichol et al. 2022 hence requiring separate modality
e.g., boxes, scribble, etc., for better text-to-image alignment.
Text-to-image generative models with box control. With alternate input control such as simple user-defined boxes, several methods Zheng et al. (2023); : Chang al. (2023) have demonstratod that the spatial composition can be controlled more faithfully. For example, Directed diffusson Ma et al. (2024a) directs the placements of objects by introducing activations at desired positions in text-to-image diffusion model, leading better generation outcome. BoxDiff Xie et al. (2023) introduces spatial constraints such as inner-box, outer-box, and corner constraints in an optimization framework, for controlling objects and context in generated images. These methods follow a forward guidance approach. Allernatively, other work Chen et al. (2024b) demonstrates the superiority of backward guidance over forward guidance for robust layout control. Their method optimizes the latent, allowing both guided and unguided tokens to influence the generation outcome, However, these methods are limited to single images and do not investigate their effectiveness temporal data, iee,, video.
3

---

Published in Transactions on Machine Learning Research (03/2026)
Text-to-video generative models with box control. There are several works that investigate the
adherence to input control for videos Wang et al.
(2023); [iin et al. (2024); Jain et al. (2024); Ma et al. (2024b); Wang et al. (2024); Chen et al. (2025): Luo et al. (2025); Qiu et al. (2024); ; Lei et al. (2025).
None, however, look into how slight changes in controls can lead to diverse final generation outcomes.
Peekaboo Jain et al. (2024) uses atlention masking to guide a video generation process, therefore utilizing local context for generating individual objects. However, their use of an infinite attention injection in the background often results in missing background details Ma et al. (2024b). Trailblazer [M et al. (2024b), on the other hand, uses patrect/in-plaee replacement strategy to bias cross-attention maps towards user intent. With keyframed boxes, their method achieves controllable generation but often sacrifices better generation outcomes for more control. In other words, controlled generation outcomes are less faithful to the given prompl. Their method also relies on luning
per object, which is not scalable. Freetraj j et al. (2024) is also a training- frre method that achieves controllable video generation by imposing guidance on both noise construction and attention computation. While it achieves good results, its trajectory injection sometimes leads to poor visual quality or poor trajectory alignment.
Other methods, such as Motion-Zero Chen et al. (2025) and Boximator Wang et al. (2024), do not release implementation code or evaluation data, making fair experimental comparison difficult. Ctrl-V requires task- specific training and conditions video generation on an initial frame, whereas our method is training-free free and operates directly on text-to-video diffusion backbones. Generally, other methods also differ substantially in supervision regime or underlying model architecture, which limits fair comparison.
To the best of our knowledge, we are the first to explore the benefit of adjustments to box-controlled generation outcomes. We build the work of Trailblazer Ma et al. (2024b), and apply our method to two text-to-video models Ma et al. (2024b Li et al. 2025)). We demonstrate our core contributions in the following sections.
3
Method
Our core idea is to modify user input, i.e., the bounding boxes, aligning them to the internal attention mechanisms within the video diffusion model. As shown in Figure 2, we implement this by introducing a differentiable attention map editing method, which we then optimize through to adjust the bounding boxes. When adjusting, as we do not have any specific measure for what a good trajectory is-after all, we are generating video from scratch- we look into how the attention maps evolve through the network layers. Specifically, we encourage the edits to be inline with what how the neural network creates further attention maps, that is, we regularize it such that the attention map, after editing, causes the attention map of the next layer within the neural network to focus within the box.
To explain our method, and for completeness, we first review how control injection is done for a training-froe baseline, then discuss our work.
3.1 Preliminary: Trailblazer Ma et al. (2024b)
Our goal is generate video with F frames, where desired object of interest, e.g., 'cat', adheres faithfully to the motion and control of the bounding boxes B E Fx4 provided by the user. We build upon Trailblazer Ma et al. (2024b). which 'injects' user control in the form of bounding boxes, by directly adjusting the internal attention maps of the video diffusion model. A strong benefit in doing so is that there is no nced to train or finc-tune the video diffusion model, and any off-tho-shelf vidco diffusion model can be used as long as it has cross-allention layers.
Specifically, at inference, we feed an input text embedding p, noisy latent code 2, and timestep t, as input to video diffusion model θg. After each cross-attention layer, extract and edit cross-attention maps As E RCXHxWXN temporal cross-allention maps Ar E RCXH*WxFXF, Then, denoting the predicted noise sample as E,, we can write
As, =0pe= 2,(p),11).
(1)
4

---

Published in Transactions on Machine Learning Research (03/2026)
(a) Generated frame and (b) Attention map for to- (c) Attention map edited by (d) Attention map edited by user control
ken 'Swan'
Trailblazer Ma et al. (0024b) our method
Figure 3: Example of attention map editing- We show an example of the generated video frame with the desired user control bounding box, and the associated attention map edits. While effective, Trailblazer Ma ctt al. (024b)) relics on a replaccment operation that not diffcrentiable with respect to the box paramoters. Our method, on the other hand, performs a smooth differentiable edit.
where is the number of text tokens, and = 77 when using CLIP tokenizer Radford et al. (202)). Here, the control signal is then injected by modifying the attention maps As and Ar. Note here that As encodes the relationship between the feature V and text embedding p, while Ar encodes the relationship between pixels among different frames, without any association to the text.
In more detail, the attention maps are modified by directly weakening the attention map outside of the bounding box via multiplying a constant weakening factor, and strengthening the attention map within the box by adding a Gaussian map. Consider now the masks M8. E RCxH*wxX E [o, 1] and Mg. E RCCHxWXFXX E [0, 1] for the spatial and lemporal layers, respectively, which encodes whether the pixels involved in the attention maps are either within (1) the control bounding box or not (0). These maps Mg are anchored by box parameters b (bi, by, by, by) R representing top, left, right and bottom coordinates. C, H, W stands for the number of channels, the height, and the width respectively. Trailblazer Ma et al.) 2024bb then modifies the attention maps as
A. A. (Nw M.) + M.)+ X, o M.
(2)
weakening outside
strengthening inside
where subscript denotes both spatial and temporal, is the element-wise multiplication, Aw is the weak- ening factor which we set to =0.001 and X, is the strengthening factor which set to X =0.155 15 and Mg is Gaussian map, defined as?
(Fcoodd - c)² 20
-c)²
Mg exp
(3)
where Icoord is a tensor of image coordinates, c= (bl+br be-b6) E IR? is the conter the box, and standard deviation a E R22 is typically set to one-third of the box height and width we 3
While effective demonstrated in Ma to the box parameters it contains discrete borders, especially where the mask M, transitions either from 0 to 1 or vice versa. Moreover, the shape of the edit, as shown in Figure 3. has sharp discontinuities at the edges. While the Gaussian map allows focusing the attention map to the center of the box, the standard
(2024b), the edit in Equation (2) is not differentiable with respect
'Trailllazer Ma et al. (2024b) uses per-animal hyperparameter setting, but this is impractical for our evaluation scenario with hundreds of animals. We thus this value fixed value that found empirically and for all evaluations.
including method.
PIn the of the temporal attention map, there centers, for each frame, and simply take the maximum value each pixel overlapping regions.
The original paper claims one-half, but the official code release uses one-third, which what use.

---

Published in Transactions on Machine Learning Research (03/2026)
deviation of the Gaussian is set to large enough value to ensure that the attention map is spread over the entire box, but this then leads to truncated shape with clipped edges. Thus, it is non-trivial to optimize the box parameters with respect to any objective.
3.2 Differentiable attention map editing
To address this, we introduce differentiable attention map editing method that does not rely on replacement. Specifically, use the Gaussian edit map Mg as starting point, make its borders smooth that is differentiable. We then propose to edit without relying on the binary masks Ms and Mr.
Smooth masks. To prevent the discontimuities shown in Figure 3 we smooth out the borders of the Gaussian edit map Mg, with 1D smooth step functions, both the horizontal and vertical directions. Formally, we write
M5 = Mg @ M2 @ My.
(4)
where M, and My are the smooth step functions defined as
Tu bt
br
Ia
M, Sig
Sig
fi
K
(5)
Lu bt
by L
M, Sig
Sig
K
K
where к controls the strength of the smooth edge transition, Sig denotes the standard sigmoid function, and Xedge vh2 + w2, is calculated fraction of the bounding box's
Iu, Io H W. Formally, strength diagonal length, where h, w are the height, and width of the box. In practice, we set Aedge 0.03. We show an example of our attention map edit in Figure 3d.
Differentiable editing.
important to note that differentiability becomes limited as long as the discrete mask M, is utilized. The first term in Eq. (2) does not provide useful gradient for optimizing the box coordinates, since the binary mask M. represents hard selection. As result, the derivative of M, with respect to the box coordinates is zero almost everywhere and undefined at the mask boundaries.
Although, the second term may at first glance appear differentiable, it is still evaluated under multiplication by M.. Since Eq. (2) involves a product between M. and Mg, applying the product rule yields one term involving the derivative of M., which vanishes almost everywhere, and another term involving derivatives of Mg. While the latter term is non-zero, only affects how values are distributed within the current box and provides limited gradient signal for moving or resizing the box itself.
Though gradients (or subgradients) may flow through the selected values, they do not provide faithful signal for learning the selection boundaries without an explicit relaxation Xie et al. (2020). ) In contrast, our masks Mg are fully differentiable and of similar shape as the binary ones, and can thus be used instead of M, Equation (2). We thus write:
A. A. 1 M8) M&) +
A
(6)
weakening outside
strengthening inside
where again, Aw and A are the weakening and strengthening factors, respectively, which we set empirically as =0.000 and A,=0.15 same as in the case of Trailblazer Ma et al. (2024b). While the difference between Equation (6) and Equation i2 subtle, Equation (6) has no discrete borders, and thus safe to differentiate with respect. to the box parameters.

---

Published in Transactions on Machine Learning Research (03/2026)
3.3 Optimizing bounding boxes to align with attention maps
With the attention map editing now being differentiable with respect to the box parameters, can oplimize the bounding box such that the attention map of the next layer is maximized within the box R The difficulty in doing so, however, is how to define what is a good bounding box odit. Our hypothesis is that a good edit would be one that is 'easy' for the neural network to follow, thus being close to what the neural network itself would do. This is because both attention map edits in [Section 3.2 are heuristically designed and there is no guarantee that such edit would not derail the video generation process. In fact, shown earlier in Figure I. this does happen.
We thus propose to look into how the edited attention maps propagate through the network layers, and optimize the bounding boxes such that this propagation follows the user's intent. Specifically, without loss of generality, let us denote the transformation induced by the next attention layer as f, the 'values" in the typical attention operation as V, we can then write
(A!+1, 1 S(A', V.),,
(7)
where the superscript denotes the layer index, the subscript * again denotes both spatial and temporal, and A, is the edited attention map of layer I. We then look A which now represents how the network is utilizing edited attention, and aim to encourage the atlention to be within the user-specified bounding box mask M..
Inspired by ross-attention guidance (Chen et al. 2024b] we define the loss to encourage that the sum of all attention values Σ A is maximized within the box. This can be expressed in various ways, but as the magnitude of attention may differ from one layer to another, we define it such that the sum the attention values solely come from within the box, that Σ A = Σ M The summation is taken over spatial positions and selected tokens; the resulting scalar squared and averaged over the batch and across layers. We thus write our loss function as
Σ AAA o M++1
Cattn =
(8)
ΣA.
Note that this loss then scales between 0 and 1, with 0 indicating the attention map is purely concentrated to be within the bounding box.
Preventing attention from completely ignoring outside the box. Solely relying on the maximization of attention inside the bounding box causes the model to ignore the background, resulting in poor generation quality and less alignment to the prompt. Thus, we introduce balancing loss, which allows our method to attend to the outside region, leading to an effective whole generation. We write
1- ΣAA o (1 - (1-M
C nttn
ΣA. (+1
which now enforces outside of the box to retain attention.
Regularizing to remain close to user intent. As we wish to preserve as much of the user intent as possible, we regularize to keep the box close the original onc. We write
Creg b buser 1
(10)
where b would be the optimized bounding box, and buser the original box provided by the user.
Final optimization objective. The final loss is then,
Listal = Luttn + a-attaf-att + Aregfrog
(11)
where A-atta and Azeg are the regularization strength, which we empirically set to A x VA, where A is the number of pixels in the image and X =10.
NNoe that this alone causes the model highly ignore outside the bounding box, which balance; for ease explanation,
omit. this for now.
7

---

Published in Transactions on Machine Learning Research (03/2026)
Question 20
Description: the lion is walking
Play All Videos
Select the video(s) where the generation quality aligns with the description (multiple choices allowed):
Question 20:
No preference
Figure User study interface We show sample of the user study interface for prompt 'the lion is walking'. Users are asked to select their preference over a set of five video generations, provided in a preference. In this example, only the
random order. Users allowed to select multiple choices generation results are shown without user control to isolate quality vs. control. For this question, C and E look preferable, whereas the control is located at the bottom-left where the lion is at A, B, and D. C and E completely ignore user control, yet generate a preferred view of the prompt. As we are interested in controlled generation, we consider both answers to the quality preference and the trajectory faithfulness when evaluating the quality generations.
3.4 Implementation details
We implement our method in PyTorch Pasıke al. (0019 based the official code of Trailblazer Ma et a.l (2024b). We further apply our method from Trailblazer Ma et al. (2024b) to T2V-Turbo Li et al. (2025, a different vidco generator. For a fair comparison, we use the same video length of 24 frames, and the default setting for all other byperparameters associated with Trailblazer Ma et al. (024b). Because of our resource constraints- memory of our GPU cards is 32GB we use resolution of 256 x 320 and 320 320 for T2V-Turbo Li et al. (2025) and Trailblazer Ma et al. (2024b). respectively. We also use 320 320 resolution for the Peckaboo baseline, and the native resolution 320 x 512 for Freetraj baseline. We find that ===0.3 works well for the T2V-Turbo Li et al. (2025) backbone. Following Ma et al. (2024b),
only edit the spatial attention map. We run 5 optimization iterations for 5 editing steps, Icading to 25 itcrations in total. We use the Adam optimizer Kingma Ba 2015 for box adjustments and provide further details on our optimization procedure the supplemental material.
4 Experiments
4.1 Experimental setup
Evaluating controllable T2V models is challenging as generated videos lack ground truth and quality metrics only form a proxy to numan-perceived quality. is also non-trivial to generate large and diverse, input control trajectories without violating physics, as the generation outcome highly sensitive the input human-preferent preference metrics. control trajectories from animal videos. In the following subsections, will first For the latter,
control, as we will demonstrate. For the former, we run user study alongside human-

---

Published in Transactions on Machine Learning Research (03/2026)
Peekaboo (1)
Ours (1)
Peekaboo (2)
Ours (2)
The vild dog is
running iigi
The firebrat insect is noving
Figure 5: Qualitative results comparing Peekaboo Jain et al. (2024) and our method (full).
Each row shows two representative frames per method (left to right: Peekaboo, Ours). Our method yields better generation quality and improved alignment with the text prompt.
discuss the experimental setup, including the dataset and how we create the control trajectories. We also discuss the baselines, the quantitative metrics, and the user study protocol.
Dataset and bounding- box trajectories. As in Ma et al. 2024b), we use the Animal Kingdom dataset Ng et al. (2022) as a reference dataset for actual videos. This datasot contains several wild animals such as cheetah and hippotamus, with corresponding text describing activity, e.g., the cheetah is running. This dataset contains 18,74 744 video clips.
We keep only video clips with a single object using NLTK library Bird et al. (2009), and moving verbs indicating motion. We then use OWL large-scale open vocabulary delector Minderer et al. 2022 to detect object boxes, PySceneDetect [Castellano (2144-2044) library to split seenes to extract trajectories, i.e., se- quence of bounding boxes. To obtain trajectories that are useful, we drop videos that have less than two frames with the object being detected, or with only two detections across scene cuts. Also, videos that show discontinuous trajectories such abrupt camera motions or frame cuts dropped. These are trajectories that have Intersection over Union (IoU) between the consecutive frames being less than 50. The above results in 1,980 videos (157 unique animals) for training and 526 videos (96 unique animals) for testing, We interpolate detected boxes for the missing frames to obtain a complete trajectory over the video. We further filter out trajectories that are of boxes that are too small, i.e., maximum box width and height smaller than 10% the width and height of the image, and then randomly sample an interval of 24 frames from the trajectory. This results in 377 trajectories from the training set that can be used for method design and validation, and 226 trajectories exclusively for testing.
We compare our method against Peekaboo Jain et al. (2024) showcasing the quality improve- ment of our method. We also compare against Trailblazer Ma et al. (2024b). with its original backbone and we further adapt it to a different recent text-to-video model T2V-Turbo Li et al. (2025). We also compare our method against different variations of our method, specifically, our method without box optimization, our method without Eqquatin (9) and using optimized boxes with Trailblazer Ma et al. (2024b).
Baselines.
Quantitative metrics. We quantify the performance of each method using metrics. We report PickScore Kirstain al. (2023) and HPS v2 (Human Preference Score) Wu (2023) We report the mean intersection over union (mIOU) between the detected objects via an open-vocabulary detector Minderer et al. (2022) and the user control, using the animal kingdom dataset. reference.
alignment
9

---

Published in Transactions on Machine Learning Research (03/2026) Baselines
Ours w/o Box Opt.
Baselines + Our boxes
Our method
The hyena is valking
iiee iguans walking
Figure 6: Qualitative results with T2V-Turbo Li et al. (2025) (top row) and Trailblazer Ma al. (2024b) (bottom row) backbones. We display the original control in blue and optimized boxes in orange. As shown, ours provides improved rendering quality and adherence to control and the prompt, shown in the top and bottom row. See video results for best viewing.
User study evaluation. We also conduct user study with 20 participants selecting their preference over a set of 5 different methods; see Figure 4. Users were asked to perform two different tasks: (1) to evaluate the quality of the generations without being provided any bounding-box annotations to solely evaluate quality; and (2) to evaluate the adherence to the bounding box. These were provided as two separate questions and the questions were grouped by tasks. Multiple choices were allowed, including no preference. Each
participant was asked to answer 40 questions (20 for T2V-Turbo Li al. (025 and 20 for Trailblazer Ma ct al. (024bb) which takes about 20-30 minutes.
To rule out generations that completely failed due to the limitations of the video diffussio model, we chose only those which provide an loU value of at least 60% for at least one of the methods being compared. Candidate questions were then selected randomly and anonymized. A final set with overall quality is used for the study. This random selection was fixed for all participants.
Finally, as the first task ignores the user control completely (following user study design), the preference recorded for this tasks can be irrelevant to the controlled generation task as shown in Figure 4 Thus, we mark preferred quality only when user selected the video for both tasks. This study was approved by the Institutional Review Board.
4.2 Results
Quantitative results. We report the standard quantitative results demonstrates consistent improvements across different architectures Ma et al. (2024b:: Li et al. (2025) (top and bottom row). Also, our method outperforms baselines such as Peekaboo Jain al. (2024), Trailblazer Ma et al. (2024b), and Freetraj Qiu al. (2024) in terms of human preference scores using PickScore and HPSv2 (bottom row). Additionally, "Our bores + Trailblazer Ma et al. (0024)) validates the benefits of applying our adjusted boxes (top row). In terms of control accuracy (mIOU), the results show that method matches the same performance as the baselines. Though the benefit of adjustments is well highlighted in the user studies, this results reveals that our adjustments do not degrade control.
Table
Overall, our method
Qualitative results.
10

---

Published in Transactions on Machine Learning Research (03/2026)
Model
PickScore f
HPSv2 t
mIOU
Trailblazer Ma et al. (2024b)
0.244
0.222
0.37
Our boxes + Trrilblazer backbone Our method w/o Box Opt.
0.257
0.223
0.36
0.243
0.221
0.37
Our method (full)
0.257
0.225
0.37
Peekaboo Jain et al.
2024
0.125
0.189
0.30
Trailblazer Ma et al. 2024b
0.146
0.222
0.37
Freetraj Qiu et al.
2024
0.178
0.223
0.34
Trailblazer + T2V Turbo backbone
0.234
0.253
0.41
Our method using T2V-Tuboo backbone
0.317
0.263
0.41
Table 1: Quantitative results with human-preference and control metrics Our full method outperforms baseline and demonstrates consistent preference across different architectures, while achieving competitive control.
Quality preference (controlled gen. only)
Baselines
Ours w/o Bax Opt.
Our boxes
0.14
Baselines
method
Our methed wie Bal. loss
0.12
0.10
0.08
0.06
0.20
0.25
0.30
0.35
0.40
Trajectory faithfulness
Figure 7: User study results We report the user preference for trajectory faithfulness on the x-axis, and quality of generation (for generations that adhere to control) on the y-axis. The 95% confidence intervals are drawn green dashed lines. Our method significantly outperforms Trailblazer Ma et al. (024b) with both backbones.
We show qualitative examples of video frames generated by our method comparod basclines in Figure Figure 6 and Figure 8 Notably, our method performs siguifieantly better than Peekaboo Jain et al. (2024) and Freetraj Qiu et al. (2024), in terms of generation quality and adherence control, in F Figure and
Fig Figure 8 respectively.
Also, when compared with T2V-Turbo and Trailblazer backbones, our method, courtesy of adjustments, produces better generation that adhere with the prompt in Figure This is especially visible the second row, where generation follows motion the prompt "The marine iguana walking". Interestingly, applying our adjusted bounding boxes also provides better prompt following for the baseline method. Though this benefit. is not always transferable, the baseline still uses discrete editing strategy. Best performance
11

---

Published in Transactions on Machine Learning Research (03/2026)
Freetraj (1)
Ours (1)
Freetraj (2)
Ours (2)
The drosophi la me melanogaas ter fruit fly is moving
wild dog running
Figure Qualitative results comparing Freetraj Qiu et al. (2024) and our method on T2V- Turbo Li et al. (0025). Frames shown at common display height (aspect ratio preserved); frame resolutions
differ (Freetraj: 310x512, Ours: 256x320). Ours still delivers belter generation quality and adherence to control.
with adjustment is achieved with our method. Finally, in Figure 9 and Figure 10 we show additional qualitative results of our method compared to baselines.
User evaluation. We report user study results in Figure 7 by large margin. It is also worth noting that using our adjusted boxes also benefits Trailblazer Ma et al. (2024b), shown by the slight increase in preference. We hypothesize that this is because our boxes are optimized for the same underlying model. This further demonstrates that small adjustments matter, not only for the method that we have introduced, but also for the base model. Finally, Ours w/o balancing loss (i.e., Equation (9) being left out) show that taking the overall generation into account is effective. Simply optimizing the bounding box, without considering the whole generation results in a middle ground between the baseline and the non-optimized version. However, when both are enabled, we can achieve significant performance increase.
As shown, our method the most preferred
5 Conclusion
In this work, we demonstrated that small adjustments to user- aroviido bounding boxes can lead to significant improvements in controlled vidco gencration. By optimizing the bounding boxes better align with the internal attention maps of video diffusion models while maintaining proximity to user inputs, we achieved both higher quality generations and better adherence control signals. Through extensive experiments and user studies, we validated that simple yet effective approach outperforms existing methods for controlled video generation. We believe our findings open up new possibilities for improving user control in text-to-video generation by considering how control signals can be optimized to work better with pretrained models.
Limitations and future work A limitation of our method is that the quality of generated images be bound by the underlying video model, shown in Figure 11 With the rapid progress in this area, believe this limitation will be alleviated naturally. While in theory our method can be applied to other forms of control, we have only validated our idea to bounding boxes. But beyond bounding boxes, our work highlights a broader principle: control signals themselves are often misaligned with a model's internal representations, and modest optimization can substantially improve outcomes. This perspective naturally extends to future work on optimizing other conditional inputs, such as trajectories, sketches, depth cues. Finally, our method requires partial pack-propagation during optimization, thus slows down generation. With the T2V-Turbo Li
12

---

Published in Transactions on Machine Learning Research (03/2026)
Frame 1
Frame 4
Frame 16
Frame 24
Trailblazer Ma et a
(2024b
Our method
Traibllazer Ma el al
2024b
Our method
Figure 9: Additional qualitative results with Trailblazer Ma et al. (2024b) We show additional examples of our edits. As shown, ours is more consistent with user intent and of higher quality. The prompt for the top 2 rows is "The firebrat insect is moving', while the prompt for the bottom 2 rows is "The jellyfish is swimming". Video results are available in the supplementary local HTML page.
13

---

Published in Transactions on Machine Learning Research (03/2026)
Frame
Frame
Frame 16
Frame 24
Trailblazer Ma ct a..
T2V- T2V-Tubbe TV-Turo Li et a..
(2024b on
2025
Our method on T2V-Turbo Li et al. 2025
Trailblazer Ma al.
T2V-Turbo Li et al.
(2024b on
20023
T2V-Turbo Li et al. (2025
Our method on
Figure 10: Additional qualitative results with T2V-Turbo Li et al. (2025) backbone We show additional examples of our edits. Ours show more consistency with user intent and of higher quality within the box. The prompt for the top rows is "The surgeonfish swimming" " while the prompt for the bottom rows is 'The tiger is walking'. :. Video results also available in the supplementary local HTML page.
14

---

Published in Transactions on Machine Learning Research (03/2026)
(a) Trailblazer Ma et al. (2024b)
(b) Our method
Figure 11: Failure cases We show example failure cases when the vidco model fails to generate the
desired output for the prompt the orca is swimming. Here, regardless of the bounding box input, the model unable generate the content it has no knowledge of the target prompt. We hypothesize that
more advanced video models, which are being made increasingly available Research 2023 Wan et al. 2022 will alleviate this problem.
et al. 2025) backbone, our generations take minute and 37 seconds, while Trailblazer + original T2V Turbo takes 48 seconds, both on an NVIDIA RTX A6000. While our edits bring significant enhancements, optimizing this would also be an interesting extension. Conclusively, our main novelty demonstrates that small adjustments matter, and we therefore wish to draw the attention of the community to the sensitivity of control signals and hope that our work will serve as a foundational ground.
Broader impact statement Our method is applicable beyond animals, and generations can be abused with malicious intent. For example, controlling the movement of celebrities for personal gain. We discourage such practices and hope that providing proper awareness can alleviate or minimize variations such intent.
Acknowledgement This work supported by Borealis AI through the Borealis AI Global Fellowship Award. We acknowledge the computational resources provided by the University of British Columbia and thank the volunteers who participated in the user study.
References
Anurag Arnab, Mostala Dehghani, Georg Heigold, Chen Sun, Mario Lučić, and Cordelia Schmid. Vivit: A video vision transformer. In ICCV, 2021.
Steven Bird, Edward Loper, and Ewan Klein. Natural language toolkit, 2009.
Brandon Castellano. Pyscenedetect: Video scene cut detection and analysis tool, 2014-2024.
Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming- Hsuan Yang, Kevin Murphy, William T Froeman, Michael Rubinstein, al. Muse: Text-to-image generation via maskod generative transformers. In ICML, 2023.
Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or, dikend-and--xcitee. Attention-basel semantic guidance for text-to-image diffusion models. ACM Trans. Graph., 2023.
Changgu Chen, Junwei Shu, Lianggangxu Chen, Gaoqi He, Changbo Wang, and Yang Li. Motion-zero: Zero-shot moving object control framework for diffusion-based video generation. In AAAI, 2025.
Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, and Ying Shan. Videocrafter2: Overcoming data limitations for high-quality video diffusion models. In CVPR, 2024a.
15

---

Published in Transactions on Machine Learning Research (03/2026)
Minghao Chen, Iro Laina, and Andrea Vedaldi. Training- free layout control with cross-attention guidance. In WACV, 2024b.
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfr Aberman, Yacl Pritch, and Daniel Cohen-Or. Prompt-to- prompt image editing with cross attention control. In TCLR, 2023.
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint 0rXiv:210.002003 2022a.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusioo models. In NewrIPS, 2022b.
Yash Jain, Anshul Nascry, Vibhav Vineet, and Harkirat Behl. Poekaboo: Interactive video generation via masked--iifuusion. In CVPR, 2024.
Diederik Kingına and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick-a-pic: An open dataset of user preferences for text-to-imann generation. In NeurlPS, 2023.
Cheng Lei, Jiayu Zhang, Yue Ma, Xinyu Wang, Long Chen, Liang Tang, Yiqiang Yan, Fei Su, and Zhicheng Zhao. Ditraj: training-free free trajectory control for video diffusion transformer. arXiv preprint wwXiv:2509.218%0 2025.
Jiachen Li, Qian Long, Jian Zheng, Xiaofeng Gao, Robinson Piramuthu, Wenhu Chen, and William Yang Wang. T2r-tubbo-v2: Enhancing video generation model post-training through data, reward, and condi- tional guidance design. In ICLR, 2025.
Long Lian, Baifeng Shi, Adam Yala, Trevor Darrell, and Boyi Li. Lmm- Aim-grouned video diffusion models. In ICL.R, 2024.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig, Pre-train, prompt, and predict: systematic survey of prompting methods in natural language processing. ACM Comput. Surv., 2023.
Ge Ya Luo, ZhiHla Luo, Anthony Gosselin, Alexia Jolicoeur-Martinenn and Christopher Pal. Ctrl-v; Higher fidelity autonomous vehicle video generation with bounding- box controlled object motion. Trans. Mach. Learn. Res., 2025.
Wan-Duo Kurt Ma, Avisek Lahiri, John P Lewis, Thomas Leung, and W Bastiaan Kleijn. Directed diffusion: Direct control of object placement through attention guidance. In AAAI, 2024a.
Wan-Duo Kurt Ma, John P Lewis, and W Bastiaan Kleijn. Trailblazer: Trajectory control for diffusion- vidco gencration. In SIGGRAPH Asia, 2024b.
Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Doso- vitskiy, Aravindh Mahendran, Amurag Arnab, Mostafa Dehghani, Zhuoran Shen, et al. Simple open- vocabulary object detection. In ECCV, 2022.
Xun Long Ng, Kian Eng Ong, Qichen Zheng, Yun Ni, Si Yong Yeo, and Jun Liu. Animal kingdom: A large and diverse dataset for animal behavior understanding. In CVPR, 2022.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, llya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. In ICML, 2022.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-perfornance deep learning library. In NewrIPS, 2019.
16

---

Published in Transactions on Machine Learning Research (03/2026)
Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski. Styleelip: Text-driven manipulation of stylegan imagery. In TCCV, 2021.
Haonan Qiu, Zhaoxi Chen, Zhouxia Wang, Yingqing He, Menghan Xia, and Ziwei Liu. Freetraj: Tuming-fres trajectory control in video diffusion models. arXiv preprint arXiu:2406. 16863, 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. ai preprint arXiv:220..06155 2022.
Runway Research. Genmo: Text-to-video generation with diffusion models, 2023.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Texl-to-video generation without text-video data. In ICLR, 2023.
Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusion features for text-driven image-to-image translation. In CVPR, 2023.
Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-War Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, et al. Wan: Open and advanced large-scale vidco generative models. arXie preprint arXiv:8503.2000 20314, 2025.
Jiawei Wang, Yuchen Zhang, Jiaxin Zou, Yan Zeng, Guoqiang Wei, Liping Yuan, and Hang Li. Boximator: Generating rich and controllable motions for video synthesis. In ICML, 2024.
Xiang Wang, Hangjie Yuan, Shiwei Zhang, Dayou Chen, Jiuniu Wang, Yingya Zhang, Yujun Shen, Deli Zhao, and Jingren Zhou. Videocomposer: Compositional video synthesis with motion controllability. NewrIPS, 2023.
Dirk Weissenborn, Oscar Täckström, and Jakob Uszkorcit. Scaling autoregressive vidco models. In ICLR, 2020.
Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: A solid benchmark for evaluating human preferences of text-to- text-to-image synthesis. arXiv, 2023.
Jinheng Xie, Yuexiang Li, Yawen Huang, Haozhe Liu, Wentian Zhang, Yefeng Zheng, and Mike Zheng Shou. Boxdiff: Text-to-image synthesis with training box-constrained diffusion. In ICCV, 2023.
Yujia Xie, Hanjun Dai, Minshuo Chen, Bo Dai, Tuo Zhao, Hongyuan Zha, Wei Wei, and Tomas Pfister. Differentiable top-k with optimal transport. NeurIPS, 2020.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In ICCV, 2023.
Guangcong Zheng, Xianpan Zhou, Xuewei Li, Zhongang Qi, Ying Shan, and Xi Li. Layoutdiffusion: Con- trollable diffusion model for layout-lo-inage generation. In CVPR, 2023.
17

---

Published in Transactions on Machine Learning Research (03/2026)
Supplementary Material
We provide additional information to the main content, including an ablation study, failure cases, further implementation details, and results with complex patterns and difficult trajectories. We also include more results, in terms of videos, as well as an interactive slider that visualizes how internal attention maps evolve during optimization, on the project webpage.
A Ablation study
Regularization strength for deviation. We ablate the impact of the regularizer strength in Equation (11) for how much the box deviates from user intent. We show our results with Trailblazer Ma et al. (2024b)
in Figure 12] As shown, stronger regularization leads to bounding boxes that are more faithful to the user input, but their generation results are poor in quality. Too low values create much deviation. 0.1 strikes good balance between user intent and generation quality.
Linear trajectories. We also show our method applied to linear trajectories to simulate a user input. Figure 13 use T2V-Turbo Li et al. (2025) with our method and provide bounding-bow -bx that lincarly interpolates between start and an end bounding box. As shown, our method provides enhancement.
Attention map
Attention reshaped # Edit channels
Edited copy (lower channels)
Layer info
240, 1600, 77
240, 40, 40, 77
120
120, 40, 40, 77
down blocks.0.0.0
240, 1600, 77
240, 40, 40, 77
120
120, 40, 40, 77
down
blocks.0.1.0
480, 400, 77
480, 20, 20, 77
240
240, 20, 20, 77
down blocks.
480, 400, 77
480, 20, 20,
77
240
240, 20, 20, 77
down blocks.1.1.0
960, 100, 77
960, 10, 10,
480
480, 10, 10, 77
down blocks.2.00
960, 100, 77
960, 10,
480
480, 10, 10, 77
down blocks.2.1.0
960, 25, 77
960, 5,
480
480, 5, 5, 77
mid block.0.0
960, 100, 77
960, 10, 10, 77
480
480, 10, 10, 77
up up_blocks.l
blocks.1.0.00
960, 100, 77
960, 10, 10, 77
480
480, 10, 10, 77
up 12__blocks.11111
960, 100, 77
960, 10, 10, 77
480
480, 10, 10, 77
up blocks.1.2.0
480, 400, 77
480, 20, 20, 77
240
240, 20, 20, 77
up blocks.2.0.
480, 400, 77
480, 20, 20, 77
240
240, 20, 20,
77
up blocks.2.11.
480, 400, 77
480, 20, 20,
77
240
240,
20, 20,
77
up blocks.2.2.0
240, 1600, 77
240, 40, 40,
77
120
120, 40, 40,
77
up blocks.3.0.0
240, 1600, 77
240,
40,
40,
77
120
120,
40,
40,
77
up
blocks.3.1.0
240, 1600, 77
240,
40,
40,
77
120
120, 40, 40,
77
up blocks.3.2.0
As in Trailblazer Ma et al. (2024b). we edit only the lower channels across all layers. Each layer follows the spatial cross-attention naming where down blocks. can be fully written as down_blockee blocks. olltemtioons 0. O.transformer blocks. altmn2.
Table 2: Edited cross attention maps
B Failure cases
Additionally, 11% and 33% of the users did not have a preference for the quality and trajectory faithfulness questions respectively. This can be attributed to the limitations of the underlying video model both in terms of inability to faithfully generate according the prompt and at desired locations. Still, with more advancod video models becoming increasingly available Research (2023): Wan et al. (2025), we expect this problem be mitigated in the future.

---

Published in Transactions on Machine Learning Research (03/2026)
C
Further implementation details
The wolf is exploring
1.0
0.1
0.01
0.001
We show example frame with original bounding boxes as blue and the adjusted boxes orange. A higher penalty (1.0) allows less movement, while too loose penalty (0.001) causes generation to stray away from user intent. We set the penalty to 0.1 for all our experiments, which strikes a good balance.
Figure 12: Effect of regularization strength/penale
Optimization procedure We present our optimization procedure in Algorithm tion interleaves with the diffusion process.
1 and show how optimiza-
Algorithm Trajectory optimization
1: Input: Video diffusion model Os, decoder D, initial latents 2, prompt embedding p, denoise steps K, spatial edit steps Ks, temporal edit steps Kr, inner optimization steps O, learning rate 7, and trajectory boxes B
for 1 to K do r+0 t TiiestepSSchchule())
inner optimization step counter
while (i < Ks or Kr) and O do
6:
/* extract attention maps and noise prediction */
7:
(As, Ar, t O,(p,
8:
1* Apply editing (Eq. 6), compute loss (Eq. 11) and gradients */ (As, Ar) +- edit (As, Ar)
b Eq. 6 aggregatod over frames and layers; Eq. 11
10:
L [(As, Ár, B)
11:
9 + VsL
12:
/ Update bounding boxes */
13:
B AdamStep(B, n)
14:
/ Project boxes to valid boundaries */
15:
B + (Boj(B)
16:
17:
end while
18:
/ standard denoise step (no editing) */
19:
6 + (,(p,z,t)
20: + ScheduulrStep((z.en
21: end for
22: return D(2)
Keeping bounding boxes within images. To prevent optimization from making bounding boxes go out of images, we simply clip the optimized bounding box coordinates.
Additional hyperparameters. Besides the hyperparameters noted in the main paper, for others, we follow Trailblazer Ma et al. (2024b) to be comparable. Specifically for Trailblazer Ma et. al. (2024b).
2

---

Published in Transactions on Machine Learning Research (03/2026)
Zeroscope v2 576w the underlying text-to-video model and the same DPM Solver for both methods, with a guidance scale of 9.0. We run 40 denoising steps, applying the editing (and our optimization) on the first five steps. For Peekaboo baseline, we run the same number of denoising steps with the default number of frozen steps set to 2. For Frectraj baseline, we run with 50 denoising steps. We also use their DDIM cta of O, with edit steps, and set the video length to 24 frames, matching the impul. trajectory.
For T2V-Turbo Li et al, (202)), we use the official implementation, with the default parameters and 16 denoising steps. We apply the edits, both of our method and Trailblazer Ma al. (0224b) for only the first five steps.
Control metric. For computing the mloU, we simply choose, among the delected bounding boxes, one that matches most closely the user control bounding box. Note that this is different from Trailblazer [a al.) (2024b) which relies the object dctection score. Our strategy takes into account that generated vidcos are not constrained to generate only single object, there may be distractors. We found that Trailblazer Ma et al. (2024b) strategy of using the detection scores does get easily distracted by distractors, thus providing an unfair comparison. Our strategy fixes this.
Model architecture. For Trailblazer Ma et al. (024b), initial gaussian noise with shape x F 40 40 is generated where F 24 frames. We edit only the spatial attention with larget video resolution of 320 x 320. The corresponding attention maps are computed from the queries and keys, while the total number of tokens is 77 Radford et al. (2021); each token has dimension of 1024. The progression of the edited attention maps with their respoctive dimensions is detailed in Table
the U-Net architecture of Zeroscope. An
For T2V-Turbo Li al. (2025). the base model utilizes the same U-Net-style architecture as Trailblazer Ma et al. (0024b), hence edit in the same way for Trailblazer Ma et al. (2024b). We use PyTorch Paszke
et al. 2019) gradient checkpointing in our implementations.
D Ablations on component design choices
We include comprehensive ablations key design choices in Tables to 6
Deviation penalty. Table shows that excessive deviation leads to loss of user intent (lower mloU), while overly restrictive penalties limit quality improvements. A penalty 0.1 balances control and generation quality.
Smoothing kernel normalization. Table 4 demonstrates that normalizing the smoothing kernel to pro- duce consistent peak attention across layers significantly improves PickScore, HPSv2, and mloU.
Background preservation term Table 5 confirms that the background attention term is necessary for
maintaining scene coherence.
Edge strength. Table shows that very low edge strength behaves similarly to discontinuous masks (e.g., Trailblazer); degrading quality, while overly strong edges reduce control accuracy. Intermediate values (0.001 0.03) provide the best trade-off.
E Complex patterns and difficult trajectories
We show results for complex patterns and difficult trajectories such morphing, zig-zag, U-turn, and stationary-to-move in Figures 14 to 16
wot.c//aaapeeeeee/arososoaaeeeee "Cheng et.al. DPM-Solver: A ODE Salver Diftusion Probabilistie Model Sampling in Around 10 Steps, Advances in Neural Information Processing Systems, 2022
Zeroscope v2 576w model. http://///ggggioooooooooeeaaaaar
3

---

Published in Transactions on Machine Learning Research (03/2026)
The cuttlefish swimming
Trailblazer Ma
(b) Our method
(c) Trailblazer Ma
(d) Our method
et al. 2024b on
on
et al. 2024b) on
on
12v-Tur Turbo Li et al
Li al.
T2V Turo Li al.
T2V RVr-uubb Li et al.
2025
2025
2025
2025
The horse is running
(e)
Trailblazer
Ma
(f) Our method
(g)
Trailblazer Ma
(h) Our method
et al 2024b
al. 2024b
T2V-1 Turbo et 12V-Turbo
Li et al.
T2r-Turbo Li et al.
T2r-Turbo Li et al
(2025
2025
2025
2025
Figure 13: Application to perspective and linear trajectorics (top and bottom respectively) We show example frames of applying method to T2V-Turbo Li et al. 2025 with trajectory that linearly interpolates two bounding boxes (start and end). Ours still delivers improvements, both in terms of generation quality within the bounding box and adherence.
PickScore
HPSv2 1
mIOU 1
0
0.210
0.230
0.31
0.001
0.211
0.230
0.32
0.01
0.199
0.228
0.36
0.1
0.194
0.225
0.38
1
0.187
0.221
0.37
Table 3: Ablation results for deviation penalty term Are= X VA
Normalization
PickScore t HPSv2 f
mIOU
True
0.509
0.225
0.38
False
0.491
0.223
0.38
Table Ablation results for smoothing kernel normalization
Xtatt PickScore 1
HPSv2
miout 1
0.228
0.219
0.36
0.231
0..21
0.39
0.256
0.225
0.38
100
0.285
0.228
0.31
Table 5: Ablation results for background proservation term X-atta

---

Published in Transactions on Machine Learning Research (03/2026)
Aedge
PickScore f
HPSv2
mIOU
0.0001
0.240
0..224
0.40
0.001
0.248
0.226
0..8
0.03
0.243
0.2255
0.38
0.1
0.268
0.232
0.36
Table 6: Ablation results for edge strength Aedge
Frame
Frame
Frame 13
Trailblazer Ma et al
Traillaazer back
Our boxes +
Figure 14: Additional qualitative results for morphing task (Parrot Penguin)- We show
transition frames, where our optimized boxes provide benefit for better control of the morphing task.
Frame
Frame 4
Frame 16
Frame 24
Our method
Our method
Figure 15: Additional qualitative results with stationary-to-mov and U-turn trajectories
We
show example frames where our method is applied complex patterns and preserves difficult motions. Best viewed in motion the project webpage.

---

Published in Transactions on Machine Learning Research (03/2026)
Frame
Frame 4
Frame 16
Frame 24
Our method
Figure 16: Extreme scenario case with very small boxes and in zig-zag trajectory We present extreme scenario with zig-zag motion using very small boxes, different from the object size. Our method compensates and generates motion that follows user intent. Best viewed in motion on the project webpage.


Reviews:
[{"review_id": "jEFXh3AozE", "reviewer": "Reviewer_mm2Q", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a training-free method to improve bounding-box–based control in text-to-video diffusion models by optimizing the user-provided bounding boxes themselves rather than modifying the model. The core idea is to slightly adjust bounding boxes so that they better align with internal attention maps of the diffusion model.\n\nTechnically, the method replaces the discrete binary masking used in prior work with a smooth, differentiable mask, enabling gradients with respect to bounding-box parameters. The authors then optimize the box locations using an objective that maximizes next-layer attention within the box, while enforcing a balancing term to preserve background attention and a regularizer to remain close to the original user input. The method is evaluated on two video diffusion backbones and compared primarily against Trailblazer and Peekaboo.\n\n**Strengths.**\n\n- The method is training-free and easy to integrate into existing attention-editing pipelines.\n\n- Empirical results suggest that small box perturbations can affect perceived generation quality.\n\n**Weaknesses.**\n\n- The conceptual novelty over prior attention-guided control methods is limited.\n\n- The paper lacks clarity on core algorithmic details and optimization semantics.\n\n- Experimental evidence is insufficient to convincingly support the claimed mechanism.\n\n- Visualization and ablation studies are weak relative to the central claims.", "claims_and_evidence_yn": "No", "claims_explanation": "The paper claims that (i) small adjustments to bounding boxes significantly improve generation quality, and (ii) this improvement arises from better alignment between user intent and internal attention dynamics. However, the provided evidence falls short in several key aspects:\n\n**Unclear and under-specified optimization process.**\n\nThe paper does not clearly explain whether bounding boxes are optimized per frame or jointly across time, whether they are static or dynamic during sampling, or how optimization interleaves with diffusion steps. As a result, it is difficult to understand what is actually being optimized and when.\n\n**Weak mechanistic validation.**\n\nAlthough the method is motivated by attention alignment, the paper provides minimal visualization of attention evolution or box trajectories over optimization. There is no direct evidence that the proposed objective improves attention behavior in a meaningful or stable way, beyond indirect downstream effects.\n\n**Limited ablations on critical design choices.**\n\nSeveral central components—including the smoothing kernel design, the choice of layer at which attention is maximized, and the relative weighting of loss terms—are fixed heuristically with little justification. Without proper ablations, it is unclear whether the reported gains are robust or incidental.\n\n**Baseline coverage is narrow.**\n\nComparisons are largely limited to Trailblazer and Peekaboo. The absence of stronger or more recent trajectory-control baselines weakens the empirical claims of superiority.\n\nOverall, while the experiments suggest that box perturbations can matter, the paper does not convincingly demonstrate that the proposed method is principled, necessary, or broadly effective.", "audience_yn": "No", "audience_explanation": "The idea that user-provided control signals may be misaligned with a model’s internal representations is not new, and similar themes have appeared in prior work on attention guidance and inference-time control. While the specific instantiation of optimizing bounding boxes may interest a narrow audience working directly on video layout control, the lack of conceptual depth and weak empirical grounding limit the broader appeal.\n\nWithout stronger theoretical insight, clearer algorithmic exposition, or more compelling experimental evidence, the paper’s contribution is incremental and unlikely to significantly influence future work.", "requested_changes": "**Substantially clarify the method and optimization semantics.**\n\nPrecisely specify how bounding boxes are optimized across frames and timesteps, and how this interacts with the diffusion process.\n\n**Provide direct evidence for the claimed attention-alignment mechanism.**\n\nInclude visualizations of attention maps and box trajectories throughout optimization to demonstrate that the method behaves as intended.\n\n**Add comprehensive ablations of key components.**\n\nThis should include kernel choice, loss terms, loss weights, and layer selection. Without these, it is not possible to assess robustness.\n\n**Strengthen baseline comparisons.**\n\nInclude additional, more recent trajectory or layout control methods, or clearly justify their exclusion.\n\n**Other details**\n\nImprove writing clarity and organization; the current presentation is difficult to follow and contributes to confusion about the method; evaluate on more challenging prompts and multi-object scenarios; discuss computational cost more critically, especially given the modest gains.", "broader_impact_concerns": "This paper does not require a Broader Impact Statement, since it only leverages pretrained generative models and does not introduce additional ethical complications.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "PAzffCvtz3", "reviewer": "Reviewer_pt2F", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper addresses the challenge of enforcing spatial control in text-to-video synthesis. The authors propose a training-free optimization method that adjusts user-provided bounding boxes (spatiotemporal tubelets) to better align with the video model's internal cross-attention maps. To achieve this, the authors introduce a differentiable smooth masking technique and an optimization objective that maximizes attention within the box while preserving background attention. The results demonstrate that these \"minor adjustments\" to the input coordinates can significantly improve video quality and subject fidelity without requiring model fine-tuning. A user study confirms preferences for this method over baselines like Trailblazer and Peekaboo.", "claims_and_evidence_yn": "No", "claims_explanation": "The claims are partially supported. The paper successfully demonstrates that \"optimizing\" the box coordinates improves generation quality metrics (PickScore, HPSv2) and user preference compared to rigid control baselines.\n\nHowever, the evidence regarding \"user intent\" is limited by the experimental design:\n\n- **Dataset Limitation**: The experiments rely exclusively on trajectories extracted from the Animal Kingdom dataset using object detection (OWL-ViT). Automated detections are often jittery or noisy compared to genuine human inputs. It is unclear if the method’s success comes from \"correcting\" noisy dataset labels rather than adhering to deliberate user intent.\n\n- **Lack of Diverse Trajectories**: Real-world user intent often involves smooth curves, stationary-to-moving transitions, or complex paths (e.g., zig-zag)[1]. The current evaluation filters out discontinuous trajectories, potentially ignoring the difficult cases where user control is most needed.\n\n- **Missing Baselines for Complex Scenarios**: As noted in the Trailblazer project page, users often request complex compositions or morphing. The paper does not demonstrate how this optimization loop handles scenarios where the user strictly requires a specific path like compositing, morphing, and multiple[1].\n\n[1]https://hohonu-vicml.github.io/Trailblazer.Page/", "audience_yn": "Yes", "audience_explanation": "Yes. The paper is relevant to researchers in generative media and controllable synthesis. The proposed \"training-free\" approach  and the insight that rigid adherence to control signals can degrade quality are valuable takeaways for the community. The finding that differentiable attention editing can replace heuristic masking is of particular interest to algorithm designers.", "requested_changes": "To strengthen the paper, I request the following additions:\n\n- **Evaluation on Synthetic/Manual Trajectories**: Please include qualitative examples using manually drawn, smooth trajectories (similar to the original Trailblazer demos). This is crucial to prove the method works on intended motion, not just on filtered dataset detections.\n\n- **Complex Movement Analysis**: Include examples of more challenging movements (e.g., a \"U-turn\" or zig-zag, similar to the original Trailblazer demos) to see if the optimization \"smooths out\" the user's intended complexity to satisfy the model, or if it retains the complex shape.\n\n- **Clarification on \"Adjustment\" Limits**: Please add a discussion or ablation on how much the box is allowed to deviate. If the user draws a specific box and the optimization moves it significantly to improve quality, does this violate \"user intent\"? A visual comparison of the degree of drift for difficult prompts would be helpful.", "broader_impact_concerns": "While the experiments focus on animals, the method improves the controllability of video generation generally. The authors should briefly acknowledge the standard implications of controllable media synthesis, such as the potential for creating deepfakes or misleading content involving humans, rather than stating there are no implications.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "SEXdg9C7iH", "reviewer": "Reviewer_tqFk", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "### **Summary**\nThis paper proposes a neat and practical idea: instead of strictly enforcing user-provided bounding boxes for text-to-video (T2V) models, slightly *optimizing* these boxes can significantly improve both generation quality and control adherence. The key challenge is that prior box-based attention editing methods are non-differentiable with respect to box parameters. To address this, the authors introduce a differentiable attention edi and formulate an optimization objective that adjusts bounding boxes to better align with the model’s internal cross-attention behavior. Importantly, the method stays close to the original user intent while yielding noticeable gains. Experiments on the Animal Kingdom dataset, including user studies, show consistent improvements over prior work.\n\n### **Strengths**\n* The motivation is clear and intuitive: small mismatches between user control and the model’s internal \"comfort zone\" can hurt generation quality, and adjusting the control itself is a reasonable fix.\n* While optimization-based ideas are not new (e.g., latent optimization), applying them directly to *conditional inputs* like bounding boxes is a fresh and interesting direction, especially for video.\n* The proposed differentiable attention editing is simple yet effective, and integrates well with existing training-free control methods.\n* Qualitative results and user studies convincingly demonstrate improved visual quality and better prompt adherence compared to baselines.\n\n### **Weaknesses**\n* Although the paper compares against strong baselines (e.g., Trailblazer), several closely related works mentioned in the related work section, such as Motion-Zero or Ctrl-V, are not experimentally discussed. It would help to clarify why these methods are not directly comparable, or to provide a more explicit discussion of how the proposed approach differs from or improves upon them.\n* The claim that Equation (2) is non-differentiable with respect to the box parameters could be explained more clearly. While the discreteness can introduce discontinuities, this is not immediately obvious from the equations alone, since the box coordinates also appear in the Gaussian term. A more explicit explanation of where and why gradients break (e.g., due to hard mask boundaries) would improve clarity.\n* Some notations and implementation details are underspecified. For example, the meaning of $\\text{Sig}$ in Equation (5) is not explicitly defined. Additionally, the optimization setup is somewhat vague: Equation (11) depends on the layer index $l$ and diffusion timestep $t$ (although not annotated), but it is unclear which layers are optimized, over which denoising steps, and whether this is applied uniformly across the network. Clarifying these points would make the method easier to reproduce and better understood by readers.", "claims_and_evidence_yn": "No", "claims_explanation": "Overall, the paper provides solid empirical evidence for most of its claims, but not all of them are equally convincing.\n\nThe authors list five main contributions:\n\n1. Small adjustments to user-provided bounding boxes can significantly improve generation quality.\n2. These adjustments remain close to the original user intent.\n3. Existing attention-based box editing formulations are non-differentiable.\n4. A new differentiable attention editing formulation is proposed to enable backpropagation.\n5. Experiments and user studies demonstrate improved performance over prior work.\n\nFor claims (1), (2), (4), and (5), the evidence is generally clear and convincing. The qualitative results (e.g., Figures 2 and 6), quantitative metrics (Table 1), and user study outcomes (Figure 7) consistently support the claim that small box adjustments lead to better visual quality and control adherence without deviating much from the original inputs.\n\nHowever, the claim regarding the non-differentiability of the existing attention formulation (claim (3)) is less convincing as currently presented. While the paper states that Equation (2) is non-differentiable due to discrete boundaries, this point is not explained in enough detail to be fully persuasive", "audience_yn": "Yes", "audience_explanation": "Controllable video generation is a pretty active topic right now, both in research and in real applications. This paper’s main angle, improving controllability and quality by *optimizing the control signal itself*, feels like a simple but intriguing idea. I think at least part of the TMLR audience (especially folks working on diffusion-based generative modeling, controllability, and training-free guidance/editing) would be interested in these findings, and it could also spark follow-up work on optimizing other kinds of conditional inputs beyond boxes.", "requested_changes": "### **Critical**\n* **Strengthen the positioning vs. prior work.** The paper mentions several related methods in Section 2, but the experimental/storyline focus is mostly on Trailblazer/Peekaboo. It would really help to add a more explicit discussion of why some of the other works (already cited) aren’t directly comparable, or what specific advantage this “control input optimization” view brings compared to them. Even a short, direct paragraph would make the paper feel much more complete.\n\n* **Clarify and better justify the non-differentiability claim.** The paper states the original attention edit (Equation (2)) is non-differentiable w.r.t. box parameters due to discrete mask boundaries, but the explanation isn’t fully convincing as written. I’d strongly recommend adding a clearer justification (e.g., explicitly pointing out the hard thresholding / step behavior in the binary mask and why that blocks or destabilizes gradients), so the motivation for the differentiable reformulation is rock-solid.\n\n* **Fix missing/unclear definitions and setup details.** Some notation is introduced without definition (e.g., $\\text{Sig}$ in Equation (5)). Also, the optimization procedure would benefit from being spelled out more concretely: which layers are included in the objective (one specific layer vs. multiple/all), and over which denoising steps the updates are applied. Making these details explicit will make the method easier to understand and reproduce.\n\n### **Minor**\n* **Citation / venue naming consistency.** Some venue names are formatted inconsistently (e.g., AAAI appearing both as the full organization name and as the conference proceedings name, and casing varies). It’d be good to clean this up so the bibliography looks consistent and polished.", "broader_impact_concerns": "No explicit broader impact or ethical concerns raised by this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)