VideoGrain: Modulating Space-Time Attention for Multi-Grained Video Editing
Authors: Xiangpeng Yang
Linchao Zhu
Hehe Fan
Yi Yang
Venue: ICLR 2025 Poster (2025)
Keywords: diffusion model
video editing
Primary area: generative models
TL;DR: zero-shot method for class-level, instance-level and part-level video editing

Abstract: Recent advancements in diffusion models have significantly improved video generation and editing capabilities. However, multi-grained video editing, which encompasses class-level, instance-level, and part-level modifications, remains a formidable challenge. The major difficulties in multi-grained editing include semantic misalignment of text-to-region control and feature coupling within the diffusion model. To address these difficulties, we present VideoGrain, a zero-shot approach that modulates space-time (cross- and self-) attention mechanisms to achieve fine-grained control over video content. We enhance text-to-region control by amplifying each local prompt's attention to its corresponding spatial-disentangled region while minimizing interactions with irrelevant areas in cross-attention. Additionally, we improve feature separation by increasing intra-region awareness and reducing inter-region interference in self-attention. Extensive experiments demonstrate our method achieves state-of-the-art performance in real-world scenarios. Our code, data, and demos are available on the [project page](https://knightyxp.github.io/VideoGrain_project_page/).

Published as a conference paper at ICLR 2025
VIDEOGRAIN: MODULATING SPACE-TIME ATTEN- TION FOR MULTI-GRAINED VIDEO EDITING
Xiangpeng Yang
Linchao Zhu Hehe Fan
Yi Yang 2+
ReLER Lab, AAlI, University of Technology Sydney 2 ReLER Lab, CCAL Zhejiang University
Project Page: attps::////......... http:::/tttsseeeeeee //tpgg//////igtxppppppp github. Lo.VideoGrair wtwpe/orran.oooeee..poopo. project__aage
Class level
Instance level
Editing to same class objects
Separated editing to different instances
ewz.ch/fjol fjota
gerseite, ewz.ch/jot der Siegerseite
AEG
gerseite, ewz.ch/jok der Siegerseite
ewz.ch/ joos
ABG
left Samoyed, right Tiger, background - sunrise
man Iron Man
ewz.ch/jols
gerselte, wwj.ch/jot der Siegerselte
man Batman,
left cat Panda, right cat Toy poodle, ground + grassy meadow, background starry night
clay
stone wall iced wall
Part level
Adding new objects or modifying existing attributes at the part-level
mGn
Superman
Superman
cap
gray, half-sleeve
gray blue half-sleeve-ffe suit
Figure VideoGrain enables multi multi-grained video editing across class, instance, and part levels.
ABSTRACT
Recent advancements in diffusion models have significantly improved video gen- eration and editing capabilities. However, multi-grained video editing, which encompasses class-level, instance-level, and part-level modifications, remains a formidable challenge. The major difficulties in multi-grained editing include se- mantic misalignment of text-to-region control and feature coupling within the diffusion model. address these difficulties, we present VideoGrain, a zero- shot approach that modulates space-time (cross- and self-) attention mechanisms to achieve fine-grained control over video content. We enhance text-to-region control by amplifying each local prompt's attention its corresponding spatial- disentangled region while minimizing interactions with irrelevant areas in cross- attention. Additionally, we improve feature separation by increasing intra-region awareness and reducing inter-region interference in self-attention. Extensive ex- periments demonstrate our method achieves onte-of-thhear performance real- world scenarios. Our code, data, and demos are available on the project page.
"Corresponding author

---

Published as a conference paper at ICLR 2025
INTRODUCTION
Source Video
Class Level
Instance Level
Part Level
T21 class-level: TakenFlow
T21 instanco-level:
T2V based:
T2V based:
Ground-A-
DMT
hot Spidermon
left mote Spidermann right mole bear +sunglosses
Polar bear
*A Spidermon and polor bear ore jecging grassy meadow
before cherry
Figure 2:
Definition of multi-grained video editing and comparison on instance editing
Recent advances in Text-to-lmage (T2I) and Lext-to-Vided (T2V) diffusion models (Rombach et al., 2022; Wang al., 2023a; Brooks al., 2024) have enabled video manipulation through natural language prompts. In practical applications, enabling users to edit regions at various levels of gran- ularity based on textual prompts offers greater flexibility. To investigate this, we introduce new task called multi- multi-grained video editing, which encompasses class-level, instance-level, and part- level editing, as shown in Fig. left. Class-level editing refers to modifying objects within the same class. Instance-level editing means editing different instances into distinct objects. Part-level going further, requires adding new objects or modifying existing attributes at part-level.
While existing methods employ various visual consistency techniques, such as optical flow (Cong et al., 2023; Yang et al., 2023), control signals (Zhang et al., 2023b), or feature correspondence (Geyer et al.. 2023). These methods remain instance-agnostic, often mixing features of different instances during editing (see Fig. right). Ground-A- A-Vide- (Jeong & Ye, 2023), which inherits text-to-bounding box generation priors (Li et al., 2023), should be instance-level editing but still suffer from artifacts. Similarly, recent T2V-based methods like DMT (Yatim et al., 2024) and Pika (pik), although equipped with video generation priors, struggle with multi-grained edits. We find that the core issue is that diffusion models tend to treat different instances as the same class segments, leading to strong feature coupling across instances, as illustrated in Figure 3.
To address this problem, our primary insight is to 1) enable rext-to-region control and 2) keep feature separation between regions. In the typical diffusion models, the cross-attention layer serves key component to update textual features control over each spatial region, while the self-attention layer generates globally coherent structures by connecting each frame token across time. Therefore, we propose Spatial-Tempores Layout-Guided Attention (ST-Layout Attn), which modulates both space- time cross- and self-attention in a unified manner to achieve the above goals.
In the cross-attention layer, the uniform application of global text prompts across all frame tokens leads to severe semantic misalignment, which reduces the precision of multi-grained text-to-region control. To address this, we modulate cross-attention to amplify each local prompt's focus its corresponding spatial-disentane
region while suppressing attention to irrelevant areas. In the self-attention layer, pixels from one region may attend outside or similar regions within the same class, leading to feature coupling and texture mixing, which is an inherent limitation of diffusion models that complicates multi-graine enhance feature separation by increasing intra-region focus and reducing inter-region interactions, ensuring each query attends only to its target region.
video editing. To mitigate this, we modulate self-attention to
Our key contributions can be summarized as follows:
To the best of our knowledge, this is the first attempt multi mellii-grained video editing. Our method enables both class-level, instance-level and part-level editing.
We propose a novel framework, dubbed VideoGrain, which modulates spatial-temporal cross- and self-attention for text-to-region control and feature separation between regions.
Without tuming any parameters, we achieve state-of of results on existing benchmarks and real-world videos both qualitatively and quantitatively.
2

---

Published as a conference paper at ICLR 2025
2 RELATED WORK
2.1 TEXT-TO-IMAGE EDITING/GENERATION
In the realm of single attribute text-to-image editing, various approaches have been explored, from manipulating attention maps in Pix2Pix-Zero (Parmar et al., 2023) and Prompt2Prompt (Hertz et al., 2022) to employing masks in DiffEdit (Couairon et al., 2023) and Latent Blend (Avrahami et al., 2022; 2023) for foreground modifications while preserving the background.
For multi-grained editing, efforts like Attention and Excite (Chefer et al., 2023) and DPL. (Wang et al., 2023b) focus on maximizing attention scores for each subject token and reducing attention leakage. In image generation, (Kim et al., 2023) modulates attention based on layout masks and dense captions, while (Phung et al., 2023) proposed an attention refocus loss for regularization. However, using sinle-frame layout masks and dense captioning alone is insufficient for video edit- ing, as it fails to maintain the original video's s integrity and temporal consistency.
2.2 TEXT-TO-VIDEL EDITING
Video Editing based on Image Diffusion Models. Tune-A-Vidhe (TAV) (Wu et al., 2022) is the first work to extend latent diffusion models to the spatial-temporal domain and encode the source motion implicitly by one-shot tuning but still fails to preserve local details. Fatezero (Qi et al., 2023) and Pix2Video (Ceylan et al., 2023) fuse self- or cross-attention maps in the inversion pro- cess for temporal consistency. However, (Qi et al., 2023) requires extensive RAM usage and suffers from layout preservation even when equipping TAV for local object editing. (Chai et al., 2023) and (Ouyang et al., 2023), following the Neural Atlas (Kasten al., 2021) dynamic Nerf's defor- mation field (Pumarola et al., 2021), struggle with non-grid human motion. Subsequent methods like Rerender-A- Video (Yang et al., 2023), FLATTEN (Cong et al., 2023) ControlVideo (Zhang et al., 2023b) achieve strict temporal consistency via optical-flow, depth/edge maps, but failed in multi-grained editing while preserving original layouts. Tokenflow (Geyer et al., 2023) enforces a linear mix of nearest key-frame features to ensure consistency but results in detail loss. Ground-A- Video (Jeong& & Yc, 2023) leverages groundings for multi multi-grained editing, but suffers from feature mixing when bounding boxes overlap.
Video Editing based on Video Diffusion Models. Previous video editing work primarily utilized text-to-imane SD model (Rombach et al., 2022). Recent advancements in video foundation mod- els (Yu et al., 2023; Guo et al., 2023; Wang et al., 2023a; Yang et al., 2024e) have led efforts like VideoSwap (Gu al., 2023) employ temporal priors for customized motion transfer or motion editing (Mou et al., 2025). Yet, current video foundation models are limited to fixed views and struggle with non-grid human motions. Additionally, these editing methods require tuning parame- ters, which poses challenge for real-time vidco editing applications. In contrast, our VideoGrain method requires no parameter tuning, enabling zero-shot, multi-grain gelliiggraned video editing.
METHOD
3.1 MOTIVATION
To investigate why previous methods failed in instance-level video editing (see Fig. 2), we begin with basic analysis of the self-attention and cross-attention features within the diffusion model.
As shown in Fig. 3 (b), we apply K-Means clustering to the per-frame self-attention features during DDIM Inversion. Although the clustering captures a clear semantic layout, it fails to distinguish be- tween distinct instances (c.g., "left man" and "right man"). Increasing the number of clusters leads to finer segmentation at the part level but does not resolve this issue, indicating that feature homo- geneity across instances limits the diffusion model's 's effectiveness in multi- calli--rraned video editing.
Next, attempt edit the same class of two men into different instances using SDEdit (Meng et al., 2022). However, Fig. 3 (d) shows that the weights for "Iron Man" and "Spiderman" overlap on the left man, and "blossoms" weight leaks onto the right man, resulting in the failed edit in (c).
3

---

Published as a conference paper at ICLR 2025
Frame
Frame 2
Frame 3
segment
segment =
segment =
OCK
lock
(a) Source video input
(b) K-Means cluster Self-Attentio feature
iran
spiderman
cherry
blossors
(c) Instance-level failed case (d) Cross-Attention Map: "An from Man and Spiderman are jogging umder cherry blossoms"
Figure 3: Analysis of why the diffusion model failed in instance-level video editing. Our goal is to edit left man into "Iron Man," right man into "Spiderman." and trees into "cherry blossoms." In (b), we apply K-Means on self-attention, and in (d), we visualize the 32x32 cross-attention map.
Thus, for effective multi-graing editing, we pose the following question: Can we modulate attention to ensure that each local edit's 's attention weights are accurately distributed in the intended regions?
To answer this, propose VideoGrain with two key designs: (1) Modulate cross-attention to induce textual features to congregate in corresponding spatial-disentanlled regions, thereby enabling text- to-region control. (2) Modulate self-attention across the spatial-temporal axis to enhance intra- region focus and reduce inter-region interference, avoiding feature coupling within diffusion model.
3.2 PROBLEM FORMULATION
The purpose this work to perform multi-grained video editing across multiple regions based on the given prompts. This involves three hierarchical levels:
(1) Class-level editing: Editing objects within the same class. (e.g., changing two men to "Spider- man," where both belong to the human class, seen in Fig. 2 second column)
(2) Instance-level editing: Editing each individual instance to distinct object. (e.g., editing left man to "Spiderman," right man to "Polar Bear," shown in Fig. 2 third column).
(3) Part-level editing: Applying part-level edit to specific elements of individual instances. (e.g., adding "sunglasses "when editing the right man to "Polar Bear" in Fig. 2 fourth column).
Given a source video V Nx3x where N the number of frames, our goal is to obtain an edited video V based on specified edits. We aim to improve multi-graine' control in video editing by conditioning on each region's 's location and its text prompt. More formally, we optimize a video editing model f (Tg. (71, m1), .... (Tk, m))). where Tg is global prompt, and (Te, m&) are the ku. region's 's prompt and corresponding location.
3.3 OVERALL FRAMEWORK
The proposed zero-shot multi-grained video editing pipeline is illustrated Fig. 4 top. Initially, to retain high fidelity, we perform DDIM Inversion (Song et al., 2021) over the clean latent æo to get the noisy latent Ty. After the inversion process, we cluster the self-attention features to get the semantic layout as in Fig. (b). Since self-attention features alone cannot distinguish between individual instances, we further employ SAM-Track (Cheng et al., 2023) to segment each instance. Finally, in the denoising process, we introduce ST-Layout Attn to modulate cross- and self-attention for text-to-region control and keep feature separation between regions, as detailed in Sec. 3.4.
Different from global text prompt control of all frames, VideoGrain allows paired instance- or part-level prompts and their locations to be specified in the denoising process. Our method is also versatile to ControlNet condition e, which can be depth pose maps to provide structure conditions.
4

---

Published as a conference paper at ICLR 2025
Sparial Temporal Layont Attensoon Intergration
Coor liok
Coa aow.Aliie
DDIM deneisingx2
Xy-1
lnverted Laicats
DDIM
prodictor R
Spidermon and Polor jegging on grossy cherry
Cross-Attention Layer
Modulating Cross Attention
Belore Linss-Atintion
Medisaation
Query
nt
Bos
Tokens of
Key
So
Polle
Kogsing
posittive Lneguntive
Medultation
Tokeus of Interest
reigiaa atuicia polles soes. d6 Engalvancee
Spedenran
Poler (er
Boar
Self-Atteation Layer
Modulating Self Attention
Befuzz Saff-Atrration
KevtH
Medslbetion
X
M
H
After Selt Amontion Modoolation
"pasittine Loegantive
cripial teataiii
16 wwg.kie SOTCE
Figure 4: VideoGrain pipeline. (1) we integrate ST-Layout Attn into the frozen SD for multi-grained editing, where we modulate self- and cross-asd coossatteention in a unified manner. (2) In cross-attention, we view each local prompt and its location positive pairs, while the prompt and outside-location areas are negative pairs, enabling text-to-region control. (3) In self-attention, we enhance positive awareness within intra-regions and restrict negative interactions between inter-regions across frames, making each query only attend to the target region and keep feature separation. In the bottom two figures, p denotes original attention score and w, denotes the word and frame index.
3.4
LAYOUT-GUIDED ATTENTION
Based on the observation in Sec.3.1, 1,, cross-attention weight distribution adheres to the edit result. Meanwhile, self-attention also erucial to generate temporal consistent video. However, the pixels in one region may attend to outside similar regions, which poses obstacle for multi-grained video editing. Therefore, we need to modulate both self- and cross-attention to make each pixel or local prompt only focus the correct region.
To achieve this goal, we modulate both cross- and self-attention mechanisms via a unified increase positive and decrease negative manner. Specifically, for the ish frame of the query feature, modulate the query-key QKT condition map as follows:
QKT + *MMeetf/cross
A self/cross softmax
Vd
M R. O M. (1 R.) M
where R₁ E [|gueries]x[Kkpz indicates the query-key pair condition map at frame i, manipulating
whether to increase decrease the attention score for particular pair. And X E(t) (1 - (1) is regularization term. We follow the conclusion from (Kim et al., 2023), the {()) controls the modu- lation intensity across time-steps, allowing for gradual refinement of shape and appearance details. The latter is a size regulation term, making smaller region my subjected to larger modulation, en- abling dynamic attention weight adjustments layout size variations.
5

---

Published as a conference paper at ICLR 2025
Modulate Cross-Attention for Text-to-Region Control. In the cross- cots-atttntion layer, the textual feature serves as key and value, and interacts with the query feature from the video latent. Since each instance's appearance and location are closely related to the cross-attention weight distribution, we aim to encourage each instance's textual features to congregate in the corresponding location.
As shown in Fig. 4 mid, given the layout condition (Tk, my). For example, for T1 Spiderman, within the query-key cross-attention map, we can manually specify that the portion the query feature corresponding m is positive, while all the remaining parts are designated negative. Therefore, for each frame i, we can set the modulation value in cross attention layer as:
MrT max (QK -KK" M m = QK - mii((K])
(2)
R Jr, y) = { mi,k: y T 0, otherwise
(3)
where x and y are the query and key indices, and R. is the query-key condition map in the cross attention layer. We regularize this condition map by initially broadcasting each region's 's mask m, to its corresponding text key embedding K resulting in condition map Itçross E (H Each sub-region intensity then adjusts gradually the generation process. We set M. based on the gap between max/min values and the original scores, to keep modulated values within the original range. Our modulation applied to all frames to achieve spatial-temponts region control.
As shown in Fig. 4 (mid right), after adding positive and subtracting negative values, the original cross-attn weight of "Spiderman" (e.g., p) is amplified and focused on the left man. While the dis- tract weight of "polar" "bear" become concentrated on the right man. These indicate our modulation redistributes each prompt's weight align with target areas, cnabling precise text-to-region control.
to Keep Feature Separation. To adapt the T2I model for T2V editing, we treat the full video as "a larger picture," replacing spatial attention with spatial-temporal self- attention while retaining the pretrained weights. This enhances cross- fras-ffamme interaction and provides a broader visual context. However, naive self-attention can cause regions to attend to irrelevant or similar areas (e.g.. Fig. 4 bottom. before modulation query p attend to two-man), which leads to mixed texture. To address this, we need to strengthen positive focus within the same region and restrict negative interactions between different regions.
Modulate
As shown in Fig. 4 (bottom left), the maximum cross-frame diffusion feature indicates the strongest response among tokens within the same region. Note that DIFT (Tang et al., 2023) uses this to match different images, while we focus on cross-frame correspondences and intra-region attention modulation in the generation process. Nevertheless, negative inter-region correspondence is equally crucial for decoupling feature mixing. Beyond DIFT, we find that the minimum foss-frame dif- fusion feature similarity effectively captures the relations between tokens across different regions. Therefore, we define the spatial-temporal positive/negative values as:
M max (Q,[K, [Q K,,T)) Q,(K,, .K,lT), M5Re = [K, [.]] min (Ki,, ... ..K]
(4)
....
To ensure each patch attends intra-regions feature while avoiding interaction in inter-regions fea- ture. We define the -emporl query-key condition map:
{
0, Vj E [1: M). m, # my
Re (r. v)
(5)
1, otherwise
For frame indices and j. the value is zero when tokens belong different instances across frames.
As shown in the right part of Fig. bottom, after applying our self eeff-attention modulation, the query feature from the left man's nose (e.g.. p) attends only to the left instance, avoiding distraction to the right instance. This demonstrates that our self-attention modulation breaks the diffusion model's class-level feature correspondence, ensuring feature separation at the instance level.

---

Published as a conference paper at ICLR 2025
the forest
A finetruck and school bus are driving the road
" Spider Man and Wonder Woman playing badminton before charcoal grey wall'
" An Iron Man pushes Stormtrooper the soapbox on mossy stone bridge over lake in the forest
Spiderman are jogging under charry blassoms
Superman spins moon under
cherry blossoms
Instonces swap identity bg asphalt road with building under sky
Part-level editing: Superman sunglosses
Figure 5: Qualitative results, VideoGrain achieves multi-grained video editing, including class- level, instance-level, and part-level. We refer the reader to our project page for full-video results.
EXPERIMENTS
4.1 EXPERIMENTAL SETTINGS
In the experiment, we adopt the pretrained Stable Diffusion v1.5 the base model, using 50 steps of DDIM inversion and denoising. Our VideoGrain operates in a zero-shot manner, requiring no additional parameter tuning. To enhance memory efficiency. we re-engineer slice attention within our ST Layout Attn. ST Layout Attn is applied during the first 15 denoising steps. We set {(() 0.3.1 for self-attention and (() e for cross-attention, where the timestep
[0, normalized. All The experiments are conducted on an NVIDIA A40 GPU. We evaluate our VideoGrain using dataset of 76 vidco-text pairs, including videos from DAVIS (Perazzi al., 2016), TGVE', and the Internet? with 16-32 frames per video. Four automatic metrics employed for evaluation: CLIP- T. CLIP-F, Warp-Err, and Q-edit, following (Wu et al., 2022; Cong et al., 2023). All metrics are scaled by 100 for clarity. For baselines, compare against T2I-based methods, including FateZero (Qi et al., 2023), ControlVides (Zhang et al., 2023b), TokenFlow (Geyer et al., 2023), Ground (Jeong & Ye, 2023) and T2V-based DMT (Yatim al., 2024). To ensure temporal consistency, we employ FLATTEN (Cong et al., 2023) and PnP (Tumanyan et al., 2023). For faimesss, all T2I baselines are equipped with the same ControlNet conditions.
'attps://site http://ttss.gggllllee/ee google. www//wwwwwww//eeeeeeeeeeeeweeeoe http://eew/vveeeee///rrreeee
Phttpp: //wwwe istoo ophoto. ht www.wwpppell....o..............e a.mm

---

Published as a conference paper at ICLR 2025
Animal Instances
Human Instances
Part level
A Panda and Toy poodle playing toys starry night grassy
An Iron Man and monkey riding bikes snowy ground under blossoms
Ther sunglasses,
punching red boxing
sharmy night
FateZaro
ControlVicoo
TokenFiow
Grout A-Vidoo
DMT
VideoGrain
Figure 6: Qualitative comparisons. We refer the reader to our project page for detailed assessment.
4.2 RESULTS
We evaluate VideoGrain on videos covering class-level, instance-level, and part-level edits. Our method demonstrates versatility in handling animals, such as transforming "wolf" into "pig" (Fig. 5, top left). For instance-level editing, we can modify vehicles separately (e.g., transforming an "SUV" into "firetruck" and "van" into "school bus") in Fig. 5, top right. VideoGrain excels at editing multiple instances complex, occluded scenes, like "Spider-Man and Wonder Woman playing badminton" (Fig. 5, middle left). Previous methods often struggle with such rigid motion. addition, our method is capable of multi-region editing, where both foreground and background edited, as shown in the soap-box scene, where the background changes "a mossy stone bridge over a lake in the forest" (Fig. 5, middle right). Thanks to precise attention weight distribution, we can swap identities seamlessly, such and "Spider-Mane Man" swap identities (Fig. 5, bottom left). For part-level edits, VideoGrain excels in adjusting a character to wear a Superman suit while keeping sunglasses intact (Fig. bottom right). Overall, for multi-grained editing, our VideoGrain demonstrates outstanding performance.
the jogging scene, where "Iron Man"
4.3 QUALITATIVE AND QUANTITATIVE COMPARISONS
Qualitative Comparison. Figure shows a comparison between VideoGrain and baseline meth- ods, including T2I-based and T2V-ba approaches, for instance-level and part-level editing. For fairness, all T21-based methods use ControlNet conditioning. (1) Animal instances:
the left column, T21-based methods like FateZero, ControlVideo, and TokenFlow edit both cats into pandas due to same- clas- feature coupling diffusion models, failing to perform separate edits. DMT, even with video generation priors, still blends the panda and toy poodle features. In contrast, VideoGrain successfully edits one into panda and the other into a toy poodle. (2) Human instances: In the middle column, baselines struggle with same-class feature coupling, partially editing both men into
8

---

Published as a conference paper at ICLR 2025
Automatic Metric
Human Evaluation
Method
CLIP-F f
CLIP-T t
Warp-Err
Q-edit t
Edit-Ace t
Temp-Con t
Overall
FateZero
95.75
33.78
3.08
10.96
59.8
78.6
59.6
ControlVideo
97.71
34.41
4.73
7.27
53.2
50.0
43.6
TokenFlow
96.48
34.59
2.82
12.28
45.4
50.4
39.8
Ground-A-Viden Video
95.17
35,09
4.43
7.92
69.0
72.0
63.2
DMT
96.34
34.09
2.05
16.63
58.7
79.4
64.5
VideoGraln(ours)
98.63
36.56
1.42
25.75
88.4
85.0
83.0
Table 1: Quantitative comparison of automatic metrics and human evaluation. The best results are bolded.
Iron Man. DMT and Ground-A- A-ieeo also fail to follow user intent, incorrectly editing the left and right instances. VideoGrain, however, correctly transforms the right man into monkey, breaking the human-class limitation. (3) Part-level editing: In the third column, VideoGrain manages part- level edits, such as sunglasses and boxing gloves. ControlVides edits the gloves but struggles with sunglasses and motion consistency. TokenFlow and DMT edit the sunglasses but fail to modify the gloves or background. comparison, VideoGrain achieves both instance-level and part-level edits, significantly outperforming previous methods.
Quantitative Comparison. We compare the performance of different methods using both automatic metrics and human evaluation. CLIP-T calculates the average cosine similarity between the input prompt and all video frames, while CLIP-F measures the average cosine similarity between consec- utive frames. Additionally, Warp-Err captures pixel-level differences by warping the edited video frames according to the optical flow of the source video, extracted using RAFT-Large (Teed & Deng, 2020). To provide more comprehensive measure of video editing quality, we follow (Cong al., 2023) and use Q-edit, defined as CLIP-T/ Warp-Err. For clarity, we scale all automatic metrics by 100. In terms of human evaluation, we assess three key aspects: Edit-Accuracy (whether each local edit is accurately applied), Temporal Consistency (evaluated by participants for coherence between video frames), and Overall Edit Quality. We invited 20 participants to rate 76 video-text pairs on scale of 20 to 100 across these three criteria, following (Jeong & Ye, 2023). As demonstrated in Table 1. VideoGrain consistently outperforms both T2I- and T2V-based methods. This is primarily due to ST-Layout Attn's 's precise text-to-region control and maintaining feature separation between regions. As a result, our method achieves significantly higher CLIP-T and Edit-Accuracy scores compared other baselines. The improved Warp-Err and Temporal Consistency metrics further indicate that VideoGrain delivers temporally coherent video edits.
Efficiency Comparison. To evaluate efficiency, compared baselines with VideoGrain on single A6000 GPU for editing 16 video frames. The metrics include editing time (time taken to perform one edit) and both GPU and CPU memory usage. From Tab. 2, clear our method achieves the fastest editing time with the lowest memory usage, indicating its computational efficiency.
Timermin]
Memory (GiB)
RAM (GB)
WHo ayout Alla
Amla Weight Belove
Atta Weight After
ayout Alle
FateZero
8.68
27.35
144.22
jobs
ControlVidea
7.03
and
TokenFlow
4.56
17.84
5.35
Ground-A Video
5.81
17.31
9.95
DMT
5.79
27.88
8.12
VideoGrain
3,83
15.94
4,42
Table 2: Efficiency comparison.
Figure 7: Attention weight distribution.
4.4 ABLATION STUDY
To assess the contributions of different components our proposed ST-Layout Attn, we first evalu- ate whether our attention can achieve attention weight distribution, then decouple the self-attention modulation and cross-attention modulation to evaluate their individual effectiveness.
Attention Weight Distribution. We evaluate the impact of ST-Layout Attn on attention weight distribution. As shown in Fig. 7, the target prompt is "An Iron Man is playing tennis on a snow court." We visualize the cross-attention map for "man" to assess weight distribution. Without ST- Layout Attn, feature mixing occurs, with "snow" weight spilling onto "Iron Man." With ST-Layout Attn, the man's 's weight is correctly distributed. This is because we enhance positive pair scores and suppress negative pairs in both cross- and self-attention. This enables precise, separate edits for "Iron Man" and "snow"" Additional visualizations are in the Appendix.

---

Published as a conference paper at ICLR 2025
left man ~ Iron Man, right man
Spider man,
ground, trees
frosty yellow leaves
Source video
(-) Modulation
Self-Attention Modulation
(+) Cross-Aftention Modulation
(+) Cross-Attention Modulation
(-) Self-Attention Modulation
(+) Self-Attention Modulation
Figure 8: Ablation of cross- and self-modulation in ST-Layout Attn.
Method
CLIP-F
CLIP-T
Warp-Eir |
Qedit
Bascline
95.21
33.59
3.86
8.70
Baseline + Cross Modulation
96.28
36.09
2.53
14.26
Baseline Cross Modulation Self Modulation
98.63
36.56
1.42
25.75
Table 3: Quantitative ablation of cross- and self-modulation in ST-Layout Attn.
Cross-Attention Modulation. In Fig. 8 and Tab. 3, illustrate video editing results under different set up: (1) Baseline (2) Baseline + Cross-Attn Modulation (3) Baseline + Cross-Atth Modulation + Self-Attn Modulation. As shown in Fig. 8 top right, direct editing fails to discriminate between the left and right instances, leading to incorrect (left) or no edits(right). However, when equipped with cross-attention modulation, we achieve accurate text-to-region control, thereby editing left man to "Iron Man" and right man to "Spiderman" separately. The quantitative results in Tab. 3 indicate that with cross-attention modulation (second row), CLIP-T increases by 7.4%, and Q-edit increases by 63.9%. This demonstrates the effectiveness of our cross-attention modulation.
Self-Attention Modulation. However, modulating only cross-attention still leads to structure dis- tortions, such as the spider web appearing the left man. This is caused by the coupling of same class-level features (e.g., human). When using our self-attention modulation, the feature mixing significantly reduced, and the left man retains unique object features. This is achieved by decreas- ing the negative pair scores between different instances, while increasing positive scores within the part-level details, such the distinctive blue sides, are generated
same instance. As result, in the optimized areas. The quantitative decrease in Warp-Err by 43.9% and increase in Q-edit by 80.6% in Tab. 3 further prove the effectiveness of self-attention modulation.
CONCLUSION
In this paper, we aim to solve the problem of multi-grained video editing, which includes both class-level, instance-level and part-level video editing. To the best of our knowledge, this is the first attempt at this task. In this task, we find that the key problem is that the diffusion model views different instances as same-class features and direct global editing will mix different local re- gions. To wrestle with these problems, we propose VideoGrain to modulate spatial-temporal cross- and self-attention for text-to-region control while keeping feature separation between regions. In cross-attention, enhance each local prompt's 's focus its corresponding spatial-disentangled region while suppressing attention to irrelevant areas, thereby enabling text-to-region control. In self-attention, we increase intra-region awareness and reduce inter-region interactions to keep fea- ture separation between regions. Extensive experiments demonstrate that our VideoGrain surpasses previous video editing methods both class- instance-level, and part-level video editing.
10

---

Published as a conference paper at ICLR 2025
6 ETHICS STATEMENT
This project aims to solve multi-gaiined video editing. However, the potential misuse of this tech- nology, such the creation of deceptive videos by altering identities, poses risk. Strategies like incorporating invisible watermarking could explored to ensure videos are not used maliciously.
REFERENCES
https//www.pikk..rtt URL https: /wwww http://www.i.iaaa..eeaaaa
Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 15208-1821,, 2022.
Omri Avrahami, Ohad Fried, and Dani Lischinski. Blended latent diffusion. ACM Transactions on Graphics (TOG), 42(4):1-11, 2023.
Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joc Taylor, Troy Luhman, Eric Luhman, Clarence Ng. Ricky Wang, and Aditya Ramesh. Vidco generation models as world simulators. 2024. URL https: /top:ss///paaai.// wwt. rema//eeearch// video-generationn www-eodll-soooddd--------------- www.wooddddmmaaorr....
Duygu Ceylan, Chun-Hao P Huang, and Niloy J Mitra. Pix2video: Video editing using image diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 23206-23217, 2023.
Wenhao Chai, Xun Guo, Gaoang Wang, and Yan Lu. Stablevideo: Text-driven consistency-aware diffusion video editing. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 23040-23050, 2023.
Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or. Attend-and-exctt Attention-based semantic guidance for text-to-image diffusion models. ACM Transactions on Graphics (TOG), 42(4):1-10, 2023.
Yangming Cheng, Liulei Li, Yuanyou Xu, Xiaodi Li, Zongxin Yang. Wenguan Wang, and Yi Yang. Segment and track anything. arXiv preprint arXiv:2305.06558. 2023.
Yuren Cong, Mengmeng Xu, Christian Simon, Shoufa Chen, Jiawei Ren, Yanping Xie, Juan- Manuel Perez-Rua, Bodo Rosenhahn, Tao Xiang, and Sen He. Flatten: optical flow-guided attention for consistent text-to-video editing. arXiv preprint 2rXiv:2310.05900 2023.
Guillaume Couairon, Jakob Verbeek, Holger Schwenk, and Matthieu Cord. Diffedit: Diffusion- based semantic image editing with mask guidance. In ICLR 2023 (Eleventh International Con- ference on Learning Representations), 2023.
Michal Geyer, Omer Bar-Tal, Shai Bagon, and Tali Dekel. Tokenflow: Consistent diffusion features for consistent video editing. arXiv preprint arXiv:2307. 10373 2023.
Yuchao Gu, Yipin Zhou, and Mike Zheng et al. Videoswap: Customized video subject swapping with interactive semantic point correspondence. arXiv preprint arXiv:23 /2.02877 2023.
Yuwei Guo, Ceyuan Yang, Anyi Rao, Yaohui Wang, Yu Qiao, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint arXiv:2307.04725. 2023.
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-prompt image editing with cross attention control. 2022.
Hyeonho Jeong and Jong Chul Ye. Ground-a-vidde: Zero-shot grounded video editing using text- to-image diffusion models. arXiv preprint arXiv:2310.01 2023.
Heng Jia. Yunqiu Xu, Linchao Zhu, Guang Chen, Yufei Wang, and Yi Yang. Mos2: Mixture of scale and shift experts for text-only video captioning. In Proceedings of the 32nd ACM International Conference Multimedia, pp. 8498-8507 2024.

---

Published as a conference paper at ICLR 2025
Yoni Kasten, Dolev Ofri, Oliver Wang, and Tali Dekel. Layered neural atlases for consistent video editing. ACM Transactions on Graphics (TOG), 40(6):1-12, 2021.
Yunji Kim, Jiyoung Lee, Jin-Hwa Kim, Jung-Woo Woo Ha, and Jun-Yan Zhu. Dense text-to-image generation with attention modulation. In ICCV, 2023.
Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded eot-to-image generation. CVPR, 2023.
Shaoteng Liu, Yuechen Zhang. Wenbo Li, Zhe Lin, and Jiaya Jia. Video-p2p: Video editing with cross-attention control. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8599-8608, 2024.
Yu Lu, Yuanzhi Liang, Linchao Zhu, and Yi Yang. Freelong: Training-free long video genera- tion with spectralblend temporal attention. In The Tiiht Annual Conference on Neural Information Processing Systems, 2024.
Yue Ma, Xiaodong Cun, Yingqing He, Chenyang Qi, Xintao Wang, Ying Shan, Xiu Li, and Qifeng Chen. Magicstick: Controllable video editing via control handle transformations. arXiv preprint arXiv:2312.030477 2023.
Yue Ma, Yingqing He, Xiaodong Cun, Xintao Wang, Siran Chen, Xiu Li, and Qifeng Chen. Follow your pose: goseeggided text-to-video generation using pose-free videos. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 4117-4125, 2024a.
Yue Ma, Hongyu Liu, Hongfa Wang. Heng Pan, Yingqing He, Junkun Yuan, Ailing Zeng, Chengfei Cai, Heung- Yeung Shum, Wei Liu, et al. Follow-your-emoji Fine-controllable and expressive freestyle portrait animation. arXiv preprint arXiv:2406.0199 00900, 2024b.
Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. SDEdit: Guided image synthesis and editing with stochastic differential equations. Interna- fional Conference on Learning Representations, 2022.
Chong Mou, Mingdeng Cao, Xintao Wang. Zhaoyang Zhang, Ying Shan, and Jian Zhang. Revideo: Remake a video with motion and content control. Advances in Neural Information Processing Systems, 37:18481-18500 2025.
Hao Ouyang, Qiuyu Wang, Yuxi Xiao, Qingyan Bai, Juntao Zhang, Kecheng Zheng, Xiaowei Zhou, Qifeng Chen, and Yujun Shen. Codef: Content deformation fields for temporally consistent video processing, arXiv preprint (rXXi:2308.07922 2023.
Gaurav Parmar, Krishna Kumar Singh, Richard Zhang. Yijun Li, Jingwan Lu, and Jun- Yan Zhu. Zero-shot image-to-imagee translation. In ACM SIGGRAPH 2023 Conference Proceedings, pp. 1-11, 2023.
Federico Perazzi, Jordi Pont-Tuset Brian McWilliams, and Van Gool et al. benchmark dataset and evaluation methodology for video object segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 724-732, 2016.
Quynh Phung, Songwei Ge, and Jia-Bin Huang. Grounded text-to-image synthesis with attention refocusing. arXiv preprint arXiv:2306.05427 2023.
Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 10318-10327, 2021.
Chenyang Qi, Xiaodong Cun, Yong Zhang, Chenyang Lei, Xintao Wang, Ying Shan, and Qifeng Chen. Fatezero: Fusing attentions for zero-shot text-based video editing. arXiv preprint arXiv:2303.09535 2023.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
12

---

Published as a conference paper at ICLR 2025
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In Interna- tional Conference on Learning Representations, 2021.
Luming Tang, Menglin Jia, Qianqian Wang. Cheng Perng Phoo, and Bharath Hariharan. Emer- gent correspondence from image diffusion. In shirth- Conference on Neural Information Processing Systems, 2023. URL pottps::/opeetee http///pppereeee.eeee/ee wwtp///t///ooet
d=ypoi jddnU.
Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In Computer Aerion-ECCY 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16, pp. 402-419. Springer, 2020.
Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusion features for text-driven image-to-image translation. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 1921-1930, 2023.
Jiuniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, and Shiwei Zhang. Mod- elscope text-to-video technical report. arXiv preprint arXiv:2308. 06571, 2023a.
Kai Wang, Fei Yang, Shiqi Yang. Muhammad Atif Butt, and Joost van de Weijer. Dynamic prompt learning: learnin: Addressing cross-attention leakage for text-based image editing. arXiv preprint arXiv:2309.15664, 2023b.
Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Weixian Lei, Yuchao Gu, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. arXiv preprint urXiv:2212.115666 2022.
Shuai Yang, Yifan Zhou, Ziwei Liu, and Chen Change Loy. Rerender video:
Zero-shot text- guided video-to-video translation. In ACM SIGGRAPH Asia Conference Proceedings, 2023.
Xiangpeng Yang. Linchao Zhu, Hehe Fan, and Yang. Eva: Zero-shot accurate attributes and 2024a.
multi-object video editing. arXiv preprint arXiv:2403.1611
Xiangpeng Yang. Linchao Zhu, Xiaohan Wang. and Yi Yang. Dgl: Dynamic global-local prompt tuning for text-video retrieval. In Proceedings of the AAAI Conference Artificial Intelligence, pp. 6540-6548, 2024b.
Yiyuan Yang, Guodong Long, Michael Blumenstein, Xiubo Geng, Chongyang Tao, Tao Shen, and Daxin Jiang. Pre-training cross-modal retrieval by expansive lexicon-patch alignment. In LREC- COLING 2024, pp. 12977-12987, 2024c.
Yiyuan Yang, Guodong Long, Tao Shen, Jing Jiang, and Michael Blumenstein. Durl-peeson adapter for federated foundation models. arXiv preprint arXiv:2403. /92/1, 2024d.
Zhuoyi Yang. Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-to-vided diffusion models with an expert transformer. arXiv preprint arXiv:2408. 06072, 2024e.
Danah Yatim, Rafail Fridman, Omer Bar-Tal, Yoni Kasten, and Tali Dekel. Space-time diffusion features for zero-shot text-driven motion transfer. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 8466-8866 8476, 2024.
Lijun Yu, Yong Cheng, Kihyuk Sohn, José Lezama, Han Zhang, Huiwen Chang, Alexander G Hauptmann, Ming-Hsuan Yang, Yuan Hao, Irfan Essa, et al. Magvit: Masked generative video transformer. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recog- nition, pp. (0459-10469, 2023.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 3836-3847, October 2023a.
Yabo Zhang, Yuxiang Wei, Dongsheng Jiang, Xiaopeng Zhang, Wangmeng Zuo, and Qi Tian. Con- trolvideo: Training-free controllable text-to-video generation. arXiv preprint arXiv:2305. /3077, 2023b.
Dewei Zhou, Ji Xie, Zongxin Yang, and Yi Yang. 3dis: Depth-driven decoupled instance synthesis for text-to-image generation. arXiv preprint 10. /2669, 2024.
13

---

Published as a conference paper at ICLR 2025
Different from multi-modee learning (Yang et al., 2024b;e:d; Jia et al., 2024), controllable generation (Ma et al., 2024b;a; Lu et al., 2024; Zhou et al., 2024) or video editing (Yang et al., 2024a; Ma et al., 2023) requires explicit control signals. Multi- Multi-grained editing further relies additional layout conditions to edit the class, instance, or part level. Therefore, in the appendix, we first evaluate the SAM-Track masks' impact in Section A, then validate whether
method work without SAM- Track masks in Section B. Continually, we show that our method can solely edit specific subjects in Section C and part-level modification example in Section D. We also evaluate our ST-Layout Attn's temporal focus in Section E and ControlNet's effect in Section F.
A EVALUATE
MASKS' IMPACT
"Spiderman ond Polar Bear are jogging under cherry blossoms"
SAM-Track Instunce masks
VideoP2P Joint
Spiderman weight
"ber" weight
Sherry weight
Ours
'piderman' weight "bear' weight
cherry weight
(I) VideoP2P Joint Edit with SAM-Track masks
VideoP2P 1st Edit
edit Imput
Spider
VideoP2P 2nd Edit
2nd edit input
2nd: grey Polar Beur
VidcoP2P 3rd Edit
2nd edit input
green trees cherry blossoms
(2) VideoP2P Sequential Edit based on Separate SAM-Track masks
Figure 9: VideoP2P joint and sequential edit with SAM-Track masks
To evaluate the impact of using SAM-Trace Track (Cheng et al., 2023) for instance segmentation, we compare VidcoGrain against VideoP2P (Liu et al., 2024), which equipped with SAM-Track instance masks. The instance masks replace cross-attents attention masks during editing. A 16-frame one- shot tuning is performed, and ControlNer conditioning Zhang al. (2023a) is added for faimeess.
14

---

Published as a conference paper at ICLR 2025
Two experiments tested: (1) jointly editing multiple areas in single denoising process and (2) sequentially editing three areas by inputting separate masks.
Results show that joint editing (Fig. 9(1)) modifies only left man into "Spiderman." leaving other areas unchanged due to inaccurate cross-attn weight distribution. Sequential editing (Fig. 9())) succeeds initially but fails later due to enoor accumulation in denoising, resulting in blurred details.
SSpdderman and Polar Bear are jogging under cherry blossoms
OCK
Ground-A Vidco Input
Ground-A- Vidco Result
Figure 10: Ground-A-Vidae Video joint edit with instance information
Additionally, as shown figure above 10, also in Figs 2 and 6, Ground-A- Aiddeo (Jeong & Ye, 2023) struggles with multi- gratiieg video editing tasks, even with instance-level grounding information (e.g., text-to-boute ho-bounding box), which is comparable to SAM-Track's masks.
These comparisons indicate that while SAM-Trach provides layout guidance, it does not guarantec successful edits. In contrast, our method enables zero-shot multi-grained editing, which was not achievable by any previous methods, when providing existing SOTA with SAM-Tracc masks.
B VIDEOGRAIN CAN WORK WITHOUT SAM-TRACK MASKS
"Batman is playing tennis on snowy court before an iced wall"
.ch/jobs
ewZ. fobs
ewz.ch/jobs
ch ch/jobs
WZ.ch jobs
ewz.ch jobs
(1) Input Video
(2) Ground-A-Video Results
.ch/jobe jjobs
Wzz.c Jobs
ewz.ch jobs
DDIM Inversion Cluster Masks
(4) Our Results
Figure 11: Our method without additional SAM-Track masks
cluster DDIM inversion self-attention features to get inaccurate coarse layouts. Our method still achieves high-quality multi-area editing results (4). In contrast, even with precise groundings (converted from SAM-Track masks in (I)), Ground-A- Video fails to edit all three regions. These comparisons indi- cate that method does not rely SAM-Track segmentation. Instead, it works effectively only using the self-attention feature inside the diffusion model, even without accurate layout guidance.
Our method is not strictly dependent SAM-Track masks. As shown in Fig. (33),
C SOLELY EDIT ON SPECIFIC SUBJECTS, WITHOUT BACKGROUND CHANGED
Our method is designed for multi-area editing and can naturally perform ieigground-preseeeve subject editing, it treats multi-area editing selecting regions restricted to the foreground. As
15

---

Published as a conference paper at ICLR 2025
OCK
ckk
(1) Input Video
22) Left man Iron Man
Right Spiderman
(4) Joint Edit
Figure 12: Soely edit on specific subjects, without background changed
shown in Fig 12, our method can separately edit the "left man" and "right man" or jointly edit both subjects while keeping the background unchanged.
D PART-LEVEL MODIFICATION EXAMPLES
Color change: Gray shirt
Blue shirt
Head color change: Black
Ginger
Color & shape change: Gray shirt
Black suit
Body color change: Black
>> Ginger
Figure 13: Part-level modifications on humans and animals
Our part-level editing supports not only adding objects but also part-level attribute modifications. In the human case (Fig. 13 left), our method changes the color of a gray shirt to blue (second row) and edits half-sleeve shirt into black suit (third row), showcasing part-level attribute and structure editing. Similarly, in the animal case, our method can change a cat's head or body color from black ginger while preserving the belt's color, demonstrating precise part-level modifications.
E TEMPORAL FOCUS OF ST-LAYOUT ATTN
Our ST-Layout Attn designed as a full-frame approach to ensure inter-frame consistency. As shown in Fig. 14, per-frame ST-Layout Attn causes feature coupling Iron Man, while the sparse- causal method results in flickering and misses Spider Man blue details due to their limited receptive fields for positive/negative value selection across different layouts. In contrast, our ST-Layout Attn
16

---

Published as a conference paper at ICLR 2025
CC
(1) Input Video
(2) Per-frame
(3) Sparse-Causal (first frame
previous frame)
(4) Ours
Figure 14: Temporal Focus of ST-Layout Attn
effectively preserves texture details and prevents flickering, achieving temporal consistent and layout unified multi multi-grained video editing.
F CONTROL ABLATION
z.ch/jobs
ewz. ch jobs
z.ch/jobs
ewz.ch/jobs
z.ch/jobs
ewz.ch/jobb
(I) Input Video
(2) Without ControlNet
(3) With ControlNet
Figure 15: ControlNet ablation
Our method utilizes ControlNet depth/pose conditioning in certain complex motion cases provide necessary structural guidance. As shown Fig. 15, even without ControlNet, our method can still achieve simultaneous multi-region editing. However, in such cases, there may be some structural in- consistencies between the edit object and source object due to the lack of explicit structure guidance.
G MORE GENERAL OBJECTS AND SHAPE EDITING
H MORE VISUALIZATION
I LATENT BLEND
To preserve areas not intended for editing (i.e., in A, {π-trr -222; 73 ... }), we employ Latent Blend (Avrahami et al., 2022; 2023), which leverages masks to direct the model focus areas requiring editing while keeping the background region identical to the source video.
For each frame the video, we first merge each attribute mask to form the global foreground mask M, by applying the logical OR operation across all layouts masks m4,e [M4,1, M,,2, -,m.
M, m4,1 V m,,2 ... V mi,k-
We aggregate the masks M, from all frames to obtain combined mask M, and then blend the latent states 2; at each timestep & during the denoising process as follows:
2, (1 M) 2 + M 24.
(7)
where ,, indicates the latent feature the DDIM inversion process and 2 is corresponding latent feature during the DDIM denoising process.
17

---

Published as a conference paper at ICLR 2025
Edit prompt: husky corgii and a zebra in autumn view lawn
Source Video
Edit prompt: A leddy bear and a Golden Retriever are on the grass
Source Video
prompt: porsche driving the
view lawn
Source Video
Edit prompt: red porsche car driving in the autumn view lawn forest
Source Video
Figure 16: More general objects instance editing (animals) and shape editing (cars) results.
The key behind employing Latent Blend for preserving the background is that, given desired area mask, the less noisy foreground latent can be guided by the target text prompt A, Mcanwhile, the latent features outside the mask (the background) can be preserved. This blending ensures that, even the latent feature within the edit area is modified, the background features stay consistent.
J EXPERIMENTAL DETAILS
For FateZero (Qi et al., 2023), we employ pompt-to-prompt(HHere et al., 2022) replace editing. To enhance the identity binding the edited object, we set the self/cross replacement steps at 0.3 and
18

---

Published as a conference paper at ICLR 2025
jobs
k.c .ch jobs
ewz. ch Jobs
beite. ewz.ch/jobe
of der Siegerseit der Siegerselte
Source Video
ST-Layour Atta
jobs
ch jobs
ewzz ch fobs
selte. ewz.chfjobe uf der Siegerseinf der Siegerseite
WO
Aftn Weight Before
ch/jobs
ewz joijobs
ff der Siegersalhi der Siegers elte selte, ewz.ch www.ch/oobs
ARLL
Labrea
ST-Layout Atta
jobs
ch Jobs
ewzz ch/jjoo/cc jobs
Geite. ewz.ch cjobs
f der Siegerseitef der Siegerseite
ABTG
Attn Weight After
Loniops
eWZC Colis
salte. www..cn.jec... cajosjo
of der Siegarselr der Siegerseite
Jaad
Figure 17: More frames ablation of ST-Layout Attn's effects on attention weight distribution.
the blending threshold at 0.7. TokenFlow (Geyer et al., 2023), we utilize SD editing and default to 4 keyframes for 16-frame videos. For other comparative methods like ControlVideo (Zhang (Jeong & Ye, 2023) and DMT (Yatim et al., 2024), we adhere
et al., 2023b) and Ground-A-Videe their default hyperparameter settings. To ensure fairnsss across all T21-based methods compared, we re-implement ControlNet (Zhang et al., 2023a) on their codebases.
K LIMITATIONS.
First, although our method can achieve multi-grained editing of video, the generation quality is still limited by the base model since we are training-free method. In scenarios where the generation prior to SD is not ideal, artifacts may occur in the editing results. Second, since our method is based on a T2I model. struggles with large shape deformations and significant appearance changes. This limitation is inherent in zero-shot methods. A potential future direction is to incorporate motion priors from T2V generation models (Yang et al., 2024e) handle such challenges.
attps: htttpe///gtttbb..cc/////////caaa
nttpaa https////tttttom///memm
fon-transter
19


Reviews:
[{"review_id": "ls4a0X4AHP", "reviewer": "Reviewer_cwR6", "summary": "This paper introduces a new task called multi-grained video editing, which includes class-level, instance-level, and part-level editing, and proposes a zero-shot approach, ST-Modulator, to address the challenge of distinguishing distinct instances (e.g., \"left man\" and \"right man\") by modulating space-time attention mechanisms for precise, fine-grained control over video content—all without additional training.", "questions": "- For part-level editing, is the method limited to adding objects, or can it also support modifications like changing the color of clothing or the color of an animal?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written and organized, making complex concepts easy to understand.\n- The Spatial-Temporal Layout-Guided Attention method in this paper effectively addresses the challenge of precise, multi-grained video editing by modulating cross- and self-attention. This approach enables accurate text-to-region control and clear feature separation, allowing independent edits to specific subjects while avoiding unintended changes in other areas, especially in complex multi-subject scenes.\n- Compared to other approaches, ST-Modulator achieves high computational efficiency, showing lower memory usage and faster processing times.", "weaknesses": "- Limited focus on background preservation. \n- In Figures 5 and 6, most examples demonstrate edits that include modifications to the background along with the main subjects. Could the authors provide additional examples where the editing focuses solely on specific subjects, allowing the background to remain unchanged? This would help illustrate the method’s capability for selective edits in multi-subject scenes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UBQaDQIbQu", "reviewer": "Reviewer_XFkN", "summary": "This paper presents a zero-shot approach that modulates attention mechanisms to achieve fine-grained control over video content. We enhance text-to-region control by amplifying each local prompt’s attention to its corresponding spatial disentangled region while minimizing interactions with irrelevant areas.  Extensive experiments demonstrate the method achieves state-of-the-art performance in real-world scenarios.", "questions": "See weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. This is the first attempt at multi-grained video editing.\n2. The results show the SOTA performance on existing benchmarks and real-world videos.\n3. The method is intuitive and is easy to follow.", "weaknesses": "1. The readability and presentation of the manuscript need to be enhanced. For example, in Figure 4:\nThe \"I\" in \"M_i^pos\" denotes the i-th frame, whereas the \"i\" in \"cross-attention modulation\" appears to refer to the i-th token (e.g., \"polar\" is the 3rd token?). Readers should not have to guess what these symbols mean; it would be beneficial to clarify these points explicitly.\nThe \"E\" in \"L × E\" is undefined within the context of this paper, and Figure 2 does not utilize this information. Consequently, the shape information in Figure 4 could be omitted to avoid confusion.\nIt is unclear whether \"p\" in Figure 4 represents the original attention score. This should be clearly stated to avoid ambiguity.\n2. The paper employs an additional model, SAM-Track, to perform instance segmentation, which might be unfair. If this is the case, one could directly use Grounding-DINO in conjunction with SAM-Track to accurately identify the edited area (mask) based on the provided text. Subsequently, methods like prompt-to-prompt or Video-P2P could be employed to control the editing process through attention maps, potentially achieving similar results.\n3. A key concept of the paper is to enhance attention scores in relevant areas while suppressing them in irrelevant areas. However, it's worth noting that methods such as prompt-to-prompt already implement mechanisms to either reduce or increase attention scores for editing purposes. This aspect should be discussed more thoroughly to highlight the novelty and added value of the proposed approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "r4X7Cm6EiZ", "reviewer": "Reviewer_T7q7", "summary": "This paper presents a new task called multi-grained video editing,  which encompasses class, instance and fine-grained video editing. The empirical study shows that the key obstacles hindering multi-grained editing in diffusion models are text-to-region misalignment and feature coupling.  This work utilizes a pretrained T2I diffusion model to address these problems without any training. More Specifically ,the method modulates cross-attention for text-to-region control and self-attention for feature separation. Effectiveness has been proven by extensive experiments and convincing qualitative results.", "questions": "Please kindly address the questions in the weakness section.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1. This paper addresses a highly significant problem of editing granularity in video editing. It proposes a new and interesting task called multi-grained editing, which includes class, instance, and part-level editing. It allows flexible editing at any granularity and breaks the inherent feature coupling within diffusion models.\n2. The empirical study is insightful, explaining why diffusion models are limited to class-level video editing.\n3. The proposed method is novel and consistent with its motivation. It adjusts cross-attention for accurate attention weight distribution, enables text-to-region control, and modulates self-attention for feature separation. The modulation process is operated in a unified manner, and the overall framework is training-free.\n4. The qualitative experiments are impressive, indicating that ST-Modulator surpasses previous approaches in multi-grained editing.", "weaknesses": "1. Since this work proposes a new and interesting task focused on multi-grained editing, a holistic evaluation would be beneficial, such as establishing a benchmark and developing new metrics to evaluate the performance of current methods on multi-grained video editing.\n2. ST-Modulator focuses on editing videos at different spatial granularities, but the temporal length is limited to 16-32 frames. I wonder whether ST-Modulator can be extended to handle longer video sequences, such as those with hundreds of frames.\n3. What is the difference between multi-grained editing and multi-attribute editing?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BNLjD04kWx", "reviewer": "Reviewer_4zxu", "summary": "This paper introduces ST-Modulator, a zero-shot framework for multi-grained video editing, enabling class-level, instance-level, and part-level modifications in diffusion models. To address semantic misalignment and feature coupling issues, ST-Modulator modulates space-time cross- and self-attention, allowing fine-grained text-to-region control and improved feature separation. Experimental results suggest that this approach performs competitively across benchmarks without additional parameter tuning, presenting a promising solution for multi-grained video editing.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The paper clearly defines the novel task of multi-grained video editing, motivated by practical needs in video editing. By enabling edits at class-level, instance-level, and part-level, it broadens the range of possible video modifications to better meet real-world demands.\n- The paper effectively uses visualizations to illustrate key points, such as in the *Analysis of why the diffusion model failed in instance-level video editing.* These visual aids provide intuitive insights into challenges with existing diffusion models and help clarify the motivation behind the proposed approach.\n- The proposed Spatial-Temporal Layout-Guided Attention focuses on self-attention and cross-attention mechanisms to address key issues in text-to-region control and inter-region feature separation. This method effectively targets the core issues in multi-grained editing within diffusion models.\n- The proposed method does not require parameter tuning, which makes it resource-efficient—a significant advantage for video editing applications where computational cost and tuning complexity are critical factors.", "weaknesses": "1. **Focus on Image Editing over Video-Specific Needs:** The approach seems more aligned with image editing, as it emphasizes spatial layout control. However, a key distinction in video editing is maintaining inter-frame consistency, which is crucial for coherent video results. While the Spatial-Temporal Layout-Guided Attention is introduced, the temporal aspect appears to receive less emphasis or detailed explanation.\n\n2. **Lack of Clarification on Additional Control Signals:** The authors mention that the method is compatible with ControlNet conditioning, but it remains unclear if the cases shown in the paper require extra control inputs. There is no ablation study on control conditions to assess the necessity or influence of these signals. This raises questions about fairness in comparisons with other models like ControlVideo, where discrepancies in control conditions might lead to potentially biased results.\n\n3. **Dependence on SAM-Track for Segmentation:** The method relies on SAM-Track for instance segmentation, yet no ablation study examines its impact. This dependency raises concerns: it is unclear if the observed performance gain primarily stems from using SAM-Track. For example, using SAM-Track segmentations as conditional input to other video editing models might yield similar improvements, suggesting a need for direct comparisons to validate the unique contributions of the proposed approach.\n\n4. **Lack of Code or Algorithm Details:** The paper does not provide code or a detailed algorithm description. Open-sourcing the code could help address concerns such as those in Weaknesses 2 and 3 by allowing for transparent validation of the method’s assumptions, control conditions, and segmentation impacts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)