Ground-A-Video: Zero-shot Grounded Video Editing using Text-to-image Diffusion Models
Authors: Hyeonho Jeong
Jong Chul Ye
Venue: ICLR 2024 poster (2024)
Keywords: Computer Vision
Diffusion Models
Video Editing
Primary area: generative models
TL;DR: Ground-A-Video edits multi-attributes of a video in a single inference without any training, using pretrained Stable Diffusion, spatially-continuous (depth maps), and spatially-discrete (groundings) conditions.

Abstract: This paper introduces a novel grounding-guided video-to-video translation framework called Ground-A-Video for multi-attribute video editing.
Recent endeavors in video editing have showcased promising results in single-attribute editing or style transfer tasks, either by training T2V models on text-video data or adopting training-free methods.
However, when confronted with the complexities of multi-attribute editing scenarios, they exhibit shortcomings such as omitting or overlooking intended attribute changes, modifying the wrong elements of the input video, and failing to preserve regions of the input video that should remain intact.
Ground-A-Video attains temporally consistent multi-attribute editing of input videos in a training-free manner without aforementioned shortcomings.
Central to our method is the introduction of cross-frame gated attention which incorporates groundings information into the latent representations in a temporally consistent fashion, along with Modulated Cross-Attention and optical flow guided inverted latents smoothing.
Extensive experiments and applications demonstrate that Ground-A-Video's zero-shot capacity outperforms other baseline methods in terms of edit-accuracy and frame consistency.
Further results and code are available at our project page ( http://ground-a-video.github.io )

Published as a conference paper at ICLR 2024
GROUND-A-VIDER ZERO-SHOT GROUNDED VIDEO EDITING USING TEXT-TO-IMAGE DIFFUSION MODELS
Hyeonho Jeong & Jong Chul Ye Kim Jaechul Graduate School of A1, KAIST (hyeonho. jeong, jong. ye a st ac.kr
Beput Videe] "A rabélt ccting notermelon za the tabée.
Dosut Video]
mon di nolking dog on the
10utpat Video]
estingg crange gread
[Cutoce
walking a
[Dulza! Viceo]
eacinu alnnd
(Outpol Videz]
easking geet the
[outtaat Video dap eabing
taok painting
Trage)
Outputt Vidne] avaciig
Figure 1: Ground-A- Video achieves multi-attribute editing, video style transfer with attribute change, and text- to-video generation with pose guidance, all in time-consistent and training-free fashion. The boxes in the right-bottom images visualize the series of pose guidance.
ABSTRACT
Recent endeavors in video editing have showcased promising results in single- attribute editing or style transfer tasks, either by training text-to-video (T2V) models text-video data adopting training-free methods. However, when confronted with the complexities of multi-attribute editing scenarios, they ex- hibit shortcomings such as omitting or overlooking intended attribute changes, modifying the wrong elements of the input video, and failing to preserve regions of the input video that should remain intact. To address this, here we present a translation framework called Ground-A-
novel grounding- soandiin-guided guided Video for multi-attribute video editing. Ground-A- Aideo attains temporally con- sistent multi-attribute editing of input videos in a training-free manner without aforementioned shortcomings. Central to our method is the introduction of Cross- Frame Gated Attention which incorporates groundings information into the latent representations in a temporally consistent fashion, along with Modulated Cross- Attention and optical flow guided inverted latents smoothing. Extensive exper- iments and applications demonstrate that Ground-A- V-dido''s s zero-shot capacity outperforms other baseline methods in terms of edit-accuracy and frame consis- tency. Further results and code are available at http://grromnda-vvieer github.io.
INTRODUCTION
Coupled with massive text-image datasets (Schubmann et al., 2022), diffusion models (Sohl- Dickstein et al., 2015; Ho et al., 2020; Song et al., 2020b) have revolutionized text-to-image (T2I)

---

Published as a conference paper at ICLR 2024
generation, making it increasingly accessible to generate high-quality images from text descriptions. Additionally, the domain has seen profound expansion into several subfields, including controlled generation and world image editing. On the other hand, the endeavor to extend the success to the video domain poses a significant computational hurdle. Attaining time-consistent and high- quality results necessitates training on expensive video datasets an endeavor beyond the means of most researchers, particularly given the absence of publicly available, generic text-to-video models.
As such, pioneering approaches exhibit promise in text-to-video generation (Ho al., 2022b;a) and video editing (Esser et al., 2023) by repurposing T2I diffusion model weights for extensive video data training. Specifically, in pursuit of cost-effective video generation. Wu et al. (2022) suggests fine-tuning the T2I model on a single video. which enables generating variations of the video. Similar to the practice of manipulating attention maps within the realm of image editing (Hertz et al., 2022; Tumanyan et al., 2023; Parmar et al., 2023), various methods guide the denoising process by self-attention maps (Ceylan al., 2023), cross-attention maps (Liu al., 2023; Wang et al., 2023), or both (Qi et al., 2023), which are obtained during the input video inversion stage. Recently, to incorporate the denoising process with additional structural cues, ControlNet (Zhang & Agrawala, 2023) has been transferred video domain, achieving in video generation (Khachatryan et al., 2023a) and translation (Hu & Xu, 2023; Chu et al., 2023; Chen et al., 2023; Zhang et al., 2023).
output frames
Nonetheless, in the scenario of fine-grained video editing involving multiple attribute changes, i.e. Ar=(Ta-Ta;, 7b T6, Te. 2-Tnew) ). where 7 represents specific attributed indexed by subscript, they encounter issues of degraded frame consistency, severe semantic misalignment (Park al., 2023), or both, depicted in Fig. 2-Left and See. E. In specific, instances of neglecting in- tended attribute edits (Ta-Ta), modifying the wrong elements (Ta-T6)) mixing two separate edits (Ta+Ta--Tr), and struggling to preserve regions that should remain unchanged (Te-Ter) are ob- served. This is because in the existing models, the Gross-Attentin the complex semantic changes wield their influence, where the list of intricate changes being entan- gled as a form of "one-sentence target prompt" makes the problem worse
layer is the single domain where
The key to address this issue lies in ppatially-disentanglae layout information, comprising bounding box coordinates and textual captions, namely 'groundings'. Groundings disentangle the complex semantic combination by localizing each semantic element with precise location. Recently, ground- ing has been successfully employed to text-to-image generation tasks. Li et al. (2023b) and Yang et al. (2023) finetune existing T2I models to adhere to grounding conditions using box box-image paired datasets, while Xie et al. (2023) achieves training-free box-constrained image generation by inject- ing binary spatial masks into the cross-attention space. However, unlike the literature on single- image synthesis, guiding video generation process with groundings alone could significantly detri- ment frame consistency. One major problem is that localizing the bounding box is insufficient for the smooth frame transition (Fig. 2-right), which calls the need for additional structural guidance. Consequently, we propose to integrate two distinct modalities: 1) apatially-continung including depth and optical flow maps, are employed to maintain consistent structure across frames; conditions, specifically 'groundings', enable precise localization and editing
conditions,
2) spatially- of each attribute within the source video. The principal contributions of Ground-A- A-Vi video are summarized as follows:
To our knowledge, we present the first groundings-driven video editing framework, also marking the first instance of integrating both spatially and discrete conditions. We propose a novel Modulated Cross-Attention mechanism which efficiently enables in- teractions between differently optimized unconditional embeddings.
To further enhance consistency, we suggest smoothing inverted latent representations using optical flow information, which can be employed following any type of inversion.
Extensive experiments and applications demonstrate the effectiveness of our method in achieving time-consistent and precise
editing of input videos.
2
BACKGROUND
Stable Diffusion. Distinet from traditional diffusion models (Ho et al., 2020), Stable Diffusion (SD) functions within a low-dimensional latent space. which is accessed via VAE autoencoder E, D (Kingma & Welling, 2013). More precisely, once the latent representation 20 is obtained by com-
2

---

Published as a conference paper at ICLR 2024
Figure 2: Left: Failure cases of multi-attribute video editing by various methods, driven by the target "A squirrel is eating an orange on the grass, under the aurora." Ground-A A- Vidco's successful result for the same lask is shown in the second row of Fig. 1. Right: Input video reconstruction with and without ControlNet.
pressing input image f E "xxxx through the encoder E, ie. 2g E(),, diffusion forward process gradually adds Gaussian noise to 2n to obtain 2, through Markov transition with the transi- tion probability:
((a12111) = (2 VI- 1,1) t=
where the noise schedule {,3,T=m is an increasing sequence of and T the number of diffusion timesteps. Then, the backward denoising process is given by the transition probability:
p6(2)-121) = (o0 of
t
(2)
Here, the mean (2 t) can be represented using the noise predictor eø which is learned by the minimization of the MSE loss with respect to 0: N.,.,. ||e eg (2t, (,)) where refers the zero mean unit variance Gaussian noise vector, and T ((T) is the embedding of text T.
Specifically, a prevalent approach diffusion- image editing is to use the deterministic DDIM scheme (Song al., 2020a) to accelerate the sampling process. Within this scheme, the noisy latent 2T can be transformed into a fully denoised latent zp:
0t-1
2--1
Zt
EB
t T.T....,
Oz
at--
Oit
where is reparameterized noise scheduler.
Null-text Optimization. To augment the effect of text conditioning, Ho & Salimans (2022) have presented the classifier-free guidance technique (cfgg,, where the noise prediction by eg is also carried out unconditionally, namely by 'null text'. Then the cull-conditioned prediction is extrapolated with the text-conditioned prediction to produce the classifier free guidance prediction:
Eo (2t, L, T, w c0(2,,(,T) +(1 w) (0(2,,1,2)
L,
(4)
conditional prodiction
unconditionl prodiction
where @ = =0("") denotes the embedding of a null text and w denotes the guidance scale. However, when coupled with cfg featuring large guidance scale w > 1, DDIM inversion accumulates errors during the denoising process, resulting in flawed image reconstruction. In order to fix these errors, Mokady al. (2023) optimizes the embeddings of a null text. First, DDIM inversion trajectory ( Vi and the predicated backward trajectory { are computed. Then, at each timestep
starting from T,, unconditional embeddings (0.)7= are tuned towards minimizing | 2
METHOD
GROUND-A-VIDERR OVERVIEW
Given series of source video frames f fI:N and a list of intended edits Ar, goal is to accurately edit various attributes of the source video while avoiding unwanted modifications, zero-shot yet time-consistent manner. Fig. 3 illustrates the overall architecture of Gound-A-Video achieve
3

---

Published as a conference paper at ICLR 2024
this. Initially, we automatically acquire grounding information through GLIP (Li et al., 2022). The groundings and the source prompt are manually refined to form target groundings and target prompt, commonly following AT. On the other branch, the input frames undergo individual DDIM inversion and null optimization, followed by optical flow-based smoothing to form smoothed latent features. The smoothed latents separately fed into the inflated SD and ControlNet, which are modified with a sequence of attentions to achieve the temporal consistency of the video editing. The inflated ControlNet takes an additional input of depth maps. Subsequently, the target groundings are directed to the Cross-Frame Gated Attentions of the SD model, while the target prompt and optimized null- embeddings are channeled into the Modulated Cross Attentions of both SD and ControlNet models.
In Sec. 3.2, we discuss our inflated SD with per-fram frame inversion approach and proposed attention mechanisms (Modulated Cross-Attention & Cross- Frame Gated Attention). In Sec. 3.2, we intro- duce the inflated ControlNet to incorporate a citiialll--oonnuouu prior of input video: a sequence of depth maps. Lastly in Sec. 3.4, we present optical flow guided smoothing which is applied to the inverted latent features before they are fed into the SD and ControlNet models, to further improve the consistency across frames.
Input Preparation
Denoising procoss
Depth Eslimllr
df sage
Depa maps
J
slapa
DCIM Iraendion
Inflated ControlNet
Optical Flow Semootting
IncatPrones (1N
Swooded Libera 2
Optimization
Diftuston
sarnng wosteraiiin
carig
Source Grounding
BE EEOTER
CETAER
Figure 3: Left: Input preparation. We automatically obtain video groundings of input video frames flaN via GLIP. This is followed by a handcraft editing phase for both the groundings and the source prompt. The input frames undergo individual inversion and null optimization, followed by optical flow-based smoothing. Furthermore, ControlNet input d==N are obtained via ZoeDepth estimator. Right: Denoising process. The smoothed latents x I:N are fed into the inflated SD and ControlNet. The target grounding tokens U are directed to Cross-Frame Gated Attention, while context vectors 1. g } are directed to Modulated Cross Attention. The series of attentions inflated SD's transformer blocks includes Frass-Frame Gated Attention, and Modulated Cross Attention, whereas Cross-Frais Frame Gated Attention layers are not appended in inflated ControlNet. If a binary mask, the intersection of common outer spaces of target bounding boxes, exists, it is utilized for inpainting before each denoising step. This process helps preserve regions that not the target of editing (see Sec. 4.4). For brevity, we omitted timestep in all variables.
Temporal Attention,
3.2 INFLATED STABLE DIFFUSION BACKBONE
Attention Inflation with Opatial-Temporee Self-Attention. To exploit pretrained SD which is trained without temporal considerations, recent video editing methods (Wu et al., 2022; Qi et al., 2023; Chen et al., 2023) commonly inflate Spatial Self-Attentio along the temporal frame axis. In similar vein, Ground-A- Video enables inter-frame interaction by refactoring the Spatial Self- Attention into Temporal Self-Attention while retaining the pretrained weights. In specific, for the latent representation 2f of source frame f², query features are derived from spatial fea- tures of 2/. while key and value features are computed from spatial features of concatenated latents [e] 27 Specifically, this process can be written by the following mathematical representation:
Q W 2/ K = W [=!..... NM V VVWV [2] 12t...., NN1,
where Wº, W W are projection matrices.
Per-frame Inversion with Modulated Cross-Attention. Although model inflation, such as the Spatial-Temppora Attention discussed above or the Sparse-Causal Attention in (Wu et al., 2022), can contribute to preserving global semantic consistency across frames, prior research on the model inflation (Liu et al., 2023) has shown that it adversely degrades the generation quality of the original SD model, leading to imprecise reconstruction. This occurs because Spatial Self-Attention parame- ters are employed to compute frame correlations, which have not been taught during the pre-training.
4

---

Published as a conference paper at ICLR 2024
Hence, the inflated SD model falls short for the approximate inversion. To mitigate this issue, they fine-tune the attention projection matrices on the input video.
In contrast, our pipeline adopts a per-frame approach video inversion using the original, non- inflated SD model without any training. In this process, for each of the N frames within the video fluN, carry out DDIM inversion and null-text inversion (Sec. 2) individually to obtain inverted latents 2/.1 and optimized null-embedding trajectories {@!l:N Y The latents 1 are then sub- jected to Optical flow guided latents smoothing (Sec. 3.4), before proceeding to the denoising pro- cess. Additionally, to guide the separately optimized embeddings {Ø1:>) )/=1 to restore the temporal correlation, we reengineer the calculates correspondence between latent pixels and context vectors. Specifically, when performing cfg (Eq. equation 4), unlike conditional prediction where context vectors 11:N are uniform across frames, unconditional prediction employs unique context vectors 2 Since even minor variations in the context vectors across frames adversely impact the global frame consistency (see Fig. 5), we propose reprogramming the
I::
mechanism within the transformer blocks of SD that
mechanism into Modulated Cross-Attention which QK" vd V), with
can be formulated as AQt K, V) = Softmax(
if cond if uncond (5) Here, 2 denotes a spatial latent feature of frame f at timestep t, while 'cond' and 'uncond' refer to conditional and unconditional predictions, respectively. In the process of computing the uncon- ditional prediction branch of cfg (Eq. (4)), the proposed modulation produces attention maps that correlate with the similarity between of and the merged null-embeddings [ct...., J J. thus opening a path for interaction between variant context vectors.
W of W
if cond cN if uncond
W WV cN.
Q - we. 26 K =
V -
Video Groundings with Cross-Frame Gated Attention. Li et al. (2023b) proposed GLIGEN that achieves open-world grounded text-to-image generation via continual learning. In specific, they extend the SD model by adding gated self-attention layer and fine-tuning this layer to incorporate layout information, while freezing the original SD) weights. Ground-A-Vicee adopts GLIGEN's gated attention module and refactor the operation to accommodate the video-grounding in time- consistent fashion, which has not been explored before.
Following the same notations, we denote the semantic information associated with the i-th ground- ing entity as e; and a set of bounding box coordinates for the i-th grounding entity as 4. Next, we define the concept of 'grounding' for single image, expressed as g= [(e],4] ,.... (ear; 4x)) with M denoting the number of entities to ground. Expanding from image space to vidco space, we ag- gregate groundings across N frames of the input video, yielding g². = [g ...., g. Subsequently, we disentangle the complex editing objectives and introduce handeraft modifications, transitioning e, to eç for i=1,..., M, resulting in g -lt:N-the -the revised grounding input for our model. For exam- ple, consider the first frame's s grounding, denoted as g², within the input video in Figure 3, which takes the form of [( ('rabbbit', (0.0, 0.0, 0.7, 0.9)) ..... ('table,, (0.0, 0.8, 1.0, 1.0)).. Then, the editing ... "table -Table transforms grounding g² to grounding input g² which is written as [[ ('kangaroo', (0.0, 0.0, 0.7, 0.9)), ( ('snow', (0.0, 0.8, 1.0, 1.0))] The prepared grounding input requires post-processing prior proceeding to the reverse diffu- sion. The semantic information e/ is directed to the CLIP text encoder, where is transformed into text tokens. Mcanwhile, the layout information E is encoded using Fourier embedding, follow- ing Mildenhall et al. (2021), to produce layout tokens. The text tokens and layout tokens are then projected to the single 'grounding' space using an MI.,, resulting in grounding tokens:
objectives AT= "Trabbir-Taangann
/f = (I) Fourier(?))
(6)
Given the grounding tokens for frame i and an intermediate latent representation of frame i at time t, denoted as zi. our goal is to project the grounding information onto the visual latent representation. However, frame-individual projection leads to temporal incoherence as, for example, the same text conditioning for 'kangaroo' is projected differently onto the i-th latent 2/ and the j-th latent 2/ unless zt is identical to z/. Therefore, we propose the Cross-Frame Gated Attention which globally
5

---

Published as a conference paper at ICLR 2024
integrates grounding features onto the latent representation via TS (S(fftmax OK"
va V)) with:
Q- W [z] U'], = W [2], U' 1
= U. = WV
[2] U ..... U*
where U !u]..... uhsl for the grounding tokens from equation 6, and the operation TS denotes the token slicing, which ensures that the output shape of the proposed attention operation remains consistent with the input shape.
3.3 INFLATED CONTROLNET
ControlNet (Zhang & Agrawala, 2023) starts with a trainable copy of SD UNet, purposefully de- signed to complement the SD. The trainable branch is then fine-tuned to accommodate task-specific visual conditions, extending the input of Eg (ze, t, c) to (z,,, c, d), where d represents the addi- tional conditions such as depth maps or edge maps. This combined locked-copy and trainable-copy framework preserves the original synthesis capabilities of SD while enabling precise control over the structural attributes of the generated images.
To incorporate structural guidance into the video generation process, we employ inflated ControlNet architecture and depth condition. Initially, we estimate depth maps from the source video, converting d to di:N, and apply the Self-Attention inflation and Net while retaining the fine-tuned weights. During the denoising stage, the residual latent I:N features 2res E R N Xxhxxxx of ControlNet are first scaled by 'ControlNet Scale" hyperparameter,
modulation (Sec. 3.2) on
then transmitted to the inflated SD UNet, as illustrated Figure 3. The scaling parameter regulates the degree of structural preservation between the input and output (see Fig. 8, 26).
Algorithm Optical Flow guided Inverted Latents Smoothing
Require: Number of frames N, Source video frames fI:N, Timesteps T, Text-Image Diffusion Model (SD), Optical flow estimator (RAFT), Threshold for magnitude difference Mihres i run inversion (e.g. DDIM inversion)
27 < DDIM fN T, SD)
for all = 2, .... N do
map'pt + RAFT (f f*)
obtain optical flow map > compute magnitude map b. obtain binary mask denoting static region > downsample to latent-level size > flow guided smoothing
map'nay normalize (I II)
map mack
mapinag < Mehres
map mask 2T
downsample(mapp (mak)
* map + 2)-- * (1 - map
2T
end for Return 27
1:N
> temporally smoothed latents
3.4 OPTICAL FLOW GUIDED INVERTED LATENTS SMOOTHING
As a video consists of a temporal series of images with a considerable overlap of nearly identical pixels, contemporary video compression techniques (Hu al., 2021; 2022) harness motion infor- mation to mitigate temporal redundancy instead of saving individual pixels for every frame of a video. Inspired by this, Chen et al. (2023) introduces pixel-level residuals of the source video into the diffusion process, while Hu & Xu (2023) leverages motion prior to prevent the regeneration of redundant areas for frame consistency.
In our framework, latent codes of the input video are individually computed by performing inversion on each frame through the T2I model. Hence, the approach of aggregating these latent representa- tions and directly feeding them into the denoising process suboptimal with respect to preserving consistency on static regions. Also inspired by the video codecs, we propose to refine the inverted latents, guided by optical flow information extracted from the input video frames, which accurately captures the motion changes across frames. The pseudo algorithm is shown in Algorithm 1.
Specifically, we first acquire an optical flow map map' between the consecutive frames (fi-1, using an optical flow estimation network. This flow map is represented as a two-channel image, with each channel capturing vertical and horizontal motion movement. Subsequently, we incorpo- rate the two different motion channels into one-channel magnitude map map'mag by calculating the Euclidean distance at each pixel location, followed by channel-wise normalizations. The result- ing mapinag represents the comprehensive motion prior between fi-1 and f². After thresholding

---

Published as a conference paper at ICLR 2024
Videos
Syaiid camoa!
Brem making in. cosh ugutst um!!
Thre V-dea
Control Video
dwo
Cure
actlng emssge
org ranig
Figure 4: Qualitative comparison with baseline methods: Our results exhibit superior temporal consisteney, no mutated body parts, accurate structural preservation, and the highest Edit-Accuracy without omitting or mixing components of edits, Best viewed high zoom levels.
map'mag on pre-configurd threshold Morres, which generates binary mask (map we per- form smoothing inverted latents (zf-1, , 24-) using the obtained mask. This processing guarantees that static regions share the same pixel values between frames.
4 EXPERIMENTS
4.1
IMPLEMENTATION DETAILS
We leverage pretrained weights of Stable Diffusion v1.4 (Rombach al., 2022) and ControlNet- Depth (Zhang & Agrawala, 2023) in addition to self gated attention weights from GLIGEN (Li al., 2023b). We use subset of 20 videos from DAVIS dataset (Pont-Tuset al., 2017). Generated videos are configured to consist of 8 frames, unless explicitly specified, with uniform resolution of 512x512. We benefit from BLIP-2 (Li et al., 2023a) for the automated generation of video caption- ings. We then feed-forware video frames and captionings to GLIP (Li et al., 2022) model to obtain bounding boxes of the target objects. For the applications video style transfer, bounding box coor- dinates are uniformly set as [0.0. 0.0, 1.0, 1.0) which covers the whole frame. RAFT-Large network (Teed & Deng, 2020) and ZoeDepth (Bhat et al., 2023) are employed for estimating optical flow maps and depth maps, respectively. In the Appendix (Sec. D), we provide detailed configuration of hyperparameters related to the forward and reverse diffusion processes.
4.2 BASELINE COMPARISONS
Qualitative Evaluation. We offer a visual comparison against various state-of-the-art video edit- ing approaches in Fig. 4. VontrolVideo (CV) (Zhang et al., 2023) stands out as the most relevant work to ours, as it introduces training-fre -freen video editing model that is also conditioned on Con- troINet. Control-A-Vides (CAV) (Chen et al., 2023) translates video with ControlNet guidance as well, with first-frame conditioning strategy. Tune-A-Vides (TAV) (Wu et al., 2022) efficiently fine-tunes their inflated SD model on the input video. To ensure a fair evaluation, as TAV is not provided structural guidance, apply their inflation logic to ControlNet and fine-tune the inflated- SD-ControlNet on the input video. Gen-1 (Esser et al., 2023) presents a video diffusion architecture with additional structure and content guidance specifically designed for video editing. It's worth not- ing that CAV and Gen- train their models with video datasets and that the methods with ControlNet uniformly employed depth guidance for fair comparison.

---

Published as a conference paper at ICLR 2024
Quantitative Evaluation. We evaluate the proposed method against aforementioned baselines using both automatic metrics and a user study. The results are outlined in Tab 1.
(a) Automatic metrics. We use CLIP (Radford et al., 2021) for automatic metrics. For textual alignment (Hessel et al., 2021). we calculate av- erage cosine similarity between the target prompt and the edited frames. For frame consistency, we compute CLIP image features for all frames in the output video then calculate the average co- sine similarity between all pairs of video frames. As shown in Tab. I, our method outperforms baselines in both textual alignment and temporal con- sistency. (b) User study. We surveyed 28 participants to evaluate accuracy of editing and consistency of frames in the edited videos, utilizing a rating scale ranging from 1 to 5. Specifically, to measure editing accuracy, we divided the evaluation into two questions: i. WeWe all the elements in the in- put video that needed to be edited accurately edited?" ii. Were the elements in the output video that needed to be preserved accurately preserved?" Tab. 1 shows that our method surpasses the baselines in all three aspects, particularly with a significant lead in Edit-Accuracy and Freserve-Accurae
Methed
Text-Align
Frame-Con
Edit-Aco
Preserve-Ace
Frame-Con
TAV
0.810
0,959
2,99
3.13
3.05
CAV
0.801
0.955
2.25
2:30
2,88
CV
0.822
0.963
2.36
2.02
Gen-1
0.833
0.939
2.41
2.51
2.56
Ours
0.837
0.970
4.13
4.24
4.01
Table 1: Summary of quantitative evaluations using
CLIP (Left) and user study (Right).
"rae bot maling 1861 io the nodd" 14 ber car drving the real sndi the apoc By
Figure 5: Left: Comparison of Modulated Ctoss-Attentionn with the original Cross Attention, in the context of individually optimized null- embeedend.. Right: Comparison of Cross-Frame Gated Attention against original crame-independere gated mechanism and the absence of gated attention layer. The rightmost case shows edit- ing with frame-independent groundings can yield subpar results compared to editing without any groundings.
4.3 ABLATION STUDIES
Attentions. We compare the use of the original Cross- Attention with the Modulated Cross-Attention in Fig. 5-Left. The results reveal variations in unconditional context vectors lead to distinct appearances of the subject within a video and the Modulated mechanism promotes the coherency of the sub- ject's appearance. In Fig. 5-Right, we compare our proposed Cross-Frame Gated Attention scenarios with no gated at- tention (no grounding conditions) and with the direct applica- tion of GLIGEN's Gated Attention. The example of 'Iron man' Iron man' is neglected in the absence of groundings guidance and that frame-independent Gated Attention causes discrepancy in the appearance of the shoulder area of 'Iron man' The 'yellow car' example reveals that grounding guidance can result in inferior results compared to the generation with no grounding conditions at all, if not applied in a cross-frame manner. These results under- score the pivotal role of Cross-Frame Gated Attention in both edit-accuracy and cons-oonsittency. Moreover, we provide a quantitative amalysis detailing the impact of each module in Tab.2.
Text-Align
Frame-Con
w/o Modulated CA
0.835
0.967
w/o Grundingg
0.802
0.960
w/o Cross-Frume GA
0.829
0.956
w/o ContreiNet
0.823
0.948
Fill components
0.837
0.970
Table 2: Quantitative assessments on
pipeline components using CL.IP.
illustrates the semantic edit of 'man
ControlNet. We ablate the utilization of ControlNet-Deplle stances of inaccurate reconstruction of the input video. ControlNet's structure guidance is necessi- tated to draw accurate structure of 'rabbit" and 'penguin' inside the bounding boxes, respectively. To further validate the usage of ControlNet, we conducted a quantitative analysis to assess the impact of the proposed inflated ControlNet, as presented in Table 2.
Fig. 2- and Fig. 6-Right depict in-
Optical Flow Smoothing. To assess the impact of optical flow-guided inverted latents smoothing, we ablate the smoothing using three threshold values: O (no smoothing applied), 0.2 and 0.6. As revealed in Fig. 6, our flow- flow-guided smoothing effectively eliminates artifacts within static regions and enhance consistencies. To find optimal threshold value, computed Frame Consistencies using thresholds of 0.2, 0.3, and 0.4. Notably, the 0.2 threshold resulted in superior frame consistency (0.970 for threshold 0.2 > 0.968 for threshold 0.3, 0.964 for threshold 0.4).
8

---

Published as a conference paper at ICLR 2024
02 Thadhed
0.
Figure 6: Left: Editing outcomes using different thresholds for Optical Flow Smoothing. Right: Input video reconstruction results with and without ControlNet guidance.
4.4 APPLICATIONS OF SOBUND-A-VIDEES
Groundings-guided Editing with Inpainting. Employing grounding condition offers signifi- cant advantage, as it facilitates the creation of an inpainting mask. This mask is readily obtained by intersecting the shared outer areas of bounding boxes. By utilizing this acquired mask during edit- ing, which identifies regions in the source video that should remain unaltered, Preserve-Accurac further enhanced. For instance, in Fig. 7-Middle, the common outer spaces of red bounding boxes and blue bounding boxes form a binary mask which is used for inpainting. Yet, in Fig. 7-Bottom, there is no intersection of common outer spaces, and thus, inpainting is not applied in this scenario.
is
Video Style Transfer & Text-fo-video Generation with Pose Control. In the video style transfer task of 7-Middle, target style texts are injected to UNet backbone in both Cross-Frand Frame Gated Atten- tion and Modulated Cross Attention layers. The second row showcases the application of style trans- fer, while the third row shows style transfer combined with attributes editing. Our method adeptly translates input video into the desired style, all while incorporating semantic edits when necessary. Fig. 7-Right illustrates the use of Ground-A- -ideeo for zero-shot text-to-video generation with pose map guidance. The pose map images are sourced from Ma et al. (2023). These spatially-continan pose map conditions are integrated into the diffusion reverse process via inflated ControlNet.
Figure 7: Left: Video editing with and without inpainting. Middle: Video style transfer with attribute change, Right: Text-to-video generation with pose maps. All three results are generated with zero-shot inference.
CONCLUSION
In our work, we addressed the problem of complex multi- attribute editing of video and proposed an answer of utilizing both spatially-continuuus and discrete conditions.
offers precise video editing capabilities without the need for fine-tuning off-the-shelf diffusion models on any video data. We demonstrated the power of our method on various input videos and applications. We provided detailed comparisons to existing baselines along with an extensive user study, demonstrating its superiority in terms of consistency and accuracy. Limitations. Since video groundings play a crucial role in our pipeline, misleading groundings (e.g., incorrect bounding box coordinates) may result in inaccurate editing outcomes. Although the use of ControlNet inherently brings the issue of structural flexibility between the input and output, this can be effectively controlled through the 'ControlNet Scale' hyperparameter (see Fig. 8, 26).
Bround-A-Viden
Figure 8: Video editing results with different ControlNet Scales.

---

Published as a conference paper at ICLR 2024
Ethics & Reproducibility. The use of T2I foundation models brings forth several ethical con- siderations. These models possess the potential for malicious applications, such as the creation of misleading counterfeit content, which could yield adverse societal consequences. Our work heav- ily relies on one such model, making it susceptible to these concerns. Furthermore, the models were trained on dataset of noternet-sourced images (Schuhmann et al., 2022), which may encompass inappropriate content and inherent biases. Consequently, these models might perpetuate such biases (Mishkin al., 2022) and generate inappropriate imagery. To address these potential concerns and foster reproducibility, we will release our source code, model, and data under license that encour- ages ethical and legal usage. Additional information regarding experiments, implementations and the code base can be found in the Appendix (see Sec. D).
Acknowledgements. This research was supported by the National Research Foundation of Ko- rea (NRF) (KS-202300026222),, Field-oriented Technology Development Project for Customs Ad- ministration funded by the Korean government (the Ministry of Science & ICT and the Ko- Customs Service) through the National Research Foundation (NRF) of Korea under Grant NRF2021MM1144 1097910 10 & PRF202IM3IIT A 1097938, Korea Medical Device Development Fund grant funded by the Korea government (the Ministry of Science and ICT, the Ministry of Trade, In- dustry, and Energy, the Ministry of Health Welfare, the Ministry of Food and Drug Safety) (Project Number: 171 137899, KMDF PR 20200901 0015), and Culture, Sports, and Tourism R&D Pro- gram through the Korea Creative Content Agency grant funded by the Ministry of Culture, Sports and Tourism in 2023.
REFERENCES
Shariq Farooq Bhat, Reiner Birkl, Diana Wofk, Peter Wonka, and Matthias Müller. Zoedepth: Zero- shot transfer by combining relative and metric depth. arXiv preprint arXiv:2302.12288 2023.
Mingdeng Cao, Xintao Wang, Zhongang Qi, Ying Shan, Xiaohu Qie, and Yinqiang Zheng. Mas- actrl: Tuning-free mutual self-attention control for consistent image synthesis and editing. arXiv preprint arXiv:2304.0844 2023.
Duygu Ceylan, Chun-Hao Paul Huang, and Niloy J Mitra. Pix2video: Video editing using image diffusion. arXiv preprint arXiv:2303. 12688, 2023.
Weifeng Chen, Jie Wu, Pan Xie, Hefeng Wu, Jiashi Li, Xin Xia, Xuefeng Xiao, and Liang Lin. Control-a-video: Controllable text-to-video generation with diffusion models. arXiv preprint arXiv:2305.13840. 2023.
Ernie Chu, Shuo-Ye Yen Lin, and Jun Jun-Cheng Chen. Video controlnet: Towards temporally consis- tent synthetic-to-red real video translation using conditional image diffusion models. arXiv preprint 1911 2023.
Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Ger- manidis. Structure and content-guided video synthesis with diffusion models. arXiv preprint arXiv:2302.030/1, 2023.
Michal Geyer, Omer Bar-Tal, Shai Bagon, and Tali Dekel. Tokenflow: Consistent diffusion features for consistent video editing. arXiv preprint arXiv:2307. 10373, 2023.
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-pree to-mmtpppttopttttoo image editing with cross attention control. arXiv preprint arXiv:2208.01626 2022.
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free free evaluation metric for image captioning. arXiv preprint arXhv:2104.08718 2021.
Jonathan Ho and Tim Salimans.
Classifier-free diffusion guidance.
arXiv preprint
arXiv:2207. 12.2007.2229... 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
10

---

Published as a conference paper at ICLR 2024
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.023003 2022a.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David Fleet. Video diffusion models. urXiv:2204.03458. 2022b.
Zhihao Hu and Dong Xu. Videocontrolnet: A motion-guide video-to-video translation framework by using diffusion model with controlnet. arXiv preprint arXiv:2307. 14073, 2023.
Zhihao Hu, Guo Lu, and Dong Xu. Fve: A new framework towards deep video compression in feature space. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1502-1511, 2021.
Zhihao Hu, Dong Xu, Guo Lu, Wei Jiang, Wei Wang, and Shan Liu. Fve: An end-to-end framework towards deep video compression in feature space. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(4):4569 4585, 2022.
Hyeonho Jeong, Geon Yeong Park, and Jong Chul Ye. Vme: Video motion customization using temporal attention adaption for text-to-video diffusion models. arXiv preprint 23X002231 2023.
Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-based real image editing with diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 6007-6017, 2023.
Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Text2video-zero: Text-to-image diffusion models are zero-shot video generators. arXiv preprint arXiv:2303. 2023a.
Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Text2video-zero: Text-to-image diffusion models are zero-shot video generators. arXiv preprint arXiv:2303.11433 2023b.
Diederik P Kingma and Max Welling. a3Xiv:1312.61 2013.
Auto-encoding variational bayes. arXiv preprint
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-
image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597, 2023a.
Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang. Chunyuan Li, Yiwu Zhong, Li- juan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded language- image pre- pre-tranning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
10965-10973 2022.
Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang. Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22511-22521, 2023b.
Shaoteng Liu, Yuechen Zhang, Wenbo Li, Zhe Lin, and Jiaya Jia. Video-p2p: Video editing with cross-attention control. arXiv preprint arXiv:2303.04761. 2023.
Yue Ma, Yingqing He, Xiaodong Cun, Xintao Wang, Ying Shan, Xiu Li, and Qifeng Chen. Fol- low your pose: Pose-guided text-to-video generation using pose-free videos. arXiv preprint arXiv:2304.01186, 2023.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1)999-106, 2021.
Mishkin, L Ahmad, M Brundage, Krueger, and G Sastry. Dall- e preview-ring risks and limitations. Noudettu, 28:2022, 2022.
11

---

Published as a conference paper at ICLR 2024
Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Or. Null-text inversion for editing real images using guided diffusion models. In Proceedings of the IEEE/CVE Conference Computer Vision and Pattern Recognition, pp. 6038-6047 2023.
Chong Mou, Xintao Wang, Liangbin Xie, Jian Zhang. Zhongang Qi, Ying Shan, and Xiaohu Qie. 12i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. arXiv preprini SKKv:23002.08453. 2023.
Geon Yeong Park, Jeongsol Kim, Beomsu Kim, Sang Wan Lee, and Jong Chul Ye. tnergy-baseds cross attention for bayesian context update in text-to-image diffusion models. arXiv preprint arXiv:2306. 09869, 2023.
Gaurav Parmar, Krishna Kumar Singh, Richard Zhang. Yijun Li, Jingwan Lu, and Jun- Yan Zhu. Zero-shot image-to-image translation. In ACM SIGGRAPH 2023 Conference Proceedings, pp- 1-11, 2023.
Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alex Porkine-Hornung and Lue Van Gool. The 2017 davis challenge on video object segmentation. arXiv preprint arXiv:1704. 00675, 2017.
Chenyang Qi, Xiaodong Cun, Yong Zhang, Chenyang Lei, Xintao Wang, Ying Shan, and Qifeng Chen. Fatezero: Fusing attentions for zero-shot text-based video editing. arXiv preprint WrXiv:2303.09535. 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning. pp. PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 1(2):3, 2022.
Robin Rombach, Andreas Blattmann, Dominík Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp.
2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Informa- tion Processing Systems, 35:36479-36494 2022.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 15525277--22944 2022.
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-vide generation without tixt-video data. arXiv preprint arXiv:2209. 14792, 2022.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamies. In International conference on machine learn- ing. pp. 2266-2265. PMLR, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502. 2020a.
Yang Song. Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. bcore-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011..34456 2020b.
Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In Computer Vision-ECC -CconnECCV222 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16, pp. 402-419. Springer, 2020.
12

---

Published as a conference paper at ICLR 2024
Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusion features for text-driven image-to to-image translation. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 1921-1930, 2023.
Wen Wang, Kangyang Xie, Zide Liu, Hao Chen, Yue Cao, Xinlong Wang, and Chunhua Shen. Zero- shot video editing using off-the-shelf image diffusion models, arXiv preprint arXiv:2303. 17599, 2023.
Jay Zhangjie Wu, Yixiao Ge, Xintao Wang. Weixian Lei, Yuchao Gu, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for texi-to-video generation. arXiv preprint arXiv:2212. 11665, 2022.
Jinheng Xie, Yuexiang Li, Yawen Huang, Haozhe Liu, Wentian Zhang, Yefeng Zheng, and Mike Zheng Shou. Boxdiff: Text-to-image synthesis with training-free free box-constrained diffu- sion. arXiv preprint arXiv:2307.1081 2023.
Zhen Xing, Qijun Feng, Haoran Chen, Qi Dai, Han Hu, Hang Xu, Zuxuan Wu, and Yu-Gang Jiang. A survey on video diffusion models. arXiv preprint arXiv:2310. 10647, 2023.
Zhengyuan Yang, Jianfeng Wang, Zhe Gan, Linjie Li, Kevin Lin, Chenfei Wu, Nan Duan, Zicheng Liu, Ce Liu, Michael Zeng, et al. Reco: Region-controlled text-to-image generation. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14246 14255, 2023.
Lymin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models.
arXiv preprint arXiv:2302.05543, 2023.
Yabo Zhang, Yuxiang Wei, Dongsheng Jiang, Xiaopeng Zhang, Wangmeng Zuo, and Qi Tian. Con- trolvideo: Training-free controllable text-to-video generation. arXiv preprint arXiv:2305. 13077, 2023.
Rui Zhao, Yuchao Gu, Jay Zhangjie Wu, David Junhao Zhang, Jiawei Liu, Weijia Wu, Jussi Keppo, and Mike Zheng Shou. Motiondirector: Motion customization of text-to-video diffusion models. arXiv preprint 2rXiv:2310.08411 2023.
Daquan Zhou, Weimin Wang, Hanshu Yan, Weiwei Lv. Yizhe Zhu, and Jiashi Feng. Magicvideo: Efficient video generation with latent diffusion models. arXiv preprint arXiv:2211.11018, 2022.
13

---

Published as a conference paper at ICLR 2024
A COMPARISON TO PER-FRAME T21 EDITING METHODS
To further underscore the effectiveness and indispensability of Ground-A-Vided pipeline design, we offer comparisons with two groundbreaking T2I editing methods applied a per-frame manner, Le., per-frame GLIGEN (Li et al., 2023b) editing and per-frame ControlNet (Zhang & Agrawala, 2023) editing. Note that random seeds were consistently fixed when sampling each frame and DDIM inversion utilized give additional compositional guidance for both GLIGEN editing and Con- trolNet editing. As demonstrated in Fig. 9, straightforward extension of the T2I methods in frame-by- by-nebb-frrame fashion leads significant appearance inconsistencies in the video translation task.
Input Video
fox, grazs puppy. dezaert
Bird blue Bind
Silver +
GLIGEN
lipainlig
ControlNeet
(33pth)
White
Sirdl, loke diaeonnfly, emeraid Inke
Silver jeap + yellow jasp, fireworke on the sky
Ours
Per tame CLICEN
Inpaining
Per-iramic ControlNee (De0th)
Figure 9: Comparison with GLIGEN inpainting and ControlNet (depth) img2img on a frame-by-franee framee basis: The direct application of state-of-the-ar image editing methods to the video editing task results in substantial lack of appearance consistency among frames.
B UTILIZING OPTICAL FLOW-GUIDED INVERTED LATENTS SMOOTHING IN TUNE-A-VIDER
To further validate robustness of the proposed optical flow-guided inverted latents smoothing, we demonstrate its application within the Tune-A-Vided (TAV) (Wu et al., 2022) framework. Similar to our approach, TAV commences its denoising process from DDIM-inverte latents, Fig. 10 illustrates comparison between the application of flow smoothing and its absence within the TAV editing framework. Our smoothing technique effectively ensures consistency within static regions (e.g., the avocado the left and the that our proposed optical flow smoothing is applicable to any video editing framework utilizing DDIM inversion.
the right) and eliminates artifacts across frames. It is noteworthy
14

---

Published as a conference paper at ICLR 2024
Input Video
TAV without Flow Smoothing
TAV with Smoothing
Input
TAV without Smoothing
TAV with Smoothing
HP
Rabbit, watermelon, table Kangaroo, avocado, snow
Squirrel, carrot Kangaroo, corn, on the dune, desert Figure 10: Application of the Now-guided inverted latents smoothing in the Tune-A-
T-VieA-Video Video framework.
C RELATED WORK
C.I CONTROL OVER TEXI-TO-IMAGE GENERATION
Internet-scale datasets of image-text pairs (Schuhmann et al., 2022) have driven remarkable ad- vancements in diffusion models within the realm of text-image generation (Rombach et al., 2022; Ramesh et al., 2022; Saharia al., 2022). Consequently, the focus of text-image models natu- rally shifted towards the challenge controllable generation. Controlled image generation can be achieved through two distinet modalities: spatially-continuous conditions and discrete conditions. Notable contributions in the former include T21-Adapter (Mou et al., 2023) and ControlNet (Zhang & Agrawala, 2023), which augment pretrained T2I models with auxiliary networks. These networks are specifically trained to produce images conditioned on spatially-continouus visual cues such as depth maps and edge maps. Conversely, GLIGEN (Li al., 2023b) and Reco (Yang et al., 2023) fine-tune the T2I model to accommodate discrete layout information, comprising bounding box co- ordinates and textual captions.
Another approach to exert control over the generation process is to heavily reference the input image, with the goal of preserving structural and background attributes. Prompt-to-prome (Hertz al., 2022) and Pix2pix-zen zero (Parmar et al., 2023) achieve cructure-preservvee cross-attention maps of the input image, while Plug-and-play (Tumanyan et al., 2023) and MasaCtrl (Cao et al., 2023) manipulates spatial features of the self-attention mechanism, which are extracted from the input image diffusion trajectory. Alternate strategies (Mokady et al., 2023; Kawar et al., 2023) attain enhanced editing capabilities through optimization techniques. Yet, straightforward application of previously mentioned methods in frame-by-frame manner for video editing outputs pronounced flickering and temporal inconsistencies.
image generation by using
C.2 DIFFUSION MODELS FOR VIDEO
When juxtaposed with text-image generation, generating videos in a text-only condition poses a sig- nificantly elevated challenge due the complexity of constraining temporal consistency along with the scarcity of extensive text-video datasets, which are both resource-unfriendly. Video Diffusion Models (VDM) (Ho al., 2022b) designs a space-time factorized 3D UNet architecture trained on both image and video modalities. Imagen Video (Ho et al., 2022a) trains cascaded VDMs with v-prediction parameterization. Make-A-Vides (Singer et al., 2022) and MagicVides (Zhou et al., 2022) adopt similar approach in that they train T2V models transferring from pretrained T21 mod- els. More recently, Tex12 Video-Zro (Khachatryan al., 2023b) achieves zero-shot text-to-viden generation by enhancing the latent features of the generated frames with motion dynamics and re- structuring the Spatial Self- Attention along the frame-axis.
Simultaneously, significant research efforts have been dedicated to video editing tasks (Xing et al., 2023). Pioneering work in this field, exemplified by Tune-A- Video (Wu et al., 2022), has employed the approach of fine-tuning query projection matrices in attention layers to effectively retain infor- mation from the source video. Video-P2P (Liu et al., 2023) adopts an approach of fine-tuning a Text-to-Set model on the source video well, additionally introducing a decoupled-guidand atten-
15

---

Published as a conference paper at ICLR 2024
tion control for the inference stage. More recent one-shot video editing frameworks include Motion Director (Zhao et al., 2023) and VMC (Jeong et al., 2023), where they both aim to customize mo- tion patterns presented in the souree video. To eliminate the necessity of optimizing model weights entirely, various zero-shot editing techniques have been introduced. Vid2vid-zero (Wang et al,, 2023) focus a cross-attention maps guidance while Pix2Video (Ceylan al., 2023) employs self-attention maps injection mechanism, where the attention maps are obtained in the input video inversion stage in both methods. It's worth noting these two distinct attention control methods are orthogonal and thus can be utilized at the same time. Furthermore, FateZero (Qi et al., 2023) intro- duces training-free free video editing pipeline that utilizes cross-attention maps from the input video inversion trajectory. These maps are used to calculate blending masks, aiming to enhance consis- tency in the background regions of the generated videos. In contrast to approaches that involve no training on video or training on single video, Gen-1 (Esser et al., 2023) and Control-A-Vin Video (Chen et al., 2023) adopt a training- based approach for video translation, incorporating additional structural guidance. Notably, Control-A- Aideo achieves efficient convergence through their innova- tive first-frame conditioning method, optimizing resource utilization. Recent methodologies (Chu et al., 2023; Hu & Xu, 2023) have made significant advancements in frame consistency by integrat- ing an optical flow warping mechanism into their generation process. A promising concurrent work, TokenFloww (Geyer et al., 2023), accomplishes time-consistent video editing that strongly preserves spatial layout by enforcing consistency on the internal diffusion features across frames during the denoising process.
D EXPERIMENTAL DETAILS AND IMPLEMENTATIONS
In this section, provide experimental details of our method and the compared basclines. Ground- A-Video(Depth) utilizes DDIM inversion then performs null-text optimization with the default hy- perparameters in Mokady et al. (2023). In the flow-driven inverted latents smoothing stage, the magnitude threshold Mirres is set to 0.2. At inference, DDIM scheduler (Song et al., 2020a) with 50 steps and classifier-free guidance (Ho & Salimans, 2022) of 12.5 scale is used. The executable code base will be available at our project's repository.
For the comparison with Tune-A- Mime-A--ideo-CCnttoo(DDpppp (Wu et al., 2022), we utilize the authors' provided implementation! and also adjust ControlNet (Zhang & Agrawala, 2023) following their inflation guidelines. For fine-tuning, we train the model on each example video for 500 iterations using their default parameters. During inference, we utilize DDIM inversion with 50 steps and em- ploy diffusion reverse with the DDIM scheduler, also using 50 steps. The model architecture and training script for Nettrollet-atttcche Tune-A-Videe code repository. For the comparison with Control-A- Viitool-A (Chen et al., 2023), we utilize their official implementation" with the following hyperparameter settings: 50 steps for inference, a video scale of 1.5, and a noise threshold of 0.1. In our comparison with ControlVideoo(eppt) (Zhang et al., 2023), we leverage their open-source code repository² and demo website We per- form inference using 50 steps, and we apply inteleaved-framm smoothing with steps set at [19, 20). Although Gen-2, the successor to Gen-1 (Esser et al., 2023), has been recently announced, it is important to note that Gen-2 supports sole text-to-video generation functionality, which differs from the focus of our paper, which is centered on comparing video editing performance. Thus, we have utilized Gen-1 for our comparisons and the results from Gen-l were generated by executing its web-based product.
will also be made available in our project's s
hhttps://replicate.ccoo
16

---

Published as a conference paper at ICLR 2024
X bird flyiig above lake the forest.
crew Plynng sbose desert the foreet pice"
Video
Codtrol--VVdee
Ground-d-Viiuu
-DDD
Controlliet
Figure 11: Left: Input video and Ground-A- A-Vidoo's s editing result, driven by the target prompt "A crow flying above desert in the forest on fire." " Right: Cases of Neglected Edit and Edit Wrong Element. The editings are driven by the same target prompt, using Control-A- A-Video and Tune-A-Viiloo-CCom Video--onoonn,,,,, respectively.
SEMANTIC MISALIGNMENT & ADDITIONAL COMPARISONS
In our study, define the concept of semantic misalignment within the realm of video editing. To facilitate better understanding, we offer illustrative examples of various forms of semantic misalign- ment through baseline comparisons.
The most common types of semantic misalignment encountered in video editing are 'Neglected Edit' and 'Edit on Wrong Element'. Neglected Edit signified by the omission of one of the intended attribute edits, while Edit on Wrong Element refers to cases where semantic edit is ap- plied to an element that was not the intended target of the edit. These misalignments are typically observed in scenarios where wide range of different edits need to be performed. For instance, as shown Fig. II, editing scenario can represented as (Tbird
Tlake - Troeatt Tea ) In the first of Fig. 11-righr, the edit of Ton fire is omitted in the results. The results in the second row provide an example of Edit on Wrong Element, specifically visualizing the occurrence of "Thidd Ton fire- The next type is 'Mixed edit', which denotes the situations where two more sep- arate edits become intertwined and mutually affect each other, straying from the original intention. This issue is commonly observed in complex editing scenarios that involve object color changes. For instance, in the second row of Fig. 12-left, the edits TTea and "Tg-Taunse" become intertwined as 'red Tsunset and are applied globally to the entire image within video. Finally, we address PPesservation Failure', which refers to instances where the generated results fail to preserve the regions that are not the target of editing. This issue is not exclusive complex editing scenarios. For example, in the second row of Fig. 12-right, the edit "Teenuinn Tkungfu panda' the foreground object has been successfully performed, but has failed to preserve the background of the source video.
"A shivvng
sky."
pazgsai
Input Video
Vnput Video
"A radiar anikig the rost under the dky, in minset"
Basg Expond raaingg the ite."
Gan-1
ControlVieo
Ground-A A Video (Ours)
A-Vdee A-Vidoo (Durs)
Figure 12: Left: example of Mixed Edit.
Right: An example Preservation Failure.
17

---

Published as a conference paper at ICLR 2024
F SPATIAL CONDITIONS
F.1 DISCRETE AND CONTINCOUS CONDITIONS
Ground-A- A-iddoo framework categorizes spatial conditions into types: Spatially-disered con- ditions and opatially-continuons conditions. Spatially-disere conditions refer to modalities that do rigidly dictate the precise spatial arrangement of objects or elements within the ultimate out- put image video. In our work, we utilize blend of bounding box coordinates and textual descrip- tions that pertain to the grounded entity. This spatially-discret condition doesn't 't enforce particular layout structural constraints within the bounding box but rather determines the positions of the entities enclosed by the bounding box within the image frame. contrast, spatially continuuus conditions provide information that explicitly influences the placement, structure, or spatial distri- bution of objects throughout every part of the frame, e.g., edge maps and depth maps. To enhance understanding of these terms, visualize examples of each modalities that used in editing pipeline in Fig. 13.
DoUndinn
owsompszne
1325 ES5. 15.006 300 0K3
1:25 0.5 Thg
Depth map
Figure 13: Left: Spatially-comtimuuuu conditions. Right: Spatially-disrrete conditions.
F.2 STATIC AND DYNAMIC GROUNDINGS
Video groundings play central role in video editing pipeline, and would like classify them into two separate categories based on the video's ' main subject characteristics: Static groundings and Dynamic groundings. Static groundings refer groundings that describe subject that exhibit minimal movement or remain in a fixed region of the layout. In contrast, dynamic groundings involve groundings describing subject that transition from one region another within video. In Fig. 14, the first row presents an input video featuring a static 'penguin' subject. The subsequent two rows below the first row illustrate our method's successful editing using static groundings associated with the stationary subject. The first row of Fig. 15 showcases video with a dynamic "car' subject, and the two rows following it demonstrate our method's effectiveness in editing the dynamic subject with the assistance of dynamic groundings.
18

---

Published as a conference paper at ICLR 2024
penguin running
"A crochet penquin running the ice."
*A penguin is rumning on the beach.
crochet penguin is running on beach.
Figure 14: Input vidco static subject 'penguin' and static groundings-driven video editing results.
"A silver jeep is driving down curvy road in the countryside."
jeep driving curry countryside, sundet."
'A jeep is driving down curvy snowy winter mountaine
Figure 15: Input video of dynamic subject 'car' and dynamic groundings-driven video editing results.
19

---

Published as a conference paper at ICLR 2024
G FULL-LENGTH ADDITIONAL RESULTS
G.1 VIDEO STYLE TRANSFER
Ground-A Video is primarily designed for multi-attribute video editing but can also perform video style transfer tasks with temporal consistency. Additional video style transfer applications on various videos are presented in Fig. 16.
bird flying above lake the forest."
"A bind flying above lake the forest, Ukiya-e art'
"A robbit jumping the grass."
"A robbit is jumping the grass, Monnt style."
*A surfing
surfing Cezonne painting." *
skiing"
"A man skiing, Van Gogh starry night style." .
Figure 16: Full-length Video Style Transfer results.
20

---

Published as a conference paper at ICLR 2024
G.2 MULTI-ATTRIBUTE EDITING
We present additional qualitative results of Ground-A-Vides on multi-attribute editing. In each of the Fig. 17. 18, 19, and 20, the top rows visualize the optical flow maps estimated from input video. The second row shows the input frames, each with annotated bounding boxes around the entities. Our optical flow estimation network, RAFT (Teed & Deng, 2020), requires consecutive images for accurate estimation. Consequently, the first frame. lacking previous frame for com- parison, does not yield any estimation results. For Fig. 21, 22, 23, 24, and 25, the top two rows visualize both the estimated optical flow maps and the corresponding magnitude maps. An optical flow map comprises two-channel images, with each channel representing vertical and horizontal mo- tion residuals. The magnitude map, which combines motion information from both directions into a single-channel image, is computed using normmax (l|mapopell). Il) The normalization is performed along the frame dimension. In the examples, the visualization of magnitude maps demonstrates successful comprehension of vertical and horizontal movements. On all Fig. 17, 18, 19, 21, 22, 23, 24, 25 under the input video, we present the editing results in ascending order based the 'number of attributes to be changed' (from small changes to large number of changes). Furthermore, Fig. 26 presents the editing outcomes using various settings of the ControlNet Scale hyperparameter. It's noteworthy that a reduction in the scaling parameter leads to greater independence in the shapes of the tiger (particularly its ears) and the football from the input video and depth maps. To better assess temporal consistency, full-length results are provided for all figures.
Optical Maps
"A bird flying above lake the forest."
blue bird flying above iake in the forest."
dragonfly flying forest""
A flying above desert the forest fire.
Figure 17: Various editing results the video of "A bird flying above lake the forest."
http:////.rraaaooo.oooos////sees image.html html
flow_to_
21

---

Published as a conference paper at ICLR 2024
Optical Flow Maps
"A black alwan with a beok siiimiin in river neor a woll busihes!
pink red beak swimming river woll and
*A blue awan with red benk swimming in river near wall and snawy bushes.
"A biue swan with a red beak swimming locoon near a wall and snowy bushhes"
Figure 18; Various editing results on the video of "A black swan with a red beak swimming in a river near wall and bushes."
Optical Maps
"A white fax sitting the
"A ned fox sitting in the pink grass."
"A sitting the green gross."
"A puppy sicting the desert, sunrise."
Figure 19: Various editing results the video of "A white fox sitting in the grass."
22

---

Published as a conference paper at ICLR 2024
Optical Flow Maps
rabbit eating strowberries"
white rubbit is eating appies"
aquirnel is esting cherries.
Figure 20: Various editing results on the video of "A rabbit eating strawberries."
Optical Flow Maps
Meonitude Maps
"A roaning."
"A riger roaning."
A white puppy is rearing on the pond."
reoning snewy mountains*
"A puppy rearing the beach, under the sky..
Figure 21: Various editing results the video of "A cat is maring."
23

---

Published as a conference paper at ICLR 2024
Magnilude Maps
squirrel eating carrot."
"A rabbit eating corn on the snow.
eating a sausage
Fennac fox eating sausage the deesrrt."
"A fennec fax eating cucumber the dune, in desert."
Figure 22: Various editing results on the vidco of "A squirrel eating carrot."
24

---

Published as a conference paper at ICLR 2024
Optical Flow Mape
Magnitude Maps
"Brown bear walking the rock, egainst wall."
'Pando walking on the rock, against wall."
"Tiger wolking the against
Pink bear walking the rock, against
Blinkk bear walking on snowy alpine, next lakis, agginst wall.
PPink bear walking on snowy clpine, next to lake, against wall
walking snowy alpine, Inke, against
"Pink walking snowy clpine,
against blue woll.
Figure 23: Various editing results on the video of "Brown bear walking on the rock, against a wall."
25

---

Published as a conference paper at ICLR 2024
Ostical Flow Mapa
Magnitude Maps
"A man is walking dog on the read."
"A man is walking lego deg on the road."
'Homer Simpson walking a dog road."
*A main is walking dag on the pond.
"A main is walking dog on the ice."
"Tron man walking sheep on the road."
"Iron man walking a sheep the lake."
farmer walking a goot on snow
Figure 24: Various editing results on the vidco of "A man is walking a dog on the road."
26

---

Published as a conference paper at ICLR 2024
Optical Flow Maps
Magnitude
rabbit eating
"A rut euting walermelon the table."
"A robbit is eatiry the pink moen on the table."
eating
"A dog is eating baskstball on the moss."
'A kangaroo is eating on avocado on the snow, under the sky."
dog is eating basketball on the sand, at sunrise."
Squirrel eating crange on grazs,
Figure 25: Various editing results on the video of rabbit is eating a watermelon table." "
27

---

Published as a conference paper at ICLR 2024
rabbit eating a watermelon on table."
tiger is eating foothell on the toble."
Depth Maps Input)
ControlNet Scalls. 1.0
ControlNet Scale:
Figure 26: Full frame editing result with varying ControlNet Scales.
28


Reviews:
[{"review_id": "n7zhcGcGsg", "reviewer": "Reviewer_F2zK", "summary": "The paper focuses on an interesting problem, namely multi-attribute editing in the video domain. The proposed method relies on recent techniques, such as GLIP, and ControlNet, and integrates the grounding information to perform a sequence of editing operations. The aim is also to propose a training-free pipeline by using pre-trained models in a zero-shot setting. For video-level editing, a stable diffusion backbone is extended for video data with DDIM inversion, and optical flow and depth maps are integrated as conditional to improve video editing quality. The evaluation is conducted on a set of videos and the evaluation is performed qualitatively with a comparison to SOTA and quantitatively by a user study with 28 participants.", "questions": "What is the video set used for comparison and evaluation in Table 1? \n\nHow similar are the paper's evaluation metrics with CAV (Chen et.al 2023)?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The topic of paper, multi attribute editing in videos, is a challenging and interesting problem. \n\nThe paper seems mostly the integration and extension on available recent T2I pipelines for video domain. But, the selected models are recent and they fit well within the proposed pipeline. Moreover, the pipeline avoids additional training stages that is good for zero-shot pipeline.", "weaknesses": "The paper is a well-designed integration of mostly existing techniques for video editing pipeline. The authors explain the steps in subsections with details. However, I found the overall text flow confusing as the stable-diffusion model and the layers of it are explained in multiple  sections, e.g. 3.2 and 3.4, rather than a whole. Additionally, the integrations of the controlNet and optical flow into the whole pipeline are not clear. I think a revised figure with math representations consistent with the text may help to explain the model better. \n \nA more detailed qualitative evaluation of the model could be presented in the experimental section. For instance, to assess the impact of a particular component on the pipeline, the ablation section (Section 4.3) only includes the edited video outputs generated using with and without this component. However, this evaluation could also be conducted quantitatively (as in Table 1) to see the impact of some evaluated components on the pipeline.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rZxjV8qT0T", "reviewer": "Reviewer_vcyz", "summary": "The paper presents the ﬁrst grounding-driven video editing framework, which is intended to solve the problem of neglected editing, wrong element editing and temporal-inconsistency in context of the complexities of multi-attribute video editing scenarios. Moreover, the proposed method is training-free which overcomes the obstacle of excessive computational cost on video tasks. Spatial-Temporal Attention, Cross-Frame Gated Attention and Modulated Cross Attention are introduced to further enhance consistency, depth map is used as an additional condition to better preserve structure and 3D information and the binary mask calculated through the optical ﬂow map can help maintain the consistency of the background area. Eﬀectiveness has been proven by suﬃcient experiments and convincing qualitative results.", "questions": "See weaknesses above.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "S1. This paper presents the ﬁrst training-free grounding-driven video editing framework, which is relatively innovative.\n\nS2. The proposed method and experimental results are consistent with their motivation and eﬀectively solve the problem of multi-attribute video editing in complex scenes.\n\nS3. The method is clearly stated, and the details are comprehensive.", "weaknesses": "W1. Since the introduced depth map and optical ﬂow map both reﬂect pixel-level structural information, will this cause the structure before and after editing to be too consistent, so that if the foreground is replaced by an object with inconsistent structure, the editing result will be poor and lack of ﬂexibility? e.g. replace the “rabbit” in the phrase \"A rabbit is eating a watermelon on the table\" with an animal without long ears.\n\nW2. For similar reasons, this may also limit the editing method to the task of adding or deleting objects.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "coB6uPkEv4", "reviewer": "Reviewer_TXKT", "summary": "Ground-A-Video proposes a training-free framework for grounded multi-attribute editing. The grounding ability is achieved by introducing the pretrained GLIGEN gated self-attention module into existing video editing pipelines. The paper also proposes several techniques including \"cross-frame attention\" and \"flow-guided latent smoothing\" to improve temporal consistency.", "questions": "Please kindly address the questions in the weakness section.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Grounded video editing is a novel task. Bounding box grounding allows users to accurately select the regions to edit. It is a useful feature that allows better location controllability and content disentanglement over the existing popular sentence-level video editing works.\n\n2. The overall framework is training-free. The proposed framework is built upon several pretrained models like Stable Diffusion, Control Net, GLIGEN, and training-free inversion techniques like Null-text Inversion, so the framework itself requires no additional training on video data.\n\n3. The framework effectively associates editing prompts with the grounded areas. The results in the paper show better text alignment compared to previous sentence-level video editing approaches like Tune-A-Video, and ControlVideo.", "weaknesses": "1. Although under the same framework, the technical contributions are quite disconnected from one another and some of them might not be closely related to the grounded video editing task. Two main technical contributions (Modulated Cross-Attention and Flow-guided Latent Smoothing) in the paper lie in finding temporally smooth and faithful latent noise during inversion, which serves as a preparatory step and seems to be less relevant to the grounded editing task. On the other hand, the grounded editing capability mainly comes from the pretrained GLIGEN gated self-attention module, which brings limited addition to the previous image grounding task. \n\n2.  Although the proposed framework is training-free, it is still noteworthy that the per-frame null-text inversion requires gradient-based optimization on the null-text embeddings and could be time-consuming for longer videos. Moreover, the new Modulated Cross-Attention mechanism requires jointly optimizing all frames, which requires large memory.\n\n3. I am not very clear about the flow-guided smoothing after reading the description: does it only work on static areas? If not, why there are not any warping operations mentioned? I also find the terms \"spatially continuous\" and \"spatially discrete\" a bit confusing and hard to understand what continuous and discrete refers to.\n\n*For reproducibility, the code has not been released in the provided link at the point of this review.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RZizOHM0sn", "reviewer": "Reviewer_NYGw", "summary": "this paper aims to address the challenge of multi-attribute editing in video editing. By incorporating the outputs of grounding models like GLIP and employing a designed attention mechanism, the proposed approach enables precise and temporally consistent video attribute editing. The authors conducted comparative experiments with recent methods, demonstrating superior performance.", "questions": "1. The authors provided a visual comparison of the flow smooth effect in Figure 6, but it only includes a single example. Are there additional examples and comparisons with baseline and other methods (excluding flow smooth) demonstrating their effects under the same prompt?\n2. during the smoothing process, thresholds are introduced. Are there objective metric comparisons for different thresholds and experiments to evaluate the robustness of the thresholds?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This paper addresses a highly significant problem of achieving consistent fine-grained attribute editing in videos. The introduction of grounding conditions proves to be a direct and effective approach.\n2. In addition to the design of grounding conditions, the authors propose mechanisms such as noise smoothing and cross-attention.", "weaknesses": "This paper lacks substantial technical innovation as its main contribution lies in expanding the experiments on the input conditions, whether it is the depth or grounding results. The approach used for injecting conditions is based on ControlNet or cross-attention, which is a common practice in stable diffusion and related applications of ControlNet.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)