VidEdit: Zero-Shot and Spatially Aware Text-Driven Video Editing
Authors: Paul Couairon
Clément Rambour
Jean-Emmanuel HAUGEARD
Nicolas THOME
Venue: Accepted by TMLR ()

Abstract: Recently, diffusion-based generative models have achieved remarkable success for image generation and edition. However, existing diffusion-based video editing approaches lack the ability to offer precise control over generated content that maintains temporal consistency in long-term videos. On the other hand, atlas-based methods provide strong temporal consistency but are costly to edit a video and lack spatial control. In this work, we introduce VidEdit, a novel method for zero-shot text-based video editing that guarantees robust temporal and spatial consistency. In particular, we combine an atlas-based video representation with a pre-trained text-to-image diffusion model to provide a training-free and efficient video editing method, which by design fulfills temporal smoothness. To grant precise user control over generated content, we utilize conditional information extracted from off-the-shelf panoptic segmenters and edge detectors which guides the diffusion sampling process. This method ensures a fine spatial control on targeted regions while strictly preserving the structure of the original video. Our quantitative and qualitative experiments show that VidEdit outperforms state-of-the-art methods on DAVIS dataset, regarding semantic faithfulness, image preservation, and temporal consistency metrics. With this framework, processing a single video only takes approximately one minute, and it can generate multiple compatible edits based on a unique text prompt.

Published in Transactions on Machine Learning Research (03/2024)
VidEdit: Zero-Shot and Spatially Aware Text-Driven Video Editing
Paul Couairon
poul.couairon upmo.fr
Thales SIX GTS France, ThereSIS Lab, Palaiseau, France Sorbonne Université, CNRS, ISIR, 7505005 Paris, France
Clément Rambour
Cram, CEDRIC, Paris, 75003, France.
Haugeard Thates SIX GTS France, ThereSIS Lab, Palaiseaa, France
Nicolas Thome
nicolas. thome Disir upmn..ff
Sorbonne Université, CNRS, ISIR, F-75005 Paris, France
Reviewed on OpenReview: https: // openreview. net/ ceeeffrrut forum? id= +02A009I5a
Source Prompt: 'A silver jeep driving down a curey road"
Target Edit: "trees"
"grass"
"mountains" + 'a landscope in autumn"
Target Edit: "road" + night sky"
Target Edit: "car" fa retrowave neon jeep
Figure 1: VidEdit allows perform rich and diverse video edits precise semantic region of interest while perfectly preserving untargeted areas. The mothod is lightweight and maintains a strong temporal consistency on long-term videos.
Abstract
Recently, diffusion-based generative models have achieved remarkable success for image generation and edition. However, existing diffusion- based video editing approaches lack the ability to offer precise control over generated content that maintains temporal consistency in long-term videos. On the other hand, based methods provide strong temporal consistency but costly to edit video and lack spatial control. In this work,

---

Published in Transactions on Machine Learning Research (03/2024)
introduce VIDEDIT, novel method for zero-shot text-based video editing that guarantees robust temporal and spatial consistency. In particular, we combine an atlas-based video representation with a pre-trained text-to-image diffusion model to provide a training free and efficient video editing method, which by design fulfills temporal smoothness. To grant precise user control over generaled content, we utilize conditional information extracted from off-the- shelf panoptic segmenters and edge detectors which guides the diffusion sampling process. This method ensures a fine spatial control on targeted regions while strictly preserving the structure of the original video. Our quantitative and qualitative experiments show that VIDEDIT outperforms state-of-the-art methods on DAVIS dataset, regarding semantic faithfulness, image preservation, and temporal consistency metrics. With this framework, processing single video only takes approximately one minute, and it can generate multiple compatible edits based on a unique text prompt.
1 Introduction
Diffusion-based models (Ho et al., 2020; Song et al., 2020; Rombach ot al., 2022; Ramesh et al., 2022) have recently (aken over image generation. In contrast to generative adversarial networks (Goodfellow et al., 2020; Karras et al., 2018; Yu et al., 2021) which are notoriously difficult to train, diffusion models offer a more reliable training process and consistently generate highly convincing samples. Besides, they can also be used for editing purposes by integrating conditional modalities such as text (Rombach et al., 2022), edge maps or beyond (Zhang & Agrawala, 2023; Mou et al., 2023). Such capacities have given rise to numerous methods that assist artists in their content creation endeavor (Tumanyan et al., 2022; Kawar et al., 2022).
Yet, unlike image editing, text- video editing represents a whole new challenge. Indeed, naive frame-wir application of text-driven diffusson models leads to flickering video results that look poor to the human eye as they lack motion information and 3D shape understanding. To overcome this challenge, numerous methods introduce diverse spatiotemporal attention mechanisms that aim to preserve objects' appearance across neighboring frames while respecting the motion dynamics (Wu el al., 2022; Qi el al., 2023; Ceylan et al., 2023; Wang et al., 2023; Liu et al., 2023). However, they not only demand significant memory resources but also concentrate on a limited number of frames as the proposed spatiotemporal attention mechanisms are not reliable enough over time model long- term dependencies. On the other hand, current atsas-based video editing methods (Bar-Tal et al., 2022; Loescheke et al., 2022) require costly optimization procedures for each text query and do not enable precise spatial editing control nor produce diverse samples.
This paper introduces VIDEDIT, simple and effective zero-shot text- based video editing method that shows high temporal consistency and offers object-level control over the appearance of the video content. The rationale of the approach is shown in Fig 1. Given an input video and a target edit, e.g. "road" + "a night sky'; VidEdit precisely delineates the region of interest in the atlas space as well as the internal edges that characterize its semantic structure. The text prompt and the edge map are then passed to a pre-trained conditional diffusion model that generates an edit that matches these controls. During the generation phase, the edit seamlessly merges with the original atlas through blended diffusion process (Avrahami et al., 2022), which leaves the remainder of the video content unchanged. We hypothesize and confirm that diffusion models can effectively handle distortions in atlases, allowing us to modify these representations with little effort. Furthermore, by directing the generation process with conditional inputs, we can create compelling video edits with a fine-grained spatial control that maintain temporal consistency. To achieve this goal, the approach includes two main contributions.
Firstly, we combine the strengths of atlas-based approaches and text-to-image diffusion models. The idea is to decompose videos into a set of layered neural atlases (Kasten et al., 2021) which are designed to provide an interpretable and semantic unified representation of the content. We then apply pro-trained text-driven image diffusion model to perform zero-shot allas editing, the temporal coherence being preserved when edits are mapped back to the original frames. Consequently, the approach is training free and efficient as it can odit a full vidco about one minute. In addition, we take special care to preserve the structure and geometry in the atlas space as it not only encodes objects' temporal appearance but also their movements and spatial placement in the image space. Therefore, to constrain the edits to match as accurately as possible the semantic
2

---

Published in Transactions on Machine Learning Research (03/2024)
layout of an atlas representation, we leverage an off- panoptic segmenter (Cheng et al., 2022) as well as an edge detection model (HED) (Xie & Tu, 2015). The segmenter extracts the regions of interest whereas the HED specifies the inner and outer edges that guide the editing process for an optimal video content pneraatoon/resevattion trade-off. Hence, we adapt the utilization of a spatially grounded editing method to a conditional diffusion process that operates on atlas representations. This is achieved by extracting a crop around the area of interest and intentionally utilizing a non-invertible noising process.
We conduct extensive experiments on DAVIS dataset, providing quantitative and qualitative comparisons with respect to video baselines based or not on atlas representations, and frame- based editing methods. We show that VIDEDIT outperforms these basclines in terms of semantic matching to the target text query, original content preservation, and temporal consistency. Especially, we highlight the benefits of our approach for foreground or background object editions. We also illustrate the importance of the proposed contributions for optimal performance. Finally, we show the efficiency of VIDEDIT and its capacity to generate diverse samples compatible with given text prompt.
2 Related Work
Text-driven Image Editing. In the past few years, Text-to-Image (T2I) generation has become an increasingly hot topic. Recently, these generative models have benefited from the swelling popularity of diffusion models (Ho et al., 2020; Song et al., 2020; Ramesh et al., 2022) as well as the accurate image-text. alignment provided by CLIP (Radford et al. 2021). Latent Diffusion Models (LDMs) (Rombach et al., 2022) propose to enhance the training efficiency, memory, and runtime of such models, by taking the diffusion process into the latent. space of an autoencoder. As a result, they have taken over text-driven image generation and editing. For example, SDEdit (Meng et al., 2021) proposes to corrupt an image by adding Gaussian noise, and a text-conditioned diffusson network denoises it to generate new content. Other works aim to perform local image editing by using an edit mask (Avrahami et al., 2022; Couairon al., 2022) and combining the features of each step in the generation process for image blending. Still focusing on image-to-image translation, (Hertz et al., 2022) or (Tumaayan et al., 2022) extract attention features to constrain the editions to regions of interest. Kawar el al. (2022) or Mokady el al. (2022) refine image editing via an optimization procedure,
Text-driven Video Editing. While significant advances have been made in T2I generation, modeling strong lemporal consistency for video generation and editing is still a labor progress. Numerous works aim to generate original video content directly from an input text query with novel spatiotemporal attention mechanisms (Ho et al., 2022b;a; Villegas et al., 2022; Singer et al., 2022; Zhou et al., 2022). However, these methods still exhibit flickering artifacts and inconsistencies that alter the quality of the visual outputs. When it comes to video editing, existing approaches can be categorized into two main groups. On one side are methods that seek to adapt the structure of a frozen T21 diffusion model to perform video editing in a zero-shot manner. Tune-A-Video (Wu et al., 2022) overfits a video on a given text query and generate new content from similar prompts. Other approaches (Liu et al., 2023; Qi et al., 2023; Shin et al., 2023; Ceylan et al., 2023; Wang et al., 2023) propose spatiotemporal attention mechanisms transfer pre-trained T21 model knowledge to text-to-video. However, these methods stiil struggle to ensure reliable long-term coherence. On the other side, Neural Layered Atlases (Kasten et al. (2021)) provides method for decomposing video content into a set of 2D atlases that can be edited and mapped back to the frame space, ensuring excellent temporal consistency. Based on such atlases, 2eve (Bar-Tal et al. (2022)) facilitates coherent text-to-video editing by optimizing an edit layer over the atlas. However, its costly optimization for each prompt limits its ability to generate edits on the fly. VIDÉDIT follows the trail set by Text2Live in atlas- based video editing by harnessing the adaptability and efficiency of a pro-trained T2I diffusion model to perform atlas editing. We thereby eliminate any optinization procedure and enable precise user-control and quick inference.
3 VidEdit Framework
The high visual quality offered by T2I diffusion models as well as their effectiveness to generate samples that are aligned with provided conditional information motivate us to utilize these models to perform our video
3

---

Published in Transactions on Machine Learning Research (03/2024)
editing task in the 2D atlas space. To this end, introduce VIDEDIT, novel lightweight, and consistent video editing framework that provides object-scale control over the video content. The main steps of VIDEDrr are illustrated in Fig. 2. First, we propose to benefit from Neural Layered Atlas (NLA) (Kasten et al., 2021) to build global representations of the vidco content ensuring strong spatial and temporal coherence. Sccond, the underlying global scene encoded the allas representation processed through a zero-shot image editing diffusion procedure. Text-based editing inherently faces the difficulty of accurately identifying the region edit from the input text and may as well deteriorate neighboring regions introduce rough deformations in the object aspect (Wu et al., 2022; Ceylan et al., 2023; Liu al., 2023). We avoid these pitfalls by carefully extracting rich semantic information using HED maps and off-the-shelf segmentation models (Cheng et al., 2022) to guide the diffusion generative process. We adapt their design and utilization for atlas images.
Background Atlas
Bilinear Sampling
colorful bus"
(b) Foreground Atlas
Input Video
Layered Neural Atlases
3 Atlas Editing
4 Frame Edit Layer
5- Edited Video
Figure 2: Our VidEdit pipeline: An input video (1) is fed into NLA models (Kasten al., 2021) which learn to decompose it into 2D atlases (2). Depending the object want to edit, select atlas representation onto which apply editing diffusion pipeline (3). The edited atlas is then mapped back to frames via bilinear sampling from the associated pre-trained network MI (4). Finally, the frame edit layers are composited over the original frames to obtain our desired edited video (5).
3.1 Zero-shot Atlas-bast based video editing
Neural Layered Atlases. Neural Layered Atlases (NLA) (Kasten et al., 2021) provide a unified 2D representation of the appearance of an object or the background through time, by decomposing a video into set of 2D atlases. Formally, each pixel location p (z, y, t) E RR is fed into three mapping networks. While My and MI, map p to 2D (u, cooorniinatte in the foreground and background atlas regions respectively, M. predicts foreground opacity value:
M.(p) (uf, of), Mly(p) (uz, of), Ma a
(1)
Each of the predicted (u, v) -)-oodiiitttes are then fed into an atlas network A, which yields an RGB color at that location. Color can then be reconstructed by alpha-blending the predicted foreground of and background of colors at each position according to the corresponding opacity value a":
d = (1-a")c t.dy.
(2)
We train NLA in self-supervised manner in Kasten et al. (2021). The obtained background and foreground atlases are large 2D pixel representations disentangling the layers from the video. By utilizing these mapping and opacity networks, one can edit the RGBA pixel values and project them back onto the original video frames.
Zero-shot atlas editing. The 2D atlases obtained by disentangling the vidco are a well-posed framework edit objects while ensuring a strong temporal consistency. We propose here to perform zero-shot text- exxt-based editing of atlas images. This is in sharp contrast with Bar-Tal et al. (2022), which requires training specific
4

---

Published in Transactions on Machine Learning Research (03/2024)
generative model for each target text query. We pre-trained conditioned latent diffusion model, although our approach is agnostic to the image editing tool. As illustrated with Fig. 2, the automatic video editing task is transformed into a much straightforward, training free, and adaptable image editing task, resulting in competitive performance.
3.2 Semantic Atlas Editing with VidEdit
2D atlas representations pave the way to use powerful off- af-tthesheeff segmentation models (Xu et al., 2023; Kirillov et al., 2023; Zou et al., 2023) to precisely circumscribe the regions--f-interes.. The results then clean object-level editions maximizing the consistency with the original video and the rendering of the targeted object. In addition, we also extract HED maps they lead to rich object descriptions. We then use the extracted masks to guide the generative process of DDIM (Denoising Diffusion Implicit Model) model conditioned by both a target prompt and a HED map, the latter ensuring to preserve the semantic structure of the source image. The whole pipeline illustrated in Fig.
Step Extracting precise spatial information
Crop
Alpha Blending
Mask Generator
Mask M
"boat"
Original Atlas
Composited Atlas
HED
Step 2: Noising steps
1000
Crop & Resize
Noising step
1000
XT
X
X1
X
Original Atlas
M (1 M) ox
3: Conditional denoising
1000
"ructy boar"
"rusty boat"
DDIM step
1000
Replace
step
HED
HED
Edited Atlas
Figure 3: The three steps of our atlas editing procedure.
Step 1: Extracting precise spatial information. In order to generate edits that are meaningful and realistic once mapped back in the original image space, we have to guide the generative process toward a plausible output in the atlas representation. Our objective is then twofold. First, we want to precisely localize our region of interest in the atlas in order to only make alterations within this area. As in Avrahami et al. (2022), this edit mask will help to seamlessly blend our edits in the video content while having minimal impact out-of-intereet parts of the vidco. Recently, Couairon et al. (2022) proposed a method to automatically infer such a mask with a reference and target text queries, but it generally overshools the region that requires to be edited, compromising the integrity of the original video content. On the other hand, segmentation models have recently seen spectacular advances (Cheng et al., 2022; Xu et al., 2023; Kirillov et al., 2023; Zou al., 2023), allowing to confidently and accurately detect and recognize objects in images. When applied directly onto atlas grids, we observe that, despite the distribution shift with world images, these models
5

---

Published in Transactions on Machine Learning Research (03/2024)
generalize sufficiently well to infer mask around the targeted regions. Consequently, we choose to leverage the performance of these frameworks to perform panoptic segmentation and thus gain object-level spatial control over our future edits. Hence, we first take our original atlas representation which is composed of an RGB image and an alpha channel. In order to assist the segmentation network in providing a precise mask, we mix the RGB image with a fully white patch according to the alpha values. This step allows to enhance the contrast between the object and the background as illustrated in Appendix C. Then, we identify the object or region that we need to locate and create bounding box around the identified area. Finally, we produce a more accurate mask M on this smaller patch.
Socond, as we are interested in changing the aspect of objects while preserving their overall shapes, we have to ensure that our edits match their semantic structure in the at representation. Several works propose methods to perform maage-to-image translation (Mokady et al., 2022; Tumanyan et al., 2022; Bar-Tal et al,, 2022; Hertz et al., 2022). However, their various drawbacks terms of editing time or lack of generalization on atlas representations that are too far away from real-world images, hinder the use of such approaches directly in the atlas space. Consequently, we choose to align the internal knowledge of a generative text-to-image model with external control signal that helps preserving the semantic structure of objects. To this end, we opt to exploit the accurate and computationally efficient HED algorithm (Xie & Tu, 2015) to bring out critical edges that characterize the structure of our image.
Noising steps. We crop a patch from the original atlas at the bounding box location. This cropped patch is then encoded into an image latent via the VQ-autoencoder of the diffusion model. Starting from this latent dubbed Xo, we use a classical noising procedure with T = 1000 steps, which leads to a nearly isotropic Gaussian noise sample XT, i.e. po (xT) = (0. We denote p the noising ratio of a noisy latent x, such that p = t/T.
Step 2:
Step 3: Decoding with mask guidance. Starting from our latent yr = xr, we decode it with a pre- trained diffusion model that integrates control modalities (Zhang & Agrawala, 2023) to guide the denoising process. Specifically, at each step 6, the U-Net denoises the image latent in a direction determined by both the target prompt and the HED edge map:
yt-1 = 1 Yt vI osco Cp, Cp.C.) + (I-aa-1toly,, Cp. Ct)
(3)
Vai
where Cp and O₁ are the embeddings of the query text prompt and HED map, projected into a common representation space with yı, through dedicated cross-attention blocks. The encoder of the denoising U- Net ep is applied separately on yr, the inpul to be denoised, and the HED conditioning ea (A) with À a balancing coefficient that the decoder takes at each stage to compute a weighted sum of the activation maps. far E 1))_ i is a variance schedule that determines the step sizes.
The marginal of the forward process sample at slep L-1 1 admits a simple closed form given by xt-1 Vai-1x0 + VT- 01-14t-1- Following Avrahami al. (2022), we use this relation to retrieve the area outside the object's 's mask M during the generation process while the interior region is obtained following the standard diffusion process given in Eq. (3):
- 0
(4)
In the last step, the entire region outside the mask is replaced with the corresponding region from the input image, allowing to preserve exactly the background from the original crop. Our edited patch is finally replaced at its location within the atlas grid. Therefore, this pipeline seamlessly fuse the edited region with the unchanged parts of an atlas. Lastly, the edited atlas is used to perform bilinear sampling of frame edit layers. Once these layers are composited with their corresponding original frames, they produce an edited video that exhibits both spatial and temporal consistency.
4 Experiments
In this section, we describe our experimental setup, followed by qualitative and quantitative results.
6

---

Published in Transactions on Machine Learning Research (03/2024)
4.1 Experimental setup
Dataset. Following Bar- Tal et al. (2022); Wu et al. (2022); Qi et al. (2023), we evaluate our approach on videos from DAVTS dataset (Pont-Tuset et al., 2017) resized at 768 432 resolution. The length of these videos ranges from 20 to 70 frames. To automatically create edit prompts, we use captioning model (Li et al., 2022) to obtain descriptions of the original video content and we manually design 4 editing prompts for each video.
VidEdit setup. To control the semantic layout of our generated edits, we utilize a ControlNet variant of Stable Diffusion (Zhang & Agrawala, 2023). This model has learned to detect and to integrate HED edges as conditional information to a diffuusion model via training a copy of its layers while also maintaining locked the pre-trained parameters separately. The trainable and locked copics of the paramcters are connected at each block of the UNet decoder via "zero convolution" layers that are also optimized. We refer to the original paper for additional information. The original version of Stable Diffusion is trained at a 512 x 512 resolution on LAION-5B dataset (Schuhmann et al., 2022). We choose Mask2former (Cheng et al., 2022) as our instance segmentation network. To edit an atlas, we sample pure Gaussian noise (i.e. p= 1) and denoise for 50 steps with DDTM sampling and classifier-free guidance (Ho & Salimans, 2022). For a single 70 frames video, it takes 15 seconds to edit 512 x 512 patch in an atlas and minute reconstruct the video with the edit layer on NVIDIA TITAN RTX, a graphic card accessible to the general public. We set up the HED strength À to 1 by default.
Basclines. We compare our method with two text-to-image frame-wise editing approaches and three text-lo-video editing baselines. (1) SDEdil (Meng el al., 2021) is a framewise zero-shot editing approach that corrupts an input frame with noise and denoise it with a target text prompt. (2) ControlNet (Zhang & Agrawala, 2023) performs trsc editing with an external condition extracted from the target frame. (3) Terl2Live (Bar-Tal et al., 2022) is a Neural Layered Atlas (NLA) based method that trains a generator for each text query to optimize CLIP--asse loss. (4) Tune-a- Video (TAV) (Wu et al., 2022) fine-tunes an inflated version of a pre-trained diffusion model on video to produce similar content. (5) Pix2 Video (Ceylan et al., 2023) uses a structure- structure-uuded image diffusion model to perform text-guided edits on a key frame and propagate the changes to the future frames via selt-allention feature injection.
Metrics. video edit is expected to (1) be temporally consistent (Temporal), (2) faithfully render a target text query (Semantics). (3) preserve out-of-interess regions unaltered (Similarity). To evaluate CLIP based metrics, we used CLIP VIT-I./14 (Radford et al., 2021).
Temporal
Frame Consistency (Crrame), Measures the CLIP similarity between the image embeddings of consecutive video frames. Formally, it writes:
F
CFrame
CLIPScore([], 15t1)
5
FR
j=1
where 7/ is the j-th frame of edited video k; F& the number of frames in video and N the number of edited videos.
Warping Error (Ewarp). Measures the temporal stability of edited videos based on the flow warping error between two frames:
N
Ewarp
Warp(I 15
(6)
N
1=1
with W Warp(Th. Th Tk T/ the warping error between consecutive frames of an edited video defined as in Lai et al. (2018)
Semantics
7

---

Published in Transactions on Machine Learning Research (03/2024)
Prompt Consistency (Crrompt). Measures the average CLIP similarity between targel text query and each video frame. For unique pair (image; caption) the CLIP similarity writes: CLIPSSorreI, C) max(100 x cos(Er, Er, Ec), 0) with Ex the visual CLIP embedding for an image I, and Ec the textual CLIP embedding for a caption C. Cprompt then writes :
N
Cprompt
Σ N
1
1 FR
CLIPScore C*)
where 7 is the j-th frame of edited video k, O the edited caption of video k, Fy the number of frames in video & and N the number of edited videos.
Frame Accuracy (Aprame). Corresponds to the average percentage of edited frames that have higher CLIP similarity with the target text query than with their source caption. Formally, AFrame writes:
AFrame
L
[CLIPSccore(; ,0)) CLIPScore(T), ch)) 100 (8)
(T,,
Directional Similarity (Spir). Quantifies how closely the alterations made to original image align with the changes between source caption and target caption. For the j-th frame of video & the similarity score writes: SIMScore O CCk) 100 * cos(EBy Ery
Eig, Eoy Boy). Spir then writes:
1 N
SDir
SIMScore 15, S,, C*)
(9)
PR
k=
j=1
Similarity
Regarding content preservation, we have chosen three metrics that operate on different feature spaces in order to capture a rich description of perceptual similarities.
LPIPS (Zhang et al., 2018) operates on the deep feature space of VGG network and has been shown to match human perception well.
HaarPSI (Reisenhofer et al., 2018) While LPIPS evaluates the perceptual similarity between two images in deep feature space, HaarPSI performs Haar wavelet decomposition to assess local similaritics.
PSNR. measures the distance with an original image the pixel space.
These metrics are extensively described in the literature and we refer to it for further details.
A ggregate Score. This metric synthesizes in single score the overall performance of each model on semantic and similarity aspects, relatively to the best baseline. When dealing with metrics where a higher value is considered preferable, a coefficient in the aggregate score is computed as: max(S,)/S) i.c. the best score for metric i divided by the score of baseline j for metric i. When the objective is to minimize the metric, we take the inverse value. The minimal and best aggregate score for cach aspect 3, we aggregate three semantic similarity scores.
4.2 State-of-the-ar comparison
Quantitative results. Tab. gathers the overall comparison with respect to the chosen baselines VIDEDIT outperforms other approaches in terms of both semantic and similarity metries. Moreover, it exhibits a temporal consistency comparable to Text2Tive while largely surpassing alternative approaches. Regarding
Results bold correspond to the best methods based paired 1-test. (risk 5%).

---

Published in Transactions on Machine Learning Research (03/2024)
semantic metrics, as indicated by our best directional similarity score, VIDEDIT performs highly consistent edits with respect to the change between the target text query and the source caption.
Table 1: Stato-of-thc-art comparison.
Semantic
Similariy
Temporal
Mothod
Croompi())
Absonr (1)
Sma(t)
Agg:
(4)
LPIPS (1)
HaarPSI (1)
PSNR (1)
Agg
(1)
(raneef)
Agg: (1)
Serre
Score
Score
VidEdit (ours)
28. 1 (asti)
91.5 (=
21.7 (ca-4)
3.06
0.077 (20194)
0.750 (25.315)
22.6 (=16))
3.01
97.4 (=144)
5.2 (e0.3)
2:35
Teve2Lave
28.7
94.1 (434.
20.4
2.07
0,155
0.730 (80.088)
4.04
97.0
3.9
2.00
ControlNes
28.0 (2244)
84.8 (224.)
18.7 (=0.4)
3,30
0.647 (450001)
0.312 (13)
10.8 (21.5)
12.85
86.2 (214))
177.9 casta:
46.74
SDEale
26.1 (12.))
66.7 (431.9)
14.2 (-7-4)
3.96
0.290 (40.6G3)
0.377 (101
17.9 (-1.4)
9.37
83.9 (45.)
71.2 (120-1)
19.42
27.5 (2:11
(336.3)
13.7 (=000)
3.92
(asusTo)
0.274
(20.003)
(=2.3)
12.00
96.4 (22.4)
47.8 (127.))
Phe2Vidoo
29.0 (1.3.))
82.9 (430.0)
16.2 (-4:3)
3.47
0.540 (400670y
0.335 (40.069)
13.8 (-2.3)
10.90
34.4 (42.3)
160.9 (198-5
42:29
Even though our method is close to Text2Live in terms of frame accuracy and prompt consistency, the latter explicitly optimizes generator on CLIP-based loss, making the aforementioned metrics not reliable to assess its generalization performance and editing quality, as will be shown in the qualitative results. When it comes to image preservation evaluated with our similarity metrics, VIDEDIT outperforms all basclines in LPIPS and HaarPSI, and is similar to Text2Live on PSNR. This shows the capacity of our approach to optimally preserve the visual content of the source video while generating faithful edits to the target queries. Finally, VIDEDIT outperforms all methods in CFramer and shows that the fine spatial control of our approach also translates in improved temporal consistency.
To further analyze the fine-grained editing capacity of our method while preserving the original video content, we display for all baselines in Fig. 4, their local AFrame score computed within a ground truth mask compared to an outer LPIPS metric (denoted O-LPIPS) computed on the invert of the mask. We see that VIDEDIT reaches a very good local frame accuracy, even outperforming Text2Live. Morevover, VIDEDIT shows a huge improvement on the O-LPIPS metric compared to the basclines, including Text2Live (3 vs 8), showing clearly a better preservation of out-of-interest regions.
Additionally, when comparing the processing time of dif- ferent basclines, we found that VIDEDIT has signifi- cant advantage, with a ~ 30-fold speed-up factor over Text2Live. Focusing on the interative part of editing? in which users are interested in², Figure 5 underlines the lightweight aspect of our method. The panel on the left shows that VIDEorr can perform large number of edits 70 frame long video in significantly less time than other appoaches. As an illustration, VIDEDIT demon- strates approximately 30 times faster editing capabilities compared to Text2Live, which is the second leading base- line in terms of editing capacities. On the other hand, as depicted in the right panel, the use of VIDEDIT becomes increasingly time-efficient compared to other baselines, as the number of frames to edit growths
60-
50
VidEdit
Text2Live
SDEdit
30-
ControlNet
Pxx2 Video
20
Tune-e-Video
10
50%
60%
70%
80%
90%
Local Frame Accuracy
Figure 4: Masked LPIPS vs Local Object Accuracy. The size of cach dot is proportional to the standard deviation of the local object ac- curacy.
Qualitative results. We show in Fig. 6 a visual comparison against the baselines to qualitatively assess the improvement brought out by our method. We can see that VIDEDIT performs -ne-ggrined editing while perfectly presorving out-of-interest regions. In comparison to other baselines, the edits generated are more visually appealing and realistic. For example, VIDEDIT obtains a frame accuracy (Aprame) and prompt. consistency (Crrompt) scores of 26.5 and 30 respectively compared to Text2Live which reaches 26 and 35 respectively. However, we can see that Text2Live's 's scores do not automatically translate into high-quality edits it often struggles to render detailed textures precisely localized on targeted regions. For example, ice creams are poorly rendered and some untargeted areas are being altered. Regarding Tune-a- Video and Pix2Video baselines, the methods are unable to generate faithful edit at the exact location and completely degrade the
2Steps as DDIM inversion LNA construction being considered as pre-processing steps. Sditing the atlas and reconstructing the video for atlas based methods. Simply inferring the model for other baselines.
9

---

Published in Transactions on Machine Learning Research (03/2024)
10a
10a
VidEdit
Time (h)
Time (s)
Text2Live
SDEdit
109
ControlNet
10
Pix2Video
10
Tune Video
1510
25
50
100
10
20
30
40
50
60
70
80
90
100
Number of Queries
Number of Frames
Figure 5: Editing time. VIDEDIT can edit videos significantly faster than existing methods.
Source Prompt: 'A couple of people riding motorcycle down rood'
Target Edit: "potted plant"
iice cream"
VidEdit
Text2Live
Tune-a- Video
Vixeoideo
SDEdit
Figure 6: Qualitative comparison of VidEdit with other baselines. VIDEDIT generates higher quality textures than Tive Tune-a-Video and Pix2video completely alters untargeted regions.
original content. Despite relatively high frame consistency scores in this video (96% for Tune-a- Video and 89% for Pix2Video us 97.5% for VIDEDIT), noticeable flickering artifacts undermine the vidco content. On the other hand, naive frame-wise application of image-to-inage translation methods also leads temporally inconsistent results. For example, SDEdit is unable to both generate faithful edit and to preserve the original content it inherently faces trade-off between the two. Other visual comparisons are shown in Appendix B.
10

---

Published in Transactions on Machine Learning Research (03/2024)
4.3 Model Analysis
Ablations. We perform ablation studies to demonstrate the importance of our conditional controls once we map the edits back to the original image space. Tab. 2 compares the performance of our editing pipeline with both instance mask segmentation and HED edge conditioning against scenarios where these controls are disabled.
Table 2: Ablation study, Mask and HED map help to generate meaningful edits in the original frame space.
Controls
Semantic metrics
Similarity metrics
Mask
HED
Cprompt (1)
AFrame (1)
Spir (t)
LPIPS (4)
HaarPSI (1)
PSNR (t)
/
28.1 (43.0)
91.5 (+11.1)
21.7 (48.4)
0.077 (:0.054)
0.730 (40.109)
22.6 (43.6)
X
x
25.5 (43.1)
64.3 (+38.3)
10.6 (+7.5)
0.099 (+0.051)
0.632 (-0.131)
20.1 (+4.0)
X
26.3 (43.0)
72.4 (#31.0)
13.0 (#7.6)
0.095 (#0.019)
0.672 (=0.110)
20.8 (#3.6)
/
27.5 (42.8)
81.9 (1:4.2)
18.0 (2:.4)
0.081 (=0.042)
0.639 (=0.128)
20.7 (43.3)
In the case where conditional control is passed on to the model, we observe substantial drop in semantic metrics as the model generates edits at random locations in the atlas whose shapes don't match the structure of the target object. The introduction of edge conditioning without spatial awareness is quite similar to the previous case with the difference that the model tries to locally match the control information. This results in slightly better semantic results and similarity metrics than with no edge control. Finally, blending an edit with mask control without taking structure conditioning into account generates an edit at the right location but that is semantically incoherent once mapped back to the original images. Yel, this scenario achieves decent prompt consistency the objects still correspond to the target text query. We provide visual illustration of this ablation study in Appendix A.
Impact of hyperparameters. We analyze in Fig. 7 VIDEDrr's behaviour versus the HED conditioning strength and noising ratio, respectively À and To analyze the trade-off between semantic editing and source image preservation, we compute a local LPIPS computed within ground-truth mask, provided by DAVIS, versus a local CLIP score computed within the same mask for an edited object.
On the left panel, to guide the edits toward a plausible output the video frames. This phenomenon is illustrated in Appendix A. On the contrary, for strength values larger than 1.2, the conditioning weighs too much on the model and hinders its ability to generate faithful edits. As expected, we notice that the local LPIPS decreases as the edge conditioning increases. While the decreasing rate is substantial between 0 and 1, the marginal gain diminishes for larger values.
that for À values lower than 0.4, the edge conditioning is not strong enough
Overall, setting the HED strength be- tween 0.8 and 1.2 robustly enables to both perform faithful edits and preserve the original content. On the right panel, we see that both local CLIP score and LPIPS increase with the noising ratio. Indeod, for a null value, the region is reconstructed from the atlas, nearly identically to the original, and is then rewarded low LPIPS. However, no modification has been performed, the patch does not match the larget lext. query and gets a lower CLIP score. As the noising ratio increases, the region de- viates more from the input but also better matches the target edit. Note that for a p value of 100%, the local LPIPS is constrained below 19, which still indicates low disparity with the original image.
19922
Loral CLIPScore
27.9
26.5
Loral CLIPScore
26.8
15.9
Local LPIPS
24.1
26.9
25.0
12.5
- 21.7
19.3
25.8
23.3
9.1
CLIPScore
CLIPScore LPIPS
16,9
LPIPS
24.7
21.5
1.2
0%
60% 80%10009
Figure 7: VidEdit behavior wrt. different À and values.
11

---

Published in Transactions on Machine Learning Research (03/2024)
Diversity. Finally, illustrate in Fig. 8 the capacity of VIDEDIT produce various and sundry video edits from a unique pair (video; larget text query). In contrast, the randomness in Texi2Live's training process only comes from the generator's weights initialization. As result, method converges towards a unique solution and thus shows poor diversity in the generated samples.
Target edit: "bus" - "a relrowave bus"
Text22
VidEdit
Figure & Texture diversity. We edit each video four times with the same input text query. Compared to Text2Live, our method is able to synthesize more diverse samples in much less time.
5 Conclusion & Discussion
We roduced VIDEDIT, a lightweight algorithu for zero-shot semantic video editing based on latent diffuusin models. We have shown experimentally that this approach conserves more appearance information from the input video than other diffusion-based methods, leading to lighter edits. Neverthcless, the approach has few limitations. Common with Kasten et al. (2021), the capacity of the MLP mapping networks decreases for complex videos involving rapid movements and very long-term videos. Since our method relies the quality of such atlas representations, one possible way to expand the scope of possible video edits would be to strengthen and robustify the neural layered atlases construction approach.
References
Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, jun 2022. doi: 10. 00.119//cDp552688.22222 200000011....2..11..... URL https://doi. org/10. 11199/2F 2022.01767.
Omer Bar-Tal, Dolev Ofri-Amar, Rafail Fridman, Yoni Kasten, and Tali Dekel. Text2live: Text-driven layered image and video editing. In Compuler Vision ECCV 2022: 17th European Conference, Tel Avie, Israel, October 23-27, 2022, Proceedings, Part XV, pp. 707-723. Springer, 2022.
Duygu Ceylan, Chun-Hao Paul Huang, and Niloy J Mitra. Pix2video: Video editing using image diffusion. arXiv preprint 1226: 2023.
Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked-attention mask transformer for universal image segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 1290-1299, 2022.
Guillaume Couairon, Jakob Verbeek, Holger Schwenk, and Matthieu Cord. Diffedit: Diffusion-based semantic image editing with mask guidance. arXio preprint arXi::2210.112. 2022.
Tan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farlyy Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Commanications of the ACM, 63(11): 139 144, 2020.
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfrr Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to- prompt image editing with cross attention control. aixiv preprint arXi::2208.014 01626, 2022.
12

---

Published in Transactions on Machine Learning Research (03/2024)
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXtt:2207.12500 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Newral Information Processing Systems, 33:6840-6888 6851, 2020.
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, al. Imagen video: High definition video generation with diffusion models. Xiv preprint or Xiv:22200 023003 2022a.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. arXiv preprint orXiv:2204.03455 2022b.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. 2019 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 4396 4405, 2018.
Yoni Kasten, Dolev Ofri, Oliver Wang, and Tali Dekel. Layered neural atlases for consistent vidco editing. ACM Transactions on Gruphics (TOG), 40(6):1-115 12, 2021.
Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Trani. Imagic: Text-based real image editing with diffusson models. arXiv preprint arXim:2210.0090 09276,, 2022.
Alexander Kirillov, Fric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan Yen Lo, Piotr Dollár, and Ross B. Girshick. Segment anything. ArXio, abs/2304.02666 026443, 2023.
Wei-Sheng Lai, Jia-Bin Huang, Oliver Wang, Eli Shechtman, Ersin Yumer, and Ming-Hsuan Hsuan Yang. Learning blind video temporal consistency, 2018.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-iman pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning, PMLR, 2022.
pp.
Shaoteng Liu, Yuechen Zhang, Wenbo Li, Zhe Lin, and Jiaya Jia. Video-p2p: Video editing with cross-attention control. arXiv preprint arXiv:2303.04 2023.
Sebastian Loeschcke, Serge J. Belongie, and Sagie Benaim. Text-driven stylization of video objects. In ECCV Workshops, 2022.
Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun- Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. In International Conference on Learning Representations, 2021.
Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Null-text inversion for editing real images using guided diffusion models. arXiv preprint arXiv:2211.0979,, 2022.
Chong Mou, Xintao Wang, Liangbin Xie, Jing Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie. T2i- adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. ArXie, abs alss/2302.08453, 2023.
Jordi Pont Pont-Tuset, Federico Perazzi, Sergi Caclles, Pablo Arbeláez, Alex Gorkine-Horming and Luc Van Gool. The 2017 davis challenge on video object segmentation. arXiv preprint arXiv:1704..000 00675, 2017.
Chenyang Qi, Xiaodong Cun, Yong Zhang, Chenyang Lei, Xintao Wang, Ying Shan, and Qifeng Chen. Fatezero: Fusing attentions for zero-shot text-based video editing. arXiv preprint XiXi::3333009533 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishknn, Jack Clark, et al. Learning transferable visual models from natural language supervision. Tn International conference on machine learning, pp. 8748 8763. PMLR, 2021.
13

---

Published in Transactions on Machine Learning Research (03/2024)
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hicrarchical text-conditional image generation with clip latents. arXiv preprint or Xix:::200 20:2224406125, 2022.
Rafacl Reisenhofer, Scbastian Bosse, Gitta Kutyniok, and Thomas Wiegand. A haar wavelet-based perceptual similarity index for image quality assessment. Signal Processing: Image Communicatiin, 61:33-43 2018. ISSN 0923-5965. doi:
htw.:////......./...a.ee.eee.. iww.am.20...11111... URL httpss///ww..sceneeeeeeed. http://aaac.arrttaaeeaaa//////2/
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pollern Recognition, pp. 10684 10695, 2022.
Christoph Schuhmann, Romain Beaumont, Richard Veneu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. orXiv preprinl arXiv:2210.084000 2022.
Chachun Shin, Hocsoung Kim, Che Hyun Lec, Sang-gil Lee, and Sungroh Yoon. Edit-a-video: Single vidco editing with object-aware consistency. arXiv preprinl vxXiv:2303.09443 2023.
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209. 14992, 2022.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprinl W:Xiv22010002000 2020.
Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusion features for text-driven image-to-image translation. arXiv preprinl or Xrx:v:2211. 12572, 2022.
Ruben Villegas, Mohammad Babacizadch, Picter-Jan Kindermans, Hernan Moraldo, Han Zhang, Moham- mad Taghi Saffar, Santiago Castro, Julius Kunze, and Dumitru Erhan. Phenaki: Variable length video generation from open domain textual description. arXiv preprint arXin:2210. 2xX::2210.00390, 2022.
Wen Wang, Kangvang Xie, Zide Liu, Hao Chen, Yue Cao, Xinlong Wang, and Chunhua Shen. Zero-shot video editing using off- shellf image diffusion models. arXiv preprint arXiv:2303. 17599, 2023.
Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Weixian Lei, Yuchao Gu, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. arXiv preprint arXim:2212. 11565, 2022.
Saining Xie and Zhuowen Tu. Holistically-neste edge detection. Proceedings of the IEEE international conference on computer vision, pp. 1395-1403, 2015.
Jiarui Xu, Sifci Liu, Arash Vahdat, Wonmin Bycon, Xiaolong Wang, and Shalini De Mello. Open-vocabulary panoptic segmentation with text-lo-image diffusion models. ArXiv, abs/2303.04803, 2023.
Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved vggan. arXiv preprint Xrxiv 0. 0.04422, 2021.
Lvmin Zhang and Mancesh Agrawala. Adding conditional control to text-to-image diffusion models. arXiv preprint arX:::2302. 05543, 2023.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 580-595, 2018.
Daquan Zhou, Weimin Wang, Hanshu Yan, Weiwei Lv, Yizh Zhu, and Jiashi Feng. Magicvideo: Efficient. video generation with latent diffusion models. arXiv preprint trXiv2211.11011. 2022.
Xucyan Zou, Jianwei Yang, Hao Zhang, Feng Li, Linjie Li, Jianfeng Gao, and Yong Jac Lee. Segment everything everywhere all once. Arxio, abs//2004.06718, 2023.
14


Reviews:
[{"review_id": "8zgQAGFFMo", "reviewer": "Reviewer_xGev", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\nBoth quantitative and qualitative evaluations are performed. The results look promising.\n\nCons:\n\n- The abstract section is not well written. The authors claim that “Recently, diffusion-based generative models have achieved remarkable success for image generation and edition. However, their use for video editing still faces important limitations.” However, the authors fail to mention what the specific limitations are, which would look confusing to the readers.\n\n- Also, the authors claim in the abstract section that “With this framework, processing a single video only takes approximately one minute, and it can generate multiple compatible edits based on a unique text prompt.” However, the authors did not explain the equipment used for the “one minute”.\n\n- Furthermore, I do have concerns regarding the novelty of this manuscript. It seems that the proposed framework is an ensemble of existing algorithms, such as atlas-based approaches, pre-trained text-to-image diffusion models, panoptic segmenter, and HED. \n\n- In the experiments, only the DAVIS dataset is used to validate the effectiveness of the proposed method. The authors are suggested to consider using more datasets to make the evaluation more convincing.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors propose VidEdit, a novel method for zero-shot and spatially aware text-driven video editing. The authors aim to address the limitations of existing methods in video editing and introduce a new approach that combines atlas-based and pre-trained text-to-image diffusion models. By leveraging off-the-shelf panoptic segmenters and edge detectors, VidEdit enables precise spatial control over targeted regions while preserving the original video's structure. The experiments conducted on the DAVIS dataset demonstrate the superiority of VidEdit over state-of-the-art methods in terms of semantic faithfulness, image preservation, and temporal consistency metrics. Moreover, the proposed framework is lightweight and time-efficient, with a processing time of approximately one minute per video. It can generate multiple compatible edits based on a single text prompt.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please kindly refer to the weakness section.", "broader_impact_concerns": "NA.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "4wBIbJFU3b", "reviewer": "Reviewer_2v7z", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I think this is a reasonable combination of components, perhaps with exception of the edge map estimator from HED, which might be redundant since the model has access to panoptic segmentation from Mask2former. \n\nI am not sure that the method, as proposed, is complete. Using the M_f and M_b networks, we can apparently map the video's xyt coordinates onto uv coordinates, and these uv coordinates can be fed to an A network, and an alpha map for blending, but there is no learning described here -- no optimization, no reconstruction, no actual learning of the M and A networks. To learn about these steps, we need to read a different paper, Kasten et al 2021. Looking into that paper, it seems like maybe two A networks are necessary (but not mentioned here) -- is it A_f and A_b, or just \"A\" as described? What losses are used here? Is it color+rigidity+consistency+sparsity like the referenced paper? How does the panoptic segmentation fit in here? In the original paper it was a bootstrapping process, which began with MaskRCNN.\n\nThe paper mentions modifying SD by \"incorporating semantic guidance based on high-resolution semantic masks\". How are the semantics incorporated here exactly? How exactly was the architecture modified? These seem like crucial method details, and yet the paper only provides a sentence or two.\n\nA downside with the layering approach is that it assumes that the scenes are basically planar, with very little occlusion happening. It would be nice to resolve this with the diffusion somehow.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a zero-shot video editing method, combining the Neural Layered Atlas (NLA) approach with Stable Diffusion (SD) and an edge estimator and Mask2former panoptic segmentation. The idea is to reproduce the NLA approach, and then replace the manual editing of atlases with DDIM-based editing, where the DDIM is aided slightly by the edge map, and guided by a language prompt via classifier-free guidance.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Overall, I would like to see the technical details clarified, specifically with respect to the implementation choices in the atlas/layering parts (especially highlighting differences from the original work), and also clarifying the modifications to SD. Beyond these major issues, I have a long list of minor issues I uncovered while reading. \n\n\n\"VidEdit modifies the original content by precisely delineating the regions of interest\" This claim should be clarified. It sounds like the modification is to delineate the region, but I think this is not what the method actually achieves.\n\nThe paper says that \"spatially grounded\" editing \"is achieved by extracting a crop around the area of interest and intentionally utilizing a non-invertible noising process.\" What is meant here? I can't imagine something like this being implemented unintentionally, but maybe I am not reading this right. \n\nThe paper claims that diffusion-based models, unlike GANs, \"can be reliably trained on massive amounts of data and produce convincing samples\". I think this is basically a false claim. GANs can also be trained on massive amounts of data, and they can also produce convincing samples. \nThe paper claims, in 3.1, that \"The 2D atlases obtained by disentangling the video are a well-posed framework to edit objects while ensuring a strong temporal consistency.\" I am not convinced about this. In general this seems like an empirical claim that needs evidence, but also I am not sure what the authors mean by \"well-posed framework\". \n\nIn the related work, it would be great to connect the works to the current effort somehow. What are the specific similarities and differences, with respect to the current work?\n\n\"these methods still suffer from annoying flickering artifacts\" It is not nice to say that the outputs of other methods are \"annoying\" to you. About Text2Live, the related work section reports that \"its training procedure for each prompt makes it impractical to use\". I checked that paper and it does not look impractical -- it's a zero-shot method, and it's an ECCV 2022 oral, and it uses a similar layered \"atlas\" representation as this work.\n\n\n\"The ergonomy and performance of text-conditioned diffusion models are a great motivation to perform those editions based on text queries.\" I am not sure about many parts of this sentence. Ergonomy? Perform which editions, and editions of what?\n\n\n\"the automatic video editing charge\" -- What is this? I think this is only mentioned once, but it seems important. \n\n\n\"First, the inverting procedure is relatively time-consuming. We thus opt to limit the computational overhead in order to increase our editing efficiency.\" I do not follow this. Why is the computation-saving motivated by a time-consuming procedure? \n\n\n\"we choose not to perform inverse DDIM in the latent space to encode x0 for two main reasons.\" As far as I can tell, this is the first occurrence of x0. What is x0?\n\nThe paper mentions that the approach leads to \"lighter edits\" than prior work, but I am not sure this is a good thing.  \n\nThe paper says \"the capacity of the MLP mapping networks decreases for complex videos involving rapid movements and very long-term videos\". Why does the capacity decrease? Capacity seems like something you can control, and for longer videos, you should probably increase it instead of decrease it. Perhaps the authors can clarity this strategy. \n\nThe paper claims that \"one possible way to expand the scope of possible video edits would be to strengthen and robustify the neural layered atlases construction approach.\" I do not know what this means. How would the strengthening be done exactly, and what robustness is being referred to, and how would these things affect the scope of the possible edits?\n\nThe paper says that they propose a model \"which by design fulfills temporal smoothness\", but I think the temporal smoothness comes from the atlas component, which is not a contribution of this paper really.\n\nThe paper concludes with a quote from the Stable Diffusion paper, about ethical and legal use of stable diffusion. I do not think this is a useful conclusion to the paper.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "fswBw9wcPL", "reviewer": "Reviewer_n9p4", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n-\tSimple method using existing pretrained modules in an effective way. No further training needed.\n-\tDoes not require training at test-time and therefore allows for very fast inference.\n-\tThe visual results are convincing (video is missing). Method is able to generate diverse edits.\n-\tThe proposed conditioning is highly effective as the ablation shows.\n-\tAddresses the challenge of evaluating editing with some proposals of quantitative metrics, however, these metrics could be better motivated and evaluated that they measure the right thing (see below).\n\n**Weaknesses**\n\n*Evaluation metrics:* I appreciate the attempt to provide some quantitative measurements, however, I am not convinced by their suitability: \n\n-\tThe measurement for temporal consistency is calculated by the clip similarity between two frames. Besides that it is unclear, whether it uses the image or textual embedding, it is also unclear to me, how the similarity of a single clip score per frame corresponds to spatial temporal stability of a video. That seems a very coarse measurement to me and more evidence needs to be provided that this novel metric can capture temporal consistency. Alternatives on using e.g., warping errors are more detailed. Better would be also to attach a video and user study, as qualitative analysis is still very important to evaluate perceptually, whether a video is temporal stable. Given that obtaining strong temporal consistency is one of the main claims of the paper I would expect more evidence for that. \n-\tContent preservation is measured as the similarity between the edited and original. However, in the main description it is unclear whether this is applied to the full image or weighted by the editing mask M, otherwise the best results can be obtained with no editing at all and therefore can be misleading.\n\n*Lacks clarity:*\n \n-\tHow is the atlas to be edited chosen?\n-\tWill the code be made available?\n\n*Improvements in presentation:*\n\n-\tNot all variables are introduced/defined, e.g., t, alpha in Eq. 1, M in Eq. 2, variables in the frame consistency measurement, …\n-\tFigures and Captions are missing references of used methods, e.g., in Fig. 2 reference to NLA models by Kasten et al. (2021), or used mask generator, HED in Fig. 3.\n-\tAdding all equation numbers, makes referencing them possible.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper describes a method for text-based editing of videos. The proposed method uses the existing neural layer atlas model to decompose a video. A specific atlas can then be edited with the proposed editing diffusion pipeline which is the main contribution of the paper. A pretrained text-to-image diffusion model is used and additionally conditioned based on the extracted segmentation mask and edge detection. The way how the atlas is edited distinguishes the method from the previous work. It is simple and fast and does not require any additional training. The resulting images are diverse and visually convincing.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- The main weakness of the paper is the evaluation. The evaluation metrics, especially the temporal consistency needs further explanation and justification. Further, visual results as a video would be required to assess the perceptual and temporal quality and ideally supplemented with a user study.\n- Further, the overall presentation of the paper can be improved (see comments/suggestions above).", "broader_impact_concerns": "is already addressed in the paper", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The AE believes that the remaining concerns are fixable with a minor revision on the clarity of the method. The authors should make sure to take all reviewer comments into account and submit an ewer version.
Decision: Accept (poster)